Task: Monitor (AST)

Method of operation

The principal and most difficult task of the test manager is the monitoring of the execution of the plan.

While this is described in the section below as mainly an instrumental activity, it is equally a communication activity. The biggest part of the test manager’s task perhaps consists of “monitoring” the employees on the team. This includes everything, from recruiting new testers during the testing process, delegating the work, holding work consultations/team meetings, supporting, coaching and assessing employees, up to and including the conducting of exit interviews. Another very important task for the test manager in this same connection is the maintaining of contact with the world surrounding the test team, also known as stakeholder management or expectation management. Do the expectations of the test clients still correspond with what the test is going to deliver? Are there developments in the project that will influence the test process? It should be obvious that highly developed social and communication skills would not go amiss here.

In general, the activities described in the plan – such as preparing, specifying and executing the tests – should be carried out according to a particular timeline and in a particular sequence. To do this, the test manager has the necessary people at his disposal (including himself). He sets out a detailed planning for the coming period, outlining who will do what, in how many hours. This is necessary, as the planning within the test plan is not detailed to the extent that tester A knows that, in the coming week, she should specify test units X and Z, and tester B knows that he is to carry out test scripts Y1 and Y2 for test unit Y. Experience shows that such detailed planning only works for the initial short period, after which changes are always taking place, requiring the planning to be revised. Most obvious periods for which a detailed planning can be set up are the phases: Setting up and maintaining infrastructure, Preparation, Specification, Execution and Completion. With iterative system development, the test manager also makes a detailed planning per iteration. In setting up a detailed planning, the test manager makes allowance for all the aspects of planning, such as priorities, availability and skills (see also Determine The Planning (AST)).

Another of the test manager’s tasks is to fill in “blank spaces” in the test plan during the course of the test process. This is the case when, at the time of setting up the plan, certain information is missing or there is no time to carry out a particular activity.

Towards the end of the test execution, the monitoring becomes even more important, as the test manager must then be able to answer the question of whether stopping testing is justified. The exit criteria formulated in the plan are the deciding factors here, but if they are absent or no longer current, there are some rules of thumb available:

Have all the planned tests been executed (in accordance with the latest test strategy)? This emphatically does not concern the original strategy, but the latest, amended version. This contains the most recent insights of the client and test manager into the balance between risk and test coverage.
Are the number and degree of severity of the outstanding defects at an acceptable level? And to this may be added: have the costs of the testing during this period risen higher than the returns (“damage prevention”, see below)?
Has the number of newly found defects as well as the number of solved and retested defects been reduced to a minimum during the latest period (e.g. week)? This last point says something about the stability of the system. Sometimes, in the last period, so much has been reworked and retested that the system has several releases per day. If only both of the above points were to be considered, it could be decided to stop testing. However, the system is still anything but stable, and a regression test is strongly to be advised.

Only when a positive answer can be given to these questions does it make sense to recommend ending the testing.

After the test team has completed all its tasks, including the Completion phase, the test manager asks the client to terminate the assignment and to discharge the test team. The team is then disbanded.

Practice also teaches that the original plan is bound to be amended. The amendments can have both internal and external causes, i.e. from both within and beyond the test process. It is up to the test manager to flag these events or trends as early as possible. Measures for redirecting a negative trend can then be adopted promptly. This is almost always better, cheaper and faster.

Information in respect of the events or trends comes from the internally managed data and information from outside of the test process, e.g. minutes or memos, but not least also from verbal interchange, such as the project consultation, stand-up meetings, bilateral discussions, etc. This is where a good social (project) network shows its worth to the test manager. Using this information, the test manager analyses possible trends and tries to apprehend threats (or indeed opportunities) in time: will the trend continue? What needs to happen to prevent it?

For this purpose, the test manager carries out the following steps:

Analysing the event, estimating risks and defi ning countermeasures
Coordinating with the client and other stakeholders (optional, depending on tolerance)

Products

Proposed management measures.

Tools

Defect management tool
Testware management tool
Workflow tool
Planning and progress monitoring tool.

1. Analysing the event, estimate risks and define countermeasures

The test manager analyses the cause of the event and determines the consequences for the test process. He also examines the significance of the event explicitly in respect of the risks that are covered at this stage of the testing. Events can influence the testing positively or negatively, and the test manager determines the possible countermeasures, depending on the timing of the event, the analysis and the consequences for the test process.

2. Coordinating with the client and other stakeholders (optional, depending on tolerance)

Depending on the measures to be carried out, the test manager can carry them out independently, or prior agreement with the client and possibly other stakeholders may be necessary. The form that this coordination takes depends on the organisation. In practice, use is often made here of reports.

The margin that the test manager has for taking measures independently is determined by the following factors:

The degree to which the difficulties of the test process can be solved within the set assignment, the product risk analysis, the test strategy, budget, planning and other preconditions. In other words: the degree to which the client is to be left out of it.
The degree to which the difficulties can be solved within the limits of tolerance, which were agreed in the planning phase. (see Determine The Planning (AST))

In practice, the test manager should generally ask permission if the measures would influence the agreements that were made at the planning phase. In other words, if adjustments have to be made to the formulation of the assignment, product risk analysis, test strategy, budget and/or plan.