Test Management Risk And Testing

To see all articles of ISTQB-ISEB Foundation guide, see here:

Software Testing-ISTQB ISEB Foundation Guide

It is not possible to talk about test management without first looking at risk and how it affects the fundamental test process.

If there were no risk of adverse future events in software or hardware development then there would be no need for testing. In other words, if defects did not exist then neither would testing.

Risk can be defined as the chance of an event, hazard, threat or situation occurring and its undesirable consequences:

Risk—a factor that could result in future negative consequences, usually expressed as impact and likelihood.

In a project a test leader will use risk in two different ways: project risks and product risks. In both instances the calculation of the risk will be:

Level of risk = probability of the risk occurring × impact if it did happen

Project Risks

Whilst managing the testing project a test leader will use project risks to manage the capability to deliver.

Project risks include:

1. Supplier issues:

• Failure of a third party to deliver on time or at all.
• Contractual issues, such as meeting acceptance criteria.

2. Organizational factors:

• Skills, training and staff shortages.
• Personal issues.
• Political issues, such as problems that stop testers communicating their needs and test results.
• Failure by the team to follow up on information found in testing and reviews (e.g. not improving development and testing practices).
• Improper attitude toward or expectations of testing (e.g. not appreciating the value of finding defects during testing).

3. Technical issues:

• Problems in defining the right requirements.
• The extent that requirements can be met given existing project constraints.
• Test environment not ready on time.
• Late data conversion, migration planning and development and testing data conversion/migration tools.
• Low quality of the design, code, configuration data, test data and tests.

For each risk identified a probability (chance of the risk being realized) and impact (what will happen if the risk is realized) should be identified as well as the identification and management of any mitigating actions (actions aimed at reducing the probability of a risk occurring, or reducing the impact of the risk if it did occur).

So, for example if there was a risk identified that the third-party supplier may be made bankrupt during the development, the test manager would review the supplier's accounts and might decide that the probability of this is medium (3 on a scale of 1 to 5, 1 being a high risk and 5 a low one). The impact on the project if this did happen would be very high (1 using the same scale). The level of risk is therefore 3 × 1 = 3. Thus, the lower the number, the more the risk. With 3 being in the medium risk area the test leader would now have to consider what mitigating actions to take to try to stop the risk becoming a reality. This might include not using the third party, or ensuring that payment for third-party deliverables is made efficiently.

When analyzing, managing and mitigating these risks the test manager is following well-established project management principles provided within project management methods and approaches. The project risks recognized during test planning should be documented in the IEEE 829 test plan.

Product Risks

When planning and defining tests a test leader or tester using a risk-based testing approach will be managing product risks.

Potential failure areas (adverse future events or hazards) in software are known as product risks, as they are a risk to the quality of the product. In other words, the potential of a defect occurring in the live environment is a product risk. Examples of product risks are:

• Failure-prone software delivered.
• The potential that a defect in the software/hardware could cause harm to an individual or company.
• Poor software characteristics (e.g. functionality, security, reliability, usability, performance).
• Poor data integrity and quality (e.g. data migration issues, data conversion problems, data transport problems, violation of data standards).
• Software that does not perform its intended functions.

Risks are used to decide where to start testing in the software development life cycle, e.g. the risk of poor requirements could be mitigated by the use of formal reviews as soon as the requirements have been documented at the start of a project. Product risks also provide information enabling decisions regarding how much testing should be carried out on specific components or systems, e.g. the more risk there is, the more detailed and comprehensive the testing may be. In these ways testing is used to reduce the risk of an adverse effect (defect) occurring or being missed.

Mitigating product risks may also involve non-test activities. For example, in the poor requirements situation, a better and more efficient solution may be simply to replace the analyst who is writing the poor requirements in the first place.

As already stated, a risk-based approach to testing provides proactive opportunities to reduce the levels of product risk starting in the initial stages of a project. It involves the identification of product risks and how they are used to guide the test planning, specification and execution. In a risk-based approach the risks identified:

• will determine the test techniques to be employed, and/or the extent of testing to be carried out, e.g. the Motor Industry Software Reliability Association (MISRA) defines which test techniques should be used for each level of risk: the higher the risk, the higher the coverage required from test techniques;
• prioritize testing in an attempt to find the critical defects as early as possible, e.g. by identifying the areas most likely to have defects (the most complex) the testing can be focused on these areas;
• will determine any non-test activities that could be employed to reduce risk, e.g. to provide training to inexperienced designers.

Risk-based testing draws on the collective knowledge and insights of the project stakeholders, testers, designers, technical architects, business reps and anyone with knowledge of the solution to determine the risks and the levels of testing to address those risks.

To ensure that the chance of a product failure is minimized, risk management activities provide a disciplined approach: To assess continuously what can go wrong (risks). Regular reviews of the existing and looking for any new product risks should occur periodically throughout the life cycle.

• To determine what risks are important to deal with (probability × impact). As the project progresses, owing to the mitigation activities risks may reduce in importance, or disappear altogether.
• To implement actions to deal with those risks (mitigating actions).

Testing supports the identification of new risks by continually reviewing risks of the project deliverables throughout the life cycle; it may also help to determine what risks are important to reduce by setting priorities; it may lower uncertainty about risks by, for example, testing a component and verifying that it does not contain any defects; and lastly by running specific tests it may verify other strategies that deal with risks, such as contingency plans.

Testing is a risk control activity that provides feedback about the residual risk in the product by measuring the effectiveness of critical defect removal (see below) and by reviewing the effectiveness of contingency plans.

To check your understanding, I would again like to ask you some questions:

What are the two types of risks that have to be considered in testing?
Compare and contrast these two risk types.
How early in the life cycle can risk impact the testing approach?
What does MISRA determine when the level of risk is understood?

You may follow the complete series of Test Management articles here:

Test Management Risk And Testing
Software Test Organization
Software Test Approaches Test Strategies
Software Test Planning And Estimation
Software Test Progress Monitoring & Control
Software Testing Incident Management
Software Test Configuration Management

To see all articles of ISTQB-ISEB Foundation guide, see here:

Software Testing-ISTQB ISEB Foundation Guide