In our previous article, we have discussed General Software Testing Principles?
The most visible part of testing is running one or more tests: test execution. We also have to prepare for running tests, analyze the tests that have been run, and see whether testing is complete. Both planning and analyzing are very necessary activities that enhance and amplify the benefits of the test execution itself. It is no good testing without deciding how, when and what to test. Planning is also required for the less formal test approaches such as exploratory testing.
The fundamental test process consists of five parts that encompass all aspects of testing (See below Figure):
- Planning and control.
- Analysis and design.
- Implementation and execution.
- Evaluating exit criteria and reporting.
- Test closure activities.
Fundamental test process
Although the main activities are in a broad sequence, they are not undertaken in a rigid way. An earlier activity may need to be revisited. A defect found in test execution can sometimes be resolved by adding functionality that was originally not present (either missing in error, or the new facilities are needed to make the other part correct). The new features themselves have to be tested, so even though implementation and execution are in progress, the ‘earlier’ activity of analysis and design has to be performed for the new features.
Iteration of activities
We sometimes need to do two or more of the main activities in parallel. Time pressure can mean that we begin test execution before all tests have been designed.
Test Planning and Control
Planning is determining what is going to be tested, and how this will be achieved. It is where we draw a map; how activities will be done; and who will do them. Test planning is also where we define the test completion criteria. Completion criteria are how we know when testing is finished. Control, on the other hand, is what we do when the activities do not match up with the plans. It is the ongoing activity where we compare the progress against the plan. As progress takes place, we may need to adjust plans to meet the targets, if this is possible. Therefore we need to undertake both planning and control throughout the testing activities. We plan at the outset, but as testing progresses, undertake monitoring and control activities (monitoring to measure what has happened, control to adjust future activities in the light of experience). Monitoring and control feed back into the continual activity of planning.
Test Analysis and Design
Analysis and design are concerned with the fine detail of what to test (test conditions), and how to combine test conditions into test cases, so that a small number of test cases can cover as many of the test conditions as possible. The analysis and design stage is the bridge between planning and test execution. It is looking backward to the planning (schedules, people, what is going to be tested) and forward to the execution activity (test expected results, what environment will be needed).
A part of the design process needs to consider the test data that will be required for the test conditions and test cases that have been drawn up.
Test design involves predicting how the software under test should behave in a given set of circumstances. Sometimes the expected outcome of a test is trivial: when ordering books from an online book retailer, for instance, under no circumstances should money be refunded to the customer's card without intervention from a supervisor. If we do not detail expected outcomes before starting test execution, there is a real danger that we will miss the one item of detail that is vital, but wrong.
The main points of this activity are as follows:
- Reviewing requirements, architecture, design, interface specifications and other parts, which collectively comprise the test basis.
- Analyzing test items, the specification, behavior and structure to identify test conditions and test data required.
- Designing the tests, including assigning priority.
- Determining whether the requirements and the system are testable.
- Detailing what the test environment should look like, and whether there are any infrastructure and tools required.
- Highlighting the test data required for the test conditions and test cases.
- Creating bi-directional traceability between test basis and test cases.
The test implementation and execution activity involves running tests, and this will include where necessary any set-up/tear-down activities for the testing. It will also involve checking the test environment before testing begins. Test execution is the most visible part of testing, but it is not possible without other parts of the fundamental test process. It is not just about running tests. As we have already mentioned, the most important tests need to be run first. How do we know what are the most important tests to run? This is determined during the planning stages, and refined as part of test design.
One important aspect undertaken at this stage is combining test cases into an overall run procedure, so that test time can be utilized efficiently. Here the logical ordering of tests is important so that, where possible, the outcome of one test creates the preconditions for one or more tests that are later in the execution sequence.
As tests are run, their outcome needs to be logged, and a comparison made between expected results and actual results. Whenever there is a discrepancy between the expected and actual results, this needs to be investigated. If necessary a test incident should be raised. Each incident requires investigation, although corrective action will not be necessary in every case.
When anything changes (software, data, installation procedures, user documentation, etc.), we need to do two kinds of testing on the software. First of all, tests should be run to make sure that the problem has been fixed. We also need to make sure that the changes have not broken the software elsewhere. These two types are usually called retesting and regression testing, respectively. In retesting we are looking in fine detail at the changed area of functionality, whereas regression testing should cover all the main functions to ensure that no unintended changes have occurred. On a financial system, we should include end of day/end of month/end of year processing, for example, in a regression test pack.
Test implementation and execution is where the most visible test activities are undertaken, and usually have the following parts:
- Developing and prioritizing test cases, creating test data, writing test procedures and, optionally, preparing test harnesses and writing automated test scripts.
- Collecting test cases into test suites, where tests can be run one after another for efficiency.
- Checking the test environment set-up is correct.
- Running test cases in the determined order. This can be manually or using test execution tools.
- Keeping a log of testing activities, including the outcome (pass/fail) and the versions of software, data, tools and test ware (scripts, etc.).
- Comparing actual results with expected results.
- Reporting discrepancies as incidents with as much information as possible, including if possible causal analysis (code defect, incorrect test specification, test data error or test execution error).
- Where necessary, repeating test activities when changes have been made following incidents raised. This includes re-execution of a test that previously failed in order to confirm a fix (retesting), execution of a corrected test and execution of previously passed tests to check that defects have not been introduced (regression testing).
Remember that exit criteria were defined during test planning and before test execution started. At the end of test execution, the test manager checks to see if these have been met. If the criterion was that there would be 85 per cent statement coverage (i.e. 85 per cent of all executable statements have been executed, and as a result of execution the figure is 75 per cent, there are two possible actions: change the exit criteria, or run more tests. It is possible that even if the preset criteria were met, more tests would be required. Also, writing a test summary for stakeholders would say what was planned, what was achieved, highlight any differences and in particular things that were not tested.
The fourth stage of the fundamental test process, evaluating exit criteria, comprises the following:
- Checking whether the previously determined exit criteria have been met.
- Determining if more tests are needed or if the specified exit criteria need amending.
- Writing up the result of the testing activities for the business sponsors and other stakeholders.
Testing at this stage has finished. Test closure activities concentrate on making sure that everything is tidied away, reports written, defects closed, and those defects deferred for another phase clearly seen to be as such.
At the end of testing, the test closure stage is composed of the following:
- Ensuring that the documentation is in order; what has been delivered is defined (it may be more or less than originally planned), closing incidents and raising changes for future deliveries, documenting that the system has been accepted.
- Closing down and archiving the test environment, test infrastructure and testware used.
- Passing over testware to the maintenance team.
- Writing down the lessons learned from this testing project for the future, and incorporating lessons to improve the testing process (‘testing maturity’).
What are the stages in the fundamental test process (in the correct sequence)?
Briefly compare regression testing and retesting.
When should the expected outcome of a test be defined?
You may follow the complete series of Fundamentals of Testing articles here:
Why a Software Fails?
Keeping Software Test Under Control
What Testing is and What Testing Does
Software Testing Principles
Fundamental Software Test Processes
Psychology of Software Testing
Testers Code of Ethics
ISTQB Sample Questions
To see all articles of ISTQB-ISEB Foundation guide, see here:
Software Testing-ISTQB ISEB Foundation Guide
Post a Comment