To see all articles of ISTQB-ISEB Foundation guide, see here:
Software Testing-ISTQB ISEB Foundation Guide
A review is a systematic examination of a document by one or more people with the main aim of finding and removing errors. Giving a draft document to a colleague to read is the simplest example of a review, and one which can usually yield a larger crop of errors than we would have anticipated.
Reviews can be used to test anything that is written or typed; this can include documents such as requirement specifications, system designs, code, test plans and test cases. Reviews represent the first form of testing that can take place during a software development life cycle, since the documents reviewed are normally ready long before the code has been written. The practice of testing specification documents by reviewing them early on in the life cycle helps to identify defects before they become part of the executable code, and so makes those defects cheaper and easier to remove. The same defect, if found during dynamic test execution, would incur the extra cost of initially creating and testing the defective code, diagnosing the source of the defect, correcting the problem and rewriting the code to eliminate the defect. Reviewing code against development standards can also prevent defects from appearing in test execution, though in this case, as the code has already been written, not all the additional costs and delays are avoided.
Important as cost and time saving are, though, there are also other important benefits of finding defects early in the life cycle, among them the following:
- Development productivity can be improved and timescales reduced because the correction of defects in early work-products will help to ensure that those work-products are clear and unambiguous. This should enable a developer to move more quickly through the process of writing code. Also, if defects are removed before they become executable code there will be fewer errors to find and fix during test execution.
- Testing costs and time can be reduced by removing the main delays in test execution, which arise when defects are found after they have become failures and the tester has to wait for a fix to be delivered. By reviewing the code and removing defects before they become failures the tester can move more swiftly through test execution.
- Reduction in lifetime costs can be achieved because fewer defects in the final software ensure that ongoing support costs will be lower.
- Improved communication results as authors and their peers discuss and refine any ambiguous content discovered during review to ensure that all involved understand exactly what is being delivered.
The types of defects most typically found by reviews are:
- Deviations from standards either internally defined and managed or regulatory/legally defined by Parliament or perhaps a trade organization.
- Requirements defects—for example, the requirements are ambiguous, or there are missing elements.
- Design defects—for example, the design does not match the requirements.
- Insufficient maintainability—for example, the code is too complex to maintain.
- Incorrect interface specifications—for example, the interface specification does not match the design or the receiving or sending interface.
All reviews aim to find defects, but some types of review find certain types of defects more effectively and efficiently than others.
Review Process
Review processes can vary widely in their level of formality, where formality relates to the level of structure and documentation associated with the activity. Some types of review are completely informal, while others are very formal. The decision on the appropriate level of formality for a review is usually based on combinations of the following factors:
- The maturity of the development process: the more mature the process is, the more formal reviews tend to be.
- Legal or regulatory requirements. These are used to govern the software development activities in certain industries, notably in safety-critical areas such as railway signalling, and determine what kinds of review should take place.
- The need for an audit trail. Formal review processes ensure that it is possible to trace backwards throughout the software development life cycle. The level of formality in the types of review used can help to raise the level of audit trail.
Reviews can also have a variety of objectives, where the term ‘review objective’ identifies the main focus for a review. Typical review objectives include:
- Finding defects.
- Gaining understanding.
- Generating discussion.
- Decision making by consensus.
The way a review is conducted will depend on what its specific objective is, so a review aimed primarily at finding defects will be quite different from one that is aimed at gaining understanding of a document.
Basic Review Process
All reviews, formal and informal alike, exhibit the same basic elements of process:
- The document under review is studied by the reviewers.
- Reviewers identify issues or problems and inform the author either verbally or in a documented form, which might be as formal as raising a defect report or as informal as annotating the document under review.
- The author decides on any action to take in response to the comments and updates the document accordingly.
This basic process is always present, but in the more formal reviews it is elaborated to include additional stages and more attention to documentation and measurement.
Activities of a Formal Review
Reviews at the more formal end of the spectrum, such as technical reviews and inspections, share certain characteristics that differentiate them from the less formal reviews, of which walkthroughs are a typical example.
Below figure shows the key stages that characterize formal reviews.
Stages of a formal review
The following list explains the key stages in more detail:
1.
Planning:
- a. Selecting the personnel—ensuring that those selected can and will add value to the process. There is little point in selecting a reviewer who will agree with everything written by the author without question. As a rule of thumb it is best to include some reviewers who are from a different part of the organization, who are known to be ‘picky’, and known to be dissenters.
- b. Reviews, like weddings, are enhanced by including ‘something old, something new, something borrowed, something blue’. In this case ‘something old’ would be an experienced practitioner; ‘something new’ would be a new or inexperienced team member; ‘something borrowed’ would be someone from a different team; ‘something blue’ would be the dissenter who is hard to please. At the earliest stage of the process a review leader must be identified. This is the person who will coordinate all of the review activity.
- c. Allocating roles—each reviewer is given a role to provide them with a unique focus on the document under review. Someone in a tester role might be checking for testability and clarity of definition, while someone in a user role might look for simplicity and a clear relationship to business values. This approach ensures that, although all reviewers are working on the same document, each individual is looking at it from a different perspective.
- d. Defining the entry and exit criteria, especially for the most formal review types (e.g. inspection).
- e. Selecting the parts of documents to be reviewed (not always required; this will depend on the size of the document: a large document may need to be split into smaller parts and each part reviewed by a different person to ensure the whole document is reviewed fully).
2.
Kick-off: distributing documents; explaining the objectives, process and documents to the participants; and checking entry criteria (for more formal review types such as inspections). This can be run as a meeting or simply by sending out the details to the reviewers. The method used will depend on timescales and the volume of information to pass on. A lot of information can be disseminated better by a meeting rather than expecting reviewers to read pages of text.
3.
Review entry criteria: this stage is where the entry criteria agreed earlier are checked to ensure that they have been met, so that the review can continue—this is mainly used in the more formal review types such as inspections.
4.
Individual preparation: work done by each of the participants on their own before the review meeting, which would include reading the source documents, noting potential defects, questions and comments. This is a key task and may actually be time-boxed, e.g. participants may be given two hours to complete the preparation.
5.
Noting incidents: in this stage the potential defects, questions and comments found during individual preparation are logged.
6.
Review meeting: this may include discussion regarding any defects found, or simply just a log of defects found. The more formal review types like inspections will have documented results or minutes. The meeting participants may simply note defects for the author to correct; they might also make recommendations for handling or correcting the defects. The approach taken will have been decided at the kick-off stage so that all participants are aware of what is required of them. The decision as to which approach to take may be based on one or all of the following factors:
- a. Time available (if time is short the meeting may only collect defects).
- b. Requirements of the author (if the author would like help in correcting defects).
- c. Type of review (in an inspection only the collection of defects is allowed—there is never any discussion).
7.
Examine: this includes the recording of the physical meetings or tracking any group electronic communications.
8.
Rework: after a review meeting the author will have a series of defects to correct; correcting the defects is called rework.
9.
Fixing defects: here the author will be fixing defects that were found and agreed as requiring a fix.
10.
Follow-up: the review leader will check that the agreed defects have been addressed and will gather metrics such as how much time was spent on the review and how many defects were found. The review leader will also check the exit criteria (for more formal review types such as inspections) to ensure that they have been met.
11.
Checking exit criteria: at this stage the exit criteria defined at the start of the process are checked to ensure that all exit criteria have been met so that the review can be officially closed as finished.
Roles and Responsibilities
The role of each reviewer is to look at documents belonging to them from their assigned perspective; this may include the use of checklists. For example, a checklist based on a particular perspective (such as user, maintainer, tester or operations) may be used, or a more general checklist (such as typical requirements problems) may be used to identify defects.
In addition to these assigned review roles the review process itself defines specific roles and responsibilities that should be fulfilled for formal reviews. They are:
Manager: the manager decides on what is to be reviewed (if not already defined), ensures there is sufficient time allocated in the project plan for all of the required review activities, and determines if the review objectives have been met. Managers do not normally get involved in the actual review process unless they can add real value, e.g. they have technical knowledge key to the review.
Moderator: the moderator is sometimes known as the review leader. This is the person who leads the review of the document or set of documents, including planning the review, running the meeting, and follow-ups after the meeting. If necessary, the moderator may mediate between the various points of view and is often the person upon whom the success of the review rests. The moderator will also make the final decision as to whether to release an updated document.
Author: The author is the writer or person with chief responsibility for the development of the document(s) to be reviewed. The author will in most instances also take responsibility for fixing any agreed defects.
Reviewers: These are individuals with a specific technical or business background (also called checkers or inspectors) who, after the necessary preparation, identify and describe findings (e.g. defects) in the product under review. As discussed above, reviewers should be chosen to represent different perspectives and roles in the review process and take part in any review meetings.
Scribe (or recorder): The scribe attends the review meeting and documents all of the issues and defects, problems and open points that were identified during the meeting.
An additional role not normally associated with reviews is that of the tester. Testers have a particular role to play in relation to document reviews. In their test analysis role they will be required to analyze a document to enable the development of tests. In analyzing the document they will also review it, e.g. in starting to build end-to-end scenarios they will notice if there is a ‘hole’ in the requirements that will stop the business functioning, such as a process that is missing or some data that is not available at a given point. So effectively a tester can either be formally invited to review a document or may do so by default in carrying out the tester's normal test analysis role.
Types of Review
A single document may be subject to many different review types: for example, an informal review may be carried out before the document is subjected to a technical review or, depending on the level of risk, a technical review or inspection may take place before a walkthrough with a customer.
Figure below shows the different levels of formality by review type.
Formality of reviews
Each type of review has its own defining characteristics. We identify four review types to cover the spectrum of formality. These are usually known as:
1.
Informal review (least formal). Key characteristics:
- a. There is no formal process underpinning the review.
- b. The review may be documented but this is not required; many informal reviews are not documented.
- c. There may be some variations in the usefulness of the review depending on the reviewer, e.g. the reviewer does not have the technical skills but is just available to check quickly and ensure that the document makes sense.
- d. The main purpose is to find defects and this is an inexpensive way to achieve some limited benefit.
- e. The review may be implemented by pair programming (where one programmer reviews the code of the other ‘pair programmer’) or by a technical lead reviewing designs and code.
2.
Walkthrough. Key characteristics:
- a. The meeting is led by the author of the document under review and attended by members of the author's peer group.
- b. Review sessions are open-ended and may vary in practice from quite informal to very formal.
- c. Preparation by reviewers before the walkthrough meeting, production of a review report or a list of findings, and appointment of a scribe who is not the author are all optional components that are sometimes present.
- d. The main purposes are to enable learning about the content of the document under review, to help team members gain an understanding of the content of the document, and to find defects.
- e. Walkthroughs typically explore scenarios, or conduct dry runs of code or process.
3.
Technical review. Key characteristics:
- a. Technical reviews are documented and use a well-defined defect detection process that includes peers and technical experts.
- b. The review is usually performed as a peer review without management participation and is ideally led by a trained moderator who is not the author.
- c. Reviewers prepare for the review meeting, optionally using checklists, and prepare a review report with a list of findings.
- d. Technical reviews may vary in practice from the quite informal to very formal and have a number of purposes, including: discussion, decision making, evaluation of alternatives, finding defects, solving technical problems and checking conformance to specifications and standards.
4.
Inspection (most formal). Key characteristics:
- a. Inspections are led by a trained moderator who is not the author and usually involve peer examination of a document; individual inspectors work within defined roles.
- b. The inspection process is formal, based on rules and checklists, and uses entry and exit criteria.
- c. Pre-meeting preparation is essential, which would include reading of any source documents to ensure consistency.
- d. An inspection report, with a list of findings, is produced, which includes metrics that can be used to aid improvements to the process as well as correcting defects in the document under review.
- e. After the meeting a formal follow-up process is used to ensure that corrective action is completed and timely.
The main purpose is to find defects, and process improvement may be a secondary purpose.
In reality the lines between the review types often get blurred and what is seen as a technical review in one company may be seen as an inspection in another. The above is the ‘classic view’ of reviews. The key for each company is to agree the objectives and benefits of the reviews that they plan to carry out.
Success Factors for Reviews
When measuring the success of a particular review the following suggested success factors should be considered:
- Each review should have a clearly predefined and agreed objective and the right people should be involved to ensure the objective is met. For example, in an inspection each reviewer will have a defined role and therefore needs the experience to fulfil that role; this should include testers as valued reviewers.
- Any defects found are welcomed, and expressed objectively.
- The review should be seen as being conducted within an atmosphere of trust, so that the outcome will not be used for the evaluation of the participants, and that the people issues and psychological aspects are dealt with (e.g. making it a positive experience for the author and all participants).
- Review techniques (both formal and informal) that are suitable to the type and level of software work-products and reviewers (this is especially important in inspections).
- Checklists or roles should be used, where appropriate, to increase effectiveness of defect identification; for example, in an inspection, roles such as data entry clerk or technical architect may be required to review a particular document.
- Management support is essential for a good review process (e.g. by incorporating adequate time for review activities in project schedules).
- There should be an emphasis on learning and process improvement.
Other more quantitative approaches to success measurement could also be used:
- How many defects found.
- Time taken to review/inspect.
- Percentage of project budget used/saved.
In his original paper on the benefits of inspections in 1976, Michael Fagan of IBM, who developed the Fagan Inspection Method, reported a 23 per cent increase in ‘coding productivity alone’ by using inspections. Success can be measured in many ways; however, the key is to keep measuring to ensure success is still being achieved and, more importantly, reported to a wider audience.
You may follow the complete series of
Static Testing articles here:
Background to Static Techniques
Software Testing Reviews Test Process
Static Analysis By Tools
To see all articles of ISTQB-ISEB Foundation guide, see here:
Software Testing-ISTQB ISEB Foundation Guide
