What to Do When Requirements are Nonexistent or Poor?

The following section provides an overview of how to create test cases when "good" requirements do not exist.

Depending on the project and organization, requirements may be very well written and satisfy the requirements quality factors described earlier. On the other hand, it is often the case that requirements are not clear, unambiguous, and present. In this case, other alternatives need to be considered.

Ad Hoc Testing

The Art of Ad Hoc Testing

Ad hoc testing is the least formal of test techniques. It has been criticized because it is not structured. This testing type is most often used as a complement to other types of testing. Ad hoc testing finds a place during the entire testing cycle. Early in the project, ad hoc testing provides breadth to testers' understanding of your program, thus aiding in discovery. In the middle of a project, the data obtained helps set priorities and schedules. As a project nears the ship date, ad hoc testing can be used to examine defect fixes more rigorously, as described earlier.

However, this is also a strength; that is, important things can be found quickly. Ad hoc testing is performed with improvisation in which the tester seeks to find defects with any means that seem appropriate. It is different from regression testing, which looks for a specific issue with detailed reproducible steps, with a clear expected result.

Ad hoc testing is in many ways similar to jazz improvisation. Jazz musicians sometimes use a fake book consisting of lead sheets for the songs on which they will improvise. After playing the recognizable melody once, the musicians take turns playing extemporaneous solos. Sometimes they will also vary the rhythm of the piece while improvising; for example, by playing behind the beat. These improvisational solos may be recognizable as related to the original tune, or they may not. However, toward the end of the song, the players typically return to the original melody.

There is a parallel to software testing. Testers often start with a documented test design that systematically describes all the cases to be covered. One of the more productive ways to perform improvisational testing is to gather a group of two or more skilled testers in the same room, and ask them to collaborate on extemporaneous testing. The defect-finding power of testers collaborating with improvisational testing is very similar to the power of collaboration exhibited in jazz sessions.

One approach to improvisational testing is to use existing documented tests as the basis, and then invent variations on that theme.

Advantages and Disadvantages of Ad Hoc Testing

One of the best uses of ad hoc testing is for discovery. Reading the requirements or specifications (if they exist) often does not provide a good sense of how a program behaves. Ad hoc testing can find holes in your test strategy, and can expose relationships between subsystems that would otherwise not be apparent. In this way, it serves as a tool for checking the completeness of your testing.

Missing cases may be found that would not otherwise be apparent with formal test cases, as these are set in concrete. Defects found while doing ad hoc testing are often examples of entire classes of forgotten test cases.

Another use for ad hoc testing is to determine the priorities for your other testing activities. Low-level housekeeping functions and basic features often do not make it into the requirements and thus have no associated test cases.

A disadvantage of ad hoc testing is that these forms of tests are not documented and, therefore, not repeatable. This limits ad hoc tests from the regression testing suite.

Exploratory Testing

The Art of Exploratory Testing

Exploratory testing is extra suitable if requirements and specifications are incomplete, or if there is lack of time. The approach can also be used to verify that previous testing has found the most important defects. It is common to perform a combination of exploratory and scripted testing, i.e., a written set of test steps to test software, where the choice is based on risk.

Exploratory testing as a technique for testing computer software does not require significant advanced planning and is tolerant of limited documentation. It relies on the skill and knowledge of the tester to guide the testing, and uses an active feedback loop to guide and calibrate the effort. According to James Bach, "The classical approach to test design, i.e., scripted testing, is like playing ‘20 Questions’ by writing out all the questions in advance."

Exploratory testing is the tactical pursuit of software faults and defects driven by challenging assumptions. It is an approach in software testing with simultaneous learning, test design, and test execution. While the software is being tested, the tester learns things that together with experience and creativity generates new good tests to run.

Exploratory testing has been performed for a long time, and has similarities to ad hoc testing. In the early 1990s, ad hoc was too often synonymous with sloppy and careless work. This new terminology was first published by Cem Kaner in his book Testing Computer Software. Exploratory testing is more structured than classical ad hoc testing and can be as disciplined as any other intellectual activity.

Exploratory testing seeks to find out how the software actually works, and to ask questions about how it will handle difficult and easy cases. The testing is dependent on the tester's skill of inventing test cases and finding defects. The more the tester knows about the product and different test methods, the better the testing will be.

When performing exploratory testing, there are no exact expected results; it is the tester who decides what will be verified, critically investigating the correctness of the result.

In reality, testing almost always is a combination of exploratory and scripted testing, but with a tendency toward either one, depending on the context.

According to Cem Kaner and James Bach, exploratory testing is more a [mindset] or "… a way of thinking about testing" than a methodology. The documentation of exploratory testing ranges from documenting all tests performed to documenting just the bugs.

Exploratory Testing Process

The basic steps of exploratory testing are as follows:

1.Identify the purpose of the product.
2.Identify functions.
3.Identify areas of potential instability.
4.Test each function and record problems.
5.Design and record a consistency verification test.

According to James Bach, "Exploratory Testing, as I practice it, usually proceeds according to a conscious plan. But not a rigorous plan… it is not scripted in detail. To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing. We become more exploratory when we can't tell what tests should be run in advance of the test cycle."

Test cases themselves are not preplanned:

* Exploratory testing can be concurrent with product development and test execution.
* Such testing is based on implicit and explicit (if they exist) specifications as well as the "as-built" product.
* Exploratory testing starts with a conjecture as to correct behavior, followed by exploration for evidence that it works/does not work.
* It is based on some kind of mental model.
* "Try it and see if it works."

Advantages and Disadvantages of Exploratory Testing

The main advantage of exploratory testing is that less preparation is needed, important bugs are found fast, and it is more intellectually stimulating than scripted testing.

Another major benefit is that testers can use deductive reasoning based on the results of previous tests to guide their future testing on the fly. They do not have to complete a current series of scripted tests before focusing in on or moving on to exploring a more target-rich environment. This also accelerates bug detection when used intelligently.

Another benefit is that, after initial testing, most bugs are discovered by some kind of exploratory testing. This can be demonstrated logically by stating, "Programs that pass certain tests tend to continue to pass the same tests and are more likely to fail other tests or scenarios that are yet to be explored."

Disadvantages are that the tests cannot be reviewed in advance (and thus cannot prevent errors in code and test cases), and that it can be difficult to show exactly which tests have been run.

When repeating exploratory tests, they will not be performed in precisely the same manner, which can be a disadvantage if it is more important to know what exact functionality.