What is a Test Tool?

To see all articles of ISTQB-ISEB Foundation guide, see here:

Software Testing-ISTQB ISEB Foundation Guide

The ISTQB Glossary of Testing Terms defines a test tool as:

A software product that supports one or more test activities, such as planning and control, specification, building initial files and data, test execution and test analysis.

Therefore a test tool can be thought of as a piece of software that is used to make the testing process more effective or efficient. In other words, anything that makes testing easier, quicker, more accurate, etc.

This article will focus on those test tools that are listed in the ISTQB syllabus. These are, in general, the test tools that are most commonly used in the testing process and designed primarily for the testing process.

Benefits and Risks of Using Any Type of Tool

Let us consider the building of a new hotel and examine the similarities with the introduction and use of test tools. Test tools need to be thought of as long-term investments that need maintenance to provide long-term benefits. Similarly, building a hotel requires a lot of upfront planning, effort and investment. Even when the hotel is ready for use, there is still a continual long-term requirement for the provision of services such as catering, cleaning, building maintenance, provision of staff to provide adhoc services to customers, etc. The long-term benefit is that this upfront investment and ongoing maintenance and support can provide substantial income in return.

In addition, there are risks that over a period of time, the location of the hotel will become less attractive, resulting in lower demand, lower usage and a maintenance cost that is greater than the income received. Therefore the initial investment is wasted because the ongoing need/requirement did not exist.

The graph in below Figure demonstrates a typical payback model for implementing a test execution tool. The same principle applies to the majority of test tools. Note that there is an ongoing maintenance cost of using the tool, but that this ongoing maintenance cost needs to be less than the cost of performing testing activities without the tool if the investment is to be worthwhile.

Test tool payback model

The same advantages and disadvantages apply to the use of most types of test tool. However, there are exceptions to this generalization (and to the same generalization made in the ISTQB syllabus). Some tools, such as comparators, can be used virtually straight out of the box. A comparator can check whether one large test file is the same as another. If it is different it can identify and report upon the differences. This would be very difficult and time-consuming to do manually. In addition, incident management tools are fairly intuitive and easy for both experienced and novice testers to use. They are also likely to provide a ‘quick win’.

Other tools can be built by developers in-house as the need arises. For instance, test harnesses, test oracles or test data preparation tools may be relatively easy to produce for developers with a good understanding of the tool requirements and the systems and databases in the test environment.

Benefits

The main benefit of using test tools is similar to the main benefit of automating any process. That is, the amount of time and effort spent performing routine, mundane, repetitive tasks is greatly reduced. For example, consider the time and cost of making consumer goods by hand or in a factory.

This time saved can be used to reduce the costs of testing or it can be used to allow testers to spend more time on the more intellectual tasks of test planning, analysis and design. In turn, this can enable more focused and appropriate testing to be done—rather than having many testers working long hours, running hundreds of tests.

Related to this is the fact that the automation of any process usually results in more predictable and consistent results. Similarly, the use of test tools provides more predictable and consistent results as human failings such as manual keying errors, misunderstandings, incorrect assumptions, forgetfulness, etc., are eliminated. It also means that any reports or findings tend to be objective rather than subjective. For instance, humans often assume that something that seems reasonable is correct, when in fact it may not be what the system is supposed to do.

The widespread use of databases to hold the data input, processed or captured by the test tool, means that it is generally much easier and quicker to obtain and present accurate test management information, such as test progress, incidents found/fixed, etc.

Risks

Most of the risks associated with the use of test tools are concerned with over-optimistic expectations of what the tool can do and a lack of appreciation of the effort required to implement and obtain the benefits that the tool can bring.

For example, consider the production environments of most organizations considering using test tools. They are unlikely to have been designed and built with test tools in mind. Therefore, assuming that you want a test environment to be a copy of production (or at least as close to it as possible), you will also have a test environment that is not designed and built with test tools in mind.

Consider the test environments used by vendors to demonstrate their test tools. If you were the vendor would you design the environment to enable you to demonstrate the tool at its best or to demonstrate the shortcomings it may encounter in a typical test environment?

Therefore, unless detailed analysis and evaluation is done, it is likely that test tools will end up as something that seemed a good idea at the time but have been largely a waste of time and money.

After a test tool has been implemented and measurable benefits are being achieved, it is important to put in sufficient effort to maintain the tool, the processes surrounding it and the test environment in which it is used. Otherwise there is a risk that the benefits being obtained will decrease and the tool will become redundant. Additionally, opportunities for improving the way in which the tool is used could also be missed.

For example, the acquisition of various test tools from multiple vendors will require interfaces to be built or configured to import and export data between tools. Otherwise much time may be spent manually cutting and pasting data from one tool to another. If this is not done, then data inconsistencies and version control problems are likely to arise. Similar problems may arise when testing with third-party suppliers or as a result of mergers and acquisitions.

Maintenance effort will also be required to upgrade and re-configure tools so that they remain compliant with new platforms or operating systems.

An example of a hotel chain with several UK-based hotels will be used throughout this chapter. The systems that comprise the organization's system architecture are shown in below figure.

The general public can book rooms at any of the chain's hotels by:

• Contacting staff in the hotel, who then use a GUI front-end to make the booking.
• Telephoning customer services who then use a GUI front-end to make the booking.
• Using the company's website to make the booking online.

In all cases, communication with the mainframe computer is done via a middle-ware layer of XML messages.

There is a document production system that produces paper and electronic versions of customer correspondence such as booking confirmations, bills, invoices, etc.

Direct debit and credit card payments are made via BACS. Files are transmitted and confirmation and error messages are received back.

Validation of bank account details is performed by sending XML messages to and from a third-party system.

Validation and enquiry of address and postcode is also performed by sending XML messages to and from a third-party system.

A new release of the system is planned for six months' time. This will include:

• Code changes to improve performance in the XML middle-ware layer and on the mainframe. Mainframe changes will be performed by an outsourced development team in India.
• Various changes to website screens to improve usability.
• The introduction of a new third-party calendar object from which dates can be selected.
• The ability for customers to pay by cheque.
• The automatic production of cheques for refunds, cancellations, etc.
• An amended customer bill plus two other amended documents.
• Two new output documents.
• Fixes to various existing low- and medium-severity defects.

To check your understanding, I would again like to ask you some questions:

Would you expect a quick return on your investment in test tools? Why?
Describe three potential benefits of using test tools.
Describe two risks of using test tools.

You may follow the complete series of Tool Support for Testing articles here:

What is a Test Tool?
Introducing Test Tool In Organization

To see all articles of ISTQB-ISEB Foundation guide, see here:

Software Testing-ISTQB ISEB Foundation Guide