Introduction to Software Testing

Regression testing is the process of re-testing software that has been modified. Regression testing constitutes the vast majority of testing effort in commercial software development and is an essential part of any viable software development process. Large components or systems tend to have large regression test sets. Even though many developers do not want to believe it, (even when faced with indisputable evidence!), small changes to one part of a system often cause problems in distant parts of the system. Regression testing is used to find this kind of problem. In practice, regression testing is often incorporated into a continuous integration service, as discussed in Chapter 4.

It is worth emphasizing that regression tests must be automated. Indeed, it could be said that unautomated regression testing is equivalent to no regression testing. A wide variety of commercially available tools are available. The current most common are JUnit and its derivative cousins (HTMLUnit, Selenium, SUnit, CUnit, TestNG, NUnit, CPPUnit, PHPUnit, etc.).

Capture/replay tools automate testing of programs that use graphical user interfaces. Version control software, already in use to manage different versions of a given system, effectively manages the test sets associated with each version. Scripting software manages the process of obtaining test inputs, executing the software, marshaling the outputs, comparing the actual and expected outputs, and generating test reports.

The aim of this section is to explain what kinds of tests should be in a regression test set, which regression tests to run, and how to respond to regression tests that fail. We treat each of these issues in turn. We direct the reader interested in detail on any of these topics to the bibliographic notes.

The test engineer faces a Goldilocks problem1 in determining which tests to include in a regression test set. Including every test set possible results in a regression test set that is too large to manage. The result is that the test set cannot be run as often as changes are made to the software. For traditional development processes, this period often amounts to a day; regression tests run at night to evaluate software changed that day, with developers reviewing the results the following morning. For agile development processes the period is significantly shorter, for reasons discussed in Chapter 4.