Introduction to Software Testing

Chapter 3 covered the foundations for how to automate tests. A key requirement of an automated test is that the test must encode the expected results, commonly known as the test oracle. When tests are run by hand, the expected results could be explicitly written in the textual test script or simply left up to the human tester to decide at run time. But an automated test must have that knowledge encoded as an explicit check such as with a JUnit assertion.

Most of this book is concerned with designing effective test input values, but it turns out that many mistakes can and are made with test oracles. Test oracles must also exhibit effective revealability by striking a balance between checking too much (unnecessary cost) and checking too little (perhaps not revealing failures). Test oracles must solve problems with observability to obtain values that may not be readily available to the test script.

This chapter first addresses the revealability problem by presenting knowledge on what values should be checked by an oracle in section 14.1. Then section 14.2 presents ideas on how to determine what the correct results are.

WHAT SHOULD BE CHECKED?

When tests used to be run by hand, the human tester sat at the computer and could immediately observe the results. The tester would use his or her judgment to directly decide whether the software behaved correctly, based on a well-informed understanding of the software requirements. If the tester could not see not enough outputs to make a confident decision about whether the software's behavior was correct, a tester with modest programming skills could improve observability by adding additional print statements, much like debugging. Naturally, testers sometimes missed failures, and this became one of the motivations for test automation. The idea is that an automated test does not blink, so is less likely to miss a failure when it occurs.

Although a clear win for software quality, automation also meant that the test designer had to decide correct behavior for each test before the test was run, and then encode this correct behavior into the automated test. Chapter 3 introduced JUnit. It and its “xUnit” cousins are widely used in both academia and industry, thus we will use its oracle mechanism as a model.