Standardized tests, whether to evaluate student performance in coursework, to choose among applicants for college admission, or to license candidates for various professions, are often marathons. Tests designed to evaluate knowledge of coursework typically use the canonical hour, admissions tests are usually two to three hours, and licensing exams can take days. Why are they as long as they are? The first answer that jumps immediately to mind is the inexorable relationship between a test's length and its reliability. And so to know how long a test should be we must first ask, “How much reliability is needed?” Which brings us to Henny Youngman's quip, compared to what? There is a long answer to this, but let us begin with a short one, “a shorter test.”

Although a test score always gets more reliable as the test generating it gets longer, ceteris paribus, the law of diminishing returns sets in very quickly and so the marginal gain in reliability with increases in test length gets small in a big hurry. In Figure 17.1 we show the reliability of a typical professionally prepared test as a function of its length. It shows that the marginal gain of moving from a thirty-item test to a sixty- or even ninety-item one is not worth the trouble unless such additional reliability is required. As we will show shortly, it is hard to think of a situation where that would be the case, so why are tests as long as they are? What is to be gained through the excessive length? What is lost?

A Clarifying Example – the U.S. Census

Our intuitions can be clarified with an example, the Decennial U.S. Census. On midnight of January 1, 2010, the year of the last Census, it was estimated that there were 308,745,538 souls within the borders of the United States. The error bounds were ± 31,000. The budget of the 2010 Census was $13 billion, or approximately $42 per person. Is it worth this amount of money to just get a single number? Before answering this question, note that a single clerk, with access to an adding machine in a minute or two, could estimate the change from the previous Census (to an accuracy of ±0.1 percent).