Summary

In this chapter we addressed the central issues involved in test consistency: reliability, dependability, and fit. We argued that these consistency issues arise because there is always some error in test scores. Estimating just how much that error contributes to the scores of examinees revolves around notions of reliability in norm-referenced testing, dependability in Criterion-referenced testing, and fit in item response theory.

NRT reliability was shown to be heavily dependent upon correlational approaches. Thus we necessarily provided a brief explanation of how the Pearson product-moment correlation coefficient is calculated and interpreted. Three basic correlational approaches were then explained: test-retest reliability, equivalent forms reliability, and internal consistency reliabilities (including split-half adjusted by the Spearman-Brown prophecy formula, alpha, K-R20, K-R21, and standard error of measurement).

CRT dependability estimation, which typically focuses on the consistency of test-based decisions about examinees’ absolute knowledge, was classified into two general approaches: the threshold-loss methods and generalizability theory approaches. The threshold-loss methods included the original agreement and kappa coefficients based on two administrations of a CRT, and Subkoviak’s (1980) methods that allowed for estimating the agreement and kappa coefficients from a single administration of a test. The generalizability theory approaches included discussion of the domain score approach, including phi (F) and the confidence interval. The squared-error loss approaches covered Livingston’s statistic, symbolized as k2(XT), and the more effective phi(lambda), symbolized as F(l).

IRT concerns with consistency were conceptualized somewhat differently from those of NRT and CRT. For IRT, the key concept was unidimensionality. In essence, the idea was that a single trait should be measured in each test (or subtest). This was shown to be a useful concept for CRT in that CRT is generally concerned with measuring well-defined domains of homogeneous content. Factor analysis was shown to be one approach for establishing unidimensionality in tests. Several approaches to estimating data-to-model fit were also described.

The chapter ended with a brief discussion of some of the relationships among NRT reliability estimates and CRT dependability indexes in terms of what they mean for quick-down-and-dirty estimates of CRT dependability.