Levels and Tasks

We depict the relationship between concepts and observations in terms of four levels, as shown in Figure 16.1. At the broadest level is the background concept, which encompasses the constellation of potentially diverse meanings associated with a given concept. Next is the systematized concept, the specific formulation of a concept adopted by a particular researcher or group of researchers. It is usually formulated in terms of an explicit definition. At the third level are indicators, which are also routinely called measures. This level includes any systematic scoring procedure, ranging from simple measures to complex aggregated indexes. It encompasses not only quantitative indicators but also the classification procedures employed in qualitative research. At the fourth level are scores for cases, which include both numerical scores and the results of qualitative classification.

Figure 16.1

Conceptualization measurement: levels and tasks.

Figure 16.1 Long description

Level 1 labeled “background concept” as an arrow pointing downwards to the task “conceptualization” that then connects to level 2 labeled “ systematized concept”. Level 2 has an arrow pointing to the task “operationalization” that connects to level 3 labeled “indicators”. Level 3 has an arrow pointing to the task labeled “scoring cases” that connects to level 4 labeled “scores for cases”. Now going upwards, level 4 points to the task labeled “ refining indicators” that connects to level 3. Level 3 then points to the task “modifying systematized” that connects upward to level 2. Level 2 then uses the task “revisiting background concept” to connect back to level 1.

Downward and upward movement in Figure 16.1 can be understood as a series of research tasks. On the left-hand side, conceptualization is the movement from the background concept to the systematized concept. Operationalization moves from the systematized concept to indicators, and the scoring of cases applies indicators to produce scores. Moving up on the righthand side, indicators may be refined in light of scores, and systematized concepts may be fine-tuned in light of knowledge about scores and indicators. Insights derived from these levels may lead to revisiting the background concept, which may include assessing alternative formulations of the theory in which a particular systematized concept is embedded. Finally, to define a key overarching term, “measurement” involves the interaction among levels 2 to 4.

Defining Measurement Validity

Valid measurement is achieved when scores (including the results of qualitative classification) meaningfully capture the ideas contained in the corresponding concept. This definition parallels that of Bollen (Reference Bollen1989, 184), who treats validity as “concerned with whether a variable measures what it is supposed to measure.” King, Keohane, and Verba (Reference King, Keohane and Verba1994: 25) give essentially the same definition.

If the idea of measurement validity is to do serious methodological work, however, its focus must be further specified, as emphasized by Bollen (Reference Bollen1989: 197). Our specification involves both ends of the connection between concepts and scores shown in Figure 16.1. At the concept end, our basic point (explored in detail later) is that measurement validation should focus on the relation between observations and the systematized concept; any potential disputes about the background concept should be set aside as an important but separate issue. With regard to scores, an obvious but crucial point must be stressed: Scores are never examined in isolation; rather, they are interpreted and given meaning in relation to the systematized concept.

In sum, measurement is valid when the scores (level 4 in Figure 16.1), derived from a given indicator (level 3), can meaningfully be interpreted in terms of the systematized concept (level 2) that the indicator seeks to operationalize. It would be cumbersome to refer repeatedly to all these elements, but the appropriate focus of measurement validation is on the conjunction of these components.

Measurement Error, Reliability, and Validity

Validity is often discussed in connection with measurement error and reliability. Measurement error may be systematic – in which case it is called bias – or random. Random error, which occurs when repeated applications of a given measurement procedure yield inconsistent results, is conventionally labeled a problem of reliability. Methodologists offer two accounts of the relation between reliability and validity. (1) Validity is sometimes understood as exclusively involving bias, that is error that takes a consistent direction or form. From this perspective, validity involves systematic error, whereas reliability involves random error (Carmines and Zeller Reference Carmines and Zeller1979: 14–15; see also Babbie Reference Babbie2001: 144–45). Therefore, unreliable scores may still be correct “on average” and in this sense valid. (2) Alternatively, some scholars hesitate to view scores as valid if they contain large amounts of random error. They believe validity requires the absence of both types of error. Therefore, they view reliability as a necessary but not sufficient condition of measurement validity (Kirk and Miller Reference Kirk and Miller1986: 20; Shively Reference Shively1998: 45).

Our goal is not to adjudicate between these accounts but to state them clearly and to specify our own focus, namely, the systematic error that arises when the links among systematized concepts, indicators, and scores are poorly developed. This involves validity in the first sense stated earlier. Of course, the random error that routinely arises in scoring cases is also important, but it is not our primary concern.

A final point should be emphasized. Because error is a pervasive threat to measurement, it is essential to view the interpretations of scores in relation to systematized concepts as falsifiable claims (Messick Reference Messick and Linn1989: 13–14). Scholars should treat these claims just as they would any casual hypothesis, that is, as tentative statements that require supporting evidence. Validity assessment is the search for this evidence.

Conceptual Choices: Forming the Systematized Concept

We view systematized concepts as the point of departure for assessing measurement validity. How do scholars form such concepts? Because background concepts routinely include a variety of meanings, the formation of systematized concepts often involves choosing among them. The number of feasible options varies greatly. At one extreme are concepts such as triangle, which are routinely understood in terms of a single conceptual systematization; at the other extreme are “contested concepts” (Gallie Reference Gallie1956), such as democracy. A careful examination of diverse meanings helps clarify the options, but ultimately choices must be made.

These choices are deeply intertwined with issues of theory, as emphasized in Kaplan’s (Reference Kaplan1964: 53) paradox of conceptualization: “Proper concepts are needed to formulate a good theory, but we need a good theory to arrive at the proper concepts. … The paradox is resolved by a process of approximation: the better our concepts, the better the theory we can formulate with them, and in turn, the better the concepts available for the next, improved theory.” Various examples of this intertwining are explored in recent analyses of important concepts, such as Laitin’s (Reference Laitin2000) treatment of language community and Kurtz’s (Reference Laitin2000) discussion of peasant. Fearon and Laitin’s (Reference Laitin2000) analysis of ethnic conflict, in which they begin with their hypothesis and ask what operationalization is needed to capture the conceptions of ethnic group and ethnic conflict entailed in this hypothesis, further illustrates the interaction of theory and concepts.

In dealing with the choices that arise in establishing the systematized concept, researchers must avoid three common traps. First, they should not misconstrue the flexibility inherent in these choices as suggesting that everything is up for grabs. This is rarely, if ever, the case. In any field of inquiry, scholars commonly associate a matrix of potential meanings with the background concept. This matrix limits the range of plausible options, and the researcher who strays outside it runs the risk of being dismissed or misunderstood. We do not mean to imply that the background concept is entirely fixed. It evolves over time, as new understandings are developed and old ones are revised or fall from use. At a given time, however, the background concept usually provides a relatively stable matrix. It is essential to recognize that a real choice is being made, but it is no less essential to recognize that this is a limited choice.

Second, scholars should avoid claiming too much in defending their choice of a given systematized concept. It is not productive to treat other options as self-evidently ruled out by the background concept. For example, in the controversy over whether democracy versus nondemocracy should be treated as a dichotomy or in terms of gradations, there is too much reliance on claims that the background concept of democracy inherently rules out one approach or the other (Collier and Adcock Reference Collier and Adcock1999: 546–50).Footnote ⁴ It is more productive to recognize that scholars routinely emphasize different aspects of a background concept in developing systematized concepts, each of which is potentially plausible. Rather than make sweeping claims about what the background concept “really” means, scholars should present specific arguments, linked to the goals and context of their research, that justify their particular choices.

A third problem occurs when scholars stop short of providing a fleshed-out account of their systematized concepts. This requires not just a one-sentence definition, but a broader specification of the meaning and entailments of the systematized concept. Within the psychometrics literature, Shepard (Reference Shepard1993: 417) summarizes what is required: “both an internal model of interrelated dimensions or subdomains” of the systematized concept, and “an external model depicting its relationship to other [concepts].” An example is Bollen’s (Reference Bollen1990: 9–12; see also Bollen Reference Bollen1980) treatment of political democracy, which distinguishes the two dimensions of “political rights” and “political liberties,” clarifies these by contrasting them with the dimensions developed by Dahl and explores the relation between them. Bollen further specifies political democracy through contrasts with the concepts of stability and social or economic democracy. In the language of Sartori (Reference Sartori1984: 51–54), this involves clarifying the semantic field.

One consequence of this effort to provide a fleshed-out account may be the recognition that the concept needs to be disaggregated. What begins as a consideration of the internal dimensions or components of a single concept may become a discussion of multiple concepts. In democratic theory an important example is the discussion of majority rule and minority rights, which are variously treated as components of a single overall concept of democracy, as dimensions to be analyzed separately, or as the basis for forming distinct subtypes of democracy (Dahl Reference Dahl1956; Lijphart Reference Lijphart1984; Schmitter and Karl Reference Schmitter, Lynn Karl and Volten1992). This kind of refinement may result from new conceptual and theoretical arguments or from empirical findings of the sort that are the focus of the convergent/discriminant validation procedures discussed later.

Measurement Validity and the Systematized Versus Background Concept

We stated earlier that the systematized concept, rather than the background concept, should be the focus in measurement validation. Consider an example. A researcher may ask: “Is it appropriate that Mexico, prior to the year Reference Laitin2000 (when the previously dominant party handed over power after losing the presidential election), be assigned a score of 5 out of 10 on an indicator of democracy? Does this score really capture how ‘democratic’ Mexico was compared to other countries?” Such a question remains underspecified until we know whether “democratic” refers to a particular systematized concept of democracy, or whether this researcher is concerned more broadly with the background concept of democracy. Scholars who question Mexico’s score should distinguish two issues: (1) a concern about measurement – whether the indicator employed produces scores that can be interpreted as adequately capturing the systematized concept used in a given study and (2) a conceptual concern – whether the systematized concept employed in creating the indicator is appropriate vis-à-vis the background concept of democracy.

We believe validation should focus on the first issue, whereas the second is outside the realm of measurement validity. This distinction seems especially appropriate in view of the large number of contested concepts in political science. The more complex and contested the background concept, the more important it is to distinguish issues of measurement from fundamental conceptual disputes. To pose the question of validity we need a specific conceptual referent against which to assess the adequacy of a given measure. A systematized concept provides that referent. By contrast, if analysts seek to establish measurement validity in relation to a background concept with multiple competing meanings, they may find a different answer to the validity question for each meaning.

By restricting the focus of measurement validation to the systematized concept, we do not suggest that political scientists should ignore basic conceptual issues. Rather, arguments about the background concept and those about validity can be addressed adequately only when each is engaged on its own terms, rather than conflated into one overly broad issue. Consider Schumpeter’s (Reference Schumpeter1947: chap. 21) procedural definition of democracy. This definition explicitly rules out elements of the background concept, such as the concern with substantive policy outcomes, that had been central to what he calls the classical theory of democracy. Although Schumpeter’s conceptualization has been very influential in political science, some scholars (Harding and Petras Reference Harding and Petras1988; Mouffe Reference Mouffe1992) have called for a revised conception that encompasses other concerns, such as social and economic outcomes. This important debate exemplifies the kind of conceptual dispute that should be placed outside the realm of measurement validity.

Recognizing that a given conceptual choice does not involve an issue of measurement validity should not preclude considered arguments about this choice. An example is the argument that minimal definitions can facilitate causal assessment (Linz Reference Linz, Greenstein and Polsby1975: 181–82; Sartori Reference Sartori, Riggs and Teune1975: 34; Karl Reference Karl1990: 1–2; and Alvarez et al. Reference Alvarez, Antonio Cheibub, Limongi and Przeworski1996: 4). For instance, in the debate about a procedural definition of democracy, a pragmatic argument can be made that if analysts wish to study the causal relationship between democracy and socioeconomic equality, then the latter must be excluded from the systematization of the former. The point is that such arguments can effectively justify certain conceptual choices, but they involve issues that are different from the concerns of measurement validation.

Fine-Tuning the Systematized Concept with Friendly Amendments

We define measurement validity as concerned with the relation among scores, indicators, and the systematized concept, but we do not rule out the introduction of new conceptual ideas during the validation process. Key here is the back-and-forth, iterative nature of research emphasized in Figure 16.1. Preliminary empirical work may help in the initial formulation of concepts. Later, even after conceptualization appears complete, the application of a proposed indicator may produce unexpected observations that lead scholars to modify their systematized concepts. These “friendly amendments” occur when a scholar, out of a concern with validity, engages in further conceptual work to suggest refinements or make explicit earlier implicit assumptions. These amendments are friendly because they do not fundamentally challenge a systematized concept but instead push analysts to capture more adequately the ideas contained in it.

A friendly amendment is illustrated by the emergence of the “expanded procedural minimum” definition of democracy (Collier and Levitsky Reference Collier and Levitsky1997: 442–44).Footnote ⁵ Scholars noted that, despite free or relatively free elections, some civilian governments in Central and South America to varying degrees lacked effective power to govern. A basic concern was the persistence of “reserved domains” of military power over which elected governments had little authority (Valenzuela Reference Valenzuela, Mainwaring, O’Donnell and Valenzuela1992: 70). Because procedural definitions of democracy did not explicitly address this issue, measures based upon them could result in a high democracy score for these countries, but it appeared invalid to view them as democratic. Some scholars therefore amended their systematized concept of democracy to add the differentiating attribute that the elected government must to a reasonable degree have the power to rule (Karl Reference Karl1990: 2; Valenzuela Reference Valenzuela, Mainwaring, O’Donnell and Valenzuela1992: 70; Loveman Reference Loveman1994: 108–13). Debate persists over the scoring of specific cases (Rabkin Reference Rabkin1992: 165), but this innovation is widely accepted among scholars in the procedural tradition (Huntington Reference Huntington1991: 10; Markoff Reference Markoff1996: 102–04; Mainwaring, Brinks, and Pérez-Liñán Reference Mainwaring, Brinks and Pérez-Liñán2001). As a result of this friendly amendment, analysts did a better job of capturing, for these new cases, the underlying idea of procedural minimum democracy.

Contextual Specificity in Political Research

Contextual specificity affects many areas of political science. It has long been a problem in cross-national survey research (Sicinski Reference Sicinski1970; Verba Reference Verba and Vallier1971; Verba, Nie, and Kim Reference Verba, Nie and Kim1978: 32–40; Verba et al. Reference Verba, Kelman, Orren, Miyake, Watanuki, Kabashima and Ferree1987: Appendix). An example concerning features of national context is Cain and Ferejohn’s (Reference Cain and Ferejohn1981) discussion of how the differing structure of party systems in the US and Great Britain should be taken into account when comparing party identification. Context is also a concern for survey researchers working within a single nation, who wrestle with the dilemma of “inter-personally incomparable responses” (Brady Reference Brady1985: 269). For example, scholars debate whether a given survey item has the same meaning for different population subgroups – which could be defined, for example, by region, gender, class, or race. One specific concern is whether population subgroups differ systematically in their “response style” (also called “response sets”). Some groups may be more disposed to give extreme answers, and others may tend toward moderate answers (Greenleaf Reference Greenleaf1992). Bachman and O’Malley (Reference Bachman and O’Malley1984) show that response style varies consistently with race. They argue that apparently important differences across racial groups may in part reflect only a different manner of answering questions. Contextual specificity also can be a problem in survey comparisons over time, as Baumgartner and Walker (Reference Baumgartner and Walker1990) point out in discussing group membership in the US.

The issue of contextual specificity of course also arises in macrolevel research in international and comparative studies (Bollen, Entwisle, and Anderson Reference Bollen, Entwisle and Anderson1993: 345). Examples from the field of comparative politics are discussed later. In international relations, attention to context, and particularly a concern with “historicizing the concept of structure,” is central to “constructivism” (Ruggie Reference Ruggie1998: 875). Constructivists argue that modern international relations rest upon “constitutive rules” that differ fundamentally from those of both medieval Christendom and the classical Greek world (873). Although they recognize that sovereignty is an organizing principle applicable across diverse settings, the constructivists emphasize that the “meaning and behavioral implications of this principle vary from one historical context to another” (Reus-Smit Reference Reus-Smit1997: 567). On the other side of this debate, neorealists such as Fischer (Reference Fischer1993: 493) offer a general warning: If pushed to an extreme, the “claim to context dependency” threatens to “make impossible the collective pursuit of empirical knowledge.” He also offers specific historical support for the basic neorealist position that the behavior of actors in international politics follows consistent patterns. Fischer (Reference Fischer1992: 463, 465) concludes that “the structural logic of action under anarchy has the character of an objective law,” which is grounded in “an unchanging essence of human nature.”

The recurring tension in social research between particularizing and universalizing tendencies reflects in part contrasting degrees of concern with contextual specificity. The approaches to establishing equivalence discussed later point to the option of a middle ground. These approaches recognize that contextual differences are important, but they seek to combine this insight with the quest for general knowledge.

The lessons for political science are clear. Any empirical assessment of measurement validity is necessarily based on a particular set of cases, and validity claims should be made, at least initially, with reference to this specific set. To the extent that the set is heterogeneous in ways that may affect measurement validity, it is essential to (1) assess the implications for establishing equivalence across these diverse contexts and, if necessary, (2) adopt context-sensitive measures. Extension to additional cases requires similar procedures.

Establishing Equivalence: Context-Specific Domains of Observation

One important means of establishing equivalence across diverse contexts is careful reasoning, in the initial stages of operationalization, about the specific domains to which a systematized concept applies. Well before thinking about particular scoring procedures, scholars may need to make context-sensitive choices regarding the parts of the broader polity, economy, or society to which they will apply their concept. Equivalent observations may require, in different contexts, a focus on what at a concrete level might be seen as distinct types of phenomena.

Some time ago, Verba (Reference Verba1967) called attention to the importance of context-specific domains of observation. In comparative research on political opposition in stable democracies, a standard focus is on political parties and legislative politics, but Verba (122–23) notes that this may overlook an analytically equivalent form of opposition that crystallizes, in some countries, in the domain of interest group politics. Skocpol (Reference Skocpol1992: 6) makes a parallel argument in questioning the claim that the US was a “welfare laggard” because social provision was not launched on a large scale until the New Deal. This claim is based on the absence of standard welfare programs of the kind that emerged earlier in Europe but fails to recognize the distinctive forms of social provision in the US, such as veterans’ benefits and support for mothers and children. Skocpol argues that the welfare laggard characterization resulted from looking in the wrong place, that is, in the wrong domain of policy.

Locke and Thelen (Reference Locke and Thelen1995, Reference Locke and Thelen1998) have extended this approach in their discussion of “contextualized comparison.” They argue that scholars who study national responses to external pressure for economic decentralization and “flexibilization” routinely focus on the points at which conflict emerges over this economic transformation. Yet these “sticking points” may be located in different parts of the economic and political system. With regard to labor politics in different countries, such conflicts may arise over wage equity, hours of employment, workforce reduction, or shopfloor reorganization. These different domains of labor relations must be examined in order to gather analytically equivalent observations that adequately tap the concept of sticking point. Scholars who look only at wage conflicts run the risk of omitting, for some national contexts, domains of conflict that are highly relevant to the concept they seek to measure.

By allowing the empirical domain to which a systematized concept is applied to vary across the units being compared, analysts may take a productive step toward establishing equivalence among diverse contexts. This practice must be carefully justified, but under some circumstances it can make an important contribution to valid measurement.

Establishing Equivalence: Context-Specific Indicators and Adjusted Common Indicators

Two other ways of establishing equivalence involve careful work at the level of indicators. We will discuss context-specific indicators,Footnote ⁶ and what we call adjusted common indicators. In this second approach, the same indicator is applied to all cases but is weighted to compensate for contextual differences.

An example of context-specific indicators is found in Nie, Powell, and Prewitt’s (Reference Nie, Bingham Powell and Prewitt1969: 377) five-country study of political participation. For all the countries, they analyze four relatively standard attributes of participation. Regarding a fifth attribute – membership in a political party – they observe that in four of the countries party membership has a roughly equivalent meaning, but in the US it has a different form and meaning. The authors conclude that involvement in US electoral campaigns reflects an equivalent form of political participation. Nie, Powell, and Prewitt thus focus on a context-specific domain of observation (the procedure just discussed) by shifting their attention, for the US context, from party membership to campaign participation. They then take the further step of incorporating within their overall index of political participation context-specific indicators that for each case generate a score for what they see as the appropriate domain. Specifically, the overall index for the US includes a measure of campaign participation rather than party membership.

A different example of context-specific indicators is found in comparative-historical research, in the effort to establish a meaningful threshold for the onset of democracy in the nineteenth and early twentieth centuries, as opposed to the late twentieth century. This effort in turn lays a foundation for the comparative analysis of transitions to democracy. One problem in establishing equivalence across these two eras lies in the fact that the plausible agenda of “full” democratization has changed dramatically over time. “Full” by the standards of an earlier period is incomplete by later standards. For example, by the late twentieth century, universal suffrage and the protection of civil rights for the entire national population had come to be considered essential features of democracy, but in the nineteenth century they were not (Huntington Reference Huntington1991: 7, 16). Yet if the more recent standard is applied to the earlier period, cases are eliminated that have long been considered classic examples of nascent democratization in Europe. One solution is to compare regimes with respect to a systematized concept of full democratization that is operationalized according to the norms of the respective periods (Russett Reference Russett1993: 15; R. Collier Reference Collier1999: chap. 1; see also Johnson Reference Johnson1999: 118). Thus, a different scoring procedure – a context-specific indicator – is employed in each period in order to produce scores that are comparable with respect to this systematized concept.Footnote ⁷

Adjusted common indicators are another way to establish equivalence. An example is found in Moene and Wallerstein’s (Reference Laitin2000) quantitative study of social policy in advanced industrial societies, which focuses specifically on public social expenditures for individuals outside the labor force. One component of their measure is public spending on health care. The authors argue that in the US such health care expenditures largely target those who are not members of the labor force. By contrast, in other countries health expenditures are allocated without respect to employment status. Because US policy is distinctive, the authors multiply health care expenditures in the other countries by a coefficient that lowers their scores on this variable. Their scores are thereby made roughly equivalent – as part of a measure of public expenditures on individuals outside the labor force – to the US score. A parallel effort to establish equivalence in the analysis of economic indicators is provided by Zeitsch, Lawrence, and Salemian (Reference Zeitsch, Lawrence and Salernian1994: 169), who use an adjustment technique in estimating total factor productivity to take account of the different operating environments, and hence the different context, of the industries they compare. Expressing indicators in per capita terms is also an example of adjusting indicators in light of context. Overall, this practice is used to address both very specific problems of equivalence, as with the Moene and Wallerstein example, as well as more familiar concerns, such as standardizing by population.

Context-specific indicators and adjusted common indicators are not always a step forward, and some scholars have self-consciously avoided them. The use of such indicators should match the analytic goal of the researcher. For example, many who study democratization in the late twentieth century deliberately adopt a minimum definition of democracy in order to concentrate on a limited set of formal procedures. They do this out of a conviction that these formal procedures are important, even though they may have different meanings in particular settings. Even a scholar such as O’Donnell (Reference O’Donnell1993: 1355), who has devoted great attention to contextualizing the meaning of democracy, insists on the importance of also retaining a minimal definition of “political democracy” that focuses on basic formal procedures. Thus, for certain purposes, it can be analytically productive to adopt a standard definition that ignores nuances of context and apply the same indicator to all cases.

In conclusion, we note that although Przeworski and Teune’s (Reference Przeworski and Teune1970) and Verba’s arguments about equivalence are well known, issues of contextual specificity and equivalence have not received adequate attention in political science. We have identified three tools – context-specific domains of observation, context-specific indicators, and adjusted common indicators – for addressing these issues, and we encourage their wider use. We also advocate greater attention to justifying their use. Claims about the appropriateness of contextual adjustments should not simply be asserted; their validity needs to be carefully defended. Later, we explore three types of validation that may be fruitfully applied in assessing proposals for context-sensitive measurement. In particular, content validation, which focuses on whether operationalization captures the ideas contained in the systematized concept, is central to determining whether and how measurement needs to be adjusted in particular contexts.

Evolving Understandings of Validity

In the psychometric tradition, current thinking about measurement validity developed in two phases. In the first phase, scholars wrote about “types of validity” in a way that often led researchers to treat each type as if it independently established a distinct form of validity. In discussing this literature we follow its terminology by referring to types of “validity.” As noted earlier, in the rest of this chapter we refer instead to types of “validation.”

The first pivotal development in the emergence of a unified approach occurred in the 1950s and 1960s, when a threefold typology of content, criterion, and construct validity was officially established in reaction to the confusion generated by the earlier proliferation of types.Footnote ⁸ Other labels continued to appear in other disciplines, but this typology became an orthodoxy in psychology. A recurring metaphor in that field characterized the three types as “something of a holy trinity representing three different roads to psychometric salvation” (Guion Reference Guion1980: 386). These types may be briefly defined as follows.

1. 1. Content validity assesses the degree to which an indicator represents the universe of content entailed in the systematized concept being measured.

2. 2. Criterion validity assesses whether the scores produced by an indicator are empirically associated with scores for other variables, called criterion variables, which are considered direct measures of the phenomenon of concern.

3. 3. Construct validity has had a range of meanings. One central focus has been on assessing whether a given indicator is empirically associated with other indicators in a way that conforms to theoretical expectations about their interrelationship.

These labels remain very influential and are still the centerpiece in some discussions of measurement validity, as in the latest edition of Babbie’s (Reference Babbie2001: 143–44) widely used methods textbook for undergraduates.

The second phase grew out of increasing dissatisfaction with the “trinity” and led to a “unitarian” approach (Shultz, Riggs, and Kottke Reference Shultz, Riggs and Kottke1998: 269–71). A basic problem identified by Guion (Reference Guion1980: 386) and others was that the threefold typology was too often taken to mean that any one type was sufficient to establish validity (Angoff Reference Angoff, Wainer and Braun1988: 25). Scholars increasingly argued that the different types should be subsumed under a single concept. Hence, to continue with the prior metaphor, the earlier trinity came to be seen “in a monotheistic mode as the three aspects of a unitary psychometric divinity” (25).

Much of the second phase involved a reconceptualization of construct validity and its relation to content and criterion validity. A central argument was that the latter two may each be necessary to establish validity, but neither is sufficient. They should be understood as part of a larger process of validation that integrates “multiple sources of evidence” and requires the combination of “logical argument and empirical evidence” (Shepard Reference Shepard1993: 406). Alongside this development, a reconceptualization of construct validity led to “a more comprehensive and theory-based view that subsumed other more limited perspectives” (Shultz, Riggs, and Kottke Reference Shultz, Riggs and Kottke1998: 270). This broader understanding of construct validity as the overarching goal of a single, integrated process of measurement validation is widely endorsed by psychometricians. Moss (Reference Moss1995: 6) states “there is a close to universal consensus among validity theorists” that “content-and criterion-related evidence of validity are simply two of many types of evidence that support construct validity.”

Thus, in the psychometric literature (e.g., Messick Reference Messick1980: 1015), the term “construct validity” has become essentially a synonym for what we call measurement validity. We have adopted measurement validity as the name for the overall topic of this chapter, in part because in political science the label “construct validity” commonly refers to specific procedures rather than to the general idea of valid measurement. These specific procedures generally do not encompass content validation and have in common the practice of assessing measurement validity by taking as a point of reference established conceptual and/or theoretical relationships. We find it helpful to group these procedures into two types according to the kind of theoretical or conceptual relationship that serves as the point of reference. Specifically, these types are based on the heuristic distinction between description and explanation.Footnote ⁹ First, some procedures rely on “descriptive” expectations concerning whether given attributes are understood as facets of the same phenomenon. This is the focus of what we label “convergent/discriminant validation.” Second, other procedures rely on relatively well-established “explanatory” causal relations as a baseline against which measurement validity is assessed. In labeling this second group of procedures we draw on Campbell’s (Reference Campbell1960: 547) helpful term “nomological” validation, which evokes the idea of assessment in relation to well-established causal hypotheses or lawlike relationships. This second type is often called construct validity in political research (Berry et al. Reference Berry, Ringquist, Footing and Hanson1998; Elkins Reference Elkins2000).Footnote ¹⁰ Out of deference to this usage, the headings and summary statements that follow will refer to nomological/construct validation.

Types of Validation in Political Analysis

A baseline for the revised discussion of validation presented here is provided in work by Carmines and Zeller, and by Bollen. Carmines and Zeller (Reference Carmines and Zeller1979: 26; Zeller and Carmines Reference Zeller and Carmines1980: 78–80) argue that content validation and criterion validation are of limited utility in fields such as political science. While recognizing that content validation is important in psychology and education, they argue that evaluating it “has proved to be exceedingly difficult with respect to measures of the more abstract phenomena that tend to characterize the social sciences” (Carmines and Zeller Reference Carmines and Zeller1979: 22). For criterion validation, these authors emphasize that in many social sciences, few “criterion” variables are available that can serve as “real” measures of the phenomena under investigation, against which scholars can evaluate alternative measures (19–20). Hence, for many purposes it is simply not a relevant procedure. Although Carmines and Zeller call for the use of multiple sources of evidence, their emphasis on the limitations of the first two types of validation leads them to give a predominant role to nomological/construct validation. In relation to Carmines and Zeller, Bollen (Reference Bollen1989: 185–86, 190–94) adds convergent/discriminant validation to their three types and emphasizes content validation, which he sees as both viable and fundamental. He also raises general concerns about correlation-based approaches to convergent and nomological/construct validation, and he offers an alternative approach based on structural equation modeling with latent variables (192–206). Bollen shares the concern of Carmines and Zeller that, for most social research, “true” measures do not exist against which criterion validation can be carried out, so he likewise sees this as a less relevant type (188).

These valuable contributions can be extended in several respects. First, with reference to Carmines and Zeller’s critique of content validation, we recognize that this procedure is harder to use if concepts are abstract and complex. Moreover, it often does not lend itself to the kind of systematic, quantitative analysis routinely applied in some other kinds of validation. Yet, like Bollen (Reference Bollen1989: 185–86, 194), we are convinced it is possible to lay a secure foundation for content validation that will make it a viable, and indeed essential, procedure. Our discussion of this task derives from our distinction between the background and the systematized concept.

Second, we share the conviction of Carmines and Zeller that nomological/construct validation is important, yet given our emphasis on content and convergent/discriminant validation, we do not privilege it to the degree they do. Our discussion will seek to clarify some aspects of how this procedure actually works and will address the skeptical reaction of many scholars to it.

Third, we have a twofold response to the critique of criterion validation as irrelevant to most forms of social research. On the one hand, in some domains criterion validation is important, and this must be recognized. For example, the literature on response validity in survey research seeks to evaluate individual responses to questions, such as whether a person voted in a particular election, by comparing them to official voting records (Clausen Reference Clausen1968; Katosh and Traugott Reference Katosh and Traugott1980; Anderson and Silver Reference Anderson and Silver1986). Similarly, in panel studies it is possible to evaluate the adequacy of “recall” (i.e., whether respondents remember their own earlier opinions, dispositions, and behavior) through comparison with responses in earlier studies (Niemi, Katz, and Newman Reference Niemi, Katz and Newman1980). On the other hand, this is not one of the most generally applicable types of validation, and we favor treating it as one subtype within the broader category of convergent validation. As discussed later, convergent validation compares a given indicator with one or more other indicators of the concept – in which the analyst may or may not have a higher level of confidence. Even if these other indicators are as fallible as the indicator being evaluated, the comparison provides greater leverage than does looking only at one of them in isolation. To the extent that a well-established, direct measure of the phenomenon under study is available, convergent validation is essentially the same as criterion validation. Finally, in contrast both to Carmines and Zeller and to Bollen, we will discuss the application of the different types of validation in qualitative as well as quantitative research, using examples drawn from both traditions. Furthermore, we will employ crucial distinctions already introduced, including the differentiation of levels presented in Figure 16.1, as well as the contrast between specific procedures for validation, as opposed to the overall idea of measurement validity.

Content Validation

Basic Question. In the framework of Figure 16.1, does a given indicator (level 3) adequately capture the full content of the systematized concept (level 2)? This “adequacy of content” is assessed through two further questions. First, are key elements omitted from the indicator? Second, are inappropriate elements included in the indicator?Footnote ¹¹ An examination of the scores (level 4) of specific cases may help answer these questions about the fit between levels 2 and 3.

Discussion. In contrast to the other types considered, content validation is distinctive in its focus on conceptual issues, specifically, on what we have just called adequacy of content. Indeed, it developed historically as a corrective to forms of validation that focused solely on the statistical analysis of scores, and in so doing overlooked important threats to measurement validity (Sireci Reference Sireci1998: 83–87).

Because content validation involves conceptual reasoning, it is imperative to maintain the distinction we made between issues of validation and questions concerning the background concept. If content validation is to be useful, then there must be some ground of conceptual agreement about the phenomena being investigated (Cronbach and Meehl Reference Cronbach and Meehl1955: 282; Bollen Reference Bollen1989: 186). Without it, a well-focused validation question may rapidly become entangled in a broader dispute over the concept. Such agreement can be provided if the systematized concept is taken as given, so attention can be focused on whether a particular indicator adequately captures its content.

Examples of Content Validation. Within the psychometric tradition (Angoff Reference Angoff, Wainer and Braun1988: 27–28; Shultz, Riggs, and Kottke Reference Shultz, Riggs and Kottke1998: 267–68), content validation is understood as focusing on the relationship between the indicator (level 3) and the systematized concept (level 2), without reference to the scores of specific cases (level 4). We will first present examples from political science that adopt this focus. We will then turn to a somewhat different, “case-oriented” procedure (Ragin Reference Ragin1987: chap. 3), identified with qualitative research, in which the examination of scores for specific cases plays a central role in content validation.

Two examples from political research illustrate, respectively, the problems of omission of key elements from the indicator and inclusion of inappropriate elements. Paxton’s (Reference Laitin2000) article on democracy focuses on the first problem. Her analysis is particularly salient for scholars in the qualitative tradition, given its focus on choices about the dichotomous classification of cases. Paxton contrasts the systematized concepts of democracy offered by several prominent scholars – Bollen, Gurr, Huntington, Lipset, and Muller, as well as Rueschemeyer, Stephens, and Stephens – with the actual content of the indicators they propose. She takes their systematized concepts as given, which establishes common conceptual ground. She observes that these scholars include universal suffrage in what is in effect their systematized concept of democracy, but the indicators they employ in operationalizing the concept consider only male suffrage. Paxton thus focuses on the problem that an important component of the systematized concept is omitted from the indicator.

The debate on Vanhanen’s (Reference Vanhanen1979, Reference Vanhanen1990) quantitative indicator of democracy illustrates the alternative problem that the indicator incorporates elements that correspond to a concept other than the systematized concept of concern. Vanhanen seeks to capture the idea of political competition that is part of his systematized concept of democracy by including, as a component of his scale, the percentage of votes won by parties other than the largest party. Bollen (Reference Bollen1990: 13, 15) and Coppedge (Reference Coppedge1997, 6) both question this measure of democracy, arguing that it incorporates elements drawn from a distinct concept, the structure of the party system.

Case-Oriented Content Validation. Researchers engaged in the qualitative classification of cases routinely carry out a somewhat different procedure for content validation, based on the relation between conceptual meaning and choices about scoring particular cases. In the vocabulary of Sartori (Reference Sartori1970: 1040–46), this concerns the relation between the “intension” (meaning) and “extension” (set of positive cases) of the concept. For Sartori, an essential aspect of concept formation is the procedure of adjusting this relation between cases and concept. In the framework of Figure 16.1, this procedure involves revising the indicator (i.e., the scoring procedure) in order to sort cases in a way that better fits conceptual expectations, and potentially fine-tuning the systematized concept to better fit the cases. Ragin (Reference Ragin1994: 98) terms this process of mutual adjustment “double fitting.” This procedure avoids conceptual stretching (Sartori Reference Sartori1970; Collier and Mahon Reference Collier and Mahon1993),Footnote ¹² that is, a mismatch between a systematized concept and the scoring of cases, which is clearly an issue of validity.

An example of case-oriented content validation is found in O’Donnell’s (Reference O’Donnell1996) discussion of democratic consolidation. Some scholars suggest that one indicator of consolidation is the capacity of a democratic regime to withstand severe crises. O’Donnell argues that by this standard, some Latin American democracies would be considered more consolidated than those in southern Europe. He finds this an implausible classification because the standard leads to a “reductio ad absurdum” (43). This example shows how attention to specific cases can spur recognition of dilemmas in the adequacy of content and can be a productive tool in content validation.

In sum, for case-oriented content validation, upward movement in Figure 16.1 is especially important. It can lead to both refining the indicator in light of scores and fine-tuning the systematized concept. In addition, although the systematized concept being measured is usually relatively stable, this form of validation may lead to friendly amendments that modify the systematized concept by drawing ideas from the background concept. To put this another way, in this form of validation both an “inductive” component and conceptual innovation are especially important.

Limitations of Content Validation. Content validation makes an important contribution to the assessment of measurement validity, but alone it is incomplete, for two reasons. First, although a necessary condition, the findings of content validation are not a sufficient condition for establishing validity (Shepard Reference Shepard1993: 414–15; Sireci Reference Sireci1998: 112). The key point is that an indicator with valid content may still produce scores with low overall measurement validity, because further threats to validity can be introduced in the coding of cases. A second reason concerns the trade-off between parsimony and completeness that arises because indicators routinely fail to capture the full content of a systematized concept. Capturing this content may require a complex indicator that is hard to use and adds greatly to the time and cost of completing the research. It is a matter of judgment for scholars to decide when efforts to further improve the adequacy of content may become counterproductive.

It is useful to complement the conceptual criticism of indicators by examining whether particular modifications in an indicator make a difference in the scoring of cases. To the extent that such modifications have little influence on scores, their contribution to improving validity is more modest. An example in which their contribution is shown to be substantial is provided by Paxton (Reference Paxton2000). She develops an alternative indicator of democracy that takes female suffrage into account, compares the scores it produces with those produced by the indicators she originally criticized, and shows that her revised indicator yields substantially different findings. Her content validation argument stands on conceptual grounds alone, but her information about scoring demonstrates the substantive importance of her concerns. This comparison of indicators in a sense introduces us to convergent/discriminant validation, to which we now turn.

Convergent/Discriminant Validation

Basic Question. Are the scores (level 4) produced by alternative indicators (level 3) of a given systematized concept (level 2) empirically associated and thus convergent? Furthermore, do these indicators have a weaker association with indicators of a second, different systematized concept, thus discriminating this second group of indicators from the first? Stronger associations constitute evidence that supports interpreting indicators as measuring the same systematized concept – thus providing convergent validation; whereas weaker associations support the claim that they measure different concepts – thus providing discriminant validation. The special case of convergent validation in which one indicator is taken as a standard of reference, and is used to evaluate one or more alternative indicators, is called criterion validation, as discussed earlier.

Discussion. Carefully defined systematized concepts, and the availability of two or more alternative indicators of these concepts, are the starting point for convergent/discriminant validation. They lay the groundwork for arguments that particular indicators measure the same or different concepts, which in turn create expectations about how the indicators may be empirically associated. To the extent that empirical findings match these “descriptive” expectations, they provide support for validity.

Empirical associations are crucial to convergent/discriminant validation, but they are often simply the point of departure for an iterative process. What initially appears to be negative evidence can spur refinements that ultimately enhance validity. That is, the failure to find expected convergence may encourage a return to the conceptual and logical analysis of indicators, which may lead to their modification. Alternatively, researchers may conclude that divergence suggests the indicators measure different systematized concepts and may reevaluate the conceptualization that led them to expect convergence. This process illustrates the intertwining of convergent and discriminant validation.

Examples of Convergent/Discriminant Validation. Scholars who develop measures of democracy frequently use convergent validation. Thus, analysts who create a new indicator commonly report its correlation with previously established indicators (Bollen Reference Bollen1980: 380–82; Coppedge and Reineke Reference Coppedge and Reinicke1990: 61; Przeworski et al. Reference Przeworski, Alvarez, Antonio Cheibub and Limongi1996: 52; Mainwaring et al. Reference Mainwaring, Brinks and Pérez-Liñán2001: 52). This is a valuable procedure, but it should not be employed atheoretically. Scholars should have specific conceptual reasons for expecting convergence if it is to constitute evidence for validity. Let us suppose a proposed indicator is meant to capture a facet of democracy overlooked by existing measures; then too high a correlation is in fact negative evidence regarding validity, for it suggests that nothing new is being captured.

An example of discriminant validation is provided by Bollen’s (Reference Bollen1980: 373–74) analysis of voter turnout. As in the studies just noted, different measures of democracy are compared, but in this instance the goal is to find empirical support for divergence. Bollen claims, based on content validation, that voter turnout is an indicator of a concept distinct from the systematized concept of political democracy. The low correlation of voter turnout with other proposed indicators of democracy provides discriminant evidence for this claim. Bollen concludes that turnout is best understood as an indicator of political participation, which should be conceptualized as distinct from political democracy.

Although qualitative researchers routinely lack the data necessary for the kind of statistical analysis performed by Bollen, convergent/discriminant validation is by no means irrelevant for them. They often assess whether the scores for alternative indicators converge or diverge. Paxton, in the example discussed earlier, in effect uses discriminant validation when she compares alternative qualitative indicators of democracy in order to show that recommendations derived from her assessment of content validation make a difference empirically. This comparison, based on the assessment of scores, “discriminates” among alternative indicators. Convergent/discriminant validation is also employed when qualitative researchers use a multimethod approach involving “triangulation” among multiple indicators based on different kinds of data sources (Campbell and Fiske Reference Campbell and Fiske1959; Webb et al. Reference Webb, Campbell, Schwartz and Sechrest1966; Brewer and Hunter Reference Brewer and Hunter1989). Orum, Faegin, and Sjoberg (Reference Orum, Feagin, Sjoberg, Feagin, Orum and Sjoberg1991: 19) specifically argue that one of the great strengths of the case study tradition is its use of triangulation for enhancing validity. In general, the basic ideas of convergent/discriminant validation are at work in qualitative research whenever scholars compare alternative indicators.

Concerns about Convergent/Discriminant Validation. A first concern here is that scholars might think that in convergent/discriminant validation empirical findings always dictate conceptual choices. This frames the issue too narrowly. For example, Bollen (Reference Bollen, Entwisle and Anderson1993: 1208–9, 1217) analyzes four indicators that he takes as components of the concept of political liberties and four indicators that he understands as aspects of democratic rule. An examination of Bollen’s covariance matrix reveals that these do not emerge as two separate empirical dimensions. Convergent/discriminant validation, mechanically applied, might lead to a decision to eliminate this conceptual distinction. Bollen does not take that approach. He combines the two clusters of indicators into an overall empirical measure, but he also maintains the conceptual distinction. Given the conceptual congruence between the two sets of indicators and the concepts of political liberties and democratic rule, the standard of content validation is clearly met, and Bollen continues to use these overarching labels.

Another concern arises over the interpretation of low correlations among indicators. Analysts who lack a “true” measure against which to assess validity must base convergent validation on a set of indicators, none of which may be a very good measure of the systematized concept. The result may be low correlations among indicators, even though they have shared variance that measures the concept. One possible solution is to focus on this shared variance, even though the overall correlations are low. Standard statistical techniques may be used to tap this shared variance.

The opposite problem also is a concern: the limitations of inferring validity from a high correlation among indicators. Such a correlation may reflect factors other than valid measurement. For example, two indicators may be strongly correlated because they both measure some other concept; or they may measure different concepts, one of which causes the other. A plausible response is to think through, and seek to rule out, alternative reasons for the high correlation.Footnote ¹³

Although framing these concerns in the language of high and low correlations appears to orient the discussion toward quantitative researchers, qualitative researchers face parallel issues. Specifically, these issues arise when qualitative researchers analyze the sorting of cases produced by alternative classification procedures that represent different ways of operationalizing either a given concept (i.e., convergent validation) or two or more concepts that are presumed to be distinct (i.e., discriminant validation). Given that these scholars are probably working with a small N, they may be able to draw on their knowledge of cases to assess alternative explanations for convergences and divergences among the sorting of cases yielded by different classification procedures. In this way, they can make valuable inferences about validity. Quantitative researchers, by contrast, have other tools for making these inferences, to which we now turn.

Convergent Validation and Structural Equation Models with Latent Variables. In quantitative research, an important means of responding to the limitations of simple correlational procedures for convergent/discriminant validation is offered by structural equation models with latent variables (also called LISREL-type models). Some treatments of such models, to the extent that they discuss measurement error, focus their attention on random error, that is, on reliability (Hayduk Reference Hayduk1987: e.g., 118–24; Reference Hayduk1996).Footnote ¹⁴ However, Bollen has made systematic error, which is the concern of the present chapter, a central focus in his major methodological statement on this approach (Reference Bollen1989: 190–206). He demonstrates, for example, its distinctive contribution for a scholar concerned with convergent/discriminant validation who is dealing with a data set with high correlations among alternative indicators. In this case, structural equations with latent variables can be used to estimate the degree to which these high correlations derive from shared systematic bias, rather than reflect the valid measurement of an underlying concept.Footnote ¹⁵

This approach is illustrated by Bollen (Reference Bollen, Entwisle and Anderson1993) and Bollen and Paxton’s (Reference Bollen and Paxton1998, Reference Laitin2000) evaluation of eight indicators of democracy taken from data sets developed by Banks, Gastil, and Sussman.Footnote ¹⁶ For each indicator, Bollen and Paxton estimate the percent of total variance that validly measures democracy, as opposed to reflecting systematic and random error. The sources of systematic error are then explored. Bollen and Paxton conclude, for example, that Gastil’s indicators have “conservative” bias, giving higher scores to countries that are Catholic, that have traditional monarchies, and that are not Marxist-Leninist (Bollen Reference Bollen, Entwisle and Anderson1993: 1221; Bollen and Paxton Reference Bollen and Paxton2000: 73). This line of research is an outstanding example of the sophisticated use of convergent/discriminant validation to identify potential problems of political bias.

In discussing Bollen’s treatment and application of structural equation models we would like to note both similarities, and a key contrast, in relation to the practice of qualitative researchers. Bollen certainly shares the concern with careful attention to concepts, and with knowledge of cases, that we have already emphasized, and that is characteristic of case-oriented content validation as practiced by qualitative researchers. He insists that complex quantitative techniques cannot replace careful conceptual and theoretical reasoning; rather they presuppose it. Furthermore, “structural equation models are not very helpful if you have little idea about the subject matter” (Bollen Reference Bollen1989: vi, see also 194). Qualitative researchers, carrying out a case by case assessment of the scores on different indicators, could of course reach some of the same conclusions about validity and political bias reached by Bollen. A structural equation approach, however, does offer a fundamentally different procedure that allows scholars to assess carefully the magnitude and sources of measurement error for large numbers of cases and to summarize this assessment systematically and concisely.

Nomological/Construct Validation

Basic Question. In a domain of research in which a given causal hypothesis is reasonably well established, we ask: Is this hypothesis again confirmed when the cases are scored (level 4) with the proposed indicator (level 3) for a systematized concept (level 2) that is one of the variables in the hypothesis? Confirmation is treated as evidence for validity.

Discussion. We should first reiterate that because the term “construct validity” has become synonymous in the psychometric literature with the broader notion of measurement validity, to reduce confusion we use Campbell’s term “nomological” validation for procedures that address this basic question. Yet, given common usage in political science, in headings and summary statements we call this nomological/construct validation. We also propose an acronym that vividly captures the underlying idea: AHEM validation; that is, “Assume the Hypothesis, Evaluate the Measure.”

Nomological validation assesses the performance of indicators in relation to causal hypotheses in order to gain leverage in evaluating measurement validity. Whereas convergent validation focuses on multiple indicators of the same systematized concept, and discriminant validation focuses on indicators of different concepts that stand in a “descriptive” relation to one another, nomological validation focuses on indicators of different concepts that are understood to stand in an explanatory, “causal” relation with one another. Although these contrasts are not sharply presented in most definitions of nomological validation, they are essential in identifying this type as distinct from convergent/discriminant validation. In practice the contrast between description and explanation depends on the researcher’s theoretical framework, but the distinction is fundamental to the contemporary practice of political science.

The underlying idea of nomological validation is that scores which can validly be claimed to measure a systematized concept should fit well-established expectations derived from causal hypotheses that involve this concept. The first step is to take as given a reasonably well-established causal hypothesis, one variable in which corresponds to the systematized concept of concern. The scholar then examines the association of the proposed indicator with indicators of the other concepts in the causal hypothesis. If the assessment produces an association that the causal hypothesis leads us to expect, then this is positive evidence for validity.

Nomological validation provides additional leverage in assessing measurement validity. If other types of validation raise concerns about the validity of a given indicator and the scores it produces, the analysts probably do not need to employ nomological validation. When other approaches yield positive evidence, however, then nomological validation is valuable in teasing out potentially important differences that may not be detected by other types of validation. Specifically, alternative indicators of a systematized concept may be strongly correlated and yet perform very differently when employed in causal assessment. Bollen (Reference Bollen1980: 383–84) shows this, for example, in his assessment of whether regime stability should be a component of measures of democracy.

Examples of Nomological/Construct Validation. Lijphart’s (Reference Lijphart1996) analysts of democracy and conflict management in India provides a qualitative example of nomological validation, which he uses to justify his classification of India as a consociational democracy. Lijphart first draws on his systematized concept of consociationalism to identify descriptive criteria for classifying any given case as consociational. He then uses nomological validation to further justify his scoring of India (262–64). Lijphart identifies a series of causal factors that he argues are routinely understood to produce consociational regimes, and he observes that these factors are present in India. Hence, classifying India as consociational is consistent with an established causal relationship, which reinforces the plausibility of his descriptive conclusion that India is a case of consociationalism.

Another qualitative example of nomological validation is found in a classic study in the tradition of comparative-historical analysis, Perry Anderson’s Lineages of the Absolutist State.Footnote ¹⁷ Anderson (Reference Anderson1974: 413–15) is concerned with whether it is appropriate to classify as “feudalism” the political and economic system that emerged in Japan beginning roughly in the fourteenth century, which would place Japan in the same analytic category as European feudalism. His argument is partly descriptive, in that he asserts that “the fundamental resemblance between the two historical configurations as a whole [is] unmistakable” (414). He validates his classification by observing that Japan’s subsequent development, like that of post-feudal Europe, followed an economic trajectory that his theory explains as the historical legacy of a feudal state. “The basic parallelism of the two great experiences of feudalism at the opposite ends of Eurasia, was ultimately to receive its most arresting confirmation of all, in the posterior destiny of each zone” (414). Thus, he uses evidence concerning an expected explanatory relationship to increase confidence in his descriptive characterization of Japan as feudal. Anderson, like Lijphart, thus follows the two-step procedure of making a descriptive claim about one or two cases, and then offering evidence for the validity of this claim by observing that it is consistent with an explanatory claim in which he has confidence.

A quantitative example of nomological validation is found in Elkins’ evaluation of the proposal that democracy versus nondemocracy should be treated as a dichotomy, rather than in terms of gradations. One potential defense of a dichotomous measure is based on convergent validation. Thus, Alvarez and colleagues (Reference Alvarez, Antonio Cheibub, Limongi and Przeworski1996: 21) show that their dichotomous measure is strongly associated with graded measures of democracy. Elkins (Reference Elkins2000: 294–96) goes on to apply nomological validation, exploring whether, notwithstanding this association, the choice of a dichotomous measure makes a difference for causal assessment. He compares tests of the democratic peace hypothesis using both dichotomous and graded measures. According to the hypothesis, democracies are in general as conflict prone as nondemocracies but do not fight one another. The key finding from the standpoint of nomological validation is that this claim is strongly supported using a graded measure, whereas there is no statistically significant support using the dichotomous measure. These findings give nomological evidence for the greater validity of the graded measure because they better fit the overall expectations of the accepted causal hypothesis. Elkins’ approach is certainly more complex than the two-step procedure followed by Lijphart and Anderson, but the basic idea is the same.

Skepticism about Nomological Validation. Many scholars are skeptical about nomological validation. One concern is the potential problem of circularity. If one assumes the hypothesis in order to validate the indicator, then the indicator cannot be used to evaluate the same hypothesis. Hence, it is important to specify that any subsequent hypothesis-testing should involve hypotheses different from those used in nomological validation.

A second concern is that, in addition to taking the hypothesis as given, nomological validation also presupposes the valid measurement of the other systematized concept involved in the hypothesis. Bollen (Reference Bollen1989: 188–90) notes that problems in the measurement of the second indicator can undermine this approach to assessing validity, especially when scholars rely on simple correlational procedures. Obviously, researchers need evidence about the validity of the second indicator. Structural equation models with latent variables offer a quantitative approach to addressing such difficulties because, in addition to evaluating the hypothesis, these models can be specified so as to provide an estimate of the validity of the second indicator. In small-N, qualitative analysis, the researcher has the resource of detailed case knowledge to help evaluate this second indicator. Thus, both qualitative and quantitative researchers have a means for making inferences about whether this important presupposition of nomological validation is indeed met.

A third problem is that, in many domains in which political scientists work, there may not be a sufficiently well-established hypothesis to make this a viable approach to validation. In such domains, it may be plausible to assume the measure and evaluate the hypothesis, but not the other way around. Nomological validation therefore simply may not be viable. Yet it is helpful to recognize that nomological validation need not be restricted to a dichotomous understanding in which the hypothesis either is or is not reconfirmed, using the proposed indicator. Rather, nomological validation may focus, as it does in Elkins (Reference Elkins2000; see also Hill, Hanna, and Shafqat Reference Hill, Hanna and Shafqat1997), on comparing two different indicators of the same systematized concept, and on asking which better fits causal expectations. A tentative hypothesis may not provide an adequate standard for rejecting claims of measurement validity outright, but it may serve as a point of reference for comparing the performance of two indicators and thereby gaining evidence relevant to choosing between them.

Another response to the concern that causal hypotheses may be too tentative a ground for measurement validation is to recognize that neither measurement claims nor causal claims are inherently more epistemologically secure. Both types of claims should be seen as falsifiable hypotheses. To take a causal hypothesis as given for the sake of measurement validation is not to say that the hypothesis is set in stone. It may be subject to critical assessment at a later point. Campbell (1977/Reference Campbell1988: 477) expresses this point metaphorically: “We are like sailors who must repair a rotting ship at sea. We trust the great bulk of the timbers while we replace a particularly weak plank. Each of the timbers we now trust we may in turn replace. The proportion of the planks we are replacing to those we treat as sound must always be small.”

Overview

This method builds on convergent−discriminant validation. It forces the researcher to distinguish between two types of relationships. Thus, associations among indicators are hypothesized to reflect some mix of (1) alternative descriptions of the same underlying concept and/or (2) causal relations among different concepts. These hypotheses are used in constructing a statistical model that is used in estimating descriptive and explanatory parameters (Bollen Reference Bollen1989; Bollen, Rabe-Hesketh, and Skrondal Reference Bollen, Rabe-Hesketh, Skrondal, Box-Steffensmeier, Brady and Collier2008).Footnote ²

SEM-L, often identified with the LISREL software package (Bollen Reference Bollen1989, passim), is central to the comparative democracy literature, given the major contributions of Kenneth Bollen. SEM-L also encompasses various kinds of factor analysis (Kim and Mueller Reference Kim and Mueller1978a, Reference Kim and Mueller1978b; Fabrigar and Wegener Reference Fabrigar and Wegener2011) and item-response theory (Reckase Reference Reckase2009). Econometric discussions of errors-in-variables models (e.g., Greene Reference Greene2000: 375−83) also fall broadly in this tradition.

SEM-L in effect weights indicators to optimally measure the concept of concern, and to deal as effectively as possible with measurement error. When creating a model, scholars make the assumption that the observed data are generated owing to the influence of unobservable latent variables,Footnote ³ which are presumed to reflect the concepts of interest. Thus, in the democracy literature, scholars assume that particular indicators imperfectly reflect an underlying “true” level of democracy as they have conceptualized it. Analysts must make assumptions about (1) the true dimensionality of democracy, (2) which dimension of each latent variable has a measurement relationship with each observed indicator, (3) the error contained in each indicator, and sometimes (4) causal relationships among the latent variables.

Some elements of these four assumptions can be tested by statistical analysis. However, notwithstanding any tests, they basically remain assumptions. Furthermore – and this reflects a dilemma of this tradition – empirical findings are difficult to interpret unless the assumptions are valid.

Structural Equations and Democracy

Efforts to evaluate and improve cross-national measures of democracy with SEM-L have been centrally concerned with evaluating measurement error, as in Shen and Williamson (Reference Shen and Williamson2005). As part of a broader study focused on perceptions of corruption, these authors estimate structural equations that incorporate a measurement model of democracy, allowing them to assess the proportion of measurement error in the Freedom House indicators of political rights, civil liberties, and press freedom. These three measures are treated as indicators of democracy, and each indicator is estimated to have a modest level of error. Shen and Williamson’s analysis thus supports confidence in the Freedom House rankings.

Bollen and Paxton (Reference Bollen and Paxton2000), following closely on Bollen (Reference Bollen1993), offer a different application of SEM-L. Rather than embedding a measurement model of democracy in a larger causal framework, they focus on evaluating the measurement quality of different cross-national democracy indicators. In addition to the Freedom House rankings, they consider the data generated by Arthur Banks (Reference Banks1971, Reference Banks1979). Furthermore, they estimate the threat to validity owing to possible bias introduced by the institutions and authors that created each measure. For example, they conclude that the Freedom House indicators of democracy are biased in favor of Catholic countries and against Marxist ones (74–77), whereas the Banks scores have the opposite bias. In comparison with Shen and Williamson, who estimate measurement error at 7 percent, Bollen and Paxton’s analysis suggests greater error in the Freedom House scores, with about 15 percent to 20 percent of the variance produced by error. The model attempts to mitigate measurement error by creating a weighted average of available indicators.

Treier and Jackman (Reference Treier and Jackman2008) analyze measurement error in the Polity data (Marshall Reference Marshall2013). They use an item-response model, based on a hypothesized unidimensional latent factor of democracy (204–205). The analysis uncovers a substantial amount of measurement error. In an attempt to remedy this error, Pemstein, Meserve, and Melton (Reference Pemstein, Meserve and Melton2010) use similar tools to create an optimal weighting of existing measures – with the goal of generating a new indicator with reduced bias and enhanced reliability. These two studies illustrate the opportunity to gain cumulative insight into measurement error.

Critiques of SEM-L by Methodologists

These excellent applications of SEM-L are undertaken by prominent scholars, yet skepticism about this tradition must be kept clearly in view. Some of the sharpest commentaries were noted in the introduction: This approach is seen as “overwhelming common sense” (Freedman Reference Freedman1987: 102), as a “disaster” (Cliff Reference Cliff1983: 116), and as “pathological science” (Michell Reference Michell2008: 10). More specifically, critics have underscored the plethora of untested, and sometimes untestable, assumptions on which these models depend. What does it mean, as a theoretical or empirical postulate, to assume a specific joint statistical distribution for a collection of unobserved latent variables? What possible evidence could demonstrate that such an assumption is correct or mistaken?Footnote ⁴ Psychometricians have devoted great attention, some of it extremely critical, to the problem of assumptions. Michell (Reference Michell2008) suggests that in his field, “the central hypothesis (that psychological attributes are quantitative) is accepted as true in the absence of supporting evidence … Psychometricians claim to know something that they do not know and have erected barriers preserving their ignorance” (Michell Reference Michell2008: 10).

In fact, findings from the empirical analysis only test central hypotheses about measurement and causation to the extent that the model’s assumptions are accurate. As in any statistical analysis, these findings are basically a product of the model, rather than a test of it. Among key elements of the model are assumptions about unobserved variables: their number, distribution, and dimensionality, and the structure of measurement relations with the observed variables. Without assumptions such as these, the modeling enterprise is impossible.

Item response theory (IRT) attempts to sidestep some of these problems. Yet, despite differences in emphasis and procedure, the two techniques have fundamentally similar assumptions (Takane and de Leeuw Reference Takane and de Leeuw1987; Treier and Jackman Reference Treier and Jackman2008: 205–06; Reckase Reference Reckase2009). Hence, although IRT pays attention to a range of interesting issues neglected in most SEM-L analyses, it does not escape the concerns discussed here.

Notwithstanding these criticisms, structural-equation modeling contributes to measurement in several ways. It (1) brings together the various measurement validation procedures developed by psychometricians, (2) enriches work on measurement by encouraging researchers to focus directly on causal as well as descriptive connections among indicators, (3) provides estimates of random error and bias, and (4) seeks to make causal inferences that are not contaminated by these problems of measurement. It thus gives researchers evidence about the quality of indicators, although the value of that evidence is conditional on the assumptions discussed earlier.

Overview

LoM is centrally concerned with logical restrictions on the statistical techniques appropriate to a given level of measurement. It stems from the long history of work on measurement growing out of the foundational contributions of Stevens (Reference Stevens1946, Reference Stevens and Stevens1951, Reference Stevens1975); for one of many recent summaries, see Gill (Reference Gill2006: 300–04), as well as the “axiomatic” measurement tradition (Krantz, Luce, and Tversky Reference Krantz, Luce and Tversky1971; Suppes et al. Reference Suppes, Krantz, Luce and Tversky1989). Given these presumed restrictions, it is also concerned with methods for “scaling up” – that is, moving to higher LoM – among nominal, ordinal, interval, and ratio data, thereby broadening the range of appropriate statistical techniques.Footnote ⁵ Thus, starting with a nominal scale, we may ask, “What attributes must the categories have, and what analytic techniques can be applied, for researches to treat them as ordinal?” If order is established, what additional criteria must be met to establish a unit of measurement, thereby yielding an interval or ratio scale?

A recurring concern here is with achieving ordinal measurement, which is the goal of Guttman (Reference Guttman, Stouffer, Guttman, Suchman, Lazarsfeld, Star and Clausen1950; Engelhard Reference Engelhard2008) scaling. This technique tests for underlying order, and when such tests are satisfied, scholars can convert nominal categories into an ordinal scale.

Guttman scaling is applied in situations in which a series of criteria – for example, attributes of democracy – are hypothesized to reflect different positions along a presumed dimension. Some of the cases under analysis may meet the more demanding criteria that correspond to higher values on the dimension, whereas others may only meet the less demanding criteria. Guttman scaling posits that if the cases that meet the more demanding criteria also meet the less demanding criteria, then a single, ordered dimension has been established.

A further goal of the LoM tradition, beyond achieving order, is establishing a meaningful measurement unit and therefore additivity, with some theorists going so far as to claim that these are the minimal criteria that must be satisfied for a particular indicator to qualify as measurement (Campbell Reference Campbell and Campbell1920; Kariya and Finkelstein Reference Kariya and Finkelstein2000; Grant Reference Grant2004). Hence, a central concern of this approach has been offering proofs to demonstrate that these criteria are met. Empirical techniques oriented toward these concerns include Rasch measurement models (Andrich Reference Andrich1988; Fischer and Molenaar Reference Fischer and Molenaar1995; Bond and Fox Reference Bond and Fox2012).

LoM and Democracy

A conceptual issue must be addressed before discussing LoM. One finds a pointed debate as to whether democracy versus nondemocracy is inherently dichotomous (Przeworski et al. Reference Przeworski, Alvarez, Cheibub and Limongi2000; Sartori Reference Sartori, Collier and Gerring2009) or continuous (Bollen and Jackman Reference Bollen and Jackman1989).Footnote ⁶ This important discussion concerns the relation between the background concept of democracy and the systematized concept adopted by these authors. As emphasized earlier, disputes of this kind involve important conceptual and normative issues, and they are viewed here as separate from questions of measurement validity.

With regard to LoM and the pursuit of measurement validity, Coppedge and Reinicke (Reference Coppedge and Reinicke1990) seek to move beyond nominal data to create an ordered scale of polyarchy using Guttman scale analysis. On their 1 to 7 index, with 1 the highest democracy score, the ranking of countries is considered cumulative if, for example, all countries that rank as Category 6 possess all the democratic traits of countries in Category 7, as well as additional democratic traits. The same should be true throughout the scale. In fact, Coppedge and Reinicke are able to locate 137 out of 170 countries on the scale. The other thirty-three countries are ambiguous, in that the particular combination of democratic and authoritarian traits does not match this cumulative pattern. Hence, a decision rule for weighting traits is needed to achieve the core LoM goal of establishing a strict ordering. Their Guttman scale is thus only a partial order.

Baker and Koesel (Reference Baker and Koesel2001) extend Coppedge and Reinicke’s (Reference Coppedge and Reinicke1990) approach by generating Guttman scales for four components of polyarchy: elections, free expression, inclusiveness, and balanced government. Using a database of annual scores for Eastern European countries from 1992 to 2000, the authors are able to classify unambiguously nearly all country-years on the first three components they consider, successfully establishing three ordinal scales. For the “balanced government” dimension, the results were more ambiguous, with 68 of 117 country-years falling in mixed categories. Hence, order is not established for that dimension. Unlike Coppedge and Reinicke, Baker and Koesel make no attempt to provide a summary polyarchy score for each country.

From a more qualitative perspective, Munck and Verkuilen (Reference Munck and Verkuilen2002) describe the ordering and aggregation techniques necessary for capturing what they regard as the key defining attributes of democracy. They insist that scholars create scales with the smallest number of categories needed to achieve within-category case equivalence – an emphasis that fits with a LoM focus on establishing meaningful relations of equal–unequal (Munck and Verkuilen Reference Munck and Verkuilen2002: 17; Munck Reference Munck2009: chaps. 2–4).Footnote ⁷ Furthermore, for aggregating indicators of separate regime traits into an overall measure of democracy, these authors argue that “First, the analyst must make explicit the theory concerning the relationship between attributes. Second, the analyst must ensure that there is a correspondence between this theory and the selected aggregation rule, that is, that the aggregation rule is actually the equivalent formal expression of the posited relationship” (Munck and Verkuilen Reference Munck and Verkuilen2002: 24).

The authors thus share the emphasis, described earlier, on transforming subindicators into a final democracy score in a way that preserves order and equality, which are standard concerns of LoM. In a similar spirit, Coppedge and Gerring (Reference Coppedge and Gerring2011; see also Coppedge Reference Coppedge2012) offer an innovative solution to this challenge of arriving at an order- and equality-preserving aggregation rule: publish disaggregated indicators, allowing scholars to adopt the aggregation rule that seems best to them.

Treier and Jackman’s (Reference Treier and Jackman2008) analysis of the Polity indicators, previously discussed in the section “SEM-L,” also draws on the LoM tradition, given that it conveys a warning about treating ordinal data as if it contained equal intervals. Their model estimates the underlying distance among the categories of the ordinal indicators from which the Polity measure is constructed. Based on these estimates, the authors conclude, “We observe from the distances between thresholds [that] many differential increments specified in the Polity calculation are not valid” (16). Some adjacent categories are estimated as being virtually identical in terms of the underlying scale, while others are dramatically distant on that scale. Thus, a more rigorous analysis should stick to the ordinal level of measurement.

Critiques of LoM by Methodologists

The introduction already noted some of the sharpest critiques: For example, LoM yields “bad data analysis and bad science” and is “misleading” (Velleman and Wilkinson Reference Velleman and Wilkinson1993: 70, 72). Tukey’s (Reference Tukey and Jones1961/1986) sharp criticisms are also centrally focused on LoM. He sees close attention to establishing ordinality or equal distances between scores as a waste of time, because standard statistical techniques work quite well even if the indicators used partially fail on these criteria (Baker, Hardyck, and Petrinovich Reference Baker, Hardyck and Petrinovich1966). These concerns, which effectively reject a central premise of LoM, are a centerpiece of the pragmatic approach, which is discussed next.

Overview

Tukey (Reference Tukey and Jones1961/1986), as noted, advocates approaching measurement sensibly rather than puristically, avoiding an “oversimplified and over purified” view. He argues that the latter approach is “dangerous,” and in the spirit of being sensible, his inventory of bad methodological advice includes the sarcastic mandate “don’t think, use statistics” (Tukey Reference Tukey and Jones1961/1986: 202, 243, 244, 247). Refreshingly, he suggests that – in place of the application of rigid standards – “a body of data can guide its own analysis” (Tukey Reference Tukey and Jones1961/1986: 207).

The pragmatic tradition thus posits that analysts should consider standards for good measurement that lie outside the confines of traditional frameworks. These criteria may override the concerns of conventional canons of measurement, and this tradition often works back from a particular application to choices about indicators. Given the wide range of substantive agendas and analytic tools in the social and statistical sciences, this can suggest quite divergent priorities in measurement.

The pragmatic approach plays an important role in quantitative research, and statements about this tradition span many decades. Among the earliest is Lord’s (Reference Lord1953) sharp critique of Stevens’ (Reference Stevens1946, Reference Stevens and Stevens1951) framework of measurement levels and corresponding permissible statistical operations. Lord mocks the idea that a given data set is inherently at a particular level of measurement, stating that “the numbers don’t remember where they came from” (751). Depending on the circumstances, what begins as a nominal scale can meaningfully be treated as ordinal or sometimes even interval.

In subsequent contributions, Tukey (Reference Tukey and Jones1961/1986: 237–43) argues that scientific research must be guided by experience: If a procedure seems to work well for the problem in question, it should be adopted, whether or not it is justified, for example, by a LoM argument. In response to a summary by Luce (Reference Luce1959) of the LoM stance regarding scale types and permissible statistical operations, Tukey pointedly responds that “the limitations discussed by Luce do not control which statistics may ‘sensibly’ be used, but only which ones may ‘puristically’ be used” (244).

Abelson and Tukey (Reference Abelson and Tukey1963) use statistical analysis of a simulated data set to consider the results of treating ordinal data as an interval. This approach seems reasonable, given that correlations between the assigned scores and the “true” scores are mostly high, and this approach appears to make relatively little difference with respect to inferences from the data. Hence, whether there is a viable argument that a particular variable meets the criteria for interval-level measurement, there may be a practical basis for operating at that level of measurement. Such a step will make relatively little difference with respect to inferences from the data.

Multidimensional scaling (MDS) often entails a pragmatic approach. This method seeks to represent similarities and differences among cases in terms of what is usually a two-dimensional space. It is true that models have been developed to justify this technique, and that scaling procedures can sometimes be shown to be statistically consistent (Brady Reference Brady1985). Yet, in general, these scaling procedures are justified more on the basis of the intuitive usefulness and heuristic value of the displays they produce, rather than on a formal statistical model. Indeed, MDS is often carried out in exploratory contexts in which no statistical model whatsoever has been postulated, leaving pragmatic arguments as the only available justification for the procedures. As Kruskal and Wish (Reference Kruskal and Wish1978: 26–27) state, “the ultimate justification is that MDS ‘works’ and is useful.”

The technique called alternating least squares, optimal scaling (Jacoby Reference Jacoby1999; Young Reference Young1981) provides another version of the pragmatic approach. Here, categorical variables are assigned an initial scoring and treated as interval level in regression analysis. The variables are then rescored with the goal of minimizing unexplained variance in the regression. This process is repeated until an optimal scoring is achieved. In effect, categorical variables are converted into interval-level variables by choosing the set of scores that produces the best fit in the regression analysis.Footnote ⁸ The traditional concern with LoM is thus abandoned, and a variant of nomological validation is pushed to an extreme.

Pragmatic Tradition and Democracy

A standard application of the pragmatic approach occurs when authors who have created a new cross-national measure justify this measure based on its high correlation with existing, generally accepted measures – that is, convergent validation. In contrast to SEM-L, no measurement model is posited; instead, convergent validation is adopted as a practical (and often insufficiently theorized) check on the indicator’s validity. For example, Przeworski et al. (Reference Przeworski, Alvarez, Cheibub and Limongi2000) evoke a pragmatic criterion to justify their worldwide indicator of democracy between 1950 and 1990, an indicator based on a dichotomous classification of democracy/nondemocracy. They argue, “in spite of all their conceptual and observational differences, the various approaches yield highly similar classifications of regimes. Hence, there is no reason to think that the results that follow depend on the particular way regimes were classified” (55).Footnote ⁹ Thus, notwithstanding their insistence on theoretical grounds that democracy should be measured in a distinctive way – that is, as a dichotomy – Przeworski et al. consider their measure a success in part based on the pragmatic criterion of correlating with other established indicators that do not use a dichotomy.Footnote ¹⁰ This approach, in a sense, abandons their arguments about a dichotomy.

In justifying their trichotomous measure of democracy, semidemocracy, and nondemocracy for nineteen Latin American countries between 1945 and 1999, Mainwaring, Brinks, and Pérez-Liñán (Reference Mainwaring, Brinks and Pérez-Liñán2001: 48) make a parallel argument. They likewise rely in part on pragmatic validation based on high correlations with other indicators that are not trichotomies, thereby in a sense setting aside their arguments in favor of working with three categories (Mainwaring et al. Reference Mainwaring, Brinks and Pérez-Liñán2001: 53). These authors take this step, even though they have made a very specific argument about why their indicator is different from others. They invoke an additional pragmatic argument as well: “Given our cost and time constraints, it would have been difficult to construct a more fine-grained measure for each country and each year since 1945” (50). Cost and time are certainly pragmatic considerations.

Casper and Tufis (Reference Casper and Tufis2003; see also Cheibub, Ghandi, and Vreeland Reference Cheibub, Ghandi and Vreeland2010) illustrate an alternative form of the pragmatic approach, in which alternative indicators that appear highly similar in fact yield different conclusions in testing hypotheses. These authors use a standard set of independent variables, but introduce different democracy indicators as the dependent variable – which sometimes produces meaningfully different results. For example, “we can see that although primary education is significantly associated with democracy when using Polyarchy [i.e., the Coppedge measure], its significance drops out when using Polity” (198–200). Thus, they use a variant of nomological validation to explore the relative plausibility of alternative indicators.

On the basis of these findings, Casper and Tufis (Reference Casper and Tufis2003) maintain that convergent validation may be an inadequate criterion for establishing that two indicators measure the same concept. Instead, these authors apply the alternative pragmatic criterion that equivalent measures should produce similar causal inferences. Both of these pragmatic criteria – which may be in conflict with one another, as in the present example – must be taken seriously. Yet if the scholar’s goal is viable causal inference, Casper and Tufis’ results would appear to have greater importance.

The pragmatic approach has also been used to provide empirical evidence about the merits of graded, as opposed to dichotomous, measures. Elkins (Reference Elkins2000) uses dichotomous and graded versions of the Przeworski et al. (Reference Przeworski, Alvarez, Cheibub and Limongi2000) democracy indicator as alternative independent variables in regressions predicting the interdemocratic peace and regime stability.Footnote ¹¹ He assesses whether it is possible, using these indicators, to replicate substantive findings about causal relationships that many scholars view as well established in prior literature. In both situations, Elkins finds that the graded measure yields a replication of earlier findings that is more nuanced, and often more statistically compelling. Hence, through nomological validation, Elkins is able to conclude, based on the pragmatic criterion of yielding causal inferences more consistent with prior theory, that the nondichotomous is preferable.

Two other examples of a pragmatic approach are found in Bollen. His 1980 article challenges the inclusion of electoral turnout in the Polyarchy indicator by showing that in this form, Polyarchy is weakly or negatively associated with other indicators of democracy. His analysis is a crucial step in questioning the appropriateness of turnout as a component of the measure. Furthermore, although Bollen (Reference Bollen1993) relies primarily on SEM-L, the analysis in effect crosses over into the pragmatic tradition when he develops a new set of democracy scores for 1980. He uses a pragmatic criterion to justify his decision to reject factor-scoring techniques in constructing his scores, relying instead on the simple average of three existing indicators. He justifies this step by arguing that the simpler technique produces scores that will be more stable from year to year. Thus, what begins as an exemplary study in the tradition of SEM-L turns to a pragmatic criterion to produce a more usable indicator.

Critiques of Pragmatic Tradition by Methodologists

The risk with the pragmatic tradition is that it can devolve into ad hoc treatments of descriptive inference, a lack of systematic attention to measurement, and in the worst scenario, selection of measures because they confirm the hypotheses under investigation. If measurement is subordinated to other agendas, analysts may lose touch with description altogether – and thereby abandon the firmest links to the empirical world. Anyone who has sought to explain to others the idea of nomological validation – to reiterate, the approach that begins with an established explanatory hypothesis and uses it as a benchmark for evaluating a measure – doubtless has more than occasionally encountered intense skepticism. Many measurement specialists react to the pragmatic approach with precisely this skepticism.

Notwithstanding these issues, the pragmatic tradition serves as a useful reminder that any interesting statistical result is worthy of additional exploration, even if the measurement assumptions behind the analysis appear difficult or impossible to justify. To take an example that reflects standard practice in political science, if a regression using an untransformed and potentially undertheorized nominal scale as an independent variable produces a statistically significant slope, then in principle the researcher has discovered a substantive puzzle that merits further exploration.

Overview

According to the case-based tradition, indicators should represent as accurately as possible the analytically relevant details of each case, and such correspondence should be tested via direct evaluation of scores in light of primary and secondary sources. This approach relies on in-depth case studies in constructing data sets (D. Collier Reference Collier1999; Bowman, Lehoucq, and Mahoney Reference Bowman, Lehoucq and Mahoney2005), and it typically takes a broad view of the relevant information (Coppedge Reference Coppedge1999). It goes beyond the other traditions, which obviously also rely on scores based on case knowledge, in gathering far more detailed information about each case and evaluating the correspondence between in-depth knowledge and the score for the case on a given indicator. In case-based analysis, involving qualitative or quantitative cross-national comparison, the indicator may be either a categorical measure or at a higher level of measurement. The score’s failure to adequately reflect the detailed information about the case calls for a recoding of that case, and may raise larger questions about the overall indicator.

In the case-based tradition, measurement error is not typically seen as a random component of an indicator, which, in the structural-equation tradition, would be addressed through statistical means. Rather, it is a misclassification of individual cases, due either to inadequate information or bias on the part of the investigator. Misclassification is to be corrected through close, and presumably “unbiased,” attention to facts about specific events, places, and people.

Elements of the case-based tradition can be found in a variety of methodological approaches. In medium-N cross-national studies, scholars may have a high level of case knowledge while also using statistical tools identified with other measurement traditions.Footnote ¹² Likewise, some qualitative methodological work gives the case-based approach pride of place in consideration of measurement. For example, the comparative case-study tradition (George and Bennett Reference George and Bennett2005), the political science and sociology literatures on comparative-historical analysis (Mahoney and Rueschemeyer Reference Mahoney and Rueschemeyer2003), and the method of contextualized comparison (Locke and Thelen Reference Locke and Thelen1995) all emphasize relying on fine-grained case detail as a foundation of comparative research.

In the case-based tradition, the level of detail with which case knowledge is conveyed varies greatly. First, following what might be seen as “best practices,” scholars working with a small N carefully discuss the evidence justifying scores for particular cases. Much comparative-historical research includes excellent examples of detailed and explicit exposition of the criteria and evidence involved in scoring, with the evidence commonly in a narrative or quasi-narrative framework. It is in part for this reason that such studies often appear as books, rather than articles. Second, other studies, using a larger N, nonetheless provide fairly detailed discussions of how at least some of the cases are coded. Third, in still other instances, the case knowledge of the authors suggests that great expertise went into scoring cases, but the presentation (perhaps owing to space constraints in a journal article) provides little evidence of the specific choices made in coding. While such measurement decisions may reflect in-depth case knowledge, failure to present the relevant detail may impede the reader’s efforts to verify the measurement leverage derived from such knowledge.

This perspective may serve as a useful, and partially compatible, alternative to the unverified assumptions required in the structural-equation tradition, the abstractions involved in LoM, and the sometimes ad hoc treatment of indicators in the pragmatic approach. More broadly, case-based measurement serves a useful role in demonstrating the limits of general-purpose measures, as they commonly show that these measures inadequately capture the details of cases.

Case-Based Tradition and Democracy

A spectrum of case-based approaches has been applied to debates about measuring democracy. Just as we have recognized best practices in structural equation modeling, we would point to one end of this spectrum, involving highly detailed, systematic presentation of the evidence justifying measurement decisions, as most fully exemplifying the strengths of the case-based tradition. For example, in her comparative-historical study of regime episodes in two different periods in Costa Rica and Guatemala (i.e., N = 4), Yashar (Reference Yashar1997) provides several chapters of evidence to support her identification of episodes of democratization in the 1940s and 1950s, as well as democratic versus authoritarian outcomes since the middle of the twentieth century. Mahoney (Reference Mahoney2001), looking at a broader set of five Central American countries, likewise presents detailed case evidence to identify democratizing episodes in the first half of the twentieth century, as well as democratic versus nondemocratic outcomes in the second half of the century.

Studies that move toward a larger N include R. B. Collier’s (Reference Collier1999) use of comparative-historical data to support coding of thirty-eight episodes of democratization in twenty-seven countries. A sketch of relevant detail is presented for every episode, yet the amount of information presented to justify the coding of each case is necessarily more limited than with a smaller N, as with the Central American examples noted earlier. In a further step along the spectrum, Rueschemeyer, Stevens, and Stevens (Reference Rueschemeyer, Stevens and Stevens1992) consider approximately a century of regime history for roughly forty countries. While these authors sometimes present specific information justifying regime codings and transition periods, the study’s scope inherently limits the case detail that the authors can feasibly present without overwhelming other analytic priorities.

Another route to a larger N is seen in studies that examine relatively few country cases over a long period of time, thereby retaining a high level of country expertise. Bowman et al.’s (Reference Bowman, Lehoucq and Mahoney2005) “Case Expertise” article thus focuses on five countries to generate a timeseries N of 500, and Mainwaring et al.’s (Reference Mainwaring, Brinks and Pérez-Liñán2001) study focuses on nineteen countries to generate a timeseries N of 855. Both studies closely scrutinize cases and point to important disagreements with previous measures. The credibility of the indicators in these studies derives from a variety of sources. To varying degrees, the authors discuss how selected cases were scored. Bowman et al., on the one hand, use evidence about selected country-years from the history of Costa Rica, El Salvador, Honduras, Guatemala, and Nicaragua (946–49) to document the problem of “inaccurate, partial, or misleading secondary sources” in constructing cross-national democracy indicators (940). For example, the Polity indicator gives Costa Rica a fully democratic score for every year of the twentieth century, despite the major role of coups and military interventions during the first half of the twentieth century. In another regional context, Berg-Schlosser (Reference Berg-Schlosser2004) carries out a similarly insightful analysis focusing on Africa.

Mainwaring et al. (Reference Mainwaring, Brinks and Pérez-Liñán2001), on the other hand, stand toward the other end of the spectrum in the case-based tradition: They offer less detailed evidence about specific cases – hardly surprising, given that they cover nineteen Latin American countries over forty-five years. The authors explicate their coding criteria for classifying country-years, and they present illustrative evidence to justify a few measurement decisions. Yet, with so many cases, it is not feasible to present the level of detail characteristic of the comparative-historical tradition – or even the amount of country-specific information presented by Bowman et al. (Reference Bowman, Lehoucq and Mahoney2005).

The credibility of Mainwaring et al.’s (Reference Mainwaring, Brinks and Pérez-Liñán2001) measures also derives from the investigators’ scholarly reputations for in-depth country knowledge, which provides indirect evidence that the cases have been scored carefully. In addition, Mainwaring et al. (Reference Mainwaring, Brinks and Pérez-Liñán2001) and Bowman et al. (Reference Bowman, Lehoucq and Mahoney2005) step outside the case-based tradition to draw on the pragmatic tradition, in that they use convergent validation to compare the new indicators with a spectrum of existing measures.

A different application of case knowledge is found in O’Donnell’s (Reference O’Donnell1996) discussion of Argentina’s regime history between 1983 and the mid 1990s. He rejects as invalid a prior cross-national assessment of democratic consolidation, which was based on the number and severity of crises survived by a given democratic regime. Specifically, to the extent that the regime overcomes more obstacles, it is evaluated as more consolidated. In roughly a decade after its transition to democracy in 1983, Argentina passed through a politically traumatic process of bringing military officers to justice for human rights violations during the previous dictatorship, multiple attempted military coups, and a protracted economic crisis. Because these obstacles were more severe than those survived by several Southern European democracies in roughly the same time period, the cross-national indicator would suggest that Argentina’s democracy was more consolidated than those of Portugal or Spain.

Yet O’Donnell (Reference O’Donnell1996) draws on in-depth knowledge of political processes in Argentina to argue that the basic institutions of democracy were in fact more tenuous in Argentina than in Southern Europe. Hence, additional information about a small number of cases leads to the rejection of the cross-national indicator, which O’Donnell characterizes as a reductio ad absurdum.

Other examples of case-based studies that seek to preserve or enhance measurement validity are presented in Collier and Levitsky’s (Reference Collier and Levitsky1997) discussion of how scholars create new subtypes of democracy.Footnote ¹³ These authors show that analysts have created nominal categories to more adequately fit a particular case or set of cases, thereby seeking to avoid conceptual stretching (Sartori Reference Sartori1970). Thus, within the evolving comparative literature on democracy in the 1980s and 1990s, researchers inductively adapted their categories on the basis of case-based information, reflecting the interplay of case knowledge and the conceptual understanding of democracy. One facet of this adaptation involves “democracy with adjectives,” that is, democratic subtypes derived by attaching an adjective before the noun. For example, in light of the judgment that democracy in countries such as Chile after 1990 was limited by the persistence of military influence in politics, scholars created the subtype of “guarded” or “tutelary” democracy. Where civil liberties were evaluated as being tenuous, scholars created the category “illiberal democracy” (Collier and Levitsky Reference Collier and Levitsky1997: 440). These democratic subtypes in effect allow researchers to create one step in an ordinal scale of democracy, in which ambiguous cases are situated in a relationship of “less than” vis-à-vis the analysts’ conception of full democracy.

Related innovations intended to avoid an invalid scoring of cases – and thus again, to avoid conceptual stretching – involve shifts in what might be called the “overarching concept” of democracy, that is to say, the broader form of political institutions of which democracy is a specific instance. Democracy might typically be thought of as a type of “regime,” but there are variations on this usage. For example, given the apparent tenuousness of the democratic regime in Brazil in the later 1980s, scholars used the terms democratic situation or democratic government, thereby suggesting a lower level of institutionalization compared with the label “regime.” By contrast, another scholar found that although Brazil had a democratic regime, the broader democratic protection of citizenship and civil rights was sufficiently weak that the country was characterized as lacking a “democratic state” (O’Donnell Reference O’Donnell1996: 447). In all of these instances, the categories used in scoring cases were adapted to incorporate insights drawn from close knowledge of cases.

Critiques of Case-Based Tradition by Methodologists

A key critique focuses on the price the case-based tradition may often pay in the trade-off between the rich detail yielded by case-based knowledge and the leverage for sharpening measurement by systematically and carefully working with a large N. One should definitely avoid a facile conclusion that better measurement is automatically achieved with a large N, given the complicated issues of contextual specificity that arise in any measurement enterprise. However, analysis that builds on a large N certainly opens the possibility of achieving measurement that is valid and also more general.

For quantitative researchers, case-based studies of a small N also have the drawback that their standard tools of causal inference simply cannot be applied. The substantive focus of scholars in the case-based tradition is often on the small number of cases they examine for improving measurement, and the hoped-for improvements are presumably of great benefit to them. This approach to improving measurement would generally not be possible with the large N.

However, with sufficient resources, research focused on a large N can nonetheless be combined with intensive examination of specific cases. Whatever the other criticisms of the Freedom House surveys, they use a very large “survey team” to carry out their scoring; in 2011, this involved a remarkable total of seventy analysts, who brought to the task considerable country expertise. Thus, the intensive study of individual countries need not be restricted to small-N case-study research.