Measurement Validity and Validation

David Collier’s work on measurement validity and methods of validation establishes a baseline understanding that still largely describes the state of theory and practice. Scholars have developed some new ideas, and real-world conditions have changed in ways that shift the picture in subtle but important ways, but the road map that Collier’s work set up largely remains a useful guide to the landscape.

In work with Robert Adcock (Adcock and Collier Reference Adcock and Collier2001),Footnote ¹ Collier built a conceptual apparatus for understanding the measurement process as a sequence of moves from conceptual systematization, operational definition as an actual plan for measurement, and then creation of an indicator as a series of measurements connected with cases. Each step involves potential pitfalls, with measurement validity relating especially to the connections among systematized concepts, operational definitions, and indicators. Adcock and Collier further provide a typology of tools for exploring the conceptual value and empirical adequacy of different measurement schemes. Content validation involves checking that the indicator includes measurement elements that cover the full range of ideas included in the systematized concept and nothing extraneous. Convergent/discriminant validation involves checking that alternative indicators of the same concept are related in appropriate ways, whereas indicators of different concepts are empirically separable from each other. Finally, nomological/construct validation involves testing measurement schemes by seeing how well they perform within the context of a theoretical model.

Building on this work, Seawright and Collier (Reference Seawright and Collier2014) offer a somewhat more elaborate typology of intellectual traditions within the domain of measurement validation.Footnote ² Seawright and Collier differentiate approaches in terms of the ways that scholars justify their knowledge claims about measurement, finding four broad categories. First, a range of scholars draw on ideas about the internal structure of an indicator and the relative ordering of possible responses to justify some analytic techniques, generating a body of work Seawright and Collier call the levels of measurement tradition. Second, scholars devise (more or less rigorous) models of the underlying measurement process to motivate analytical models that test and improve measurement properties, a body of work the article names the structural equation modeling with latent variables tradition. Third, some scholars draw on a wide range of (quantitative and qualitative) techniques based on no really rigorous fundamental justification beyond the past experience of themselves and other scholars that those techniques are helpful in solving measurement problems. While any technique can be used in this way, some are typically used like this; Seawright and Collier refer to work in this vein as the pragmatic measurement tradition. Finally, some work is based in a deep collection of evidence about one or a few cases – so much evidence that there is, in a sense, an overflow of detail beyond the requirements of any given indicator such that each indicator can be checked against the surplus details available within the case for correspondence. This tradition is referred to as the case-based tradition.

In combination, these two chapters continue to provide a substantially useful guide to the contemporary landscape of practice in terms of measurement methods. Scholars continue to use most of the same tools that Collier and his collaborators characterized, with most of the same justifications. However, certain subtle but important changes in the landscape are noticeable, owing to changes in the technology available to scholars, the methods in wide use, and the prominence of some intellectual currents within social science. This chapter explores three ways in which measurement validation in political science and related fields has evolved in subsequent years. The surge and intellectual flourishing of interpretive/ethnographic methods in political science has offered some new perspectives related to measurement validation, but also a suite of fresh approaches to validation research at the individual level, especially for contemporary questions.

Recent years have seen an explosion of hype around the slogan of “big data,” followed by a fragmentation of that slogan into much smaller but perhaps more precise research discourses. Two of those streams of thought are of relevance here. As digital data storage and distribution become the norm, researchers have become accustomed to working with data sets whose dimensionality is vastly larger than was the norm (or, perhaps, was even achievable) when Collier’s measurement articles were written. In particular, it is trivially easy to obtain data sets that include far more different variables offering competing measures of a concept than existed in the 1980s or 1990s. This proliferation of very complex data sets pushes scholars almost necessarily away from in-depth knowledge of the data-generation process behind specific indicators and toward measurement validation tools that ask and answer more general questions.

Finally, scholars within political science have embraced text-as-data methods as a tool for engaging in measurement debates. The formal mathematical properties of these techniques often closely resemble factor analysis and thus fit neatly within the concepts of convergent/discriminant validation and structural equation modeling with latent variables from Collier’s work on validation. However, because these newer methods use text, rather than simply numbers, as the input for analysis, scholars have found mixed-method approaches all but irresistible. In practice, the boundaries between the concepts mentioned earlier and the case-based measurement tradition may be eroding in this domain.

Interpretive Perspectives and Ethnographic Methods

The last decades have seen a major increase in the organization, visibility, and scope of interpretive perspectives and the use of ethnographic research methods in political science generally. This shift has had important consequences for the intellectual landscape connected with measurement and measurement validity. Here, I will highlight two major changes resulting from the surge of interpretive work: renewed and expanded attention to considerations of how power relations, social positionality, and ethical issues interact with questions of validity; and a wonderfully enriched hands-on methodological literature related to ethnographic measurement approaches.

One prominent example is Lee Ann Fujii’s influential work on qualitative interview methods, reflected most notably in her book Interviewing in Social Science Research: A Relational Approach. Fujii’s (Reference Fujii2017) book discusses interview technique and analysis with an interest in avoiding blunders, shallowness, and misinterpretations that clearly resonates with long-standing qualitative measurement validity concerns. Yet the book adds several notable new perspectives, two of which I will highlight here.

First, there is the obvious focus on in-person collection of individual-level data from living people, an important contrast from a qualitative and case-based measurement literature that has concentrated on historical measurement, often at the country level. Fujii shows how moving to the individual level opens up space for measurement debates about meaning and identity (Fujii Reference Fujii2017: 9–11), questions that have often been treated as fuzzy or inaccessible to scientific measurement.

Second, Fujii adds the distinctive concerns of interpretive methodology with social positionality (15–22), the ethical treatment of research participants (6–7, 22–28), and a full intellectual engagement with the implications of respect for the autonomy and strategic goals of the people being interviewed (12–15). These concerns deepen and, in some ways, reposition considerations regarding validity. Much of the existing literature on validity takes the goal of the measurement process as a given, once the scholar’s conceptual choices have been made, but Fujii’s work (like that of other interpretivists) reminds us that the outcome of measurement is and necessarily must be a product of the agendas of both the scholar and the person or people being interviewed. Those people have goals of presenting their activities, political movements, and societies in certain ways, and they also have their own social theories that they seek to advocate. The way interview participants present information in pursuit of those goals can distort information and introduce bias, as one might frame such considerations in much of the existing measurement literature. However, as Fujii compellingly argues, these interactions are themselves part of the social content we wish to study, and fluidly readjusting our conceptual framework to incorporate the way we see our interlocutors engaging with us allows us to better fit our social science measurement apparatus with our understanding of people as co-creators of social meaning.

For example, Fujii discusses a series of interviews she carried out with people about violence they experienced during Rwanda’s genocide. She originally focused her research on the period of time through the end of 1994, when Rwanda’s war and genocide concluded, but she recounts an interview in which a respondent effectively persuaded her to change this measurement strategy:

From one perspective, one might ask whether it is literally correct that there was no violence in the area this woman lived during 1994, or whether this response entails measurement error. Much of the existing literature works from this perspective, which is of course important. Alternatively, one might seek to understand the woman’s goal in downplaying violence from 1994, which in this case was to emphasize the salience of violence later on and to persuade her interlocutor to understand that there was not in fact a fixed endpoint to the violence in Rwanda. That intention in itself is meaningful social data – whether or not the researcher is persuaded, as Fujii was, that it would be good research practice to follow the woman’s advice and change measurement strategies. This pivot, toward treating research interactions as social interactions per se, is a contribution that interpretivists have made to the measurement literature whose broader implications are still working themselves out.

From a less interpretive perspective, Jennifer Cyr’s (Reference Cyr2019) work on focus groups offers a parallel demonstration of the expanded range of ethnographic methods in the social science measurement validation literature in recent years. Cyr positions focus groups as a way of evaluating measurement from a qualitative perspective when the concept of interest involves individual-level social interaction; whether about behavior or belief they are thus a way of looking at shared stereotypes, jointly held cultural beliefs, and similar phenomena. In the broader imagination, focus groups are associated with market research; within social science, the most common stereotype views focus groups as a tool for survey and experimental pretests.

Political scientists in particular have underused focus groups as a qualitative tool for investigating group behavior and patterns of agreement/disagreement, which are the distinctive contribution of the method for looking at measurement validation; a single-digit percentage of published articles in the discipline use the method, and around half of those articles focus on individual responses rather than the interactions, shared beliefs, and group dynamics that are the strength of the method in terms of measurement and validation (Cyr Reference Cyr2016). Thus, these approaches still have much more to offer in terms of testing hypotheses and measurement claims about the existence and content of societal consensus, which gives focus groups special relevance in measurement debates regarding culture, ideology, and the interpretation of language and meaning. As such, they remain arguably an underemphasized component of the validation toolkit.

As these illustrative examples show, the set of research tools and perspectives available for immersive qualitative measurement at the individual level in political science is far more developed than it was at the turn of the century – and a complex set of questions about how social structure and hierarchy affect the interactions that produce measurement has been brought squarely into the literature on validity in a way that is productive and worth considering in a broader way. These transformations raise the possibility that the case-based measurement tradition, once usefully regarded as a unitary alternative to multiple quantitative and statistical approaches to validation, might best be seen as divided between a (comparative) historical and an interpretive/ethnographic approach, each offering distinctive methods and raising particular methodological concerns.

How Much Does the Scope of Data Matter?

We live in the age of big data, and as a result approaches to social measurement and validation have shifted in interesting, if sometimes surprisingly subtle, ways. In fact, the idea of big data can point to at least two different shifts in the landscape of social science measurement. Digital technology has altered the kinds of data that can be created, and it has also shifted the picture of costs and benefits when scholars make choices about measurement validation.

One much-discussed change in the social science landscape over the last decade is the dramatic increase in the amount of available data. Social media, digital measurement, and other aspects of the internet era have made certain kinds of enormous data sets feasible to create and sometimes easier for the research community in general to access. For certain kinds of research tasks, measurement using cell phone data, social media behavior, and other digital interactions can create detailed depictions of real-world dynamics that would have been inaccessible in the recent past.

Alongside these changes that have created new kinds of large-scale data sets, older and more established forms of data have become far more easily accessible. This second change has important consequences for how scholars approach practical issues related to measurement and techniques of validation. Indeed, in some literatures, it has been far more consequential.

Consider, for example, the literatures about measuring democracy that served as the central focus on measurement validation in Chapters 16 and 17 by Collier and co-authors. For the most part, little value has been drawn from social media or massive data sets created with digital devices, largely because these approaches do not capture too much relevant information about whether elections are free and fair, the nature of the franchise, the rule of law, civil–military relations, and so forth. However, practical work related to measuring democracy has seen enormous changes owing in large part to the dissemination of new data sets that compile large numbers of variables in a way that would have been impractical or impossible in earlier eras.

As a central example, consider the impact of the Quality of Government (QoG) data (Teorell et al. Reference Teorell, Dahlberg, Holmberg, Rothstein, Khomenko and Svensson2018) on how scholars approach measurement projects related to democracy. Because the QoG data compile the vast majority of variables that scholars have used to measure democracy as well as related concepts such as participation, corruption, federalism, freedom of the press, and many more, the standard QoG data set includes over 2,000 variables. Until recently, no scholar would have worked with that many variables across an entire career; now, every graduate student who downloads the QoG data for use in a term paper is engaging with a data set whose internal complexity defies human understanding.

This complexity enables, but also requires, subtle shifts in measurement validation practices. Scholars simply cannot have the kinds of close familiarity with the details of cases and measurement process for each of over 2,000 indicators that were once the hallmark of expert use of factor analysis, scaling, and other statistical validation techniques. For example, it was once a common feature of the measurement literature on democracy for scholars to debate the extent to which Freedom House’s ties to the US government bias its democracy index, with a common hypothesis being that the index is particularly punitive against left-wing governments that are not US allies (e.g., Scoble and Weisberg Reference Scoble, Wiseberg, Nanda, Scarritt and Shepherd1981: 160–61; Nagle Reference Nagle1985: 95; Goldstein Reference Goldstein, Jabine and Claude1992: 47; Mainwaring et al. Reference Mainwaring, Brinks and Pérez-Liñán2001: 145). This concern about the data emerged from knowledge of the staffing and organizational history of the body that produced the data, as well as analysis of the fit between the data and the details of specific cases, and it also motivated a few carefully designed statistical tests (Bollen Reference Bollen1993; Bollen and Paxton Reference Bollen and Paxton1998, Reference Bollen and Paxton2000; Steiner Reference Steiner2016).

As the number of available measures per concept has increased, scholars tend to substitute breadth of data for depth of engagement with the fine details of calculation and institutional history behind each indicator. Broader data sets enable the use of statistical measurement models and allow for fairly open-ended processes of discovery, at the expense of the more granular knowledge of the data-generating process that was feasible when fewer variables were available.

For example, Elff and Ziaja (Reference Elff and Ziaja2018) examine twenty-three indicators drawn from four different data sources – a total of 3,863,770 different country-year observations of various subcomponents of democracy scores. It is obviously unreasonable to expect detailed knowledge of each of these measurements. Instead, the authors quite reasonably use a general-purpose statistical model to test for the overall amount of measurement error present in each component of each score, as well as to test hypotheses about whether error is due to information about long-term levels or year-to-year change. They find that the various data sources largely agree on countries’ overall level of democracy, particularly with regard to institutional features such as elections and horizontal accountability, but there is much less clarity in the data regarding the timing of change (97–102). This suggests that purely cross-sectional statistical analysis is likely to be on fairly solid measurement ground, but panel models or statistical analyses of regime transition face special measurement challenges.

These findings are very real contributions, providing grounded advice to quantitative users of democracy scores while also identifying key issues for producers of such data. At the same time, it is clear that there has been a shift from depth to breadth in comparison with the earlier literature on potential bias against left-wing governments in Freedom House scores. Elff and Ziaja (Reference Elff and Ziaja2018), along with other similar scholars, demonstrate a great deal of knowledge about democracy measurement – but the pragmatics of an ever-expanding data environment favor generalizable questions and findings that, in the language of Przeworski and Teune (Reference Przeworski and Teune1970: 75–76), substitute variables for proper names. There is nonetheless an undeniable loss of case-study-like textured knowledge in this shift, and it is worth asking whether there can be research designs that combine the excellent breadth of knowledge offered by work such as that of Elff and Ziaja with some of the nuanced local detail of earlier debates.

Text Models and Measurement Validation

While statistical approaches to the study of text have existed in a limited scope for decades, they have become part of mainstream research practice in the social sciences during roughly the last decade. We have seen publications that use text-as-data techniques to study the politics of Muslim clerics (Nielsen Reference Nielsen2017), the evolution over time of radical political rhetorics (Karell and Freedman Reference Karell and Freedman2019), and the ways that gender shapes politicians’ behavior on social media (Beltran et al. Reference Beltran, Gallego, Huidobro, Romero and Padro2021). This diversity of applications shows the breadth of impact text-as-data tools have already had on social science measurement and marks them as one of the major changes in this space since 2010. As such, it seems natural to ask whether they have altered the intellectual landscape in terms of measurement validation. I argue that they have, although the changes are surprisingly subtle. Statistical text-as-data tools largely fit within existing measurement validation categories.

While some applications are deeply pragmatic, the majority are straightforward extensions of factor analysis and other techniques from Seawright and Collier’s structural equation modeling with latent variables tradition. The data consist of dummy variables representing words (or sometimes combinations of words), and the latent variables or factors can represent different elements for different models, but in social science typically represent themes or ideas. While there are some technical nuances, the conceptual translation is ultimately not too challenging (Grimmer and Stewart Reference Grimmer and Stewart2013).

That said, these techniques do open novel opportunities for mixed-method approaches to measurement validation. Because the data for these methods are in fact text, it is easy and natural for scholars to alternate between quantitative analysis of the data using methods related to factor analysis and qualitative measurement involving in-depth reading of individual documents in relation to the broader corpus.

These mixed-method measurement designs are often undertaken without explicit methodological discussion; scholars simply run a sentiment analysis or a structural topic model, and then select a handful of interesting documents for closer scrutiny as examples of the findings or to make sense of hard-to-interpret patterns. These practices can be reconstructed and expanded in line with the general principle that mixed-method research is best when it explicitly combines methods to compensate for the weaknesses of an initially selected method.

Text-as-data methods have key strengths that are difficult to replicate in qualitative research. They can include huge bodies of text that go far beyond the time and attention constraints of any single human reader. Statistical text methods can also detect subtle patterns in text usage that human readers likely would not see. Finally, they can offer greater levels of consistency across a long project than a single reader can typically sustain – and certainly greater than a team of readers can offer.

At the same time, these methods have important assumptions and limitations. In practice, most applied work with text-as-data models pays little attention to the methods’ formal statistical assumptions, which are complex and abstract. Instead, the results are evaluated quite pragmatically; the technique is a success if the results make sense. Scholars evaluate whether the results make sense in two fundamentally different ways. First, they explore whether the quantitative findings are strong and group words together in ways that are theoretically and substantively intuitive. Second, they examine individual documents within and across groupings to determine whether the groupings that classify them together make sense, and to form a qualitative interpretation of the meaning of that grouping. This second approach takes advantage of the strength of qualitative methods in facilitating the creation and interpretation of meaning – and thus stands as a useful example of applied mixed-method measurement.

For example, consider Lacombe’s (Reference Lacombe2021) analysis of how, over the course of the last half of the twentieth century, the National Rifle Association (NRA) taught its supporters to think of themselves as politicized conservative gun activists rather than as a politically diverse collection of people who owned guns. Consistently throughout his book, Lacombe moves back and forth between statistical analysis of texts that the NRA uses to communicate with gun owners, or that those gun owners use to express their political preferences, and qualitative close reading of specific documents as case studies nested within the broader analysis.

In his analysis of the NRA’s efforts to construct a Second Amendment ideology of gun freedom, Lacombe begins with a mostly statistical analysis of a structural topic model (89–98), a standard text-as-data model that strongly resembles factor analysis. But to understand the meaning of these topics and their historical evolution over time, Lacombe also offers close readings of carefully selected example texts that represent key usages or innovations. For example, he shows the NRA’s early efforts to connect opposition to gun control with anti-communism by quoting a 1935 editorial that imagines Trotskyites saying, “When we have weakened the country by suppressing its rifle bearers … we shall be in a position to go ahead with our plan for setting up a government based on the theories of Karl Marx, Lenin and Stalin” (102). This is not a one-way flow from quantitative to qualitative; Lacombe uses the documents qualitatively to make points about measurement whenever necessary, but readily bounces back to statistical modes as convenient (e.g., 109–12, 115–16, 120–21, 126–27).

Such deeply enmeshed methodological integration is often natural in text-as-data measurement endeavors, because of the nature of the underlying evidence as documents that can in fact be read. At the moment, there is relatively little methodological reflection about such practices, and limited clarity regarding best practices for such mixed-methods measurement. This should be an active research agenda going forward.

Conclusions

The methods and ideas discussed here fit broadly within the frameworks developed by Adcock and Collier (Reference Adcock and Collier2001) and Seawright and Collier (Reference Seawright and Collier2014). The ethnographic and interpretive methods that have taken on fresh importance in recent years fit broadly within the context of the case-based tradition, and scholars working with these methods often (although not exclusively) address questions related to content and nomological/construct validation. Scholars working with large contemporary data sets often use methods drawn from the three quantitative-leaning measurement traditions, and they address tasks from all stages of the measurement process. Finally, as discussed, text-as-data methods largely fit within the structural equation modeling with latent variables tradition and often focus on convergent/discriminant and/or nomological/construct validation. Thus, the frameworks developed in Collier’s writings on measurement validation continue to offer a useful map to this domain.

At the same time, recent developments suggest potential refinements in light of technological, intellectual, and social change. There may well be sufficient differences between the ethnographic and interpretive measurement methods that have gained intellectual ground in recent years, in comparison with the more established case-based tradition, to potentially divide the tradition in two. The availability of large, complex data sets through digital distribution pushes scholars toward statistical techniques and, in particular, toward more generalized measurement hypotheses – a trend that leads to good research but also produces some losses. There may be opportunities for research that brings case-based techniques back in to ground these methods in more specificity, although the challenge of how to do this efficiently will demand serious methodological work. Finally, the text-as-data tradition shows scholars who have run ahead of the methodological literature in terms of mixing methods across measurement traditions and the qualitative/quantitative divide. Methodological research that reflects on best practices for this kind of mixture will be an important area in future.

It is illuminating to look back on the articles on conceptual choices (Collier and Adcock Reference Collier and Adcock1999) and measurement validity (Adcock and Collier Reference Adcock and Collier2001) that I coauthored with David Collier.Footnote ¹ I reflect on these articles together since our work on the first fed into the second. My main theme will be how dualisms can hinder conversation across methodological communities. Specifically, I critically examine both the qualitative/quantitative and positivist/interpretive dualisms.

Let me first restate the shared core of our articles. We started from a commonplace observation: Political scientists routinely formulate their concepts in varying and even conflicting ways. Neither bemoaning nor celebrating this variety, we took it as a given and argued that it makes unpersuasive any generic assertions by scholars that their own conceptions are somehow inherently right or better. In framing conceptual choices as “real choices … from a range of alternatives,” we encouraged political scientists to set aside judging conceptual choices as right/wrong, true/false. Instead, we prescribed justifying their choices pragmatically in light of their specific “goals and context of research” and recognizing that other scholars with other goals and in other contexts sensibly make different choices (Reference Collier and Adcock1999: 562). We wanted political scientists to distinguish, as we put it in our second article, between the often complex and contested “background concept” and the “systematized concept” they deploy in their own research and should self-consciously craft through pragmatic choices (Reference Adcock and Collier2001: 532–33).

I find the observation and prescriptions here as compelling today as I did when we wrote. But what I do see as unhelpful was our use of the qualitative/quantitative dualism. This dualism was most prominent in our measurement validity article – subtitled “A Shared Standard for Qualitative and Quantitative Research” – and was also used in our prior talk of “qualitative and quantitative analysts” (Reference Collier and Adcock1999: 537). Using this dualism was itself a conceptual choice, and I now see how, as I will explain, it obscured a key difference in the relation of each article to interpretive research. Yet I also find the positivist/interpretive dualism that many interpretivists deploy as an alternative to the quantitative/qualitative dualism to be equally obscuring. Hence, while my own substantive research on the history of political science (Adcock Reference Adcock2014) is interpretivist in methodology, my argument here is not to replace one dualism with another. I argue that recognition and conversation across methodological communities in political science is best served by leaving behind both dualisms and rethinking accordingly the pursuit of shared standards that Collier and I (Reference Adcock and Collier2001) avowed.

Methodological Debate, New Institutions, and Dueling Dualisms

To revisit the qualitative/quantitative dualism, I will situate our articles relative to intellectual and institutional shifts in political science methodology. During the late 1990s, Gary King, Robert E. Keohane, and Sidney Verba’s Designing Social Inquiry: Scientific Inference in Qualitative Research (Reference King, Keohane and Verba1994) was the focal point of debate. They contended that there is an “essential logic underlying all social scientific research,” best articulated in “discussions of quantitative research” but equally applicable and especially needed in “qualitative research” design (3). When Collier and I used qualitative/quantitative, we participated in an intellectual response to King, Keohane, and Verba that was subsequently summed up in the title of Henry Brady and Collier’s edited volume Rethinking Social Inquiry: Diverse Tools, Shared Standards (Reference Brady and Collier2004). This response replaced talk of one “essential logic” with talk of “shared standards” and favored methodological insights flowing from “qualitative” to “quantitative” as much as in the other direction.

But methodological practice is arguably shaped more by institutions than by the merit of specific positions. Between the 1960s and 1990s, generations of political science PhD students had attended the Inter-university Consortium for Political and Social Research summer training program in quantitative methods. A second key institution, since 1986, was the American Political Science Association’s (APSA) Political Methodology section, also dominated by statistical methods. To rebalance this landscape, scholars in multiple departments were planning two new institutions as we wrote: (1) the Consortium on Qualitative Research Methods (CQRM), which offered its first training institute in 2002; and (2) the Qualitative Methods section of the APSA, of which David Collier was the founding president in 2002–03.

Creating institutions to advance “qualitative methods” tied choices about the methodological positions and methods falling within the remit of “qualitative” to resource allocation decisions: What would be taught at CQRM, and represented in the section’s panels and so on? These decisions, alongside emerging doubts about the qualitative/quantitative dualism, would lead to the 2008 renaming of these institutions to reflect their welcome of multimethod work. CQRM became the Institute for Qualitative and Multi-Method Research (IQMR) and the section Qualitative and Multi-Methods Research (QMMR), as they are to this day.

But for me the most transformative conversation accompanying the new institutions concerned the relation of “qualitative” to “interpretive” research. In the section’s inaugural newsletter, its first two presidents situated “interpretive” research within “qualitative” methods (Bennett Reference Bennett2003: 1; Collier, Seawright, and Brady Reference Collier, Seawright and Brady2003: 6). From an institutional perspective, these statements supported interpretivist claims to share in the newly created resources. But from an intellectual perspective, they struck interpretive scholars as failing to acknowledge pivotal differences. In the next newsletter issue, Dvora Yanow (Reference Yanow2003: 9–10) replied that interpretive methods have “philosophical presuppositions” fundamentally different from the “positivist philosophical presuppositions” that she saw “qualitative methods” in political science as increasingly sharing with “quantitative” research.Footnote ²

The new section’s newsletter hence showcased dueling dualisms: the quantitative/qualitative dualism, which subsumed interpretivism within “qualitative,” versus an interpretive/positivist dualism, which grouped qualitative and quantitative methods together as “positivist.” At the time I questioned if this second dualism served conversation across methodological communities (Adcock Reference Adcock2003: 16). Over time my initial skepticism has only been reinforced. But I have also come to consider the quantitative/qualitative dualism just as dubious if assessed by the goal of recognition and conversation across methodological communities.

To discuss these dualisms without reproducing them requires a terminology beyond them. For this end, I will adopt Derek Beach and Jonas Gejl Kaas’ (Reference Beach and Kaas2020) trichotomy of variation-based, case-based, and interpretive research. Their label “variation-based” highlights the variables at the heart of what has been called “quantitative” work while avoiding implications about the level of measurement of those variables. “Case-based” highlights the centrality of the number and selection of cases – whether in a process tracing single-case study, a most-similar paired comparison, or a fifteen-case qualitative comparative analysis study – while avoiding implying (as “qualitative” may) that case-based researchers avoid quantitative data. The third label recognizes the interpretivist claim to distinctiveness, even as the trichotomy rejects moves to lump other researchers together under the elusive label “positivist.” My adoption of this trichotomy is, of course, itself a conceptual choice, which Table 19.1 situates alongside the alternatives provided by the dualisms whose 2003 duel I have spotlighted.

Interpretivist Responses and The Perils of Dubious Dualisms

The trajectories of the dueling dualisms have diverged since 2003. Qualitative/quantitative has lost much of its formerly taken-for-granted dominance. While I highlighted the challenge from interpretivism, the divide between variation- and case-based research has been complicated by multimethod research, and variation-based research itself subdivided as the rise of experiments and their institutionalization in APSA’s Experimental Methods section elevated another dualism: experimental/observational.

The positivist/interpretive dualism has, by contrast, secured its standing as foundational to the methodological identity of all too many interpretivists. In this section I reflect on responses from the expanding interpretivist community to my articles with Collier, engaging first the response of Mark Bevir and coauthors (Bevir and Kedar Reference Bevir and Kedar2008; Bevir and Blakely Reference Bevir and Blakely2018), who reformulate the positivist/interpretive dualism as a philosophical dualism of naturalism/anti-naturalism. Second, I engage Frederic Charles Schaffer’s (Reference Bevir and Blakely2018) response as he applies the positivist/interpretive dualism to conceptual work to distinguish “positivist reconstruction” from “interpretive elucidation.” Table 19.2 sums up the three dualisms of the interpretivist community that I critically discuss in this section.

Mark Bevir and Asaf Kedar’s (Reference Bevir and Kedar2008) “Concept Formation in Political Science: An Anti-Naturalist Critique of Qualitative Methodology” specifically critiques our conceptual choices article. They grant a potential opening to “interpretive political science” in our treatment of reification, attention to conceptual change, and recognition of the situatedness of scholars and how normative concerns inform conceptual choices (511–12). But they charge that our article “neglects the fact that objects of social inquiry have accounts of themselves” (512) and thereby forecloses this opening. It is a frustrating charge since we did explicitly, if briefly, allow that “the analyst’s treatment of democracy vis-à-vis non-democracy may need to reflect the viewpoint of the individuals who are being studied” (Reference Collier and Adcock1999: 556).

Bevir and Kedar’s critique pushed me, however, to reflect upon our openness to interpretivism and thereby uncover a key difference between our two articles. If our conceptual choices article is more open to interpretivism than Bevir and Kedar credit, our later measurement validity article is not. I missed this initially because the qualitative/quantitative dualism masked the differing reach of our use of “qualitative” in each article: In the first it encompassed both case-based and interpretive research, but in the second it focused on case-based research alone. I am comfortable with the difference in the types of research engaged in each article, since interpretivist scholars emphasize the role of concepts in their research but do not tend to see themselves as engaged in measurement. Our articulation of measurement validity as a shared standard did not, in practice, extend beyond variation and case-based research to interpretive research, and attempting such an extension would misrecognize interpretivist commitments. I do wish we had seen and explicitly stated this at the time. Realizing it retrospectively has shown me how the qualitative/quantitative dualism can undermine recognition and conversation across methodological communities.

Other moves in Bevir and Kedar (Reference Bevir and Kedar2008), reiterated in Bevir and Jason Blakely (Reference Bevir and Blakely2018: chap. 4), pushed me to clarify my thinking about “essentialism.” Both works distinguish a “strong essentialism,” holding that all social science concepts do/should have essential element/s (which Giovanni Sartori [Reference Sartori1984] illustrates for them), from “weak essentialism,” holding that this fits only some and not all such concepts (which Collier illustrates for them). Whatever our article’s position is called, Bevir and his coauthors correctly see it as incompatible with their assertion that “the particularity and contingency” of social life (Bevir and Kedar Reference Bevir and Kedar2008: 508) is so deeply pervasive that any “logic of commonality” must be avoided since the “features of social reality are never natural or essential types, with recurring nuclei of features” (Bevir and Blakely Reference Bevir and Blakely2018: 75). Their assertion is a one-size-fits-all mandate dictating how social scientists should conceive of what they study. I would rather leave scholars to judge for themselves which of their concepts they might productively conceive of in terms of a “recurring nuclei of features” and which not.

I would suggest, moreover, that interpretivist scholars do, in practice, sometimes find it useful to conceive certain concepts in terms of a “nuclei of features.” Take, for example, Peregrine Schwartz-Shea’s crisp, clear statement that “interpretive social science consists of researchers’ interpretations of actors’ interpretations” (Reference Schwartz-Shea2020: 462). If it is essentialist to see interpreting actors’ interpretations as an “essential” baseline commonality of interpretivist work, then by Bevir and his coauthors’ definition, some interpretivists might in practice be “weak essentialists” too (I certainly am one).

Lurking behind this issue is a disagreement over philosophy’s relationship to methodology. Bevir and Kedar contrast judging methods “in pragmatic terms (i.e., in terms of their substantive utility for certain lines of inquiry)” with judging them “from a philosophical standpoint” (Reference Bevir and Kedar2008: 503). Where Collier and I saw examining examples of conceptual usage in political science as a sufficient basis to advocate our pragmatic approach, they consider our article “beset by problems that arise from a lack of systematic philosophical reflection” (514).

There are at least two reasons why I do not share the faith of Bevir and his coauthors “that our discipline can only benefit” (Bevir and Kedar Reference Bevir and Kedar2008: 514) from philosophizing. First, their view of philosophy moves beyond self-reflection about one’s own assumptions to wielding assertions about the assumptions of others as a tool of critique. As much as I welcome philosophical self-clarification, I suspect that political scientists targeted by such philosophical critique are more likely to consider their methodological assumptions to have been mischaracterized than they are to be persuaded to change them. Second, I do not believe that philosophers agree to the extent Bevir and Kedar assert (Reference Bevir and Kedar2008: 504–06, 513–14), and even if they did, I do not see why we should defer to another discipline in judging how to undertake our own. Claiming that “social scientists … must catch up to philosophy” (Bevir and Blakely Reference Bevir and Blakely2018: 15) strikes me as more paternalistic (cf. Schwartz-Shea Reference Schwartz-Shea2020: 2) than persuasive.

My skepticism of overplaying the philosophy card was reinforced by Bevir and his coauthors’ naturalism/anti-naturalism dualism. Although their criticisms of some of our specific moves are clarifying, I failed to engage these criticisms initially owing to being alienated by their packaging of them under this sweeping philosophical dualism. Having been a chemistry undergraduate, and subsequently watched one college friend go on to a dissertation in theoretical nanophysics and another in geophysics field research, I do not think research aims and activities fit a single model even in one natural science discipline, let alone across them all. If there is no one natural science model, then dividing social scientists dualistically into those who accept and those who reject such a model strikes me as a grave philosophical misdirection. When Bevir and Kedar asserted that my article with Collier had “naturalist assumptions,” I felt misrecognized in a manner akin, perhaps, to how Yanow felt in reading interpretive work presented as a subpart of “qualitative” methods.

While the anti-naturalism/naturalism dualism is distinctive to Bevir and coauthors’ philosophical agenda, the interpretive/positivist dualism it reworks has become standard in interpretive methodologies and methods. It anchors Schwartz-Shea and Yanow’s Reference Schwartz-Shea and Yanow2012 Interpretive Research Design: Concepts and Processes, which inaugurated Routledge’s Series on Interpretive Methods, and is applied to conceptual work in Schaffer’s (Reference Schaffer2016) series contribution, Elucidating Social Science Concepts: An Interpretivist Guide. His book begins from a dualistic contrast of “positivist reconstruction” with “interpretive elucidation.” Since both my articles with Collier are treated as examples of “positivist reconstruction,” reflecting on Schaffer’s treatment illustrates how the positivist/interpretive dualism may also produce misrecognition.

After introducing his dualism, Schaffer (Reference Schaffer2016) argues that three problems “dog the positivist project of conceptual reconstruction” (12). I focus on “one-sidedness,” which involves “privileging those meanings of a concept that are important to the researcher while ignoring other meanings that are salient to situated actors themselves” (12). How one-sidedness “blinds the scholar to actors’ self-understandings” is ironically itself exemplified in Schaffer’s own presentation of “positivist reconstruction” (12). He begins with quotes from our measurement validity article and Gary Goertz and James Mahoney’s (Reference Goertz and Mahoney2012) A Tale of Two Cultures. The quotes suggest these works share the view that concepts should represent “actually existing reality as accurately as possible” (Goertz Reference Goertz2006: 4). But Goertz (Reference Goertz2006: 3–5) developed his “causal, ontological, and realist view of concepts” as an explicit “contrast” to the “semantic approach” of prior work on concepts. More generally, where Bevir and Kedar (Reference Bevir and Kedar2008) engaged, albeit contentiously, with differences between Collier’s work on concepts and Sartori’s, Schaffer elides these differences in presenting Sartori, Collier, and Goertz together one-sidedly, without acknowledging their self-understandings.

I do not object to Schaffer crafting a category that brings these scholars together. There is a clear line of conversation between them, and the category is potentially useful. My concern is that his choices in labeling and structuring his category undermine cross-methodological recognition and communication. His use of “reconstruction” is grounded in the conversation of scholars being categorized, and as such Schaffer’s preference for “reconstruction” over “concept formation” (Reference Schaffer2016: 13) persuades me. But adding “positivist” saddles his category with counterproductive baggage. It is ungrounded since none of the scholars self-identify in this way, and it implies a philosophical agreement that elides how much Goertz’s “causal, ontological, and realist view of concepts” differs from the pragmatist position at the core of my articles with Collier.

In terms of conceptual structure, Schaffer allows in a footnote that “individual scholars … may hold views that correspond only partially with, or perhaps differ from, the ones I have laid out” (Reference Schaffer2016: 24n2). But this leaves unaffected the lumping he carries out in the main text. Schaffer could have avoided this by structuring “reconstruction” as a prototypical concept with Sartori as the prototype. This would fit with the central role he gives Sartori as the figure relative to whom interpretivist criticisms are introduced. Or he could have introduced “reconstruction” through a narrative: starting from Sartori, following continuities and changes in Collier’s seminal 1990s works, and the shift subsequently proposed by Goertz (Reference Goertz2006). My takeaway is that a potentially productive conversation between reconstruction and elucidation as approaches to concepts is undermined by overlaying this distinction with the dubious positivist/interpretive dualism.

Conclusion: Rethinking Shared Standards

Earlier, I situated my 1999 and 2001 articles with Collier as participating in a response to King, Keohane, and Verba (Reference King, Keohane and Verba1994) that replaced their talk of one “essential logic underlying all social scientific research” with talk of “shared standards.” To conclude, I rethink shared standards in the light of my theme of dubious dualisms.

I have argued for moving beyond the qualitative/quantitative dualism to recognize interpretive research as its own methodological community. Moreover, to sidestep the freighted terms of this dualism, I adopted a trichotomy of variation-based, case-based, and interpretive research. Given our pragmatic approach to concepts, it would be self-contradictory to justify this trichotomy as somehow representing reality as accurately as possible – it obviously simplifies the complexity of political science’s methodological archipelago, as any typology must. But I do think it better serves the goal of recognition and conversation across methodological communities than either of the dueling dualisms I have judged as dubious. One payoff of switching to a trichotomy is that, without adding much complexity, it enables us to rethink shared standards since a standard could be shared by two categories without the third. Hence, tweaking our measurement validity article’s subtitle to “a shared standard for variation- and case-based research” would clarify that “measurement validity” does not extend to interpretive research and better convey the kinds of examples and techniques we integrate.

Objections here might come not from interpretivists all too willing to lump variation-based and case-based research together as “positivist” but from scholars who stress the distinctiveness of case-based from variation-based research. In their influential 2006 article and subsequent 2012 book, Goertz and Mahoney argued that these are “two cultures” with “different norms, practices, and tool kits” (Reference Goertz and Mahoney2012: vii).Footnote ³ Derek Beach and Rasmus Brun Pedersen in turn refined this line of argument to distinguish among subtypes of Causal Case-Study Methods (2016).

Without taking a position for or against (see Kuehn and Rohlfing Reference Kuehn and Rohlfing2022) the two cultures argument, I want to suggest that it can in practice reinforce rather than contradict measurement validity as a shared standard. Beach and Pedersen (Reference Beach and Brun Pedersen2016: chaps. 1 and 2) argue that variation-based and case-based research each analyze different types of causal claims using different types of evidence, and this entails different ways to define concepts. But they also rely extensively on our measurement validity framework as they treat conceptualization and measurement (chaps. 4 and 5). I find this use of our work heartening, since most manuscripts I review drawing on our article are variation-based. That some of the most eloquent proponents of the distinctiveness of case-based research find our article’s formulation of measurement validity useful suggests that our aspiration to articulate a shared standard was not a fools’ errand.

My conclusion is that shared standards can complement rather than compete with recognition of methodological differences. As another example, I would highlight that Beach and his coauthors share with interpretivists a focus on clarifying epistemological and ontological assumptions and aligning choices of methodology and method with these assumptions. If this is another standard that in practice resonates in more than one methodological community, why not call it a “shared standard”? My rethinking suggests that doing so need not imply a further claim that this standard should be accepted also by variation-based researchers, just as we can call measurement validity a shared standard without making a claim on interpretivists.

In sum, the pursuit of shared standards need not be a search for a single set of standards for our very heterogeneous discipline. Rather, it can be a search for a plurality of standards each shared in the sense that researchers in more than one methodological community recognize it as applicable to themselves. Such standards can provide (or develop from) focal points of mutually illuminating conversation between methodological communities, and no community is, I believe, so distinct as to have no standard it shares with another. To seek out such shared standards is not to deny methodological differences but to pursue a network of bridges across those differences.