Skip to main content
×
×
Home

Measurement Validity: A Shared Standard for Qualitative and Quantitative Research

  • Robert Adcock (a1) and David Collier (a1)
Abstract

Scholars routinely make claims that presuppose the validity of the observations and measurements that operationalize their concepts. Yet, despite recent advances in political science methods, surprisingly little attention has been devoted to measurement validity. We address this gap by exploring four themes. First, we seek to establish a shared framework that allows quantitative and qualitative scholars to assess more effectively, and communicate about, issues of valid measurement. Second, we underscore the need to draw a clear distinction between measurement issues and disputes about concepts. Third, we discuss the contextual specificity of measurement claims, exploring a variety of measurement strategies that seek to combine generality and validity by devoting greater attention to context. Fourth, we address the proliferation of terms for alternative measurement validation procedures and offer an account of the three main types of validation most relevant to political scientists.

Copyright
References
Hide All
Alvarez, Michael, Cheibub, Jose Antonio, Limongi, Fernando, and Przeworski, Adam. 1996. “Classifying Political Regimes.Studies in Comparative International Development 31 (Summer): 336.
American Psychological Association. 1954. “Technical Recommendations for Psychological Tests and Diagnostic Techniques.Psychological Bulletin 51 (2, Part 2): 20138.
American Psychological Association. 1966. Standards for Educational and Psychological Tests and Manuals. Washington, DC: American Psychological Association.
Anderson, Barbara A., and Silver, Brian D.. 1986. “Measurement and Mismeasurement of the Validity of the Self-Reported Vote.American Journal of Political Science 30 (November): 77185.
Anderson, Perry. 1974. Lineages of the Absolutist State. London: Verso.
Angoff, William H. 1988. “Validity: An Evolving Concept.” In Test Validity. ed. Howard Wainer and Henry I. Braun. Hillsdale, NJ: Lawrence Erlbaum. Pp. 1932.
Babbie, Earl R. 2001. The Practice of Social Research. 9th ed. Belmont, CA: Wadsworth.
Bachman, Jerald G., and O’Malley, Patrick M.. 1984. “Yea-Saying, Nay-Saying, and Going to Extremes: Black-White Differences in Response Styles.Public Opinion Quarterly 48 (Summer): 491509.
Banks, Arthur S. 1979. Cross-National Time-Series Data Archive User's Manual. Binghamton: Center for Social Analysis, State University of New York at Binghamton.
Baumgartner, Frank R., and Walker, Jack L.. 1990. “Response to Smith's ‘Trends in Voluntary Group Membership: Comments on Baumgartner and Walker’: Measurement Validity and the Continuity of Research in Survey Research.American Journal of Political Science 34 (August): 66270.
Berry, William D., Ringquist, Evan J., Footing, Richard C., and Hanson, Russell L.. 1998. “Measuring Citizen and Government Ideology in the American States, 1960–93.American Journal of Political Science 42 (January): 32748.
Bevir, Mark. 1999. The Logic of the History of Ideas. Cambridge: Cambridge University Press.
Bollen, Kenneth A. 1980. “Issues in the Comparative Measurement of Political Democracy.American Sociological Review 45 (June): 37090.
Bollen, Kenneth A. 1989. Structural Equations with Latent Variables. New York: Wiley.
Bollen, Kenneth A. 1990. “Political Democracy: Conceptual and Measurement Traps.Studies in Comparative International Development 25 (Spring): 724.
Bollen, Kenneth A. 1993. “Liberal Democracy: Validity and Method Factors in Cross-National Measures.American Journal of Political Science 37 (November): 120730.
Bollen, Kenneth A., Entwisle, Barbara, and Anderson, Arthur S.. 1993. “Macrocomparative Research Methods.” Annual Review of Sociology 19: 32151.
Bollen, Kenneth A., and Lennox, Richard. 1991. “Conventional Wisdom on Measurement: A Structural Equation Perspective.Psychological Bulletin 110 (September): 30514.
Bollen, Kenneth A., and Paxton, Pamela. 1998. “Detection and Determinants of Bias in Subjective Measures.American Sociological Review 63 (June): 46578.
Bollen, Kenneth A., and Paxton, Pamela. 2000. “Subjective Measures of Liberal Democracy.Comparative Political Studies 33 (February): 5886.
Brady, Henry. 1985. “The Perils of Survey Research: Inter-Personally Incomparable Responses.Political Methodology 11 (3– 4): 26991.
Brady, Henry, and Collier, David, eds. 2001. Rethinking Social Inquiry: Diverse Tools, Shared Standards. Berkeley: Berkeley Public Policy Press, University of California, and Boulder, CO: Roman & Littlefield.
Brewer, John, and Hunter, Albert. 1989. Multimethod Research: A Synthesis of Styles. Newbury Park, CA: Sage.
Cain, Bruce E., and Ferejohn, John. 1981. “Party Identification in the United States and Great Britain.Comparative Political Studies 14 (April): 3147.
Campbell, Donald T. 1960. “Recommendations for APA Test Standards Regarding Construct, Trait, or Discriminant Validity.American Psychologist 15 (August): 54653.
Campbell, Donald T. 1977/1988. “Descriptive Epistemology: Psychological, Sociological, and Evolutionary.” Methodology and Epistemology for Social Science: Selected Papers. Chicago: University of Chicago Press.
Campbell, Donald T., and Fiske, Donald W.. 1959. “Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix.Psychological Bulletin 56 (March): 81105.
Carmines, Edward G., and Zeller, Richard A.. 1979. Reliability and Validity Assessment. Beverly Hills, CA: Sage.
Clausen, Aage. 1968. “Response Validity: Vote Report.Public Opinion Quarterly 41 (Winter): 588606.
Collier, David. 1995. “Trajectory of a Concept: ‘Corporatism’ in the Study of Latin American Politics.” In Latin America in Comparative Perspective: Issues and Methods. ed. Smith, Peter H.. Boulder, CO: Westview. Pp. 13562.
Collier, David, and Adcock, Robert. 1999. “Democracy and Dichotomies: A Pragmatic Approach to Choices about Concepts.” Annual Review of Political Science 2: 53765.
Collier, David, and Levitsky, Steven. 1997. “Democracy with Adjectives: Conceptual Innovation in Comparative Research.World Politics 49 (April): 43051.
Collier, David, and Mahon, James E., Jr. 1993. “Conceptual ‘Stretching’ Revisited: Adapting Categories in Comparative Analysis.American Political Science Review 87 (December): 84555.
Collier, Ruth Berins. 1999. Paths Toward Democracy. Cambridge: Cambridge University Press.
Cook, Thomas D., and Campbell, Donald T.. 1979. Quasi-Experimentation. Boston: Houghton Mifflin.
Coppedge, Michael. 1997. “How a Large N Could Complement the Small in Democratization Research.” Paper presented at the annual meeting of the American Political Science Association, Washington, DC.
Coppedge, Michael, and Reinicke, Wolfgang H.. 1990. “Measuring Polyarchy.Studies in Comparative International Development 25 (Spring): 5172.
Cronbach, Lee J., and Meehl, Paul E.. 1955. “Construct Validity in Psychological Tests.Psychological Bulletin 52 (July): 281302.
Dahl, Robert A. 1956. A Preface to Democratic Theory. Chicago: University of Chicago Press.
Elkins, Zachary. 2000. “Gradations of Democracy: Empirical Tests of Alternative Conceptualizations.American Journal of Political Science 44 (April): 293300.
Elster, Jon. 1999. Alchemies of the Mind. New York: Cambridge University Press.
Fearon, James, and Laitin, David D.. 2000. “Ordinary Language and External Validity: Specifying Concepts in the Study of Ethnicity.” Paper presented at the annual meeting of the American Political Science Association, Washington, DC.
Fischer, Markus. 1992. “Feudal Europe, 800–1300: Communal Discourse and Conflictual Politics.International Organization 46 (Spring): 42766.
Fischer, Markus. 1993. “On Context, Facts, and Norms.International Organization 47 (Summer): 493500.
Freeden, Michael. 1996. Ideologies and Political Theory: A Conceptual Approach. Oxford: Oxford University Press.
Gallie, W. B. 1956. “Essentially Contested Concepts.” Proceedings of the Aristotelian Society 51: 16798.
Gastil, Raymond D. 1988. Freedom in the World: Political Rights and Civil Liberties, 1987–1988. New York: Freedom House.
George, Alexander L., and Andrew Bennett. N.d. Case Studies and Theory Development. Cambridge, MA: MIT Press. Forthcoming.
Gerring, John. 1997. “Ideology: A Definitional Analysis.Political Research Quarterly 50 (December): 95794.
Gerring, John. 1999. “What Makes a Concept Good? A Criterial Framework for Understanding Concept Formation in the Social Sciences.Polity 31 (Spring): 35793.
Gerring, John. 2001. Practical Knowledge: A Criterial Approach to Social Science Methodology. New York: Cambridge University Press.
Gould, Andrew C. 1999. “Conflicting Imperatives and Concept Formation.Review of Politics 61 (Summer): 43963.
Green, Donald Philip. 1991. “The Effects of Measurement Error on Two-Stage Least-Squares Estimates.” In Political Analysis. vol. 2, ed. Stimson, James A.. Ann Arbor: University of Michigan Press. Pp: 5774.
Green, Donald Philip, and Palmquist, Bradley L.. 1990. “Of Artifacts and Partisan Instability.American Journal of Political Science 34 (August): 872902.
Greenleaf, Eric A. 1992. “Measuring Extreme Response Style.Public Opinion Quarterly 56 (Autumn): 38251.
Guion, Robert M. 1980. “On Trinitarian Doctrines of Validity.Professional Psychology 11 (June): 38598.
Harding, Timothy, and Petras, James. 1988. “Introduction: Democratization and the Class Struggle.Latin American Perspectives 15 (Summer): 317.
Hayduk, Leslie A. 1987. Structural Equation Modeling with LISREL. Baltimore, MD: Johns Hopkins University Press.
Hayduk, Leslie A. 1996. LISREL: Issues, Debates, and Strategies. Baltimore, MD: Johns Hopkins University Press.
Hill, Kim Q., Hanna, Stephen, and Shafqat, Sahar. 1997. “The Liberal-Conservative Ideology of U.S. Senators: A New Measure.American Journal of Political Science 41 (October): 1395413.
Huntington, Samuel P. 1991. The Third Wave: Democratization in the Twentieth Century. Norman: University of Oklahoma Press.
Jacoby, William G. 1991. Data Theory and Dimensional Analysis. Newbury Park, CA: Sage.
Jacoby, William G. 1999. “Levels of Measurement and Political Research: An Optimistic View.American Journal of Political Science 43 (January): 271301.
Johnson, Ollie A. 1999. “Pluralist Authoritarianism in Comparative Perspective: White Supremacy, Male Supremacy, and Regime Classification.” National Political Science Review 7: 11636.
Kaplan, Abraham. 1964. The Conduct of Inquiry. San Francisco: Chandler.
Karl, Terry Lynn. 1990. “Dilemmas of Democratization in Latin America.Comparative Politics 22 (October): 121.
Katosh, John P., and Traugott, Michael W.. 1980. “The Consequences of Validated and Self-Reported Voting Measures.Public Opinion Quarterly 45 (Winter): 51935.
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press.
Kirk, Jerome, and Miller, Marc L.. 1986. Reliability and Validity in Qualitative Research. Beverly Hills, CA: Sage.
Kurtz, Marcus J. 2000. “Understanding Peasant Revolution: From Concept to Theory and Case.Theory and Society 29 (February): 93124.
Laitin, David D. 2000. “What Is a Language Community?American Journal of Political Science 44 (January): 14255.
Levitsky, Steven. 1998. “Institutionalization and Peronism: The Concept, the Case and the Case for Unpacking the Concept.Party Politics 4 (1): 7792.
Lijphart, Arend. 1984. Democracies: Patterns of Majoritarian and Consensus Government in Twenty-One Countries. New Haven, CT: Yale University Press.
Lijphart, Arend. 1996. “The Puzzle of Indian Democracy: A Consociational Interpretation.American Political Science Review 90 (June): 25868.
Linz, Juan J. 1975. “Totalitarian and Authoritarian Regimes.” In Handbook of Political Science. vol. 3, ed. Greenstein, Fred I. and Polsby, Nelson W.. Reading, MA: Addison-Wesley. Pp. 175411.
Lipset, Seymour M. 1959. “Some Social Requisites of Democracy: Economic Development and Political Legitimacy.American Political Science Review 53 (March): 69105.
Locke, Richard, and Thelen, Kathleen. 1995. “Apples and Oranges Revisited: Contextualized Comparisons and the Study of Comparative Labor Politics.Politics and Society 23 (September): 33767.
Locke, Richard, and Thelen, Kathleen. 1998. “Problems of Equivalence in Comparative Politics: Apples and Oranges, Again.” Newsletter of the APSA Organized Section in Comparative Politics 9 (Winter): 912.
Loveman, Brian. 1994. “‘Protected Democracies’ and Military Guardianship: Political Transitions in Latin America, 1979–1993.Journal of Interamerican Studies and World Affairs 36 (Summer): 10589.
Mainwaring, Scott, Brinks, Daniel, and Aníbal Pérez-Linan. 2001. “Classifying Political Regimes in Latin America, 1945–1999.Studies in Comparative International Development 36 (1): 3764.
Markoff, John. 1996. Waves of Democracy. Thousand Oaks, CA: Pine Forge.
Messick, Samuel. 1980. “Test Validity and the Ethics of Assessment.American Psychologist 35 (November): 101227.
Messick, Samuel. 1989. “Validity.” In Educational Measurement. ed. Linn, Robert L.. New York: Macmillan. Pp. 13103.
Moene, Karl Ove, and Wallerstein, Michael. 2000. “Inequality, Social Insurance and Redistribution.” Working Paper No. 144, Juan March Institute, Madrid, Spain.
Moss, Pamela A. 1992. “Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment.Review of Educational Research 62 (Fall): 22958.
Moss, Pamela A. 1995. “Themes and Variations in Validity Theory.Educational Measurement: Issues and Practice 14 (Summer): 513.
Mouffe, Chantal. 1992. Dimensions of Radical Democracy. London: Verso.
Nie, Norman H., Bingham Powell, G., Jr., and Prewitt, Kenneth. 1969. “Social Structure, and Political Participation: Developmental Relationships, Part I.American Political Science Review 63 (June): 36178.
Niemi, Richard, Katz, Richard, and Newman, David. 1980. “Reconstructing Past Partisanship: The Failure of the Party Identification Recall Questions.American Journal of Political Science 24 (November): 63351.
O’Donnell, Guillermo. 1993. “On the State, Democratization and Some Conceptual Problems.World Development 27 (8): 135569.
O’Donnell, Guillermo. 1996. “Illusions about Consolidation.Journal of Democracy 7 (April): 3451.
Orum, Anthony M., Feagin, Joe R., and Sjoberg, Gideon. 1991. “Introduction: The Nature of the Case Study.” In A Case for the Case Study. ed. Feagin, Joe R., Orum, Anthony M., and Sjoberg, Gideon. Chapel Hill: University of North Carolina Press. Pp. 126.
Paxton, Pamela. 2000. “Women in the Measurement of Democracy: Problems of Operationalization.Studies in Comparative International Development 35 (Fall): 92111.
Pitkin, Hanna Fenichel. 1967. The Concept of Representation. Berkeley: University of California Press.
Pitkin, Hanna Fenichel. 1987. “Rethinking Reification.Theory and Society 16 (March): 26393.
Przeworski, Adam, Alvarez, Michael, Cheibub, Jose Antonio, and Limongi, Fernando. 1996. “What Makes Democracies Endure?Journal of Democracy 7 (January): 3955.
Przeworski, Adam, and Teune, Henry. 1970. Logic of Comparative Social Inquiry. New York: John Wiley.
Rabkin, Rhoda. 1992. “The Aylwin Government and ‘Tutelary’ Democracy: A Concept in Search of a Case?Journal of Interamerican Studies and World Affairs 34 (Winter): 11994.
Ragin, Charles C. 1987. The Comparative Method. Berkeley: University of California Press.
Ragin, Charles C. 1994. Constructing Social Research. Thousand Oaks, CA: Pine Forge.
Reus-Smit, Christian. 1997. “The Constitutional Structure of International Society and the Nature of Fundamental Institutions.International Organization 51 (Autumn): 55589.
Ruggie, John G. 1998. “What Makes the World Hang Together? Neo-Utilitarianism and the Social Constructivist Challenge.International Organization 52 (Autumn): 85585.
Russett, Bruce. 1993. Grasping the Democratic Peace. Princeton, NJ: Princeton University Press.
Sartori, Giovanni. 1970. “Concept Misformation in Comparative Research.American Political Science Review 64 (December): 103353.
Sartori, Giovanni. 1975. “The Tower of Babel.” In Tower of Babel, ed. Giovanni Sartori, Fred W. Riggs, and Henry Teune. International Studies Association, University of Pittsburgh. Pp. 737.
Sartori, Giovanni, ed. 1984. Social Science Concepts: A Systematic Analysis. Beverly Hills, CA: Sage.
Sartori, Giovanni, Riggs, Fred W., and Teune, Henry. 1975. Tower of Babel: On the Definition and Analysis of Concepts in the Social Sciences. International Studies Association, University of Pittsburgh.
Schaffer, Frederic Charles. 1998. Democracy in Translation: Understanding Politics in an Unfamiliar Culture. Ithaca, NY: Cornell University Press.
Schmitter, Philippe C., and Karl, Terry Lynn. 1992. “The Types of Democracy Emerging in Southern and Eastern Europe and South and Central America.” In Bound to Change: Consolidating Democracy in East Central Europe. ed. Volten, Peter M. E.. New York: Institute for EastWest Studies. Pp. 4268.
Schrodt, Philip A., and Gerner, Deborah J.. 1994. “Validity Assessment of a Machine-Coded Event Data Set for the Middle East, 1982–92.American Journal of Political Science 38 (August): 82554.
Schumpeter, Joseph. 1947. Capitalism, Socialism and Democracy. New York: Harper.
Shepard, Lorrie. 1993. “Evaluating Test Validity.” Review of Research in Education 19: 40550.
Shively, W. Phillips. 1998. The Craft of Political Research. Upper Saddle River, NJ: Prentice Hall.
Shultz, Kenneth S., Riggs, Matt L., and Kottke, Janet L.. 1998. “The Need for an Evolving Concept of Validity in Industrial and Personnel Psychology: Psychometric, Legal, and Emerging Issues.Current Psychology 17 (Winter): 26586.
Sicinski, Andrzej. 1970. “‘Don't Know’ Answers in Cross-National Surveys.Public Opinion Quarterly 34 (Spring): 1269.
Sireci, Stephen G. 1998. “The Construct of Content Validity.Social Indicators Research 45 (November): 83117.
Skocpol, Theda. 1992. Protecting Soldiers and Mothers. Cambridge, MA: Harvard University Press.
Sussman, Leonard R. 1982. “The Continuing Struggle for Freedom of Information.” In Freedom in the World. ed. Gastil, Raymond D.. Westport, CT: Greenwood. Pp. 10119.
Valenzuela, J. Samuel. 1992. “Democratic Consolidation in Post-Transitional Settings: Notion, Process, and Facilitating Conditions.” In Issues in Democratic Consolidation. ed. Scott Mainwaring, Guillermo O’Donnell, and J. Samuel Valenzuela. Notre Dame, IN: University of Notre Dame Press. Pp. 57104.
Vanhanen, Tatu. 1979. Power and the Means to Power. Ann Arbor, MI: University Microfilms International.
Vanhanen, Tatu. 1990. The Process of Democratization. New York: Crane Russak.
Verba, Sidney. 1967. “Some Dilemmas in Comparative Research.World Politics 20 (October): 11127.
Verba, Sidney. 1971. “Cross-National Survey Research: The Problem of Credibility.” In Comparative Methods in Sociology. ed. Ivan Vallier. Berkeley: University of California Press. Pp. 30956.
Verba, Sidney, Kelman, Steven, Orren, Gary R., Miyake, Ichiro, Watanuki, Joji, Kabashima, Ikuo, and G. Donald Ferree, Jr. 1987. Elites and the Idea of Equality. Cambridge, MA: Harvard University Press.
Verba, Sidney, Nie, Norman, and Kim, Jae-On. 1978. Participation and Political Equality. Cambridge: Cambridge University Press.
Webb, Eugene J., Campbell, Donald T., Schwartz, Richard D., and Sechrest, Lee. 1966. Unobtrusive Measures: Nonreactive Research in the Social Sciences. Chicago: Rand McNally.
Zeitsch, John, Lawrence, Denis, and Salernian, John. 1994. “Comparing Like with Like in Productivity Studies: Apples, Oranges and Electricity.Economic Record 70 (June): 16270.
Zeller, Richard A., and Carmines, Edward G.. 1980. Measurement in the Social Sciences: The Link between Theory and Data. Cambridge: Cambridge University Press.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

American Political Science Review
  • ISSN: 0003-0554
  • EISSN: 1537-5943
  • URL: /core/journals/american-political-science-review
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 10
Total number of PDF views: 3156 *
Loading metrics...

Abstract views

Total abstract views: 13006 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 17th July 2018. This data will be updated every 24 hours.