Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Home
  • Home
Article
  • Cited by 78

  • Get access
    Add to cart USD25.00
    Check if you have access via personal or institutional login
    Log in Register

Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research

  • GARY KING (a1), CHRISTOPHER J. L. MURRAY (a2), JOSHUA A. SALOMON (a1) and AJAY TANDON (a2)

Abstract

We address two long-standing survey research problems: measuring complicated concepts, such as political freedom and efficacy, that researchers define best with reference to examples; and what to do when respondents interpret identical questions in different ways. Scholars have long addressed these problems with approaches to reduce incomparability, such as writing more concrete questions—with uneven success. Our alternative is to measure directly response category incomparability and to correct for it. We measure incomparability via respondents' assessments, on the same scale as the self-assessments to be corrected, of hypothetical individuals described in short vignettes. Because the actual (but not necessarily reported) levels of the vignettes are invariant over respondents, variability in vignette answers reveals incomparability. Our corrections require either simple recodes or a statistical model designed to save survey administration costs. With analysis, simulations, and cross-national surveys, we show how response incomparability can drastically mislead survey researchers and how our approach can alleviate this problem.

Copyright

© 2003 by the American Political Science Association

Corresponding author

Professor of Government, Harvard University, Center for Basic Research in the Social Sciences, Cambridge MA 02138. (http://GKing.Harvard.Edu, King@Harvard.Edu).
Executive Director, Evidence and Information for Policy, World Health Organization, Geneva, Switzerland (MurrayC@WHO.int).
Assistant Professor of International Health, Harvard School of Public Health, Center for Polulation and Development Studies, Cambridge, MA (JSalomon@hsph.harvard.edu).
Health Economist, Evidence and Information for Policy, World Health Organization, Geneva, Switzerland (TandonA@WHO.int).

References

Hide All
Aldrich John H. and Richard D. McKelvey 1977 A Method of Scaling with Applications to the 1968 and 1972 Presidential Elections. American Political Science Review 71 (March): 111–30.
Alt James Bo Sarlvik and Ivor Crewe. 1976 Individual Differences Scaling and Group Attitude Structures: British Party Imagery in 1974. Quality and Quantity 10 (October): 297 320.
Baum Lawrence 1988 Measuring Policy Change in the U.S. Supreme Court. American Political Science Review 82 (September): 905–12
Brady Henry E. 1985 The Perils of Survey Research: Inter-Personally Incomparable Responses. Political Methodology 11 (3–4): 269–90.
Brady Henry E. 1989 Factor and Ideal Point Analysis for Interpersonally Incomparable Data. Psychometrika 542 (2): 181 202.
Cantril Hadley 1965 The Pattern of Human Concerns. New Brunswick, NJ: Rutgers University Press.
Caroll J. D. and J. J. Chang. 1970 Analysis of Individual Differences in Multidimensional Scaling. Psychometrika 35 (3): 283 319.
Cheung Gordon W. and Roger B. Rensvold 2000 Assessing Extreme and Acquiescence Response Sets in Cross-Cultural Research Using Structural Equations Modeling (with Comments). Journal of Cross-Cultural Psychology 31 (2): 187 212.
Clarkson Douglas B. 2000 A Random Effects Individual Difference Multidimensional Scaling Model. Computational Statistics and Data Analysis 32 (January): 337–47.
Clinton Joshua Simon Jackman and Douglas Rivers 2002 The Statistical Analysis of Roll Call Data. Ph.D. diss. Stanford University.
Gelman Andrew Gary King. 1994 A Unified Method of Evaluating Electoral Systems and Redistricting Plans. American Journal of Political Science 38 (June): 514–54.
Green Donald P. and Alan Gerber 2001 Reclaiming the Experimental Tradition in Political Science. In Political Science: State of the Discipline, III. ed. Helen Milner and Ira Katznelson Washington, DC: APSA.
Groot Wim and Henriette Maassen van den Brink. 1999 Job Satisfaction and Preference Drift. Economics Letters 63 (June): 363–67.
Groseclose Tim Steven D. Levitt and James Snyder 1999 Comparing Interest Group Scores Across Time and Chambers: Adjusted ADA Scores for the U.S. Congress. American Political Science Review 93 (March): 33 50.
Heckman James and James Snyder 1997 Linear Probabilty Models of the Demand for Attributes with an Empirical Application to Estimating the Preferences of Legislators. Rand Journal of Economics 28 (Special Issue): 142–89.
Holland Paul W. and Howard Wainer eds. 1993 Differential Item Functioning. Hillsdale, NJ: Lawrence Erlbaum.
Johnson Timothy P. 1998 Approaches to Equivalence in Cross-Cultural and Cross-National Survey Research. ZUMA Nachrichten Spezial 3 1–40.
Johnson Valen E. and James H. Albert 1999 Ordinal Data Modeling. New York: Springer.
Kahneman Daniel David Schkade and Cass R. Sunstein 1998 Shared Outrage and Erratic Awards: The Psychology of Punitive Damages. Journal of Risk and Uncertainty 16 (April): 49 86.
Kinder Donald R. and Thomas R. Palfrey eds. 1993 Experimental Foundations of Political Science. Ann Arbor: University of Michigan Press.
King Gary 1997 A Solution to the Ecological Inference Problem. Reconstructing Individual Behavior from Aggregate Data Princeton, NJ: Princeton University Press.
King Gary James Honaker Anne Joseph and Kenneth Scheve 2001 Analyzing Incomplete Political Science Data: An Alternative Algorithm for Multiple Imputation. American Political Science Review 95 (March): 49 69.
Lewis Jeffrey B. 2001 Estimating Voter Preference Distributions from Individual-Level Voting Data. Political Analysis 9 (3): 275–97.
Linden Wim Van Der and Ronald K. Hambleton eds. 1997 Handbook of Modern Item Response Theory. New York: Springer.
Londregan John 2000 Estimating Legislator's Preferred Points. Political Analysis 8 (1): 21 34.
Martin Elizabeth A. Pamela C. Campanelli and Robert E. Fay 1991 An Application of Rasch Analysis to Questionnaire Design: Using Vignettes to Study the Meaning of ‘Work' in the Current Population Survey. The Statistician 40 (3): 265–76.
Mead A. 1992 Review of the Development of Multidimensional Scaling Methods. The Statistician 41 (1): 27 39.
Palfrey Thomas R. and Keith T. Poole 1987 The Relationship between Information, Ideology, and Voter Behavior. American Journal of Political Science 31 (September): 511–30.
Piquero Alex R. and Randall Macintosh 2002 The Validity of a Self-Reported Delinquency Scale: Comparisons across Gender, Age, Race, and Place of Residence. Sociological Methods and Research 30 (4): 492 529.
Poole Keith T. 1998 Recovering a Basic Space from a Set of Issue Scales. American Journal of Political Science 42 (September): 954–93.
Poole Keith and R. Steven Daniels 1985 Ideology, Party, and Voting in the U.S. Congress, 1959–1980. American Political Science Review 79 (June): 373–99.
Poole Keith and Howard Rosenthal. 1991 Patterns of Congressional Voting. American Journal of Political Science 35 (February): 228–78.
Przeworski Adam and Henry Teune. 1966–67 Equivalence in Cross-National Research. Public Opinion Quarterly 30 (4): 551–68.
Rossi P. H. and S. L. Nock eds. 1983 Measuring Social Judgements: The Factorial Survey Approach. Beverly Hills: CA: Sage.
Sen Amartya 2002 Health: Perception versus Observation. British Medical Journal 324 (April 13): 860–61.
Shealy R. and W. Stout 1993 A Model-Based Standardization Approach That Separates True Bias/DIF from Group Ability Differences and Detects Test Bias/DIF as Well as Item Bias/DIF. Psychometrika 58 (2): 159–94.
Sniderman Paul M. and Douglas B. Grob 1996 Innovations in Experimental Design in Attitude Surveys. Annual Review of Sociology 22 (August): 377–99.
Stewart Anita L. and Anna Napoles-Springer 2000 Health-Related Quality of Life Assessments in Diverse Population Groups in the United States. Medical Care 38 (9): II–102 II–124.
Suchman L. and B. Jordan 1990 Interactional Troubles in Face to Face Survey Interviews (with Comments and Rejoinder). Journal of the American Statistical Association 85 (409): 232–53.
Thissen David Lynn Steinberg Howard Wainer 1993 Detection of Differential Item Functioning Using the Parameters of the Item Response Models. In Differential Item Functioning. Paul W. Holland and Howard Wainer ed. Hillsdale, NJ: Lawrence Erlbaum.
Torgerson Warren S. 1958 Theory and Methods of Scaling. New York: Wiley and Sons.
Wolfe Rory and David Firth 2002 Modelling Subjective Use of an Ordinal Reponse Scale in a Many Period Crossover Experiment. Applied Statistics 51 (2): 245–55.

Related content

Powered by UNSILO
  • Cited by 78

  • Get access
    Add to cart USD25.00
    Check if you have access via personal or institutional login
    Log in Register

Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research

  • GARY KING (a1), CHRISTOPHER J. L. MURRAY (a2), JOSHUA A. SALOMON (a1) and AJAY TANDON (a2)

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed.