Skip to main content
    • Aa
    • Aa

Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research


We address two long-standing survey research problems: measuring complicated concepts, such as political freedom and efficacy, that researchers define best with reference to examples; and what to do when respondents interpret identical questions in different ways. Scholars have long addressed these problems with approaches to reduce incomparability, such as writing more concrete questions—with uneven success. Our alternative is to measure directly response category incomparability and to correct for it. We measure incomparability via respondents' assessments, on the same scale as the self-assessments to be corrected, of hypothetical individuals described in short vignettes. Because the actual (but not necessarily reported) levels of the vignettes are invariant over respondents, variability in vignette answers reveals incomparability. Our corrections require either simple recodes or a statistical model designed to save survey administration costs. With analysis, simulations, and cross-national surveys, we show how response incomparability can drastically mislead survey researchers and how our approach can alleviate this problem.

Corresponding author
David Florence Professor of Government, Harvard University, Center for Basic Research in the Social Sciences, Cambridge MA 02138. (http://GKing.Harvard.Edu, King@Harvard.Edu).
Executive Director, Evidence and Information for Policy, World Health Organization, Geneva, Switzerland (
Assistant Professor of International Health, Harvard School of Public Health, Center for Population and Development Studies, Cambridge, MA (
Health Economist, Evidence and Information for Policy, World Health Organization, Geneva, Switzerland (
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

AldrichJohn H., Richard D.McKelvey. 1977A Method of Scaling with Applications to the 1968 and 1972 Presidential Elections.” American Political Science Review 71 (March): 111–30.

AltJames, BoSarlvik, IvorCrewe. 1976Individual Differences Scaling and Group Attitude Structures: British Party Imagery in 1974.” Quality and Quantity 10 (October): 297320.

BaumLawrence. 1988Measuring Policy Change in the U.S. Supreme Court.” American Political Science Review 82 (September): 905–12.

CarollJ. D., J. J.Chang. 1970Analysis of Individual Differences in Multidimensional Scaling.” Psychometrika 35 (September): 283319.

CheungGordon W., Roger B.Rensvold. 2000Assessing Extreme and Acquiescence Response Sets in Cross-Cultural Research Using Structural Equations Modeling (with Comments).” Journal of Cross-Cultural Psychology 31 (March): 187212.

ClarksonDouglas B. 2000A Random Effects Individual Difference Multidimensional Scaling Model.” Computational Statistics and Data Analysis 32 (January): 337–47.

GelmanAndrew, and GaryKing. 1994A Unified Method of Evaluating Electoral Systems and Redistricting Plans.” American Journal of Political Science 38 (June): 514–54.

GrootWim, HenrietteMaassen van den Brink. 1999Job Satisfaction and Preference Drift.” Economics Letters 63 (June): 363–67.

GrosecloseTim, Steven D.Levitt, JamesSnyder. 1999Comparing Interest Group Scores Across Time and Chambers: Adjusted ADA Scores for the U.S. Congress.” American Political Science Review 93 (March): 3350.

HeckmanJames, JamesSnyder. 1997Linear Probabilty Models of the Demand for Attributes with an Empirical Application to Estimating the Preferences of Legislators.” Rand Journal of Economics 28 (Special Issue): 142–89.

KahnemanDaniel, DavidSchkade, and Cass R.Sunstein. 1998Shared Outrage and Erratic Awards: The Psychology of Punitive Damages.” Journal of Risk and Uncertainty 16 (April): 4986.

LewisJeffrey B. 2001Estimating Voter Preference Distri butions from Individual-Level Voting Data.” Political Analysis 9 (Summer): 275–97.

MartinElizabeth A., Pamela C.Campanelli, Robert E.Fay. 1991An Application of Rasch Analysis to Questionnaire Design: Using Vignettes to Study the Meaning of ‘Work’ in the Current Population Survey.” The Statistician 40 (September): 265–76.

MeadA. 1992Review of the Development of Multidimensional Scaling Methods.” The Statistician 41 (April): 2739.

PalfreyThomas R., Keith T.Poole. 1987The Relationship between Information, Ideology, and Voter Behavior.” American Journal of Political Science 31 (September): 511–30.

PiqueroAlex R., RandallMacintosh. 2002The Validity of a Self-Reported Delinquency Scale: Comparisons across Gender, Age, Race, and Place of Residence.” Sociological Methods and Research 30 (May): 492–529.

PooleKeith T. 1998Recovering a Basic Space from a Set of Issue Scales.” American Journal of Political Science 42 (September): 954–93.

PooleKeith R. StevenDaniels. 1985Ideology, Party, and Voting in the U.S. Congress, 1959–1980.” American Political Science Review 79 (June): 373–99.

PooleKeith, and HowardRosenthal. 1991Patterns of Congressional Voting.” American Journal of Political Science 35 (February): 228–78.

PrzeworskiAdam, and HenryTeune. 1966–67Equivalence in Cross-National Research.” Public Opinion Quarterly 30 (Winter): 551–68.

SenAmartya. 2002Health: Perception versus Observation.” British Medical Journal 324 (April 13): 860–61.

ShealyR., W.Stout. 1993A Model-Based Standardization Approach That Separates True Bias/DIF from Group Ability Differences and Detects Test Bias/DIF as Well as Item Bias/DIF.” Psychometrika 58 (June): 159–94.

SnidermanPaul M., Douglas B.Grob. 1996Innovations in Experimental Design in Attitude Surveys.” Annual Review of Sociology 22 (August): 377–99.

StewartAnita L., AnnaNapoles-Springer. 2000Health-Related Quality of Life Assessments in Diverse Population Groups in the United States.” Medical Care 38 (September): II–102– II–124.

SuchmanL., B.Jordan. 1990Interactional Troubles in Face to Face Survey Interviews (with Comments and Rejoinder).” Journal of the American Statistical Association 85 (March): 232–53.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

American Political Science Review
  • ISSN: 0003-0554
  • EISSN: 1537-5943
  • URL: /core/journals/american-political-science-review
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Altmetric attention score

Full text views

Total number of HTML views: 9
Total number of PDF views: 166 *
Loading metrics...

Abstract views

Total abstract views: 737 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 26th June 2017. This data will be updated every 24 hours.