Skip to main content

The Logic of the Survey Experiment Reexamined

  • Brian J. Gaines (a1), James H. Kuklinski (a2) and Paul J. Quirk (a3)

Scholars of political behavior increasingly embed experimental designs in opinion surveys by randomly assigning respondents alternative versions of questionnaire items. Such experiments have major advantages: they are simple to implement and they dodge some of the difficulties of making inferences from conventional survey data. But survey experiments are no panacea. We identify problems of inference associated with typical uses of survey experiments in political science and highlight a range of difficulties, some of which have straightforward solutions within the survey-experimental approach and some of which can be dealt with only by exercising greater caution in interpreting findings and bringing to bear alternative strategies of research.

Hide All
Achen, Christopher H. 1986. The statistical analysis of quasi-experiments. Berkeley, CA: University of California Press.
Ansolabehere, Stephen, and Iyengar, Shanto. 1995. Going negative: How attack ads shrink and polarize the electorate. New York: Free Press.
Brady, Henry E., and Collier, David, eds. 2000. Rethinking social inquiry: Diverse tools, shared standards. Lanham, MD: Rowman and Littlefield.
Clarke, Harold D., Kornberg, Allan, McIntyre, Chris, Bauer-Kaase, Petra, and Kaase, Max. 1999. The effect of economic priorities on the measurement of value change: New experimental evidence. American Political Science Review 93: 637–47.
Davis, Darren W., and Silver, Brian D. 2003. Stereotype threat and race of interviewer effects in a survey on political knowledge. American Journal of Political Science 47: 3345.
Druckman, James N. 2001. The implications of framing effects for citizen competence. Political Behavior 23: 225–56.
Druckman, James N., and Lupia, Arthur. 2006. Mind, will, and choice. In The Oxford handbook on contextual political analysis, ed. Tilly, Charles and Goodwin, Robert E., 97113. Oxford: Oxford University Press.
Druckman, James N., and Nelson, Kjersten R. 2003. Framing and deliberation: How citizens' conversations limit elite influence. American Journal of Political Science 47: 729–45.
Fearon, James D. 1991. Counterfactuals and hypothesis testing in political science. World Politics 43: 169–95.
Gibson, James. 1998. A sober second thought: An experiment in persuading Russians to tolerate. American Journal of Political Science 42: 819–50.
Gilens, Martin. 1999. Why Americans hate welfare: Race, media, and the politics of anti-poverty reform. Chicago, IL: University of Chicago Press.
Gilens, Martin. 2001. Political ignorance and collective policy preferences. American Political Science Review 95: 379–98.
Grant, J. Tobin, and Rudolph, Thomas J. 2003. Value conflict, group affect, and the issue of campaign finance. American Journal of Political Science 47: 453–69.
Heckman, James J., Ichimura, Hidehiko, and Todd, Petra E. 1997. Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. The Review of Economic Studies 64: 605–54.
Heckman, James J., Ichimura, Hidehiko, and Todd, Petra E. 1998. Matching as an econometric evaluation estimator. The Review of Economic Studies 65: 261–94.
Heckman, James, Ichimura, Hidehiko, Smith, Jeffrey, and Todd, Petra. 1998. Characterizing selection bias using experimental data. Econometrica 66: 1017–98.
Hovland, Carl I., and Weiss, Walter. 1951. The influence of source credibility on communication effectiveness. Public Opinion Quarterly 15: 635–50.
Hurwitz, Jon, and Peffley, Mark. 1997. Public perception of race and crime: The role of racial stereotypes. American Journal of Political Science 41: 375401.
Hyman, Herbert H., and Sheatsley, Paul B. 1950. The current status of American public opinion. In The teaching of contemporary affairs, ed. Payne, J. C., 1134. Washington, DC: National Council for the Social Studies.
Inglehart, Ronald. 1990. Culture shift in advanced industrial society. Princeton, NJ: Princeton University Press.
Kinder, Donald R., and Sanders, Lynn M. 1996. Divided by color: Racial politics and democratic ideals. Chicago, IL: University of Chicago Press.
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing social inquiry: Scientific inference in qualitative research. Princeton, NJ: Princeton University Press.
Kuklinski, James H., Cobb, Michael D., and Gilens, Martin. 1997. Racial attitudes and the “New South.” Journal of Politics 59: 323–49.
Kuklinski, James H., Riggle, Ellen, Ottati, Victor, Schwarz, Norbert, and Wyer, Robert S. Jr. 1991. The cognitive and affective bases of political tolerance judgments. American Journal of Political Science 35: 127.
Lacy, Dean. 2001. A theory of nonseparable preferences in survey responses. American Journal of Political Science 45: 239–58.
LaLonde, Robert J. 1986. Evaluating the econometric evaluations of training programs with experimental data. American Economic Review 76: 604–20.
Lieberson, Stanley. 1987. Making it count: The improvement of social research and theory. Berkeley, CA: University of California Press.
Lodge, Milton, Steenbergen, Marco R., and Brau, Shawn. 1995. The responsive voter: Campaign information and the dynamics of candidate evaluation. American Political Science Review 89: 309–26.
Lupia, Arthur, and McCubbins, Mathew D. 1998. The democratic dilemma: Can citizens learn what they need to know? Cambridge: Cambridge University Press.
Luskin, Robert C., Fishkin, James S., and Jowell, Roger. 2002. Considered opinions: Deliberative polling in Britain. British Journal of Political Science 32: 455–87.
Mondak, Jeffery J. 1993. Source cues and policy approval: The cognitive dynamics of public support for the Reagan agenda. American Journal of Political Science 37: 186212.
Mutz, Diana C. 1998. Impersonal influence: How perceptions of mass collectives affect political attitudes. New York: Cambridge University Press.
Mutz, Diana C., and Reeves, Byron. 2005. The new videomalaise: Effects of televised civility on political trust. American Political Science Review 99: 115.
Nelson, Thomas E., Clawson, Rosalee A., and Oxley, Zoe M. 1997. Media framing of a civil liberties conflict and its effect on tolerance. American Political Science Review 91: 567–83.
Peffley, Mark, Hurwitz, Jon, and Sniderman, Paul M. 1997. Racial stereotypes and whites' political views of blacks in the context of welfare and crime. American Journal of Political Science 41: 3060.
Rosenbaum, Paul R., and Rubin, Donald B. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika 70: 4155.
Schuman, Howard, and Bobo, Lawrence. 1988. Survey-based experiments on white racial attitudes toward residential segregation. American Journal of Sociology 94: 273–99.
Smith, Alastair. 1999. Testing theories of strategic choice: The example of crisis escalation. American Journal of Political Science 43: 1254–83.
Sniderman, Paul M. 1996. Innovations in experimental design in attitude surveys. Annual Review of Sociology 22: 377–99.
Sniderman, Paul M., Brody, Richard A., and Tetlock, Philip E. 1991. Reasoning and choice: Explorations in political psychology. Cambridge: Cambridge University Press.
Sniderman, Paul M., and Carmines, Edward G. 1997. Reaching beyond race. Cambridge, MA: Harvard University Press.
Sniderman, Paul M., and Piazza, Thomas. 1993. The scar of race. Cambridge, MA: Belknap Press.
Sniderman, Paul M., and Theriault, Sean. 2004. The dynamics of political argument and the logic of issue framing. In Studies in public opinion: Attitudes, nonattitudes, measurement error, and change, ed. Saris, Willem E. and Sniderman, Paul M., 133–65. Princeton, NJ: Princeton University Press.
Sniderman, Paul M., Hagendoorn, Louk, and Prior, Markus. 2004. Predispositional factors and situational triggers: Exclusionary reactions to immigrant minorities. American Political Science Review 98: 3550.
Taber, Charles S., and Lodge, Milton. 2006. Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science 50: 755–69.
Zaller, John. 1992. The nature and origins of mass opinion. New York: Cambridge University Press.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed