Skip to main content Accessibility help
×
Home
Hostname: page-component-55597f9d44-fnprw Total loading time: 0.23 Render date: 2022-08-12T04:33:29.458Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "useNewApi": true } hasContentIssue true

The Logic of the Survey Experiment Reexamined

Published online by Cambridge University Press:  04 January 2017

Brian J. Gaines
Affiliation:
Department of Political Science and Institute of Government and Public Affairs, University of Illinois at Urbana-Champaign, 702 South Wright Street, Urbana, IL 61801. e-mail: bjgaines@uiuc.edu (corresponding author)
James H. Kuklinski
Affiliation:
Department of Political Science and Institute of Government and Public Affairs, University of Illinois at Urbana-Champaign, 702 South Wright Street, Urbana, IL 61801. e-mail: kuklinsk@ad.uiuc.edu
Paul J. Quirk
Affiliation:
Department of Political Science, University of British Columbia, 2329 West Mall, Vancouver, British Columbia, Canada, V6T 1Z4. e-mail: quirk@politics.ubc.ca

Abstract

Scholars of political behavior increasingly embed experimental designs in opinion surveys by randomly assigning respondents alternative versions of questionnaire items. Such experiments have major advantages: they are simple to implement and they dodge some of the difficulties of making inferences from conventional survey data. But survey experiments are no panacea. We identify problems of inference associated with typical uses of survey experiments in political science and highlight a range of difficulties, some of which have straightforward solutions within the survey-experimental approach and some of which can be dealt with only by exercising greater caution in interpreting findings and bringing to bear alternative strategies of research.

Type
Research Article
Copyright
Copyright © The Author 2006. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Achen, Christopher H. 1986. The statistical analysis of quasi-experiments. Berkeley, CA: University of California Press.Google Scholar
Ansolabehere, Stephen, and Iyengar, Shanto. 1995. Going negative: How attack ads shrink and polarize the electorate. New York: Free Press.Google Scholar
Brady, Henry E., and Collier, David, eds. 2000. Rethinking social inquiry: Diverse tools, shared standards. Lanham, MD: Rowman and Littlefield.Google Scholar
Clarke, Harold D., Kornberg, Allan, McIntyre, Chris, Bauer-Kaase, Petra, and Kaase, Max. 1999. The effect of economic priorities on the measurement of value change: New experimental evidence. American Political Science Review 93: 637–47.CrossRefGoogle Scholar
Davis, Darren W., and Silver, Brian D. 2003. Stereotype threat and race of interviewer effects in a survey on political knowledge. American Journal of Political Science 47: 3345.CrossRefGoogle Scholar
Druckman, James N. 2001. The implications of framing effects for citizen competence. Political Behavior 23: 225–56.CrossRefGoogle Scholar
Druckman, James N., and Lupia, Arthur. 2006. Mind, will, and choice. In The Oxford handbook on contextual political analysis, ed. Tilly, Charles and Goodwin, Robert E., 97113. Oxford: Oxford University Press.Google Scholar
Druckman, James N., and Nelson, Kjersten R. 2003. Framing and deliberation: How citizens' conversations limit elite influence. American Journal of Political Science 47: 729–45.CrossRefGoogle Scholar
Fearon, James D. 1991. Counterfactuals and hypothesis testing in political science. World Politics 43: 169–95.CrossRefGoogle Scholar
Gibson, James. 1998. A sober second thought: An experiment in persuading Russians to tolerate. American Journal of Political Science 42: 819–50.CrossRefGoogle Scholar
Gilens, Martin. 1999. Why Americans hate welfare: Race, media, and the politics of anti-poverty reform. Chicago, IL: University of Chicago Press.CrossRefGoogle Scholar
Gilens, Martin. 2001. Political ignorance and collective policy preferences. American Political Science Review 95: 379–98.CrossRefGoogle Scholar
Grant, J. Tobin, and Rudolph, Thomas J. 2003. Value conflict, group affect, and the issue of campaign finance. American Journal of Political Science 47: 453–69.CrossRefGoogle Scholar
Heckman, James J., Ichimura, Hidehiko, and Todd, Petra E. 1997. Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. The Review of Economic Studies 64: 605–54.CrossRefGoogle Scholar
Heckman, James J., Ichimura, Hidehiko, and Todd, Petra E. 1998. Matching as an econometric evaluation estimator. The Review of Economic Studies 65: 261–94.CrossRefGoogle Scholar
Heckman, James, Ichimura, Hidehiko, Smith, Jeffrey, and Todd, Petra. 1998. Characterizing selection bias using experimental data. Econometrica 66: 1017–98.CrossRefGoogle Scholar
Hovland, Carl I., and Weiss, Walter. 1951. The influence of source credibility on communication effectiveness. Public Opinion Quarterly 15: 635–50.CrossRefGoogle Scholar
Hurwitz, Jon, and Peffley, Mark. 1997. Public perception of race and crime: The role of racial stereotypes. American Journal of Political Science 41: 375401.CrossRefGoogle Scholar
Hyman, Herbert H., and Sheatsley, Paul B. 1950. The current status of American public opinion. In The teaching of contemporary affairs, ed. Payne, J. C., 1134. Washington, DC: National Council for the Social Studies.Google Scholar
Inglehart, Ronald. 1990. Culture shift in advanced industrial society. Princeton, NJ: Princeton University Press.Google Scholar
Kinder, Donald R., and Sanders, Lynn M. 1996. Divided by color: Racial politics and democratic ideals. Chicago, IL: University of Chicago Press.Google Scholar
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing social inquiry: Scientific inference in qualitative research. Princeton, NJ: Princeton University Press.Google Scholar
Kuklinski, James H., Cobb, Michael D., and Gilens, Martin. 1997. Racial attitudes and the “New South.” Journal of Politics 59: 323–49.CrossRefGoogle Scholar
Kuklinski, James H., Riggle, Ellen, Ottati, Victor, Schwarz, Norbert, and Wyer, Robert S. Jr. 1991. The cognitive and affective bases of political tolerance judgments. American Journal of Political Science 35: 127.CrossRefGoogle Scholar
Lacy, Dean. 2001. A theory of nonseparable preferences in survey responses. American Journal of Political Science 45: 239–58.CrossRefGoogle Scholar
LaLonde, Robert J. 1986. Evaluating the econometric evaluations of training programs with experimental data. American Economic Review 76: 604–20.Google Scholar
Lieberson, Stanley. 1987. Making it count: The improvement of social research and theory. Berkeley, CA: University of California Press.Google Scholar
Lodge, Milton, Steenbergen, Marco R., and Brau, Shawn. 1995. The responsive voter: Campaign information and the dynamics of candidate evaluation. American Political Science Review 89: 309–26.CrossRefGoogle Scholar
Lupia, Arthur, and McCubbins, Mathew D. 1998. The democratic dilemma: Can citizens learn what they need to know? Cambridge: Cambridge University Press.Google Scholar
Luskin, Robert C., Fishkin, James S., and Jowell, Roger. 2002. Considered opinions: Deliberative polling in Britain. British Journal of Political Science 32: 455–87.CrossRefGoogle Scholar
Mondak, Jeffery J. 1993. Source cues and policy approval: The cognitive dynamics of public support for the Reagan agenda. American Journal of Political Science 37: 186212.CrossRefGoogle Scholar
Mutz, Diana C. 1998. Impersonal influence: How perceptions of mass collectives affect political attitudes. New York: Cambridge University Press.CrossRefGoogle Scholar
Mutz, Diana C., and Reeves, Byron. 2005. The new videomalaise: Effects of televised civility on political trust. American Political Science Review 99: 115.CrossRefGoogle Scholar
Nelson, Thomas E., Clawson, Rosalee A., and Oxley, Zoe M. 1997. Media framing of a civil liberties conflict and its effect on tolerance. American Political Science Review 91: 567–83.CrossRefGoogle Scholar
Peffley, Mark, Hurwitz, Jon, and Sniderman, Paul M. 1997. Racial stereotypes and whites' political views of blacks in the context of welfare and crime. American Journal of Political Science 41: 3060.CrossRefGoogle Scholar
Rosenbaum, Paul R., and Rubin, Donald B. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika 70: 4155.CrossRefGoogle Scholar
Schuman, Howard, and Bobo, Lawrence. 1988. Survey-based experiments on white racial attitudes toward residential segregation. American Journal of Sociology 94: 273–99.CrossRefGoogle Scholar
Smith, Alastair. 1999. Testing theories of strategic choice: The example of crisis escalation. American Journal of Political Science 43: 1254–83.CrossRefGoogle Scholar
Sniderman, Paul M. 1996. Innovations in experimental design in attitude surveys. Annual Review of Sociology 22: 377–99.CrossRefGoogle Scholar
Sniderman, Paul M., Brody, Richard A., and Tetlock, Philip E. 1991. Reasoning and choice: Explorations in political psychology. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Sniderman, Paul M., and Carmines, Edward G. 1997. Reaching beyond race. Cambridge, MA: Harvard University Press.Google Scholar
Sniderman, Paul M., and Piazza, Thomas. 1993. The scar of race. Cambridge, MA: Belknap Press.Google Scholar
Sniderman, Paul M., and Theriault, Sean. 2004. The dynamics of political argument and the logic of issue framing. In Studies in public opinion: Attitudes, nonattitudes, measurement error, and change, ed. Saris, Willem E. and Sniderman, Paul M., 133–65. Princeton, NJ: Princeton University Press.Google Scholar
Sniderman, Paul M., Hagendoorn, Louk, and Prior, Markus. 2004. Predispositional factors and situational triggers: Exclusionary reactions to immigrant minorities. American Political Science Review 98: 3550.CrossRefGoogle Scholar
Taber, Charles S., and Lodge, Milton. 2006. Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science 50: 755–69.CrossRefGoogle Scholar
Zaller, John. 1992. The nature and origins of mass opinion. New York: Cambridge University Press.CrossRefGoogle Scholar
290
Cited by

Save article to Kindle

To save this article to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

The Logic of the Survey Experiment Reexamined
Available formats
×

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

The Logic of the Survey Experiment Reexamined
Available formats
×

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

The Logic of the Survey Experiment Reexamined
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *