Skip to main content Accessibility help
×
×
Home

Are Survey Experiments Externally Valid?

  • JASON BARABAS (a1) and JENNIFER JERIT (a1)
Abstract

Researchers use survey experiments to establish causal effects in descriptively representative samples, but concerns remain regarding the strength of the stimuli and the lack of realism in experimental settings. We explore these issues by comparing three national survey experiments on Medicare and immigration with contemporaneous natural experiments on the same topics. The survey experiments reveal that providing information increases political knowledge and alters attitudes. In contrast, two real-world government announcements had no discernable effects, except among people who were exposed to the same facts publicized in the mass media. Even among this exposed subsample, treatment effects were smaller and sometimes pointed in the opposite direction. Methodologically, our results suggest the need for caution when extrapolating from survey experiments. Substantively, we find that many citizens are able to recall factual information appearing in the news but may not adjust their beliefs and opinions in response to this information.

Copyright
Corresponding author
Jason Barabas is Associate Professor, Department of Political Science, Florida State University, 531 Bellamy Building, Tallahassee, FL 32306 (jason.barabas@fsu.edu).
Jennifer Jerit is Associate Professor, Department of Political Science, Florida State University, 531 Bellamy Building, Tallahassee, FL 32306 (jjerit@fsu.edu).
References
Hide All
Abrajano, Marisa A. 2005. “Who Evaluates a Presidential Candidate by Using Non-policy Campaign Messages?Political Research Quarterly 58 (March): 5567.
Achen, Chris. 1982. Interpreting and Using Regression. Thousand Oaks, CA: Sage.
Arceneaux, Kevin, and Johnson, Martin. 2007. “Channel Surfing: Does Choice Reduce Videolmalaise.” Presented at the 2007 Annual Meeting of the Midwest Political Science Association, Chicago.
Arceneaux, Kevin, and Kolodny, Robin. 2009. “Educating the Least Informed: Group Endorsements in a Grassroots Campaign.” American Journal of Political Science 53 (October): 755–70.
Barabas, Jason. 2004. “How Deliberation Affects Policy Opinions.” American Political Science Review 98 (November): 687701.
Barabas, Jason, and Jerit, Jennifer. 2009. “Estimating the Causal Effects of Media Coverage on Policy-specific Knowledge.” American Journal of Political Science 53 (January): 7389.
Bennett, W. Lance. 2006. News: The Politics of Illusion. 7th ed. New York: Longman.
Benz, Matthias, and Meier, Stephan. 2008. “Do People Behave in Experiments as in the Field? Evidence from Donations.” Experimental Economics 11: 268–81.
Berinsky, Adam J. 2007. “Assuming the Costs of War: Events, Elites, and American Public Support for Military Conflict.” Journal of Politics 69 (November): 975–97.
Blalock, H.M. Jr. 1979. Social Statistics. 2nd ed. New York: McGraw-Hill.
Brader, Ted. 2005. “Striking a Responsive Chord: How Political Ads Motivate and Persuade Voters by Appealing to Emotions.” American Journal of Political Science 49 (April): 388405.
Brehm, John. 1993. The Phantom Respondents: Opinion Surveys and Political Representation. Ann Arbor: University of Michigan Press.
Cameron, A. Colin, and Trivedi, Pravin K.. 2005. Microeconometrics: Methods and Applications. New York: Cambridge University Press.
Campbell, Donald T., and Stanley, Julian C.. 1963. Experimental and Quasi-experimental Designs for Research. Chicago: Rand McNally.
Chong, Dennis, and Druckman, James N.. 2007. “Framing Public Opinion in Competitive Democracies.” American Political Science Review 101 (November): 637–56.
Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum.
Converse, Philip E. 1964. “The Nature of Belief Systems in Mass Publics.” In Ideology and Discontent, ed. Apter, David E.. New York: Free Press.
Cook, Fay Lomax, Jacobs, Lawrence R., and Kim, Dukhong. 2010. “Trusting What You Know: Information, Knowledge, and Confidence in Social Security.” Journal of Politics 72 (April): 397412.
Cook, Thomas D., and Campbell, Donald T.. 1979. Quasi-experimentation: Design & Analysis Issues for Field Settings. New York: Houghton Mifflin.
Cook, Thomas D., Shadish, William R., and Wong, Vivian C.. 2008. “Three Conditions under Which Experimental and Observational Studies Produce Comparable Causal Estimates: New Findings from Within-study Comparisons.” Journal of Policy Analysis and Management 27 (Autumn): 724–50.
Delli Carpini, Michael X. 2009. “The Psychology of Citizen Learning.” In The Political Psychology of Democratic Citizenship, eds. Borgida, Eugene, Federico, Christopher M., and Sullivan, John L.. New York: Oxford University Press, 2351.
Druckman, James N. 2004. “Political Preference Formation: Competition, Deliberation, and the (Ir)relevance of Framing Effects.” American Political Science Review 98 (November): 671–86.
Druckman, James N., Green, Donald P., Kuklinski, James H., and Lupia, Arthur. 2006. “The Growth and Development of Experimental Research in Political Science.” American Political Science Review 100 (November): 627–35.
Druckman, James N., and Nelson, Kjersten R.. 2003. “Framing and Deliberation: How Citizens’ Conversations Limit Elite Influence.” American Journal of Political Science 47 (October): 729–45.
Dunning, Thad. 2008. “Improving Causal Inference: Strengths and Limitations of Natural Experiments.Political Research Quarterly 61 (2): 282–93.
Eagly, Alice, and Chaiken, Shelly. 1993. The Psychology of Attitudes. New York: Harcourt Brace.
Gaines, Brian J., and Kuklinski, James H.. 2009. “Including Self-selection in Random Assignment Experiments.” University of Illinois at Urbana–Champaign. Unpublished manuscript.
Gaines, Brian J., Kuklinski, James H., and Quirk, Paul J.. 2007. “The Logic of the Survey Experiment Reexamined.” Political Analysis 15 (Winter): 120.
Gelman, Andrew. 2007. “Struggles with Survey Weighting and Regression Modeling.” Statistical Science 22 (2): 153–64.
Gerber, Alan, and Green, Donald P.. 2000. “The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment.” American Political Science Review 94 (September): 653–63.
Gibson, James. 1998. “A Sober Second Thought: An Experiment in Persuading Russians to Tolerate.” American Journal of Political Science 42 (July): 819–50.
Gilens, Martin. 2001. “Political Ignorance and Collective Policy Preferences.” American Political Science Review 95 (June): 379–96.
Gilens, Martin. 2002. “An Anatomy of Survey-Based Experiments.” In Navigating Public Opinion. eds. Manza, J., Cook, F. L., and Page, B. I.. New York: Oxford, 232–50.
Gramlich, Edward M. 1998. Is It Time to Reform Social Security? Ann Arbor: University of Michigan Press.
Grosser, Jens, and Schram, Arthur. 2006. “Neighborhood Information Exchange and Voter Participation: An Experimental Study.” American Political Science Review 100 (May): 235–48.
Hovland, Carl. 1959. “Reconciling Conflicting Results Derived from Experimental and Survey Studies of Attitude Change.” American Psychologist 14: 817.
Huber, Gregory A., and Arceneaux, Kevin. 2007. “Identifying the Persuasive Effects of Presidential Advertising.” American Journal of Political Science 51 (October): 957–77.
Iyengar, Shanto, and Kinder, Donald. 1987. News That Matters. Chicago: University of Chicago Press.
Jacobs, Lawrence R., Cook, Fay Lomax, and Delli Carpini, Michael X.. 2009. Talking Together: Public Deliberation and Political Participation in America. Chicago: University of Chicago Press.
Jacobs, Lawrence R., and Shapiro, Robert Y.. 1998. “Myths and Misunderstandings about Public Opinion toward Social Security.” In Framing the Social Security Debate: Values, Politics, and Economics, eds. Arnold, R. Douglas, Graetz, Michael J., and Munnell, Alicia H.. Washington, DC: Brookings Institution Press, 355–88.
Jerit, Jennifer. 2009. “How Predictive Appeals Shape Policy Opinions.” American Journal of Political Science 53 (April): 411–26.
Jerit, Jennifer, and Barabas, Jason. 2006. “Bankrupt Rhetoric: How Misleading Information Affects Knowledge about Social Security.” Public Opinion Quarterly 70 (Fall): 278303.
Jerit, Jennifer, Kuklinski, James H., and Quirk, Paul J.. 2009. “Strategic Rhetoric, Emotional Citizens, and the Rhetoric of Prediction.” In The Political Psychology of Democratic Citizenship, eds. Borgida, Eugene, Sullivan, John L., and Federico, Christopher M.. New York: Oxford University Press, 100–24.
Kam, Cindy D., Wilking, Jennifer R., and Zechmeister, Elizabeth J.. 2007. “Beyond the ‘Narrow Database’: Another Convenience Sample for Experimental Research.” Political Behavior 29 (December): 415–40.
Kellstedt, Paul M., and Whitten, Guy D.. 2009. The Fundamentals of Political Science Research. New York: Cambridge University Press.
Kennedy, Peter. 2003. A Guide to Econometrics. 5th ed. Cambridge, MA: MIT Press.
Keppel, Geoffrey. 1982. Design and Analysis: A Researcher's Handbook. 2nd ed. Englewood Cliffs, NJ: Prentice-Hall.
Kinder, Donald. 2007. “Curmudgeonly Advice.” Journal of Communication 57 (March): 155–62.
Kinder, Donald R., and Palfrey, Thomas R., eds. 1993. Experimental Foundations of Political Science. Ann Arbor: University of Michigan Press.
Kinder, Donald R., and Sanders, Lynn M.. 1996. Divided by Color: Racial Politics and Democratic Ideals. Chicago: University of Chicago Press.
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press.
Krosnick, Jon A., and Schuman, Howard. 1988. “Attitude Intensity, Importance, and Certainty and Susceptibility to Response Effects.” Journal of Personality and Social Psychology 54 (6): 940–52.
Kuklinski, James H., Quirk, Paul J., Jerit, Jennifer, and Rich, Robert F.. 2001. “The Political Environment and Citizen Competence.” American Journal of Political Science 45 (April): 410–24.
Kuklinski, James H., Quirk, Paul J., Jerit, Jennifer, Schwieder, David, and Rich, Robert F.. 2000. “Misinformation and the Currency of Citizenship.” Journal of Politics 62 (August): 790816.
Lassen, David Dreyer. 2005. “The Effect of Information on Voter Turnout: Evidence from a Natural Experiment.” American Journal of Political Science 49 (January): 103–88.
Lavine, Howard. 2002. “On-line versus Memory Based Process Models of Political Evaluation.” In Political Psychology, ed. Monroe, Kristen. Mahwah, NJ: Lawrence Erlbaum Associates, 225–47.
Levitt, Steven D., and List, John A.. 2007. “What Do Laboratory Experiments Measuring Social Preferences Reveal about the Real World?Journal of Economic Perspectives 21 (Spring): 153–71.
Loughin, Thomas M. 2004. “A Systematic Comparison of Methods for Combining p-Values from Independent Tests.” Computational Statistics and Data Analysis 47: 467–85.
Lupia, Arthur, Krupnikov, Yanna, Levine, Adam Seth, Grafstrom, Casandra, McMillian, William, and McGovern, Erin. n.d. “How ‘Point Blindness’ Dilutes the Value of Stock Market Reports.” Political Communication. Forthcoming.
Luskin, Robert C., Fishkin, James S., and Jowell, Roger. 2002. “Considered Opinions: Deliberative Polling in Britain.” British Journal of Political Science 32: 455–87.
Marmor, Theodore R. 2000. The Politics of Medicare. 2nd ed. NY: Aldine de Gruyter.
McClelland, Gary H., and Judd, Charles M.. 1993. “Statistical Difficulties of Detecting Interactions and Moderator Effects.” Psychological Bulletin 114 (2): 376–90.
McDermott, Rose. 2002. “Experimental Methods in Political Science.”Annual Review of Political Science 5: 3161.
Mondak, Jeffery J. 1995. “Newspapers and Political Awareness.” American Journal of Political Science 39 (May): 513–27.
Mutz, Diana, and Reeves, Byron. 2005. “The New Videomalaise: Effects of Televised Civility on Political Trust.” American Political Science Review 99 (February): 115.
Nadeau, Richard, and Niemi, Richard G.. 1995. “Educated Guesses: The Process of Answering Factual Knowledge Questions in Surveys.” Public Opinion Quarterly 59 (Autumn): 323–46.
Norris, Pippa, and Sanders, David. 2003. “Message or Medium? Learning during the 2001 British General Election.” Political Communication 20: 233–62.
Oberlander, Jonathan. 2003. The Political Life of Medicare. Chicago: University of Chicago Press.
O'Keefe, Daniel J. 2007. “Post Hoc Power, Observed Power, A Priori Power, Retrospective Power, Prospective Power, Achieved Power: Sorting Out Appropriate Uses of Statistical Power Analyses.” Communication Methods and Measures 1 (4): 291–99.
Page, Benjamin I. 2000. “How Public Opinion Affects Reform: Is Social Security Reform Ready for the American Public?” In Social Security and Medicare: Individual versus Collective Risk and Responsibility, eds. Burke, Shelia, Kingson, Eric, and Reinhardt, Uwe. Washington, DC: National Academy of Social Insurance, 183218.
Page, Benjamin I., and Shapiro, Robert Y.. 1992. The Rational Public: Fifty Years of Trends in Americans’ Policy Preferences. Chicago: University of Chicago Press.
Prior, Marcus, and Lupia, Arthur. 2008. “Money, Time, and Political Knowledge: Distinguishing Quick Recall and Political Learning Skills.” American Journal of Political Science 52 (January): 169–83.
Robinson, Gregory, McNulty, John E., and Krasno, Jonathan S.. 2009. “Observing the Counterfactual? The Search for Political Experiments in Nature.” Political Analysis 17: 341–57.
Roth, Alvin. 1995. “Introduction to Experimental Economics.” In The Handbook of Experimental Economics, eds. Kagel, John H. and Roth, Alvin E.. Princeton, NJ: Princeton University Press, 3109.
Shadish, William R., Cook, Thomas D., and Campbell, Donald T.. 2002. Experimental and Quasi-experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.
Shaw, Greg M., and Mysiewicz, Sarah E.. 2004. “Trends: Social Security and Medicare.” Public Opinion Quarterly 68 (Fall): 394423.
Sniderman, Paul M., and Piazza, Thomas. 1993. The Scar of Race. Cambridge, MA: Harvard University Press.
Sniderman, Paul M., and Theriault, Sean M.. 2004. “The Structure of Political Argument and the Logic of Issue Framing.” In Studies in Public Opinion: Attitudes, Nonattitudes, Measurement Error, and Change, eds. Saris, Willem E. and Sniderman, Paul M.. Princeton, NJ: Princeton University Press, 133–64.
Turgeon, Mathieu. 2009. “‘Just Thinkin’: Attitude Development, Public Opinion, and Political Representation.” Political Behavior 31 (September): 353–78.
Weisberg, Herbert F. 2005. The Total Survey Error Approach: A Guide to the New Science of Survey Research. Chicago: University of Chicago Press.
Winship, Christopher, and Radbill, Larry. 1994. “Sampling Weights and Regression Analysis.” Sociological Methods and Research 23 (2): 230–57.
Wooldridge, Jeffrey M. 2009. Introductory Economics: A Modern Approach. 4th ed. Mason, OH: South-Western Cengage Learning.
Zaller, John R. 1992. The Nature and Origins of Mass Opinion. New York: Cambridge.
Zaller, John R. 2002. “The Statistical Power of Election Studies to Detect Media Exposure Effects in Political Campaigns.” Electoral Studies 21: 297329.
Zaller, John R., and Feldman, Stanley. 1992. “A Simple Theory of the Survey Response: Answering Questions versus Revealing Preferences.” American Journal of Political Science 36 (August): 579616.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

American Political Science Review
  • ISSN: 0003-0554
  • EISSN: 1537-5943
  • URL: /core/journals/american-political-science-review
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed