Skip to main content
×
×
Home

The Generalizability of Survey Experiments*

  • Kevin J. Mullinix (a1), Thomas J. Leeper (a2), James N. Druckman (a3) and Jeremy Freese (a4)
Abstract

Survey experiments have become a central methodology across the social sciences. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, due to the expense of population-based samples, much research relies on convenience samples (e.g. students, online opt-in samples). The emergence of affordable, but non-representative online samples has reinvigorated debates about the external validity of experiments. We conduct two studies of how experimental treatment effects obtained from convenience samples compare to effects produced by population samples. In Study 1, we compare effect estimates from four different types of convenience samples and a population-based sample. In Study 2, we analyze treatment effects obtained from 20 experiments implemented on a population-based sample and Amazon's Mechanical Turk (MTurk). The results reveal considerable similarity between many treatment effects obtained from convenience and nationally representative population-based samples. While the results thus bolster confidence in the utility of convenience samples, we conclude with guidance for the use of a multitude of samples for advancing scientific knowledge.

Copyright
Footnotes
Hide All
*

The authors acknowledge support from a National Science Foundation grant for Time-Sharing Experiments in the Social Sciences (SES-1227179). Druckman and Freese are co-Principal Investigators of TESS, and Study 2 was designed and funded as a methodological component of their TESS grant. Study 1 includes data in part funded by an NSF Doctoral Dissertation Improvement Grant to Leeper (SES-1160156) and in part collected via a successful proposal to TESS by Mullinix and Leeper. Druckman and Freese were neither involved in Study 1 nor with any part of the review or approval of Mullinix and Leeper's TESS proposal (via recusal, given other existing collaborations). Only after data from both studies were collected did authors determine that the two studies were so complementary that it would be better to publish them together. The authors thank Lene Aarøe, Kevin Arceneaux, Christoph Arndt, Adam Berinsky, Emily Cochran Bech, Scott Clifford, Adrienne Hosek, Cindy Kam, Lasse Laustsen, Diana Mutz, Helene Helboe Pedersen, Richard Shafranek, Flori So, Rune Slothuus, Rune Stubager, Magdalena Wojcieszak, workshop participants at Southern Denmark University, and participants at The American Panel Survey Workshop at Washington University, St. Louis.

Footnotes
References
Hide All
Ahler, Douglas J. 2014. “Self-Fulfilling Misperceptions of Public Polarization.” The Journal of Politics 76 (3): 607–20.
Baker, Reg. et al. 2010. “Research Synthesis: AAPOR Report on Online Panels.” Public Opinion Quarterly 74 (4): 171.
Barabas, Jason and Jerit, Jennifer. 2010. “Are Survey Experiments Externally Valid?American Political Science Review 104 (May): 226242.
Benoit, Kenneth, Conway, Drew, Lauderdale, Benjamin E., Laver, Michael, and Mikhaylov, Slava. 2015. “Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data.” American Political Science Review: Forthcoming.
Berger, Arthur Asa. 2014. Media and Communications Research Methods: An Introduction to Qualitative and Quantitative Approaches. Los Angeles: Sage Publication, Inc.
Berinsky, Adam J., Huber, Gregory A., and Lenz, Gabriel S.. 2012. “Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk.” Political Analysis 20 (Summer): 351–68.
Berinsky, Adam J., Margolis, Michele F., and Sances, Michael W.. 2014. “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys.” American Journal of Political Science 58 (3): 739–53.
Bloom, Howard S. 2005. Learning More from Social Experiments. New York: Russell Sage Foundation.
Bohannon, John. 2011. “Social Science for Pennies.” Science 334 (October): 307.
Brady, Henry E. 2000. “Contributions of Survey Research to Political Science.” PS: Political Science & Politics 33 (1): 4757.
Broockman, David E. and Green, Donald P.. 2013. “Do Online Advertisements Increase Political Candidates’ Name Recognition or Favorability? Evidence from Randomized Field Experiments.” Political Behavior 36 (2): 263–89.
Callegaro, Mario, Baker, Reg, Bethlehem, Jelke, Göritz, Anja S., Krosnick, Jon A., and Lavrakas, Paul J.. 2014. “Online Panel Research: History, Concepts, Applications, and a Look at the Future.” In Online Panel Research: A Data Quality Perspective, eds. Callegaro, Mario, Baker, Reg, Bethlehem, Jelke, Göritz, Anja S., Krosnick, Jon A., and Lavrakas, Paul J.. Sussex, West, United Kingdom: John Wiley & Sons Ltd.
Campbell, Donald T. 1969. “Prospective: Artifact and Control.” In Artifact in Behavioral Research, eds. Rosenthal, Robert and Rosnow, Robert. New York: Academic Press.
Cassese, Erin C., Huddy, Leonie, Hartman, Todd K., Mason, Liliana, and Weber, Christopher R.. 2013. “Socially Mediated Internet Surveys: Recruiting Participants for Online Experiments.” PS: Political Science and Politics 46 (4): 110.
Chandler, Jesse, Mueller, Pam, and Paolacci, Gabriele. 2014. “Nonnaiveté Among Amazon Mechanical Turk Workers: Consequences and Solution for Behavioral Researchers.” Behavior Research Methods 46 (1): 112–30.
Chong, Dennis and Druckman, James N.. 2007a. “Framing Public Opinion in Competitive Democracies.” American Political Science Review 101 (4): 637–55.
Chong, Dennis and Druckman, James N.. 2007b. “Framing Theory.” Annual Review of Political Science 10 (1): 103–26.
Clifford, Scott and Jerit, Jennifer. 2015. “Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies.” Journal of Experimental Political Science 1 (2): 120–31.
Coppock, Alexander and Green, Donald P.. 2015. “Assessing the Correspondence Between Experimental Results Obtained in the Lab and Field: A Review of Recent Social Science Research.” Political Science Research and Methods 3 (1): 113–31.
Druckman, James N. 2001. “The Implications of Framing Effects for Citizen Competence.” Political Behavior 23 (3): 225–56.
Druckman, James N. 2004. “Priming the Vote: Campaign Effects in a US Senate Election.” Political Psychology 25: 577–94.
Druckman, James N. and Lupia, Arthur. 2012. “Experimenting with Politics.” Science 335 (March): 1177–79.
Druckman, James N. and Kam, Cindy D.. 2011. “Students as Experimental Participants: A Defense of the ‘Narrow Data Base’.” In Cambridge Handbook of Experimental Political Science, eds. Druckman, J. N., Green, D. P., Kuklinski, J. H., and Lupia, A.. New York: Cambridge University Press, 4157.
Druckman, James N., Green, Donald P., Kuklinski, James H., and Lupia, Arthur. 2006. “The Growth and Development of Experimental Research in Political Science.” American Political Science Review 100 (November): 627–35.
Druckman, James N., Peterson, Erik, and Slothuus, Rune. 2013. “How Elite Partisan Polarization Affects Public Opinion Formation.” American Political Science Review 107 (1): 5779.
Dynamo. 2014. “Guidelines for Academic Requesters.” Retrieved 6 October 2015 from (http://wiki.wearedynamo.org/index.php/Guidelines_for_Academic_Requesters), Accessed October 6, 2015.
Egami, Naoki and Imai, Kosuke. 2015. “Causal Interaction in High-Dimension.” Working paper.
Entman, Robert M. 1993. “Framing: Toward Clarification of a Fractured Paradigm.” Journal of Communication 43 (4): 5158.
Fowler, Anthony and Margolis, Michele. 2014. “The Political Consequences of Uninformed Voters.” Electoral Studies 34: 100–10.
Franco, Annie, Malhotra, Neil, and Simonovits, Gabor. 2014. “Publication Bias in the Social Sciences: Unlocking the File Drawer.” Science 345 (August): 1502–5.
Freese, Jeremy, Howat, Adam, Mullinix, Kevin J., and Druckman, James N.. 2015. “Limitations of Screening Methods to Obtain Representative Samples Using Online Labor Markets.” Working Paper, Northwestern University.
Gamson, William A. and Modigiliani, Andre. 1989. “Media Discourse and Public Opinion on Nuclear Power: A Constructionist Approach.” American Journal of Sociology 95 (1): 137.
Gelman, Andrew and Stern, Hal. 2006. “The Difference Between ‘Significant’ and ‘Not Significant’ is not Itself Statistically Significant.” The American Statistician 60 (4): 328–31.
Gerber, Alan S. and Green, Donald P.. 2008. “Field Experiments and Natural Experiments.” In Oxford Handbook of Political Methodology, eds. Box-Steffensmeier, J. M., Brady, H. E., and Collier, D.. New York: Oxford University Press, 357–81.
Gerber, Alan S. and Green, Donald P.. 2011. Field Experiments: Design, Analysis, and Interpretation. New York: W.W. Norton & Company.
Gerring, John. 2012. Social Science Methodology: A Unified Framework. New York: Cambridge University Press.
GfK. 2013. “Knowledge Panel Design Summary.” Available at: https://www.gfk.com/Documents/GfK-KnowledgePanel.pdf. Last accessed 20 November 2015.
Goodman, Joseph K., Cryder, Cynthia E., and Cheema, Amar. 2012. “Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples.” Journal of Behavioral Decision Making 26: 213–24.
Green, Donald P. and Kern, Holger L.. 2012. “Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees.” Public Opinion Quarterly 76 (3): 491511.
Henrich, Joseph, Heine, Steven J., and Norenzayan, Ara. 2010. “The Weirdest People in the World?Behavioral and Brain Sciences 33 (April): 6183.
Hillygus, D. Sunshine, Jackson, Natalie, and Young, McKenzie. 2014. “Professional Respondents in Nonprobability Online Panels.” In Online Panel Research: A Data Quality Perspetive, eds. Callegaro, Mario, Baker, Reg, Bethlehem, Jelke, Göritz, Anja S., Krosnick, Jon A., and Lavrakas, Paul J.. West Sussex, United Kingdom: John Wiley & Sons Ltd.
Holt, Charles A. 2006. Markets, Games, and Strategic Behavior: Recipes for Interactive Learning. Reading, MA: Addison-Wesley.
Horton, John J., Rand, David G., and Zeckhauser, Richard J.. 2011. “The Online Laboratory: Conducting Experiments in a Real Labor Market.” Experimental Economics 14 (3): 399425.
Hovland, Carl I. 1959. “Reconciling Conflicting Results Derived from Experimental and Survey Studies of Attitude Change.” The American Psychologist 14: 817.
Huber, Gregory A., Hill, Seth J., and Lenz, Gabriel S.. 2012. “Sources of Bias in Retrospective Decision-Making: Experimental Evidence of Voters’ Limitations in Controlling Incumbents.” American Political Science Review 106 (4): 720–41.
Huff, Connor and Tingley, Dustin. 2015. “‘Who are these people?’ Evaluating the demographic characteristics and political preferences of MTurk survey respondents.” Research & Politics 2 (3): 111. DOI: 10.1177/2053168015604648.
Iyengar, Shanto. 1991. Is Anyone Responsible? How Television Frames Political Issues. Chicago, IL: The University of Chicago Press.
Jerit, Jennifer, Barabas, Jason, and Clifford, Scott. 2013. “Comparing Contemporaneous Laboratory and Field Experiments on Media Effects.” Public Opinion Quarterly 77 (1): 256–82.
Kam, Cindy D., Wilking, Jennifer R., and Zechmeister, Elizabeth J.. 2007. “Beyond the ‘Narrow Data Base’: Another Convenience Sample for Experimental Research.” Political Behavior 29 (4): 415–40.
Keeter, Scott, Kennedy, Courtney, Dimock, Michael, Best, Jonathan and Craighill, Peyton. 2006. “Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey.” Public Opinion Quarterly 70 (5): 759–79.
Klar, Samara. 2013. “The Influence of Competing Identity Primes on Political Preferences.” Journal of Politics 75 (4): 1108–24.
Klar, Samara, Robison, Joshua, and Druckman, James N.. 2013. “Political Dynamics of Framing.” In New Directions in Media and Politics, ed. Ridout, Travis N.. New York: Routledge, 173192.
Klein, Richard A. et al. 2014. “Investigating Variation in Replicability: A ‘Many Labs’ Replication Project.” Social Psychology 45: 142–52.
Kraft, Peter. 2008. “Curses—Winner's and Otherwise—in Genetic Epidemiology.” Epidemiology 19 (September): 649–51.
Kriss, Peter H. and Weber, Roberto. 2013. “Organizational Formation and Change: Lessons from Economic Laboratory Experiments.” In Handbook of Economic Organization: Integrating Economic and Organizational Theory, ed. Northampton, A. Grandori.: Edward Elgar Publishing Limited, 245–72.
Krupnikov, Yanna and Levine, Adam Seth. 2014. “Cross-Sample Comparisons and External Validity.” Journal of Experimental Political Science 1 (Spring): 5980.
Lupia, Arthur. 2014. “The 2013 Ithiel de Sola Pool Lecture: What is the Value of Social Science? Challengers for Researchers and Government Funders.” PS: Political Science & Politics 47 (January): 17.
Malhotra, Neil and Kuo, Alexander G.. 2008. “Attributing Blame: The Public's Response to Hurricane Katrina.” The Journal of Politics 70 (1): 120–35.
McDermott, Rose. 2002. “Experimental Methodology in Political Science.” Political Analysis, 10: 325–42.
Morawski, Jill G. 1988. The Rise of Experimentation in American Psychology. New Haven: Yale University Press.
Mutz, Diana C. 2011. Population-Based Survey Experiments. Princeton: Princeton University Press.
Nelson, Thomas E., Clawson, Rosalee A., and Oxley, Zoe M.. 1997. “Media Framing of a Civil Liberties Conflict and Its Effect on Tolerance.” American Political Science Review 91 (3): 567–83.
Nock, Steven L. and Guterbock, Thomas M.. 2010. “Survey experiments.” In Handbook of Survey Research, eds. Marsden, P. V., and Wright, J. D.. Emerald, UK, 837–64.
Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349: 943.
Paolacci, Gabriele, Chandler, Jesse, and Ipeirotis, Panagiotis G.. 2010. “Running Experiments on Amazon Mechanical Turk.” Judgment and Decision Making, 5 (August): 411–9.
Pew. 2012. Assessing the Representativeness of Public Opinion Surveys. Available at: http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-public-opinion-surveys/. Last accessed 20 November 2015.
Rand, David G., Peysakhovich, Alexander, Kraft-Todd, Gordon T., Newman, George E., Wurzbacher, Owen, Nowak, Martin A., and Greene, Joshua D.. 2014. “Social Heuristics Shape Intuitive Cooperation.” Nature Communications 5: 112.
Redlawsk, David P., Civettini, Andrew J., and Emmerson, Karen M.. 2010. “The Affective Tipping Point: Do Motivated Reasoners Ever ‘Get It’?Political Psychology 31: 563593.
Riker, William H. 1996. The Strategy of Rhetoric: Campaigning for the American Constitution. New Haven, CT: Yale University Press.
Ross, Joel, Irani, Lily, Silberman, M. Six, Zaldivar, Andrew, and Tomlinson, Bill. 2010. “Who are the Crowdworkers? Shifting Demographics in Amazon Mechanical Turk. In CHI EA 2010, New York: ACM Press, 2863–72.
Sears, David O. 1986. “College Sophomores in the Laboratory: Influences of a Narrow Data Base on Social Psychology's View of Human Nature.” Journal of Personality and Social Psychology 51: 515530.
Shadish, William R., Cook, Thomas D., and Campbell, Donald T.. 2001. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, MA: Houghton-Mifflin.
Sniderman, Paul. 2011. The Logic and Design of the Survey Experiment: An Autobiography of a Methodological Innovation.” In Cambridge Handbook of Experimental Political Science, eds. Druckman, J. N., Green, D. P., Kuklinski, J. H., and Lupia, A.. New York: Cambridge University Press, 102–14.
Steinmetz, Stephanie, Bianchi, Annamaria, Tijdens, Kea, and Biffignandi, Silvia. 2014. “Improving Web Surveys Quality: Potentials and Constraints of Propensity Score Adjustments.” In Online Panel Research: A Data Quality Perspetive, eds. Callegaro, Mario, Baker, Reg, Bethlehem, Jelke, Göritz, Anja S., Krosnick, Jon A., and Lavrakas, Paul J.. West Sussex, United Kingdom: John Wiley & Sons Ltd.
Valentino, Nicholas A., Traugott, Michael W., and Hutchings, Vincent L.. 2002. “Group Cues and Ideological Constraint: A Replication of Political Advertising Effects Studies in the Lab and in the Field.” Political Communicaton 19 (1): 2948.
Wang, Wei, Rothschild, David, Goel, Sharad, and Gelman, Andrew. 2015. “Forecasting Elections with Non-representative Polls.” International Journal of Forecasting 31 (3): 980991.
Weinberg, Jill D., Freese, Jeremy, and McElhattan, David. 2014. “Comparing Demographics, Data Quality, and Results of an Online Factorial Survey Between a Population-Based and a Crowdsource-Recruited Sample.” Sociological Science 1: 292310.
Wright, James D., and Marsden, Peter V. 2010. “Survey Research and Social Science: History, Current Practice, and Future Prospects.” In Handbook of Survey Research, eds. Marsden, P. V., and Wright, J. D.. Emerald, UK, 326.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Experimental Political Science
  • ISSN: 2052-2630
  • EISSN: 2052-2649
  • URL: /core/journals/journal-of-experimental-political-science
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Keywords

Type Description Title
WORD
Supplementary materials

Mullinix supplementary material
Mullinix supplementary material 1

 Word (103 KB)
103 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 46
Total number of PDF views: 692 *
Loading metrics...

Abstract views

Total abstract views: 3614 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 23rd June 2018. This data will be updated every 24 hours.