Skip to main content
×
Home

The Generalizability of Survey Experiments*

  • Kevin J. Mullinix (a1), Thomas J. Leeper (a2), James N. Druckman (a3) and Jeremy Freese (a4)
Abstract
Abstract

Survey experiments have become a central methodology across the social sciences. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, due to the expense of population-based samples, much research relies on convenience samples (e.g. students, online opt-in samples). The emergence of affordable, but non-representative online samples has reinvigorated debates about the external validity of experiments. We conduct two studies of how experimental treatment effects obtained from convenience samples compare to effects produced by population samples. In Study 1, we compare effect estimates from four different types of convenience samples and a population-based sample. In Study 2, we analyze treatment effects obtained from 20 experiments implemented on a population-based sample and Amazon's Mechanical Turk (MTurk). The results reveal considerable similarity between many treatment effects obtained from convenience and nationally representative population-based samples. While the results thus bolster confidence in the utility of convenience samples, we conclude with guidance for the use of a multitude of samples for advancing scientific knowledge.

Copyright
Footnotes
Hide All
*

The authors acknowledge support from a National Science Foundation grant for Time-Sharing Experiments in the Social Sciences (SES-1227179). Druckman and Freese are co-Principal Investigators of TESS, and Study 2 was designed and funded as a methodological component of their TESS grant. Study 1 includes data in part funded by an NSF Doctoral Dissertation Improvement Grant to Leeper (SES-1160156) and in part collected via a successful proposal to TESS by Mullinix and Leeper. Druckman and Freese were neither involved in Study 1 nor with any part of the review or approval of Mullinix and Leeper's TESS proposal (via recusal, given other existing collaborations). Only after data from both studies were collected did authors determine that the two studies were so complementary that it would be better to publish them together. The authors thank Lene Aarøe, Kevin Arceneaux, Christoph Arndt, Adam Berinsky, Emily Cochran Bech, Scott Clifford, Adrienne Hosek, Cindy Kam, Lasse Laustsen, Diana Mutz, Helene Helboe Pedersen, Richard Shafranek, Flori So, Rune Slothuus, Rune Stubager, Magdalena Wojcieszak, workshop participants at Southern Denmark University, and participants at The American Panel Survey Workshop at Washington University, St. Louis.

Footnotes
References
Hide All
Ahler Douglas J. 2014. “Self-Fulfilling Misperceptions of Public Polarization.” The Journal of Politics 76 (3): 607–20.
Baker Reg. et al. 2010. “Research Synthesis: AAPOR Report on Online Panels.” Public Opinion Quarterly 74 (4): 171.
Barabas Jason and Jerit Jennifer. 2010. “Are Survey Experiments Externally Valid?American Political Science Review 104 (May): 226242.
Benoit Kenneth, Conway Drew, Lauderdale Benjamin E., Laver Michael, and Mikhaylov Slava. 2015. “Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data.” American Political Science Review: Forthcoming.
Berger Arthur Asa. 2014. Media and Communications Research Methods: An Introduction to Qualitative and Quantitative Approaches. Los Angeles: Sage Publication, Inc.
Berinsky Adam J., Huber Gregory A., and Lenz Gabriel S.. 2012. “Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk.” Political Analysis 20 (Summer): 351–68.
Berinsky Adam J., Margolis Michele F., and Sances Michael W.. 2014. “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys.” American Journal of Political Science 58 (3): 739–53.
Bloom Howard S. 2005. Learning More from Social Experiments. New York: Russell Sage Foundation.
Bohannon John. 2011. “Social Science for Pennies.” Science 334 (October): 307.
Brady Henry E. 2000. “Contributions of Survey Research to Political Science.” PS: Political Science & Politics 33 (1): 4757.
Broockman David E. and Green Donald P.. 2013. “Do Online Advertisements Increase Political Candidates’ Name Recognition or Favorability? Evidence from Randomized Field Experiments.” Political Behavior 36 (2): 263–89.
Callegaro Mario, Baker Reg, Bethlehem Jelke, Göritz Anja S., Krosnick Jon A., and Lavrakas Paul J.. 2014. “Online Panel Research: History, Concepts, Applications, and a Look at the Future.” In Online Panel Research: A Data Quality Perspective, eds. Callegaro Mario, Baker Reg, Bethlehem Jelke, Göritz Anja S., Krosnick Jon A., and Lavrakas Paul J.. Sussex West, United Kingdom: John Wiley & Sons Ltd.
Campbell Donald T. 1969. “Prospective: Artifact and Control.” In Artifact in Behavioral Research, eds. Rosenthal Robert and Rosnow Robert. New York: Academic Press.
Cassese Erin C., Huddy Leonie, Hartman Todd K., Mason Liliana, and Weber Christopher R.. 2013. “Socially Mediated Internet Surveys: Recruiting Participants for Online Experiments.” PS: Political Science and Politics 46 (4): 110.
Chandler Jesse, Mueller Pam, and Paolacci Gabriele. 2014. “Nonnaiveté Among Amazon Mechanical Turk Workers: Consequences and Solution for Behavioral Researchers.” Behavior Research Methods 46 (1): 112–30.
Chong Dennis and Druckman James N.. 2007a. “Framing Public Opinion in Competitive Democracies.” American Political Science Review 101 (4): 637–55.
Chong Dennis and Druckman James N.. 2007b. “Framing Theory.” Annual Review of Political Science 10 (1): 103–26.
Clifford Scott and Jerit Jennifer. 2015. “Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies.” Journal of Experimental Political Science 1 (2): 120–31.
Coppock Alexander and Green Donald P.. 2015. “Assessing the Correspondence Between Experimental Results Obtained in the Lab and Field: A Review of Recent Social Science Research.” Political Science Research and Methods 3 (1): 113–31.
Druckman James N. 2001. “The Implications of Framing Effects for Citizen Competence.” Political Behavior 23 (3): 225–56.
Druckman James N. 2004. “Priming the Vote: Campaign Effects in a US Senate Election.” Political Psychology 25: 577–94.
Druckman James N. and Lupia Arthur. 2012. “Experimenting with Politics.” Science 335 (March): 1177–79.
Druckman James N. and Kam Cindy D.. 2011. “Students as Experimental Participants: A Defense of the ‘Narrow Data Base’.” In Cambridge Handbook of Experimental Political Science, eds. Druckman J. N., Green D. P., Kuklinski J. H., and Lupia A.. New York: Cambridge University Press, 4157.
Druckman James N., Green Donald P., Kuklinski James H., and Lupia Arthur. 2006. “The Growth and Development of Experimental Research in Political Science.” American Political Science Review 100 (November): 627–35.
Druckman James N., Peterson Erik, and Slothuus Rune. 2013. “How Elite Partisan Polarization Affects Public Opinion Formation.” American Political Science Review 107 (1): 5779.
Dynamo. 2014. “Guidelines for Academic Requesters.” Retrieved 6 October 2015 from (http://wiki.wearedynamo.org/index.php/Guidelines_for_Academic_Requesters), Accessed October 6, 2015.
Egami Naoki and Imai Kosuke. 2015. “Causal Interaction in High-Dimension.” Working paper.
Entman Robert M. 1993. “Framing: Toward Clarification of a Fractured Paradigm.” Journal of Communication 43 (4): 5158.
Fowler Anthony and Margolis Michele. 2014. “The Political Consequences of Uninformed Voters.” Electoral Studies 34: 100–10.
Franco Annie, Malhotra Neil, and Simonovits Gabor. 2014. “Publication Bias in the Social Sciences: Unlocking the File Drawer.” Science 345 (August): 1502–5.
Freese Jeremy, Howat Adam, Mullinix Kevin J., and Druckman James N.. 2015. “Limitations of Screening Methods to Obtain Representative Samples Using Online Labor Markets.” Working Paper, Northwestern University.
Gamson William A. and Modigiliani Andre. 1989. “Media Discourse and Public Opinion on Nuclear Power: A Constructionist Approach.” American Journal of Sociology 95 (1): 137.
Gelman Andrew and Stern Hal. 2006. “The Difference Between ‘Significant’ and ‘Not Significant’ is not Itself Statistically Significant.” The American Statistician 60 (4): 328–31.
Gerber Alan S. and Green Donald P.. 2008. “Field Experiments and Natural Experiments.” In Oxford Handbook of Political Methodology, eds. Box-Steffensmeier J. M., Brady H. E., and Collier D.. New York: Oxford University Press, 357–81.
Gerber Alan S. and Green Donald P.. 2011. Field Experiments: Design, Analysis, and Interpretation. New York: W.W. Norton & Company.
Gerring John. 2012. Social Science Methodology: A Unified Framework. New York: Cambridge University Press.
GfK. 2013. “Knowledge Panel Design Summary.” Available at: https://www.gfk.com/Documents/GfK-KnowledgePanel.pdf. Last accessed 20 November 2015.
Goodman Joseph K., Cryder Cynthia E., and Cheema Amar. 2012. “Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples.” Journal of Behavioral Decision Making 26: 213–24.
Green Donald P. and Kern Holger L.. 2012. “Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees.” Public Opinion Quarterly 76 (3): 491511.
Henrich Joseph, Heine Steven J., and Norenzayan Ara. 2010. “The Weirdest People in the World?Behavioral and Brain Sciences 33 (April): 6183.
Hillygus D. Sunshine, Jackson Natalie, and Young McKenzie. 2014. “Professional Respondents in Nonprobability Online Panels.” In Online Panel Research: A Data Quality Perspetive, eds. Callegaro Mario, Baker Reg, Bethlehem Jelke, Göritz Anja S., Krosnick Jon A., and Lavrakas Paul J.. West Sussex, United Kingdom: John Wiley & Sons Ltd.
Holt Charles A. 2006. Markets, Games, and Strategic Behavior: Recipes for Interactive Learning. Reading, MA: Addison-Wesley.
Horton John J., Rand David G., and Zeckhauser Richard J.. 2011. “The Online Laboratory: Conducting Experiments in a Real Labor Market.” Experimental Economics 14 (3): 399425.
Hovland Carl I. 1959. “Reconciling Conflicting Results Derived from Experimental and Survey Studies of Attitude Change.” The American Psychologist 14: 817.
Huber Gregory A., Hill Seth J., and Lenz Gabriel S.. 2012. “Sources of Bias in Retrospective Decision-Making: Experimental Evidence of Voters’ Limitations in Controlling Incumbents.” American Political Science Review 106 (4): 720–41.
Huff Connor and Tingley Dustin. 2015. “‘Who are these people?’ Evaluating the demographic characteristics and political preferences of MTurk survey respondents.” Research & Politics 2 (3): 111. DOI: 10.1177/2053168015604648.
Iyengar Shanto. 1991. Is Anyone Responsible? How Television Frames Political Issues. Chicago, IL: The University of Chicago Press.
Jerit Jennifer, Barabas Jason, and Clifford Scott. 2013. “Comparing Contemporaneous Laboratory and Field Experiments on Media Effects.” Public Opinion Quarterly 77 (1): 256–82.
Kam Cindy D., Wilking Jennifer R., and Zechmeister Elizabeth J.. 2007. “Beyond the ‘Narrow Data Base’: Another Convenience Sample for Experimental Research.” Political Behavior 29 (4): 415–40.
Keeter Scott, Kennedy Courtney, Dimock Michael, Best Jonathan and Craighill Peyton. 2006. “Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey.” Public Opinion Quarterly 70 (5): 759–79.
Klar Samara. 2013. “The Influence of Competing Identity Primes on Political Preferences.” Journal of Politics 75 (4): 1108–24.
Klar Samara, Robison Joshua, and Druckman James N.. 2013. “Political Dynamics of Framing.” In New Directions in Media and Politics, ed. Ridout Travis N.. New York: Routledge, 173192.
Klein Richard A. et al. 2014. “Investigating Variation in Replicability: A ‘Many Labs’ Replication Project.” Social Psychology 45: 142–52.
Kraft Peter. 2008. “Curses—Winner's and Otherwise—in Genetic Epidemiology.” Epidemiology 19 (September): 649–51.
Kriss Peter H. and Weber Roberto. 2013. “Organizational Formation and Change: Lessons from Economic Laboratory Experiments.” In Handbook of Economic Organization: Integrating Economic and Organizational Theory, ed. Northampton A. Grandori.: Edward Elgar Publishing Limited, 245–72.
Krupnikov Yanna and Levine Adam Seth. 2014. “Cross-Sample Comparisons and External Validity.” Journal of Experimental Political Science 1 (Spring): 5980.
Lupia Arthur. 2014. “The 2013 Ithiel de Sola Pool Lecture: What is the Value of Social Science? Challengers for Researchers and Government Funders.” PS: Political Science & Politics 47 (January): 17.
Malhotra Neil and Kuo Alexander G.. 2008. “Attributing Blame: The Public's Response to Hurricane Katrina.” The Journal of Politics 70 (1): 120–35.
McDermott Rose. 2002. “Experimental Methodology in Political Science.” Political Analysis, 10: 325–42.
Morawski Jill G. 1988. The Rise of Experimentation in American Psychology. New Haven: Yale University Press.
Mutz Diana C. 2011. Population-Based Survey Experiments. Princeton: Princeton University Press.
Nelson Thomas E., Clawson Rosalee A., and Oxley Zoe M.. 1997. “Media Framing of a Civil Liberties Conflict and Its Effect on Tolerance.” American Political Science Review 91 (3): 567–83.
Nock Steven L. and Guterbock Thomas M.. 2010. “Survey experiments.” In Handbook of Survey Research, eds. Marsden P. V., and Wright J. D.. Emerald, UK, 837–64.
Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349: 943.
Paolacci Gabriele, Chandler Jesse, and Ipeirotis Panagiotis G.. 2010. “Running Experiments on Amazon Mechanical Turk.” Judgment and Decision Making, 5 (August): 411–9.
Pew. 2012. Assessing the Representativeness of Public Opinion Surveys. Available at: http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-public-opinion-surveys/. Last accessed 20 November 2015.
Rand David G., Peysakhovich Alexander, Kraft-Todd Gordon T., Newman George E., Wurzbacher Owen, Nowak Martin A., and Greene Joshua D.. 2014. “Social Heuristics Shape Intuitive Cooperation.” Nature Communications 5: 112.
Redlawsk David P., Civettini Andrew J., and Emmerson Karen M.. 2010. “The Affective Tipping Point: Do Motivated Reasoners Ever ‘Get It’?Political Psychology 31: 563593.
Riker William H. 1996. The Strategy of Rhetoric: Campaigning for the American Constitution. New Haven, CT: Yale University Press.
Ross Joel, Irani Lily, Silberman M. Six, Zaldivar Andrew, and Tomlinson Bill. 2010. “Who are the Crowdworkers? Shifting Demographics in Amazon Mechanical Turk. In CHI EA 2010, New York: ACM Press, 2863–72.
Sears David O. 1986. “College Sophomores in the Laboratory: Influences of a Narrow Data Base on Social Psychology's View of Human Nature.” Journal of Personality and Social Psychology 51: 515530.
Shadish William R., Cook Thomas D., and Campbell Donald T.. 2001. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, MA: Houghton-Mifflin.
Sniderman Paul. 2011. The Logic and Design of the Survey Experiment: An Autobiography of a Methodological Innovation.” In Cambridge Handbook of Experimental Political Science, eds. Druckman J. N., Green D. P., Kuklinski J. H., and Lupia A.. New York: Cambridge University Press, 102–14.
Steinmetz Stephanie, Bianchi Annamaria, Tijdens Kea, and Biffignandi Silvia. 2014. “Improving Web Surveys Quality: Potentials and Constraints of Propensity Score Adjustments.” In Online Panel Research: A Data Quality Perspetive, eds. Callegaro Mario, Baker Reg, Bethlehem Jelke, Göritz Anja S., Krosnick Jon A., and Lavrakas Paul J.. West Sussex, United Kingdom: John Wiley & Sons Ltd.
Valentino Nicholas A., Traugott Michael W., and Hutchings Vincent L.. 2002. “Group Cues and Ideological Constraint: A Replication of Political Advertising Effects Studies in the Lab and in the Field.” Political Communicaton 19 (1): 2948.
Wang Wei, Rothschild David, Goel Sharad, and Gelman Andrew. 2015. “Forecasting Elections with Non-representative Polls.” International Journal of Forecasting 31 (3): 980991.
Weinberg Jill D., Freese Jeremy, and McElhattan David. 2014. “Comparing Demographics, Data Quality, and Results of an Online Factorial Survey Between a Population-Based and a Crowdsource-Recruited Sample.” Sociological Science 1: 292310.
Wright James D., and Marsden Peter V. 2010. “Survey Research and Social Science: History, Current Practice, and Future Prospects.” In Handbook of Survey Research, eds. Marsden P. V., and Wright J. D.. Emerald, UK, 326.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Experimental Political Science
  • ISSN: 2052-2630
  • EISSN: 2052-2649
  • URL: /core/journals/journal-of-experimental-political-science
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Keywords:

Type Description Title
WORD
Supplementary Materials

Mullinix supplementary material
Mullinix supplementary material 1

 Word (103 KB)
103 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 23
Total number of PDF views: 361 *
Loading metrics...

Abstract views

Total abstract views: 2019 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 17th November 2017. This data will be updated every 24 hours.