Skip to main content
×
Home
    • Aa
    • Aa

The Generalizability of Survey Experiments*

  • Kevin J. Mullinix (a1), Thomas J. Leeper (a2), James N. Druckman (a3) and Jeremy Freese (a4)
Abstract
Abstract

Survey experiments have become a central methodology across the social sciences. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, due to the expense of population-based samples, much research relies on convenience samples (e.g. students, online opt-in samples). The emergence of affordable, but non-representative online samples has reinvigorated debates about the external validity of experiments. We conduct two studies of how experimental treatment effects obtained from convenience samples compare to effects produced by population samples. In Study 1, we compare effect estimates from four different types of convenience samples and a population-based sample. In Study 2, we analyze treatment effects obtained from 20 experiments implemented on a population-based sample and Amazon's Mechanical Turk (MTurk). The results reveal considerable similarity between many treatment effects obtained from convenience and nationally representative population-based samples. While the results thus bolster confidence in the utility of convenience samples, we conclude with guidance for the use of a multitude of samples for advancing scientific knowledge.

Copyright
Footnotes
Hide All
*

The authors acknowledge support from a National Science Foundation grant for Time-Sharing Experiments in the Social Sciences (SES-1227179). Druckman and Freese are co-Principal Investigators of TESS, and Study 2 was designed and funded as a methodological component of their TESS grant. Study 1 includes data in part funded by an NSF Doctoral Dissertation Improvement Grant to Leeper (SES-1160156) and in part collected via a successful proposal to TESS by Mullinix and Leeper. Druckman and Freese were neither involved in Study 1 nor with any part of the review or approval of Mullinix and Leeper's TESS proposal (via recusal, given other existing collaborations). Only after data from both studies were collected did authors determine that the two studies were so complementary that it would be better to publish them together. The authors thank Lene Aarøe, Kevin Arceneaux, Christoph Arndt, Adam Berinsky, Emily Cochran Bech, Scott Clifford, Adrienne Hosek, Cindy Kam, Lasse Laustsen, Diana Mutz, Helene Helboe Pedersen, Richard Shafranek, Flori So, Rune Slothuus, Rune Stubager, Magdalena Wojcieszak, workshop participants at Southern Denmark University, and participants at The American Panel Survey Workshop at Washington University, St. Louis.

Footnotes
References
Hide All
AhlerDouglas J. 2014. “Self-Fulfilling Misperceptions of Public Polarization.” The Journal of Politics 76 (3): 607–20.
BakerReg. et al. 2010. “Research Synthesis: AAPOR Report on Online Panels.” Public Opinion Quarterly 74 (4): 171.
BarabasJason and Jerit Jennifer. 2010. “Are Survey Experiments Externally Valid?American Political Science Review 104 (May): 226242.
BenoitKenneth, Conway Drew, Lauderdale Benjamin E., Laver Michael, and Mikhaylov Slava. 2015. “Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data.” American Political Science Review: Forthcoming.
BergerArthur Asa. 2014. Media and Communications Research Methods: An Introduction to Qualitative and Quantitative Approaches. Los Angeles: Sage Publication, Inc.
BerinskyAdam J., Huber Gregory A., and Lenz Gabriel S.. 2012. “Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk.” Political Analysis 20 (Summer): 351–68.
BerinskyAdam J., Margolis Michele F., and Sances Michael W.. 2014. “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys.” American Journal of Political Science 58 (3): 739–53.
BloomHoward S. 2005. Learning More from Social Experiments. New York: Russell Sage Foundation.
BohannonJohn. 2011. “Social Science for Pennies.” Science 334 (October): 307.
BradyHenry E. 2000. “Contributions of Survey Research to Political Science.” PS: Political Science & Politics 33 (1): 4757.
BroockmanDavid E. and Green Donald P.. 2013. “Do Online Advertisements Increase Political Candidates’ Name Recognition or Favorability? Evidence from Randomized Field Experiments.” Political Behavior 36 (2): 263–89.
CallegaroMario, Baker Reg, Bethlehem Jelke, Göritz Anja S., Krosnick Jon A., and Lavrakas Paul J.. 2014. “Online Panel Research: History, Concepts, Applications, and a Look at the Future.” In Online Panel Research: A Data Quality Perspective, eds. Callegaro Mario, Baker Reg, Bethlehem Jelke, Göritz Anja S., Krosnick Jon A., and Lavrakas Paul J.. Sussex West, United Kingdom: John Wiley & Sons Ltd.
CampbellDonald T. 1969. “Prospective: Artifact and Control.” In Artifact in Behavioral Research, eds. Rosenthal Robert and Rosnow Robert. New York: Academic Press.
CasseseErin C., Huddy Leonie, Hartman Todd K., Mason Liliana, and Weber Christopher R.. 2013. “Socially Mediated Internet Surveys: Recruiting Participants for Online Experiments.” PS: Political Science and Politics 46 (4): 110.
ChandlerJesse, Mueller Pam, and Paolacci Gabriele. 2014. “Nonnaiveté Among Amazon Mechanical Turk Workers: Consequences and Solution for Behavioral Researchers.” Behavior Research Methods 46 (1): 112–30.
ChongDennis and Druckman James N.. 2007a. “Framing Public Opinion in Competitive Democracies.” American Political Science Review 101 (4): 637–55.
ChongDennis and Druckman James N.. 2007b. “Framing Theory.” Annual Review of Political Science 10 (1): 103–26.
CliffordScott and Jerit Jennifer. 2015. “Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies.” Journal of Experimental Political Science 1 (2): 120–31.
CoppockAlexander and Green Donald P.. 2015. “Assessing the Correspondence Between Experimental Results Obtained in the Lab and Field: A Review of Recent Social Science Research.” Political Science Research and Methods 3 (1): 113–31.
DruckmanJames N. 2001. “The Implications of Framing Effects for Citizen Competence.” Political Behavior 23 (3): 225–56.
DruckmanJames N. 2004. “Priming the Vote: Campaign Effects in a US Senate Election.” Political Psychology 25: 577–94.
DruckmanJames N. and Lupia Arthur. 2012. “Experimenting with Politics.” Science 335 (March): 1177–79.
DruckmanJames N. and Kam Cindy D.. 2011. “Students as Experimental Participants: A Defense of the ‘Narrow Data Base’.” In Cambridge Handbook of Experimental Political Science, eds. Druckman J. N., Green D. P., Kuklinski J. H., and Lupia A.. New York: Cambridge University Press, 4157.
DruckmanJames N., Green Donald P., Kuklinski James H., and Lupia Arthur. 2006. “The Growth and Development of Experimental Research in Political Science.” American Political Science Review 100 (November): 627–35.
DruckmanJames N., Peterson Erik, and Slothuus Rune. 2013. “How Elite Partisan Polarization Affects Public Opinion Formation.” American Political Science Review 107 (1): 5779.
Dynamo. 2014. “Guidelines for Academic Requesters.” Retrieved 6 October 2015 from (http://wiki.wearedynamo.org/index.php/Guidelines_for_Academic_Requesters), Accessed October 6, 2015.
EgamiNaoki and Imai Kosuke. 2015. “Causal Interaction in High-Dimension.” Working paper.
EntmanRobert M. 1993. “Framing: Toward Clarification of a Fractured Paradigm.” Journal of Communication 43 (4): 5158.
FowlerAnthony and Margolis Michele. 2014. “The Political Consequences of Uninformed Voters.” Electoral Studies 34: 100–10.
FrancoAnnie, Malhotra Neil, and Simonovits Gabor. 2014. “Publication Bias in the Social Sciences: Unlocking the File Drawer.” Science 345 (August): 1502–5.
FreeseJeremy, Howat Adam, Mullinix Kevin J., and Druckman James N.. 2015. “Limitations of Screening Methods to Obtain Representative Samples Using Online Labor Markets.” Working Paper, Northwestern University.
GamsonWilliam A. and Modigiliani Andre. 1989. “Media Discourse and Public Opinion on Nuclear Power: A Constructionist Approach.” American Journal of Sociology 95 (1): 137.
GelmanAndrew and Stern Hal. 2006. “The Difference Between ‘Significant’ and ‘Not Significant’ is not Itself Statistically Significant.” The American Statistician 60 (4): 328–31.
GerberAlan S. and Green Donald P.. 2008. “Field Experiments and Natural Experiments.” In Oxford Handbook of Political Methodology, eds. Box-Steffensmeier J. M., Brady H. E., and Collier D.. New York: Oxford University Press, 357–81.
GerberAlan S. and Green Donald P.. 2011. Field Experiments: Design, Analysis, and Interpretation. New York: W.W. Norton & Company.
GerringJohn. 2012. Social Science Methodology: A Unified Framework. New York: Cambridge University Press.
GfK. 2013. “Knowledge Panel Design Summary.” Available at: https://www.gfk.com/Documents/GfK-KnowledgePanel.pdf. Last accessed 20 November 2015.
GoodmanJoseph K., Cryder Cynthia E., and Cheema Amar. 2012. “Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples.” Journal of Behavioral Decision Making 26: 213–24.
GreenDonald P. and Kern Holger L.. 2012. “Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees.” Public Opinion Quarterly 76 (3): 491511.
HenrichJoseph, Heine Steven J., and Norenzayan Ara. 2010. “The Weirdest People in the World?Behavioral and Brain Sciences 33 (April): 6183.
HillygusD. Sunshine, Jackson Natalie, and Young McKenzie. 2014. “Professional Respondents in Nonprobability Online Panels.” In Online Panel Research: A Data Quality Perspetive, eds. Callegaro Mario, Baker Reg, Bethlehem Jelke, Göritz Anja S., Krosnick Jon A., and Lavrakas Paul J.. West Sussex, United Kingdom: John Wiley & Sons Ltd.
HoltCharles A. 2006. Markets, Games, and Strategic Behavior: Recipes for Interactive Learning. Reading, MA: Addison-Wesley.
HortonJohn J., Rand David G., and Zeckhauser Richard J.. 2011. “The Online Laboratory: Conducting Experiments in a Real Labor Market.” Experimental Economics 14 (3): 399425.
HovlandCarl I. 1959. “Reconciling Conflicting Results Derived from Experimental and Survey Studies of Attitude Change.” The American Psychologist 14: 817.
HuberGregory A., Hill Seth J., and Lenz Gabriel S.. 2012. “Sources of Bias in Retrospective Decision-Making: Experimental Evidence of Voters’ Limitations in Controlling Incumbents.” American Political Science Review 106 (4): 720–41.
HuffConnor and Tingley Dustin. 2015. “‘Who are these people?’ Evaluating the demographic characteristics and political preferences of MTurk survey respondents.” Research & Politics 2 (3): 111. DOI: 10.1177/2053168015604648.
IyengarShanto. 1991. Is Anyone Responsible? How Television Frames Political Issues. Chicago, IL: The University of Chicago Press.
JeritJennifer, Barabas Jason, and Clifford Scott. 2013. “Comparing Contemporaneous Laboratory and Field Experiments on Media Effects.” Public Opinion Quarterly 77 (1): 256–82.
KamCindy D., Wilking Jennifer R., and Zechmeister Elizabeth J.. 2007. “Beyond the ‘Narrow Data Base’: Another Convenience Sample for Experimental Research.” Political Behavior 29 (4): 415–40.
KeeterScott, Kennedy Courtney, Dimock Michael, Best Jonathan and Craighill Peyton. 2006. “Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey.” Public Opinion Quarterly 70 (5): 759–79.
KlarSamara. 2013. “The Influence of Competing Identity Primes on Political Preferences.” Journal of Politics 75 (4): 1108–24.
KlarSamara, Robison Joshua, and Druckman James N.. 2013. “Political Dynamics of Framing.” In New Directions in Media and Politics, ed. Ridout Travis N.. New York: Routledge, 173192.
KleinRichard A. et al. 2014. “Investigating Variation in Replicability: A ‘Many Labs’ Replication Project.” Social Psychology 45: 142–52.
KraftPeter. 2008. “Curses—Winner's and Otherwise—in Genetic Epidemiology.” Epidemiology 19 (September): 649–51.
KrissPeter H. and Weber Roberto. 2013. “Organizational Formation and Change: Lessons from Economic Laboratory Experiments.” In Handbook of Economic Organization: Integrating Economic and Organizational Theory, ed. Northampton A. Grandori.: Edward Elgar Publishing Limited, 245–72.
KrupnikovYanna and Levine Adam Seth. 2014. “Cross-Sample Comparisons and External Validity.” Journal of Experimental Political Science 1 (Spring): 5980.
LupiaArthur. 2014. “The 2013 Ithiel de Sola Pool Lecture: What is the Value of Social Science? Challengers for Researchers and Government Funders.” PS: Political Science & Politics 47 (January): 17.
MalhotraNeil and Kuo Alexander G.. 2008. “Attributing Blame: The Public's Response to Hurricane Katrina.” The Journal of Politics 70 (1): 120–35.
McDermottRose. 2002. “Experimental Methodology in Political Science.” Political Analysis, 10: 325–42.
MorawskiJill G. 1988. The Rise of Experimentation in American Psychology. New Haven: Yale University Press.
MutzDiana C. 2011. Population-Based Survey Experiments. Princeton: Princeton University Press.
NelsonThomas E., Clawson Rosalee A., and Oxley Zoe M.. 1997. “Media Framing of a Civil Liberties Conflict and Its Effect on Tolerance.” American Political Science Review 91 (3): 567–83.
NockSteven L. and Guterbock Thomas M.. 2010. “Survey experiments.” In Handbook of Survey Research, eds. Marsden P. V., and Wright J. D.. Emerald, UK, 837–64.
Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349: 943.
PaolacciGabriele, Chandler Jesse, and Ipeirotis Panagiotis G.. 2010. “Running Experiments on Amazon Mechanical Turk.” Judgment and Decision Making, 5 (August): 411–9.
Pew. 2012. Assessing the Representativeness of Public Opinion Surveys. Available at: http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-public-opinion-surveys/. Last accessed 20 November 2015.
RandDavid G., Peysakhovich Alexander, Kraft-Todd Gordon T., Newman George E., Wurzbacher Owen, Nowak Martin A., and Greene Joshua D.. 2014. “Social Heuristics Shape Intuitive Cooperation.” Nature Communications 5: 112.
RedlawskDavid P., Civettini Andrew J., and Emmerson Karen M.. 2010. “The Affective Tipping Point: Do Motivated Reasoners Ever ‘Get It’?Political Psychology 31: 563593.
RikerWilliam H. 1996. The Strategy of Rhetoric: Campaigning for the American Constitution. New Haven, CT: Yale University Press.
RossJoel, Irani Lily, Silberman M. Six, Zaldivar Andrew, and Tomlinson Bill. 2010. “Who are the Crowdworkers? Shifting Demographics in Amazon Mechanical Turk. In CHI EA 2010, New York: ACM Press, 2863–72.
SearsDavid O. 1986. “College Sophomores in the Laboratory: Influences of a Narrow Data Base on Social Psychology's View of Human Nature.” Journal of Personality and Social Psychology 51: 515530.
ShadishWilliam R., Cook Thomas D., and Campbell Donald T.. 2001. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, MA: Houghton-Mifflin.
SnidermanPaul. 2011. The Logic and Design of the Survey Experiment: An Autobiography of a Methodological Innovation.” In Cambridge Handbook of Experimental Political Science, eds. Druckman J. N., Green D. P., Kuklinski J. H., and Lupia A.. New York: Cambridge University Press, 102–14.
SteinmetzStephanie, Bianchi Annamaria, Tijdens Kea, and Biffignandi Silvia. 2014. “Improving Web Surveys Quality: Potentials and Constraints of Propensity Score Adjustments.” In Online Panel Research: A Data Quality Perspetive, eds. Callegaro Mario, Baker Reg, Bethlehem Jelke, Göritz Anja S., Krosnick Jon A., and Lavrakas Paul J.. West Sussex, United Kingdom: John Wiley & Sons Ltd.
ValentinoNicholas A., Traugott Michael W., and Hutchings Vincent L.. 2002. “Group Cues and Ideological Constraint: A Replication of Political Advertising Effects Studies in the Lab and in the Field.” Political Communicaton 19 (1): 2948.
WangWei, Rothschild David, Goel Sharad, and Gelman Andrew. 2015. “Forecasting Elections with Non-representative Polls.” International Journal of Forecasting 31 (3): 980991.
WeinbergJill D., Freese Jeremy, and McElhattan David. 2014. “Comparing Demographics, Data Quality, and Results of an Online Factorial Survey Between a Population-Based and a Crowdsource-Recruited Sample.” Sociological Science 1: 292310.
WrightJames D., and Marsden Peter V. 2010. “Survey Research and Social Science: History, Current Practice, and Future Prospects.” In Handbook of Survey Research, eds. Marsden P. V., and Wright J. D.. Emerald, UK, 326.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Experimental Political Science
  • ISSN: 2052-2630
  • EISSN: 2052-2649
  • URL: /core/journals/journal-of-experimental-political-science
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Keywords:

Type Description Title
WORD
Supplementary Materials

Mullinix supplementary material
Mullinix supplementary material 1

 Word (103 KB)
103 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 20
Total number of PDF views: 315 *
Loading metrics...

Abstract views

Total abstract views: 1848 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 24th October 2017. This data will be updated every 24 hours.