Skip to main content
×
×
Home

The External Validity of College Student Subject Pools in Experimental Research: A Cross-Sample Comparison of Treatment Effect Heterogeneity

  • Danielle L. Lupton (a1)
Abstract

Are student subject experiment pools comparable across institutions? Despite repeated concerns over the “college sophomore problem,” many experiment-based studies still rely on student subject pools due to their convenience and accessibility. In this paper, I investigate whether student subject pools are comparable across universities by examining how respondents across three student subject pools at distinct educational institutions perform on the same survey experiment about crisis bargaining between states. I argue that, due to selection biases inherent in university matriculation and the self-selection of students into experimental protocols, respondents across these subject pools will exhibit key demographic differences. I also examine whether respondents across these subject pools think similarly about international politics and respond comparably to experimental treatments. I find that, while there are significant demographic differences across subject pools, subjects across institutions respond similarly to experimental treatments—with the key exception of information regarding the regime type of a state. Furthermore, there is little evidence that these demographic differences impact conditional average treatment effects across subgroups. These findings carry critical implications for the use of student samples across political science and within international relations more specifically, particularly regarding the current replication crisis in the discipline.

Copyright
Corresponding author
Footnotes
Hide All

Contributing Editor: R. Michael Alvarez

Author’s note: Thank you to Matt Luttig and Aila Mattanock for their comments on earlier drafts of this paper. Thank you also to Chris Gelpi, Tim Büthe, and Bill Boettcher for their comments on the survey instrument. This work was supported by funding from Duke University and Colgate University. This research was approved by Institutional Review Boards at Colgate University (#ER-S15-33), Duke University (#B0170), and North Carolina State University (#2999). Replication files are available at Lupton (2018b).

Footnotes
References
Hide All
Barabas, Jason, and Jerit, Jennifer. 2010. Are survey experiments externally valid? American Political Science Review 104(2):226242.
Berinsky, Adam, Huber, Gregory, and Lenz, Gabriel. 2012. Evaluating online labor markets for experimental research: Amazon.com’s mechanical turk. Political Analysis 20(3):351368.
Clifford, Scot, Jewell, Ryan, and Waggoner, Philip. 2015. Are samples drawn from mechanical turk valid for research on political ideology? Research and Politics 2(4): 2053168015622072.
Coppock, Alexander. Forthcoming. Generalizing from survey experiments conducted on Amazon mechanical turk: A replication approach. Political Science Research and Methods . Available at https://doi.org/10.1017/psrm.2018.10.
Coppock, Alexander, Leeper, Thomas, and Mullinix, Kevin. 2017. The generalizability of heterogeneous treatment effect estimates across samples. Working Paper. Available at https://acoppock.github.io/projectpages_CLM_heterogeneity.html.
Cumming, Geoff. 2012. Understanding the New Statistics . New York: Routledge.
Druckman, James, and Kam, Cindy. 2011. Students as experimental participants: A defense of the narrow base. In Cambridge Handbook of Experimental Political Science , ed. Druckman, James, Green, Donald, Kuklinski, James, and Lupia, Arthur. Cambridge: Cambridge University Press, pp. 4157.
Falk, Armin, Meier, Stephan, and Zehnder, Christian. 2013. Do lab experiments misrepresent social preferences? The case of self-selected student samples. Journal of the European Economic Association 11(4):839852.
Freedman, David. 2008. On regression adjustments to experimental data. Advances in Applied Mathematics 40:180193.
Green, Donald, and Kern, Holger. 2012. Modeling heterogeneous treatment effects in survey experiments with Bayesian additive regression trees. Public Opinion Quarterly 76(3):491511.
Grimmer, Justin, Messing, Solomon, and Westwood, Sean. 2017. Estimating heterogeneous treatment effects and the effects of heterogeneous treatment effects with ensemble methods. Political Analysis 25:413434.
Hyde, Susan. 2015. Experiments in international relations: Lab, survey, and field. Annual Review of Political Science 18:403424.
Imai, Kosuke, and Strauss, Aaron. 2011. Estimation of heterogeneous treatment effects from randomized experiments, with application to the optimal planning of the get-out-the-vote campaign. Political Analysis 19:119.
Kam, Cindy, Wilking, Jennifer, and Zechmeister, Elizabeth. 2007. Beyond the ‘narrow data base’: Another convenience sample for experimental research. Political Behavior 29(4):415440.
Kees, Jeremy, Berry, Christopher, Burton, Scot, and Sheehan, Kim. 2017. An analysis of data quality: Professional panels, student subject pools, and Amazon’s mechanical turk. Journal of Advertising 46(1):141155.
Kosinski, Michal, Matz, Sandra, Gosling, Samuel, Popov, Vesselin, and Stillwell, David. 2015. Facebook as a research tool for the social sciences: Opportunities, challenges, ethical considerations, and practical guidelines. American Psychologist 70(6):543556.
Krupnikov, Yanna, and Levine, Adam Seth. 2014. Cross-sample comparisons and external validity. Journal of Experimental Political Science 1(1):5980.
Lupton, Danielle. 2018a. Reexamining reputation for resolve: Leaders, states, and the onset of international crises. Journal of Global Security Studies 3(2):198216.
Lupton, Danielle. 2018b. Replication data for: The external validity of college student subject pools in experimental research: A cross-sample comparison of treatment effect heterogeneity, https://doi.org/10.7910/DVN/U0TYY3, Harvard Dataverse, V1, UNF:6:w4bmPwNI9Q8PJmgXvWMVJQ==.
Lupton, Danielle. 2018c. Signaling resolve: Leaders, reputations, and the importance of early interactions. International Interactions 44(1):5987.
McDermott, Rose. 2002. Experimental methods in political science. Annual Review of Political Science 5:3161.
McDermott, Rose. 2011. Internal and external validity. In Cambridge Handbook of Experimental Political Science , ed. Druckman, James, Green, Donald, Kuklinski, James, and Lupia, Arthur. Cambridge: Cambridge University Press, pp. 2740.
Mintz, Alex. 2004. Foreign policy decision making in familiar and unfamiliar settings: An experimental study of high-ranking military officers. Journal of Conflict Resolution 48(1):91104.
Mintz, Alex, Redd, Steven, and Vedlitz, Arnold. 2006. Can we generalize from student experiments to the real world in political science, military affairs, and international relations? Journal of Conflict Resolution 50(5):757776.
Mintz, Alex, Yang, Yi, and McDermott, Rose. 2011. Experimental approaches to international relations. International Studies Quarterly 55(2):493501.
Mullinix, Kevin, Leeper, Thomas, Druckman, James, and Freese, Jeremy. 2015. The generalizability of survey experiments. Journal of Experimental Political Science 2(2):109138.
Mutz, Diana, and Pemantle, Robin. 2015. Standards for experimental research: Encouraging a better understanding of experimental methods. Journal of Experimental Political Science 2(2):192215.
Paolacci, Gabriele, Chandler, Jesse, and Ipeirotis, Panagiotis. 2010. Running experiments on Amazon mechanical turk. Judgment and Decision Making 5(5):411419.
Renshon, Jonathan. 2015. Losing face and sinking costs: Experimental evidence on the judgment of political and military leaders. International Organization 69(3):659695.
Sears, David. 1986. College sophomores in the laboratory: Influences of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology 51(3):515530.
Tomz, Michael. 2007. Domestic audience costs in international relations: An experimental approach. International Organization 61(4):821840.
Tomz, Michael, and Weeks, Jessica. 2013. Public opinion and the democratic peace. American Political Science Review 107(4):849865.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax

Keywords

Type Description Title
UNKNOWN
Supplementary materials

Lupton supplementary material
Lupton supplementary material 1

 Unknown (170 KB)
170 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed