Hostname: page-component-5d84bcc8dc-dmkht Total loading time: 0 Render date: 2026-09-04T23:32:31.433Z Has data issue: false hasContentIssue false
  • English
  • Français

The External Validity of College Student Subject Pools in Experimental Research: A Cross-Sample Comparison of Treatment Effect Heterogeneity

Published online by Cambridge University Press:  19 October 2018

Danielle L. Lupton Open the ORCID record for Danielle L. Lupton [Opens in a new window]
Danielle L. Lupton*
Affiliation:
Assistant Professor of Political Science, Colgate University, 13 Oak Drive, Hamilton, NY 13346, USA. Email: dlupton@colgate.edu
*
*Email: dlupton@colgate.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Are student subject experiment pools comparable across institutions? Despite repeated concerns over the “college sophomore problem,” many experiment-based studies still rely on student subject pools due to their convenience and accessibility. In this paper, I investigate whether student subject pools are comparable across universities by examining how respondents across three student subject pools at distinct educational institutions perform on the same survey experiment about crisis bargaining between states. I argue that, due to selection biases inherent in university matriculation and the self-selection of students into experimental protocols, respondents across these subject pools will exhibit key demographic differences. I also examine whether respondents across these subject pools think similarly about international politics and respond comparably to experimental treatments. I find that, while there are significant demographic differences across subject pools, subjects across institutions respond similarly to experimental treatments—with the key exception of information regarding the regime type of a state. Furthermore, there is little evidence that these demographic differences impact conditional average treatment effects across subgroups. These findings carry critical implications for the use of student samples across political science and within international relations more specifically, particularly regarding the current replication crisis in the discipline.

Keywords

survey experimentssubject poolsgeneralizabilitystudent subjectstreatment effects

Information

Type
Articles
Information
Political Analysis , Volume 27 , Issue 1 , January 2019 , pp. 90 - 97
Copyright
Copyright © The Author(s) 2018. Published by Cambridge University Press on behalf of the Society for Political Methodology. 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Contributing Editor: R. Michael Alvarez

Author’s note: Thank you to Matt Luttig and Aila Mattanock for their comments on earlier drafts of this paper. Thank you also to Chris Gelpi, Tim Büthe, and Bill Boettcher for their comments on the survey instrument. This work was supported by funding from Duke University and Colgate University. This research was approved by Institutional Review Boards at Colgate University (#ER-S15-33), Duke University (#B0170), and North Carolina State University (#2999). Replication files are available at Lupton (2018b).

References

Barabas, Jason, and Jerit, Jennifer. 2010. Are survey experiments externally valid? American Political Science Review 104(2):226–242.Google Scholar
Berinsky, Adam, Huber, Gregory, and Lenz, Gabriel. 2012. Evaluating online labor markets for experimental research: Amazon.com’s mechanical turk. Political Analysis 20(3):351–368.Google Scholar
Clifford, Scot, Jewell, Ryan, and Waggoner, Philip. 2015. Are samples drawn from mechanical turk valid for research on political ideology? Research and Politics 2(4): 2053168015622072.Google Scholar
Coppock, Alexander. Forthcoming. Generalizing from survey experiments conducted on Amazon mechanical turk: A replication approach. Political Science Research and Methods . Available at https://doi.org/10.1017/psrm.2018.10.Google Scholar
Coppock, Alexander, Leeper, Thomas, and Mullinix, Kevin. 2017. The generalizability of heterogeneous treatment effect estimates across samples. Working Paper. Available at https://acoppock.github.io/projectpages_CLM_heterogeneity.html.Google Scholar
Cumming, Geoff. 2012. Understanding the New Statistics . New York: Routledge.Google Scholar
Druckman, James, and Kam, Cindy. 2011. Students as experimental participants: A defense of the narrow base. In Cambridge Handbook of Experimental Political Science , ed. Druckman, James, Green, Donald, Kuklinski, James, and Lupia, Arthur. Cambridge: Cambridge University Press, pp. 41–57.Google Scholar
Falk, Armin, Meier, Stephan, and Zehnder, Christian. 2013. Do lab experiments misrepresent social preferences? The case of self-selected student samples. Journal of the European Economic Association 11(4):839–852.Google Scholar
Freedman, David. 2008. On regression adjustments to experimental data. Advances in Applied Mathematics 40:180–193.Google Scholar
Green, Donald, and Kern, Holger. 2012. Modeling heterogeneous treatment effects in survey experiments with Bayesian additive regression trees. Public Opinion Quarterly 76(3):491–511.Google Scholar
Grimmer, Justin, Messing, Solomon, and Westwood, Sean. 2017. Estimating heterogeneous treatment effects and the effects of heterogeneous treatment effects with ensemble methods. Political Analysis 25:413–434.Google Scholar
Hyde, Susan. 2015. Experiments in international relations: Lab, survey, and field. Annual Review of Political Science 18:403–424.Google Scholar
Imai, Kosuke, and Strauss, Aaron. 2011. Estimation of heterogeneous treatment effects from randomized experiments, with application to the optimal planning of the get-out-the-vote campaign. Political Analysis 19:1–19.Google Scholar
Kam, Cindy, Wilking, Jennifer, and Zechmeister, Elizabeth. 2007. Beyond the ‘narrow data base’: Another convenience sample for experimental research. Political Behavior 29(4):415–440.Google Scholar
Kees, Jeremy, Berry, Christopher, Burton, Scot, and Sheehan, Kim. 2017. An analysis of data quality: Professional panels, student subject pools, and Amazon’s mechanical turk. Journal of Advertising 46(1):141–155.Google Scholar
Kosinski, Michal, Matz, Sandra, Gosling, Samuel, Popov, Vesselin, and Stillwell, David. 2015. Facebook as a research tool for the social sciences: Opportunities, challenges, ethical considerations, and practical guidelines. American Psychologist 70(6):543–556.Google Scholar
Krupnikov, Yanna, and Levine, Adam Seth. 2014. Cross-sample comparisons and external validity. Journal of Experimental Political Science 1(1):59–80.Google Scholar
Lupton, Danielle. 2018a. Reexamining reputation for resolve: Leaders, states, and the onset of international crises. Journal of Global Security Studies 3(2):198–216.Google Scholar
Lupton, Danielle. 2018b. Replication data for: The external validity of college student subject pools in experimental research: A cross-sample comparison of treatment effect heterogeneity, https://doi.org/10.7910/DVN/U0TYY3, Harvard Dataverse, V1, UNF:6:w4bmPwNI9Q8PJmgXvWMVJQ==.Google Scholar
Lupton, Danielle. 2018c. Signaling resolve: Leaders, reputations, and the importance of early interactions. International Interactions 44(1):59–87.Google Scholar
McDermott, Rose. 2002. Experimental methods in political science. Annual Review of Political Science 5:31–61.Google Scholar
McDermott, Rose. 2011. Internal and external validity. In Cambridge Handbook of Experimental Political Science , ed. Druckman, James, Green, Donald, Kuklinski, James, and Lupia, Arthur. Cambridge: Cambridge University Press, pp. 27–40.Google Scholar
Mintz, Alex. 2004. Foreign policy decision making in familiar and unfamiliar settings: An experimental study of high-ranking military officers. Journal of Conflict Resolution 48(1):91–104.Google Scholar
Mintz, Alex, Redd, Steven, and Vedlitz, Arnold. 2006. Can we generalize from student experiments to the real world in political science, military affairs, and international relations? Journal of Conflict Resolution 50(5):757–776.Google Scholar
Mintz, Alex, Yang, Yi, and McDermott, Rose. 2011. Experimental approaches to international relations. International Studies Quarterly 55(2):493–501.Google Scholar
Mullinix, Kevin, Leeper, Thomas, Druckman, James, and Freese, Jeremy. 2015. The generalizability of survey experiments. Journal of Experimental Political Science 2(2):109–138.Google Scholar
Mutz, Diana, and Pemantle, Robin. 2015. Standards for experimental research: Encouraging a better understanding of experimental methods. Journal of Experimental Political Science 2(2):192–215.Google Scholar
Paolacci, Gabriele, Chandler, Jesse, and Ipeirotis, Panagiotis. 2010. Running experiments on Amazon mechanical turk. Judgment and Decision Making 5(5):411–419.Google Scholar
Renshon, Jonathan. 2015. Losing face and sinking costs: Experimental evidence on the judgment of political and military leaders. International Organization 69(3):659–695.Google Scholar
Sears, David. 1986. College sophomores in the laboratory: Influences of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology 51(3):515–530.Google Scholar
Tomz, Michael. 2007. Domestic audience costs in international relations: An experimental approach. International Organization 61(4):821–840.Google Scholar
Tomz, Michael, and Weeks, Jessica. 2013. Public opinion and the democratic peace. American Political Science Review 107(4):849–865.Google Scholar
Supplementary material: File

Lupton supplementary material

Lupton supplementary material 1

Download Lupton supplementary material(File)
File 170.5 KB