Skip to main content Accesibility Help

Cross-Sample Comparisons and External Validity

  • Yanna Krupnikov (a1) and Adam Seth Levine (a2)

Experimentation is an increasingly popular method among political scientists. While experiments are highly advantageous for creating internally valid conclusions, they are often criticized for being low on external validity. Critical to questions of external validity are the types of subjects who participate in a given experiment, with scholars typically arguing that samples of adults are more externally valid then student samples. Despite the vociferousness of such arguments, these claims have received little empirical treatment. In this paper we empirically test for key differences between student and adult samples by conducting four parallel experiments on each of the three samples commonly used by political scientists. We find that our student and diverse, national adult sample behave consistently and in line with theoretical predictions once relevant moderators are taken into account. The same is not true for our adult convenience sample.

Hide All
Arceneaux, K. 2012. Cognitive Biases and the Strength of Political Arguments. American Journal of Political Science 56: 271–85.
Banks, A. and Valentino, N. A. 2012. Emotional Substrates of White Racial Attitudes. American Journal of Political Science 56: 289–97.
Barabas, J. and Jerit, J. 2010. Are Survey Experiments Externally Valid? American Political Science Review 104: 226–42.
Berinsky, A. J., Huber, G. A. and Lenz, G. S. 2012. Evaluating Online Labor Markets for Experimental Research:'s Mechanical Turk. Political Analysis 20: 351–68.
Brady, H. 2000. Contributions of Survey Research to Political Science. PS: Political Science and Politics 33: 4757.
Brody, J. L., Gluck, J. P. and Aragon, A. S. 2000. Participants’ Understanding of the Process of Psychological Research: Debriefing. Ethics & Behavior 10: 1325.
Brooks, D. J. and Geer, J. G. 2007. Beyond Negativity: The Effects of Incivility on the Electorate. American Journal of Political Science 51: 116.
Chong, D. and Druckman, J. N. 2007. Framing Public Opinion in Competitive Democracies. American Political Science Review 101: 637–55.
Chong, D. and Druckman, J. N. 2010. Framing Theory. Annual Review of Political Science 10: 103–26.
Cook, T. D., Bean, J. R., Calder, B. J., Frey, R., Krovetz, M. L., and Reisman, S. R. 1970. Demand Characteristics and Three Conceptions of the Frequently Deceived Subject. Journal of Personality and Social Psychology 14: 185–94.
Dalen, L. H., Stanton, N. A. and Roberts, A. D. 2001. Faking Personality Questionnaires in Personnel Selection. Journal of Management Development 20: 729–42.
Druckman, J. N., Fein, J. and Leeper, T. J. 2012. A Source of Bias in Public Opinion Stability. American Political Science Review 106: 430–54.
Druckman, J. N. and Kam, C. D. 2011. Students as Experimental Participants: A Defense of the ‘Narrow Data Base’. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.
Druckman, J. N. and Leeper, T. 2012. Learning More from Political Communication Experiments: Pretreatment and Its Effects. American Journal of Political Science 56: 875–96.
Gaines, B. J., Kuklinski, J. H. and Quirk, P. J. 2007. The Logic of the Survey Experiment Reexamined. Political Analysis 15: 120.
Gartner, S. S. 2008. The Multiple Effects of Casualties on Public Support for War: An Experimental Approach. American Political Science Review 102: 95106.
Gerber, A. S. and Green, D. P. 2008. Field Experiments and Natural Experiments. In The Oxford Handbook of Political Methodology. eds. Box-Steffensmeier, Brady, and Collier, Oxford: Oxford University Press.
Gerber, A. S., Huber, G. A., Doherty, D. and Dowling, C. M. 2011. Citizens’ Policy Confidence and Electoral Punishment: A Neglected Dimension of Electoral Accountability. Journal of Politics 73: 1206–24.
Haider-Markel, D. P. and Joslyn, M. R. 2001. Gun Policy, Opinion, Tragedy, and Blame Attribution: The Conditional Influence of Issue Frames. Journal of Politics 63: 520–43.
Henry, P. J. 2008. College Sophomores in the Laboratory Redux: Influences of a Narrow Data Base on Social Psychology's View of the Nature of Prejudice. Psychological Inquiry: An International Journal for the Advancement of Psychological Theory 19: 4971.
Hopkins, D. and King, G. 2010. Improving Anchoring Vignettes: Designing Surveys to Correct Interpersonal Incomparability. Public Opinion Quarterly 74: 201–22.
Huber, G. A. and Paris, C. 2013. Assessing the Programmatic Equivalence Assumption in Question Wording Experiments: Understanding why Americans Like Assistance to the Poor More Than Welfare. Public Opinion Quarterly 77: 385–97.
Huddy, L. and Khatib, N. 2007. American Patriotism, National Identity, and Political Involvement. American Journal of Political Science 51: 6377.
Iyengar, S. 2011. Laboratory Experiments in Political Science. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.
Jerit, J. 2009. How Predictive Appeals Affect Policy Opinions. American Journal of Political Science 53: 411–26.
Jones, E. E. and Sigall, H. 1971. The Bogus Pipeline: A New Paradigm for Measuring Affect and Attitude. Psychological Bulletin 76: 349–64.
Kam, C. D. 2007. Implicit Attitudes, Explicit Choices: When Subliminal Priming Predicts Candidate Preference. Political Behavior 29: 343–67.
Kam, C. D. and Simas, E. 2010. Risk Orientation and Policy Frames. Journal of Politics 72 (2): 381–96.
Kam, C. D., Wilking, J. R. and Zechmeister, E. J. 2007. Beyond the ‘Narrow Data Base’: Another Convenience Sample for Experimental Research. Political Behavior 29: 415–40.
Kinder, D. 2003. Communication and Politics in the Age of Information. In Oxford Handbook of Political Psychology. eds. Sears, Huddy, and Jerivs, Oxford University Press.
Kinder, D. and Palfrey, T. 1993. On Behalf of Experimental Political Science. In Experimental Foundations of Political Science. eds. Kinder and Palfrey, Ann Arbor, Michigan: University of Michigan.
Levine, M. and Ensom, M. H. H. 2001. Post hoc Power Analysis: An Idea Whose Time Has Passed? Psychometry 21: 405–09.
McDermott, R. 2002. Experimental Methodology in Political Science. Political Analysis 10: 325–42.
McDermott, R. 2011. Internal and External Validity. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.
McGraw, K. 2011. Candidate Impressions and Evaluation. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.
Miller, J. M. and Krosnick, J. A. 2000. News Media Impact on the Ingredients of Presidential Evaluations: Politically Knowledgeable Citizens are Guided by a Trusted Source, American Journal of Political Science 44: 301–15.
Morton, R. B. and Williams, K. C. 2010. Experimental Political Science and the Study of Causality. New York: Cambridge University Press.
Mutz, D. 1992. Impersonal Influence: Effects of Representations of Public Opinion on Political Attitudes, Political Behavior 14: 89122.
Mutz, D. 2011. Population-Based Survey Experiments. Princeton, NJ: Princeton University Press.
Nelson, T. E., Bryner, S. M. and Carnahan, D. 2011. Media and Politics. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.
Nelson, T. E., Clawson, R. A. and Oxley, Z. M. 1997. Media Framing of a Civil Liberties Conflict and Its Effect on Tolerance. American Political Science Review 91: 567–83.
Ostrom, T. M. 1973. The Bogus Pipeline: A New Ignis Fatuus? Psychological Bulletin 79: 252–59.
Roese, N. J. and Jamieson, D. W. 1993. Twenty Years of Bogus Pipeline Research: A Critical Review and Meta-Analysis. Psychological Bulletin 114: 363–75.
Schwarz, N., Hippler, H.-J., Deutsch, B., and Strack, F. 1985. Response Scales: Effects of Category Range on Reported Behavior and Comparative Judgments. Public Opinion Quarterly 49: 388–95.
Sears, D. 1986. College Sophomores in the Laboratory: Influences of a Narrow Data Base on Social Psychology's View of Human Nature. Journal of Personality and Social Psychology 51: 515–30.
Shadish, W. R., Cook, T. D. and Campbell, D. T. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.
Sigall, H. and Page, R. 1972. Reducing Attenuation in the Expression of Interpersonal Affect via the Bogus Pipeline. Sociometry 35: 629–42.
Steffens, M. C. 2004. Is the Implicit Association Test Immune to Faking? Experimental Psychology 51: 165–79.
Taber, C. and Lodge, M. 2006. Motivated Skepticism in the Evaluation of Political Beliefs. American Journal of Political Science 50: 755–69.
Transue, J. E., Lee, D. J. and Aldrich, J. H. 2009. Treatment Spillover Effects Across Survey Experiments. Political Analysis 17: 143–61.
Vavreck, L. and Rivers, D. 2008. The 2006 Cooperative Congressional Election Study. Journal of Elections, Public Opinion, and Parties 18: 355–66.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Experimental Political Science
  • ISSN: 2052-2630
  • EISSN: 2052-2649
  • URL: /core/journals/journal-of-experimental-political-science
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Type Description Title
Supplementary materials

Krupnikov Supplementary Material

 PDF (86 KB)
86 KB


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed