Skip to main content Accessibility help
×
Home
Hostname: page-component-59b7f5684b-hd9dq Total loading time: 0.525 Render date: 2022-09-30T22:50:19.914Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "displayNetworkTab": true, "displayNetworkMapGraph": false, "useSa": true } hasContentIssue true

Cross-Sample Comparisons and External Validity

Published online by Cambridge University Press:  20 August 2014

Yanna Krupnikov
Affiliation:
Department of Political Science, Stony Brook University, Stony Brook, NY 11794, USA; email: yanna.krupnikov@stonybrook.edu
Adam Seth Levine
Affiliation:
Department of Government, Cornell University, Ithaca, NY 14850, USA; email: ASL22@cornell.edu

Abstract

Experimentation is an increasingly popular method among political scientists. While experiments are highly advantageous for creating internally valid conclusions, they are often criticized for being low on external validity. Critical to questions of external validity are the types of subjects who participate in a given experiment, with scholars typically arguing that samples of adults are more externally valid then student samples. Despite the vociferousness of such arguments, these claims have received little empirical treatment. In this paper we empirically test for key differences between student and adult samples by conducting four parallel experiments on each of the three samples commonly used by political scientists. We find that our student and diverse, national adult sample behave consistently and in line with theoretical predictions once relevant moderators are taken into account. The same is not true for our adult convenience sample.

Type
Research Article
Copyright
Copyright © The Experimental Research Section of the American Political Science Association 2014 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Arceneaux, K. 2012. Cognitive Biases and the Strength of Political Arguments. American Journal of Political Science 56: 271–85.CrossRefGoogle Scholar
Banks, A. and Valentino, N. A. 2012. Emotional Substrates of White Racial Attitudes. American Journal of Political Science 56: 289–97.CrossRefGoogle Scholar
Barabas, J. and Jerit, J. 2010. Are Survey Experiments Externally Valid? American Political Science Review 104: 226–42.CrossRefGoogle Scholar
Berinsky, A. J., Huber, G. A. and Lenz, G. S. 2012. Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk. Political Analysis 20: 351–68.CrossRefGoogle Scholar
Brady, H. 2000. Contributions of Survey Research to Political Science. PS: Political Science and Politics 33: 4757.Google Scholar
Brody, J. L., Gluck, J. P. and Aragon, A. S. 2000. Participants’ Understanding of the Process of Psychological Research: Debriefing. Ethics & Behavior 10: 1325.CrossRefGoogle ScholarPubMed
Brooks, D. J. and Geer, J. G. 2007. Beyond Negativity: The Effects of Incivility on the Electorate. American Journal of Political Science 51: 116.CrossRefGoogle Scholar
Chong, D. and Druckman, J. N. 2007. Framing Public Opinion in Competitive Democracies. American Political Science Review 101: 637–55.CrossRefGoogle Scholar
Chong, D. and Druckman, J. N. 2010. Framing Theory. Annual Review of Political Science 10: 103–26.CrossRefGoogle Scholar
Cook, T. D., Bean, J. R., Calder, B. J., Frey, R., Krovetz, M. L., and Reisman, S. R. 1970. Demand Characteristics and Three Conceptions of the Frequently Deceived Subject. Journal of Personality and Social Psychology 14: 185–94.CrossRefGoogle Scholar
Dalen, L. H., Stanton, N. A. and Roberts, A. D. 2001. Faking Personality Questionnaires in Personnel Selection. Journal of Management Development 20: 729–42.CrossRefGoogle Scholar
Druckman, J. N., Fein, J. and Leeper, T. J. 2012. A Source of Bias in Public Opinion Stability. American Political Science Review 106: 430–54.CrossRefGoogle Scholar
Druckman, J. N. and Kam, C. D. 2011. Students as Experimental Participants: A Defense of the ‘Narrow Data Base’. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.CrossRefGoogle Scholar
Druckman, J. N. and Leeper, T. 2012. Learning More from Political Communication Experiments: Pretreatment and Its Effects. American Journal of Political Science 56: 875–96.CrossRefGoogle Scholar
Gaines, B. J., Kuklinski, J. H. and Quirk, P. J. 2007. The Logic of the Survey Experiment Reexamined. Political Analysis 15: 120.CrossRefGoogle Scholar
Gartner, S. S. 2008. The Multiple Effects of Casualties on Public Support for War: An Experimental Approach. American Political Science Review 102: 95106.CrossRefGoogle Scholar
Gerber, A. S. and Green, D. P. 2008. Field Experiments and Natural Experiments. In The Oxford Handbook of Political Methodology. eds. Box-Steffensmeier, Brady, and Collier, Oxford: Oxford University Press.Google Scholar
Gerber, A. S., Huber, G. A., Doherty, D. and Dowling, C. M. 2011. Citizens’ Policy Confidence and Electoral Punishment: A Neglected Dimension of Electoral Accountability. Journal of Politics 73: 1206–24.CrossRefGoogle Scholar
Haider-Markel, D. P. and Joslyn, M. R. 2001. Gun Policy, Opinion, Tragedy, and Blame Attribution: The Conditional Influence of Issue Frames. Journal of Politics 63: 520–43.CrossRefGoogle Scholar
Henry, P. J. 2008. College Sophomores in the Laboratory Redux: Influences of a Narrow Data Base on Social Psychology's View of the Nature of Prejudice. Psychological Inquiry: An International Journal for the Advancement of Psychological Theory 19: 4971.CrossRefGoogle Scholar
Hopkins, D. and King, G. 2010. Improving Anchoring Vignettes: Designing Surveys to Correct Interpersonal Incomparability. Public Opinion Quarterly 74: 201–22.CrossRefGoogle Scholar
Huber, G. A. and Paris, C. 2013. Assessing the Programmatic Equivalence Assumption in Question Wording Experiments: Understanding why Americans Like Assistance to the Poor More Than Welfare. Public Opinion Quarterly 77: 385–97.CrossRefGoogle Scholar
Huddy, L. and Khatib, N. 2007. American Patriotism, National Identity, and Political Involvement. American Journal of Political Science 51: 6377.CrossRefGoogle Scholar
Iyengar, S. 2011. Laboratory Experiments in Political Science. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.Google Scholar
Jerit, J. 2009. How Predictive Appeals Affect Policy Opinions. American Journal of Political Science 53: 411–26.CrossRefGoogle Scholar
Jones, E. E. and Sigall, H. 1971. The Bogus Pipeline: A New Paradigm for Measuring Affect and Attitude. Psychological Bulletin 76: 349–64.CrossRefGoogle Scholar
Kam, C. D. 2007. Implicit Attitudes, Explicit Choices: When Subliminal Priming Predicts Candidate Preference. Political Behavior 29: 343–67.CrossRefGoogle Scholar
Kam, C. D. and Simas, E. 2010. Risk Orientation and Policy Frames. Journal of Politics 72 (2): 381–96.CrossRefGoogle Scholar
Kam, C. D., Wilking, J. R. and Zechmeister, E. J. 2007. Beyond the ‘Narrow Data Base’: Another Convenience Sample for Experimental Research. Political Behavior 29: 415–40.CrossRefGoogle Scholar
Kinder, D. 2003. Communication and Politics in the Age of Information. In Oxford Handbook of Political Psychology. eds. Sears, Huddy, and Jerivs, Oxford University Press.Google Scholar
Kinder, D. and Palfrey, T. 1993. On Behalf of Experimental Political Science. In Experimental Foundations of Political Science. eds. Kinder and Palfrey, Ann Arbor, Michigan: University of Michigan.CrossRefGoogle Scholar
Levine, M. and Ensom, M. H. H. 2001. Post hoc Power Analysis: An Idea Whose Time Has Passed? Psychometry 21: 405–09.Google ScholarPubMed
McDermott, R. 2002. Experimental Methodology in Political Science. Political Analysis 10: 325–42.CrossRefGoogle Scholar
McDermott, R. 2011. Internal and External Validity. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.Google Scholar
McGraw, K. 2011. Candidate Impressions and Evaluation. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.Google Scholar
Miller, J. M. and Krosnick, J. A. 2000. News Media Impact on the Ingredients of Presidential Evaluations: Politically Knowledgeable Citizens are Guided by a Trusted Source, American Journal of Political Science 44: 301–15.CrossRefGoogle Scholar
Morton, R. B. and Williams, K. C. 2010. Experimental Political Science and the Study of Causality. New York: Cambridge University Press.CrossRefGoogle Scholar
Mutz, D. 1992. Impersonal Influence: Effects of Representations of Public Opinion on Political Attitudes, Political Behavior 14: 89122.CrossRefGoogle Scholar
Mutz, D. 2011. Population-Based Survey Experiments. Princeton, NJ: Princeton University Press.CrossRefGoogle Scholar
Nelson, T. E., Bryner, S. M. and Carnahan, D. 2011. Media and Politics. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.Google Scholar
Nelson, T. E., Clawson, R. A. and Oxley, Z. M. 1997. Media Framing of a Civil Liberties Conflict and Its Effect on Tolerance. American Political Science Review 91: 567–83.CrossRefGoogle Scholar
Ostrom, T. M. 1973. The Bogus Pipeline: A New Ignis Fatuus? Psychological Bulletin 79: 252–59.CrossRefGoogle Scholar
Roese, N. J. and Jamieson, D. W. 1993. Twenty Years of Bogus Pipeline Research: A Critical Review and Meta-Analysis. Psychological Bulletin 114: 363–75.CrossRefGoogle Scholar
Schwarz, N., Hippler, H.-J., Deutsch, B., and Strack, F. 1985. Response Scales: Effects of Category Range on Reported Behavior and Comparative Judgments. Public Opinion Quarterly 49: 388–95.CrossRefGoogle Scholar
Sears, D. 1986. College Sophomores in the Laboratory: Influences of a Narrow Data Base on Social Psychology's View of Human Nature. Journal of Personality and Social Psychology 51: 515–30.CrossRefGoogle Scholar
Shadish, W. R., Cook, T. D. and Campbell, D. T. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.Google Scholar
Sigall, H. and Page, R. 1972. Reducing Attenuation in the Expression of Interpersonal Affect via the Bogus Pipeline. Sociometry 35: 629–42.CrossRefGoogle ScholarPubMed
Steffens, M. C. 2004. Is the Implicit Association Test Immune to Faking? Experimental Psychology 51: 165–79.CrossRefGoogle ScholarPubMed
Taber, C. and Lodge, M. 2006. Motivated Skepticism in the Evaluation of Political Beliefs. American Journal of Political Science 50: 755–69.CrossRefGoogle Scholar
Transue, J. E., Lee, D. J. and Aldrich, J. H. 2009. Treatment Spillover Effects Across Survey Experiments. Political Analysis 17: 143–61.CrossRefGoogle Scholar
Vavreck, L. and Rivers, D. 2008. The 2006 Cooperative Congressional Election Study. Journal of Elections, Public Opinion, and Parties 18: 355–66.CrossRefGoogle Scholar
Supplementary material: PDF

Krupnikov Supplementary Material

Appendix

Download Krupnikov Supplementary Material(PDF)
PDF 86 KB
161
Cited by

Linked content

Please note a has been issued for this article.

Save article to Kindle

To save this article to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Cross-Sample Comparisons and External Validity
Available formats
×

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

Cross-Sample Comparisons and External Validity
Available formats
×

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

Cross-Sample Comparisons and External Validity
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *