Skip to main content
×
×
Home

Assessing the Correspondence between Experimental Results Obtained in the Lab and Field: A Review of Recent Social Science Research*

  • Alexander Coppock and Donald P. Green
Abstract

A small but growing social science literature examines the correspondence between experimental results obtained in lab and field settings. This article reviews this literature and reanalyzes a set of recent experiments carried out in parallel in both the lab and field. Using a standardized format that calls attention to both the experimental estimates and the statistical uncertainty surrounding them, the study analyzes the overall pattern of lab-field correspondence, which is found to be quite strong (Spearman's ρ = 0.73). Recognizing that this correlation may be distorted by the ad hoc manner in which lab-field comparisons are constructed (as well as the selective manner in which results are reported and published), the article concludes by suggesting directions for future research, stressing in particular the need for more systematic investigation of treatment effect heterogeneity.

Copyright
Footnotes
Hide All
*Alexander Coppock is a PhD student, Department of Political Science, Columbia University, 420 West 118th Street, Suite 710, New York, NY 10025 (ac3242@columbia.edu). Donald P. Green is Professor of Political Science, Columbia University, 420 West 118th Street, Suite 710, New York, NY 10025 (dpg2110@columbia.edu). We are grateful to Oliver Armantier, Peter Aronow, Lindsay Dolan, Uri Gneezy, John List, Jennifer Jerit, Andrej Tusicisny and two anonymous reviewers for helpful comments, Glenn Harrison and Nicholas Valentino for graciously providing replication data, and to Johannes Abeler and Ido Erev for providing supplementary information. To view supplementary material for this article, please visit http://dx.doi.org/10.1017/psrm.2014.10
Footnotes
References
Hide All
Abeler, J., Marklein, F.. 2013. ‘Fungibility, Labels, and Consumption’. Unpublished manuscript.
Arceneaux, K.Nickerson, D.W.. 2009. ‘Who Is Mobilized to Vote? A ReAnalysis of 11 Field Experiments’. American Journal of Political Science 53(1):116.
Armantier, O.Boly, A.. 2013. ‘Comparing Corruption in the Laboratory and in the Field in Burkina Faso and in Canada’. The Economic Journal 123(573):11681187.
Ashraf, N., Berry, J.Shapiro, J.M.. 2010. ‘Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia’. American Economic Review 100(December):23832413.
Baird, S., Ferreira, F., Özler, B., Woolcock, M.. 2013. ‘Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review’. Technical report, Campbell Systematic Reviews.
Banerjee, A., Kumar, S., Pande, R., Su, F.. 2010. ‘Do Informed Voters Make Better Choices? Experimental Evidence from Urban India’. Unpublished manuscript.
Benz, M.Meier, S.. 2008. Do People Behave in Experiments as in the Field? Evidence from Donations. Experimental Economics 11(3):268281.
Bond, R.Smith, P.B.. 1996. ‘Culture and Conformity: A Meta-analysis of Studies Using Asch's (1952b, 1956) Line Judgment Task’. Psychological Bulletin 119(1):111137.
Bornstein, G., Erev, I.Rosen, O.. 1990. ‘Intergroup Competition as a Structural Solution to Social Dilemmas’. Social Behaviour 5(4):247260.
Camerer, C.F. forthcoming. ‘The Promise and Success of Lab-Field Generalizability in Experimental Economics: A Critical Reply to Levitt and List’. In G. Frechette and A. Schotter (eds), Methods of Modern Experimental Economics. Oxford: Oxford University Press.
Castro, L., Scartascini, C.. 2013. ‘Tax Compliance and Enforcement in the Pampas: Evidence from a Field Experiment’. Washington, DC: Inter-American Development Bank, 2013.
Charness, G., Schram, A.. 2013. ‘Social and Moral Norms in Allocation Choices in the Laboratory’. Unpublished manuscript.
Chong, A., De La O, A.L., Karlan, D.S., Wantchekon, L.. 2010. ‘Information Dissemination and Local Governments Electoral Returns, Evidence from a Field Experiment in Mexico’. Unpublished manuscript.
Cohen, J.Dupas, P.. 2010. ‘Free Distribution or Cost-sharing? Evidence from a Randomized Malaria Prevention Experiment’. Quarterly Journal of Economics 125(1):145.
Cook, T.D.Campbell, D.T.. 1979. Quasi-Experimentation: Design & Analysis Issues for Field Settings. Boston, MA: Houghton Mifflin.
Engel, C. 2011. ‘Dictator Games: a Meta Study’. Experimental Economics 14(4):583610.
Erev, I., Bornstein, G.Galili, R.. 1993. ‘Constructive Intergroup Competition as a Solution to the Free Rider Problem: A Field Experiment’. Journal of Experimental Social Psychology 29:463478.
Falk, A.Heckman, J.J.. 2009. ‘Lab Experiments are a Major Source of Knowledge in the Social Sciences’. Science 326(5952):535538.
Fellner, G., Sausgruber, R.Traxler, C.. 2013. ‘Testing Enforcement Strategies in the Field: Threat, Moral Appeal and Social Information’. Journal of the European Economic Association 11(3):634660.
Gerber, A.S. 2004. ‘Does Campaign Spending Work?: Field Experiments Provide Evidence and Suggest New Theory’. American Behavioral Scientist 47(5):541574.
Gerber, A.S.Green, D.P.. 2012. Field Experiments: Design, Analysis, and Interpretation. New York: W.W. Norton.
Gerber, A.S., Green, D.P.Larimer, C.W.. 2008. ‘Social Pressure and Voter Turnout: Evidence from a Large-Scale Field Experiment’. American Political Science Review 102(1):3348.
Glennerster, R.Takavarasha, K.. 2013. Running Randomized Evaluations: A Practical Guide. Princeton, NJ: Princeton University Press.
Gneezy, U., Haruvy, E.Yafe, H.. 2004. ‘The Inefficiency of Splitting the Bill’. The Economic Journal 114:265280.
Gneezy, U.List, J.A.. 2006. ‘Putting Behavioral Economics to Work: Testing for Gift Exchange in Labor Markets Using Field Experiments’. Econometrica 74(5):13651384.
Gneezy, U., Niederle, M.Rustichini, A.. 2003. ‘Performance in Competitive Environments: Gender Differences’. The Quarterly Journal of Economics 118(3):10491074.
Gneezy, U.Rustichini, A.. 2000. ‘Pay Enough or Don't Pay at All’. The Quarterly Journal of Economics 115(3):791810.
Gneezy, U.Rustichini, A.. 2004. ‘Gender and Competition at a Young Age’. American Economic Review 94(2):377381.
Green, D.P., Tusicisny, A. 2013. ‘Statistical Analysis of Results from Laboratory Studies in Experimental Economics: A Critique of Current Practice’. Unpublished manuscript.
Großer, J.Schram, A. 2010. ‘Public Opinion Polls, Voter Turnout, and Welfare: An Experimental Study’. American Journal of Political Science 54(3):700717.
Habyarimana, J., Humphreys, M., Posner, D.N.Weinstein, J.M.. 2009. Coethnicity: Diversity and the Dilemmas of Collective Action. New York: Russell Sage Foundation.
Harder, V.S., Stuart, E.A.Anthony, J.C.. 2010. ‘Propensity Score Techniques and the Assessment of Measured Covariate Balance to Test Causal Associations in Psychological Research’. Psychological Methods 15(3):234249.
Harrison, G.W.List, J.A.. 2004. ‘Field Experiments’. Journal of Economic Literature 42(4):10091055.
Harrison, G.W.List, J.A.. 2008. ‘Naturally Occurring Markets and Exogenous Laboratory Experiments: A Case Study of the Winners Curse’. The Economic Journal 118(528):822843.
Henrich, J., Boyd, R., Bowles, S., Camerer, C.F., Fehr, E., Gintis, H.McElreath, R.. 2001. ‘In Search of Homo Economicus: Behavioral Experiments in 15 Small-Scale Societies’. American Economic Review 91(2):7378.
Henrich, J., Heine, S.J.Norenzayan, A.. 2010. ‘The Weirdest People in the World?’ Behavioral and Brain Sciences 33:6183.
Hotz, V.J., Imbens, G.W.Mortimer, J.H.. 2005. ‘Predicting the Efficacy of Future Training Programs using Past Experiences at other Locations’. Journal of Econometrics 125(1–2):241270.
Humphreys, M., Sanchez de la Sierra, R.van der Windt, P.. 2013. ‘Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration’. Political Analysis 21(1):120.
Humphreys, M., Weinstein, J. M.. 2010. ‘Policing Politicians: Citizen Empowerment and Political Accountability in Uganda’. Unpublished manuscript.
Jerit, J., Barabas, J.Clifford, S.. 2013. ‘Comparing Contemporaneous Laboratory and Field Experiments on Media Effects’. Public Opinion Quarterly 77(1):256282.
Johnson, N.D.Mislin, A.A.. 2011. ‘Trust Games: A Meta-analysis’. Journal of Economic Psychology 32(5):865889.
Kessler, J.Vesterlund, L.. forthcoming. ‘The External Validity of Laboratory Experiments: Qualitative rather than Quantitative Effects’. In G. Frechette and A. Schotter (eds), Methods of Modern Experimental Economics. Oxford: Oxford University Press.
King, E.B.Ahmad, A.S.. 2010. An Experimental Field Study Of Interpersonal Discrimination Toward Muslim Job Applicants. Personnel Psychology 63(4):881906.
Lalancette, M.-F.Standing, L.. 1990. Asch Fails Again. Social Behavior and Personality 18(1):712.
Levitt, S.D.List, J.A. 2007. ‘Viewpoint: On the Generalizability of Lab Behaviour to the Field’. Canadian Journal of Economics 40(2):347370.
List, J.A. 2006. ‘The Behavioralist Meets the Market: Measuring Social Preferences and Reputation Effects in Actual Transactions’. Journal of Political Economy 114(1):137.
Milgram, S. 1963. ‘Behavioral Study of Obedience’. The Journal of Abnormal and Social Psychology 67(4):371378.
Morton, R.B.Williams, K.C.. 2010. Experimental Political Science and the Study of Causality: From Nature to the Lab. New York: Cambridge University Press.
Oosterbeek, H., Sloof, R.van de Kuilen, G.. 2004. ‘Cultural Differences in Ultimatum Game Experiments: Evidence from a Meta-Analysis’. Experimental Economics 7(2):171188.
Paluck, Elizabeth Levy. 2009. ‘Reducing Intergroup Prejudice and Conflict Using the Media: A Field Experiment in Rwanda’. Journal of Personality and Social Psychology 96(3):574587.
Rondeau, D.List, J.A.. 2008. ‘Matching and Challenge Gifts to Charity: Evidence from Laboratory and Natural Field Experiments’. Experimental Economics 11(3):253267.
Rosenthal, R. 1979. ‘The File Drawer Problem and Tolerance for Null Results’. Psychological Bulletin 86(3):638641.
Schultz, W.P., Khazian, A.M.Zaleski, A.C.. 2008. ‘Using Normative Social Influence to Promote Conservation Among Hotel Guests’. Social Influence 3(1):423.
Sears, D.O. 1986. ‘College Sophomores in the Laboratory: Influences of a Narrow Data Base on Social Psychology's View of Human Nature’. Journal of Personality and Social Psychology 51(3):515530.
Shadish, W.R., Cook, T.D.Campbell, D.T.. 2002. Experimental and Quasi-experimental Designs for Generalized Causal Inference. Boston, MA: Houghton, Mifflin and Company.
Shang, J., Reed, A.Croson, R.. 2008. ‘Identity Congruency Effects on Donations’. Journal of Marketing Research 45:351361.
Stewart, R., Rooyen, C.V., Korth, M., Chereni, A., Rebelo Da Silva, N.de Wet, T.. 2012. Do Micro-credit, Micro-savings and Micro-leasing Serve as Effective Financial Inclusion Interventions Enabling Poor People, and Especially Women, to Engage in Meaningful Economic Opportunities in Low- and Middle-income Countries. London: EPPI-Centre, University of London.
Student. 1908. ‘Probable Error of a Correlation Coefficient’. Biometrika 6(2):302310.
Valentino, N.A., Traugott, M.W.Hutchings, V.L.. 2002. ‘Group Cues and Ideological Constraint: A Replication of Political Advertising Effects Studies in the Lab and in the Field’. Political Communication 19:2948.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Science Research and Methods
  • ISSN: 2049-8470
  • EISSN: 2049-8489
  • URL: /core/journals/political-science-research-and-methods
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
Type Description Title
WORD
Supplementary materials

Coppock and Green Supplementary Material
Appendix

 Word (122 KB)
122 KB
UNKNOWN
Replication Materials on Dataverse

Coppock & Green Dataset
Dataset

 Unknown
UNKNOWN
Supplementary materials

Coppock and Green Datasets
Datasets

 Unknown

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 3
Total number of PDF views: 138 *
Loading metrics...

Abstract views

Total abstract views: 576 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 25th April 2018. This data will be updated every 24 hours.