Hostname: page-component-cb9f654ff-h4f6x Total loading time: 0 Render date: 2025-08-02T05:00:38.964Z Has data issue: false hasContentIssue false
  • English
  • Français

Is a non-representative convenience sample of adults good enough? Insights from an economic experiment

Published online by Cambridge University Press:  17 January 2025

Sean F. Ellis Open the ORCID record for Sean F. Ellis [Opens in a new window] ,
Olesya M. Savchenko  and
Kent D. Messer
Sean F. Ellis*
Affiliation:
Behavior Change for Good, The Wharton School, University of Pennsylvania, 3720 Walnut St., 19104 Philadelphia, PA, USA
Olesya M. Savchenko
Affiliation:
Food and Resource Economics Department, University of Florida, 1183 McCarty Hall A, 32611 Gainesville, FL, USA
Kent D. Messer*
Affiliation:
Department of Applied Economics and Statistics, University of Delaware, 531 S. College Avenue, 19716 Newark, DE, USA
*
e-mail: sfellis@wharton.upenn.edu
messer@udel.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Recruitment of representative and generalizable adult samples is a major challenge for researchers conducting economic field experiments. Limited access to representative samples or the high cost of obtaining them often leads to the recruitment of non-representative convenience samples. This research compares the findings from two field experiments involving 860 adults: one from a non-representative in-person convenience sample and one from a representative online counterpart. We find no meaningful differences in the key behaviors of interest between the two samples. These findings contribute to a growing body of literature demonstrating that non-representative convenience samples can be sufficient in certain contexts.

Keywords

Non-representative convenience samplingField experimentsOnline recruitmentRepresentative sampling

JEL classification

C93: Field Experiments B41: Economic Methodology C83: Survey Methods • Sampling Methods

Information

Type
Methodology Paper
Information
Journal of the Economic Science Association , Volume 9 , Issue 2 , December 2023 , pp. 293 - 307
Copyright
Copyright © The Author(s), under exclusive licence to Economic Science Association 2023.

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Banerjee, A. V., Duflo, E. (2009). The experimental approach to development economics. Annual Review of Economics, 1(1), 151–178. 10.1146/annurev.economics.050708.143235Google Scholar
Boas, T. C., Christenson, D. P., Glick, D. M. (2020). Recruiting large online samples in the United States and India: facebook, mechanical turk, and qualtrics. Political Science Research and Methods, 8(2), 232–250. 10.1017/psrm.2018.28Google Scholar
Bol, D. (2019). Putting politics in the lab: a review of lab experiments in political science. Government and Opposition, 54(1), 167–190. 10.1017/gov.2018.14Google Scholar
Camerer, C. (2011). The promise and success of lab-field generalizability in experimental economics: A critical reply to Levitt and List. SSRN Electronic Journal, 10.2139/ssrn.1977749Google Scholar
Ellis, S. F., Savchenko, O. M., Messer, K. D. (2022). Mitigating stigma associated with recycled water: Aquifer recharge and trophic levels. American Journal of Agricultural Economics, 104(3), 1077–1099. 10.1111/ajae.12256Google Scholar
Frigau, L., Medda, T., Pelligra, V. (2019). From the field to the lab. an experiment on the representativeness of standard laboratory subjects. Journal of Behavioral and Experimental Economics, 78, 160–169. 10.1016/j.socec.2018.06.003Google Scholar
Gelman, A., Goal, S., Rothschild, D., Wang, W., & Schill, D. (2016). High-frequency polling with non-representative data Political communication in real time, New York: Routledge.Google Scholar
Goldberg, M., Van Der Linden, S., Ballew, M. T., Rosenthal, S. A., Leiserowitz, A. (2019). Convenient but biased? the reliability of convenience samples in research about attitudes toward climate change. Osfpreprints., 11, 4453.Google Scholar
Levitt, S. D., List, J. A. (2007). Viewpoint: on the generalizability of lab behaviour to the field. Canadian Journal of Economics/revue Canadienne D’économique, 40(2), 347–370. 10.1111/j.1365-2966.2007.00412.xGoogle Scholar
Levitt, S. D., List, J. A. (2008). Homo economicus evolves. Science, 319(5865), 909–910. 10.1126/science.1153640Google ScholarPubMed
List, J. A., Price, M. K. (2016). The use of field experiments in environmental and resource economics. Review of Environmental Economics and Policy, 10(2), 206–225. 10.1093/reep/rew008Google Scholar
Muller, S. (2014). Randomised Trials for Policy: A Review of the External Validity of Treatment Effects (Working Paper). Accessed 14 Dec 2020.Google Scholar
Palm-Forster, L. H., Ferraro, P. J., Janusch, N., Vossler, C. A., Messer, K. D. (2019). Behavioral and experimental agri-environmental research: Methodological challenges, literature gaps, and recommendations. Environmental and Resource Economics, 73(3), 719–742. 10.1007/s10640-019-00342-xGoogle Scholar
Palm-Forster, L. H., Messer, K. D., & Palm-Forster, L. H. (2021). Experimental and Behavioral Economics to Inform Agri-Environmental Programs and Policies Handbook of Agricultural Economics, USA: David Just and Chris Barrett.Google Scholar
Peth, D., Mußhoff, O. (2020). Comparing compliance behaviour of students and farmers an extra-laboratory experiment in the context of agri-environmental nudges in Germany. Journal of Agricultural Economics., 71(2), 601–615. 10.1111/1477-9552.12352Google Scholar
Roe, B. E., Just, D. R. (2009). Internal and external validity in economics research: tradeoffs between experiments, field experiments, natural experiments, and field data. American Journal of Agricultural Economics, 91(5), 1266–1271. 10.1111/j.1467-8276.2009.01295.xGoogle Scholar
Rosch, S., Raszap Skorbiansky, S., Weigel, C., Messer, K. D., Hellerstein, D. (2021). Barriers to using economic experiments in evidence-based agricultural policymaking. Applied Economic Perspectives and Policy, 43(2), 531–555. 10.1002/aepp.13091Google Scholar
Savchenko, O., Kecinski, M., Li, T., Messer, K. D. (2018). reclaimed water and food production: cautionary tales from consumer research. Environmental Research., 170, 320–331. 10.1016/j.envres.2018.12.051Google ScholarPubMed
Wang, W., Rothschild, D., Goel, S., Gelman, A. (2015). Forecasting elections with non-representative polls. International Journal of Forecasting, 31(3), 980–991. 10.1016/j.ijforecast.2014.06.001Google Scholar
Weigel, C., Paul, L. A., Ferraro, P. J., Messer, K. D. (2021). Challenges in recruiting US farmers for policy-relevant economic field experiments. Applied Economic Perspectives and Policy, 43(2), 556–572. 10.1002/aepp.13066Google Scholar