Hostname: page-component-77f85d65b8-2tv5m Total loading time: 0 Render date: 2026-03-27T02:49:22.687Z Has data issue: false hasContentIssue false
  • English
  • Français

Direct replications in the era of open sampling

Published online by Cambridge University Press:  27 July 2018

Gabriele Paolacci  and
Jesse Chandler
Gabriele Paolacci
Affiliation:
Rotterdam School of Management, Erasmus University Rotterdam, 3062 PA, Rotterdam, The Netherlands. gpaolacci@rsm.nlhttps://www.rsm.nl/people/gabriele-paolacci/
Jesse Chandler
Affiliation:
Mathematica Policy Research, Ann Arbor, MI 48104. Institute for Social Research, University of Michigan Ann Arbor, MI 48109. jjchandl@umich.eduhttps://www.jessechandler.com
Get access Rights & Permissions [Opens in a new window]

Abstract

Data collection in psychology increasingly relies on “open populations” of participants recruited online, which presents both opportunities and challenges for replication. Reduced costs and the possibility to access the same populations allows for more informative replications. However, researchers should ensure the directness of their replications by dealing with the threats of participant nonnaiveté and selection effects.

Information

Type
Open Peer Commentary
Information
Behavioral and Brain Sciences , Volume 41 , 2018 , e144
Copyright
Copyright © Cambridge University Press 2018 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Arechar, A. A., Kraft-Todd, G. T. & Rand, D. G. (2017) Turking overtime: How participant characteristics and behavior vary over time and day on Amazon Mechanical Turk. Journal of the Economic Science Association 3(1):1–11.Google Scholar
Casey, L., Chandler, J., Levine, A. S., Proctor, A. & Strolovitch, D. Z. (2017, April–June) Intertemporal differences among MTurk worker demographics. SAGE Open, 1–15. doi: 10.1177/2158244017712774.Google Scholar
Chandler, J., Mueller, P. & Paolacci, G. (2014) Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods 46(1):112–30.Google Scholar
Chandler, J., Paolacci, G., Peer, E., Mueller, P. & Ratliff, K. A. (2015) Using nonnaive participants can reduce effect sizes. Psychological Science 26(7):1131–39.Google Scholar
Chandler, J. & Shapiro, D. (2016) Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology 12:53–81.Google Scholar
DeVoe, S. E. & House, J. (2016). Replications with MTurkers who are naïve versus experienced with academic studies: A comment on Connors, Khamitov, Moroz, Campbell, and Henderson (2015). Journal of Experimental Social Psychology 67:65–67.Google Scholar
Difallah, D., Filatova, E. & Ipeirotis, P. (2018) Demographics and dynamics of mechanical Turk workers. In: Proceedings of WSDM 2018: The Eleventh ACM International Conference on Web Search and Data Mining, Marina Del Rey, CA, USA February 5–9, 2018, pp. 135–143. Available at: https://dl.acm.org/citation.cfm?doid=3159652.3159661.Google Scholar
Goodman, J. K. & Paolacci, G. (2017) Crowdsourcing consumer research. Journal of Consumer Research 44(1):196–210.Google Scholar
Krupnikov, Y. & Levine, A. S. (2014). Cross-sample comparisons and external validity. Journal of Experimental Political Science 1(1), 59–80.Google Scholar
Peer, E., Vosgerau, J. & Acquisti, A. (2014) Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods 46(4):1023–31.Google Scholar
Simonsohn, U. (2015) Small telescopes: Detectability and the evaluation of replication results. Psychological Science 26:559–69.Google Scholar
Stewart, N., Chandler, J. & Paolacci, G. (2017) Crowdsourcing samples in cognitive science. Trends in Cognitive Sciences 21(10):736–48.Google Scholar
Stewart, N., Ungemach, C., Harris, A. J., Bartels, D. M., Newell, B. R., Paolacci, G. & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making 10(5):479–91.Google Scholar
Thomson, K. S. & Oppenheimer, D. M. (2016) Investigating an alternate form of the cognitive reflection test. Judgment and Decision Making 11(1):99–113.Google Scholar
Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K. & Zeelenberg, R. (2017) Participant nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin and Review. Available at: http://doi.org/10.3758/s13423-017-1348-y.Google Scholar