Skip to main content
    • Aa
    • Aa

Amazon Mechanical Turk for Industrial and Organizational Psychology: Advantages, Challenges, and Practical Recommendations

  • Sang Eun Woo (a1), Melissa Keith (a1) and Meghan A. Thornton (a1)

We are in almost full agreement with Landers and Behrend's (2015) thoughtful and balanced critiques of various convenience sampling strategies focusing on the four most frequently used data sources in our field. In this commentary, we expand on Landers and Behrend's discussions specifically around Mechanical Turk (MTurk) by providing further supporting voice and/or clarity to the four potential concerns and relative advantages associated with MTurk. We also raise a few additional concerns and challenges to which the current literature does not yet offer definitive answers. We conclude with some practical guidelines summarizing the relative advantages and unique challenges of using MTurk.

Corresponding author
Correspondence concerning this article should be addressed to Sang Eun Woo, 703 Third Street, West Lafayette, IN 47907. E-mail:
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

J. J. Arnett (2008). The neglected 95%: Why American psychology needs to become less American. American Psychologist, 63, 602614.

T. S. Behrend , D. J. Sharek , A. W. Meade , & E. N. Wiebe (2011). The viability of crowdsourcing for survey research. Behavioral Research Methods, 43 (3), 114. doi:10.3758/s13428-011-0081-0

M. Buhrmester , T. Kwang , & S. D. Gosling (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 35.

J. Chandler , P. Mueller , & G. Paolacci (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavioral Research, 46, 112130. doi:10.3758/s13428-013-0365-7

M. J. C. Crump , J. V. McDonnell , & T. M. Gureckis (2013). Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PLos ONE, 8 (3). doi:10.1371/journal.pone.0057410

K. Fort , A. Gilles , & K. B. Cohen (2011). Amazon Mechanical Turk: Gold mine or coal mine? Computational Linguistics, 37 (2), 413420.

C. J. Holden , T. Dennie , & A. D. Hicks (2013). Assessing the reliability of the M5–120 on Amazon's Mechanical Turk. Computers in Human Behavior, 29, 17491754. doi:10.1016/j.chb.2013.02.020

J. J. Horton , D. G. Rand , & R. J. Zeckhauser (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399425.

G. J. Lautenschlager , & V. L. Flaherty (1990). Computer administration of questions: More desirable or more social desirability? Journal of Applied Psychology, 75, 310314.

Y. Lelkes , J. A. Krosnick , D. M. Marx , C. M. Judd , & B. Park (2012). Complete anonymity compromises accuracy of self-reports. Journal of Experimental Social Psychology, 48, 12911299.

A. D. Ong , & D. J. Weiss (2000). The impact of anonymity on responses to sensitive questions. Journal of Applied Social Psychology, 30, 16911708.

E. Peer , J. Vosgerau , & A. Acquisti (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46 (4), 10231031.

D. G. Rand (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172179.

D. J. Simons , & C. F. Chabris (2012). Common (mis)beliefs about memory: A replication and comparison of telephone and Mechanical Turk survey methods. PLoS ONE, 7 (12), 15. doi:10.1371/journal.pone.0051876

A. B. Van Zant , & L. J. Kray (2014). “I can't lie to your face”: Minimal face-to-face interaction promotes honesty. Journal of Experimental Social Psychology, 55, 234238.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Industrial and Organizational Psychology
  • ISSN: 1754-9426
  • EISSN: 1754-9434
  • URL: /core/journals/industrial-and-organizational-psychology
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Altmetric attention score

Full text views

Total number of HTML views: 5
Total number of PDF views: 71 *
Loading metrics...

Abstract views

Total abstract views: 303 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 22nd September 2017. This data will be updated every 24 hours.