Skip to main content

Amazon Mechanical Turk for Industrial and Organizational Psychology: Advantages, Challenges, and Practical Recommendations

  • Sang Eun Woo (a1), Melissa Keith (a1) and Meghan A. Thornton (a1)

We are in almost full agreement with Landers and Behrend's (2015) thoughtful and balanced critiques of various convenience sampling strategies focusing on the four most frequently used data sources in our field. In this commentary, we expand on Landers and Behrend's discussions specifically around Mechanical Turk (MTurk) by providing further supporting voice and/or clarity to the four potential concerns and relative advantages associated with MTurk. We also raise a few additional concerns and challenges to which the current literature does not yet offer definitive answers. We conclude with some practical guidelines summarizing the relative advantages and unique challenges of using MTurk.

Corresponding author
Correspondence concerning this article should be addressed to Sang Eun Woo, 703 Third Street, West Lafayette, IN 47907. E-mail:
Hide All
Arnett, J. J. (2008). The neglected 95%: Why American psychology needs to become less American. American Psychologist, 63, 602614.
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavioral Research Methods, 43 (3), 114. doi:10.3758/s13428-011-0081-0
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research:'s Mechanical Turk. Political Analysis, 20, 351368.
Bloxham, E. (2014, September 12). The Fortune 500 can easily afford to give low-wage workers a healthy raise. Fortune. Retrieved from
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 35.
Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavioral Research, 46, 112130. doi:10.3758/s13428-013-0365-7
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PLos ONE, 8 (3). doi:10.1371/journal.pone.0057410
Fort, K., Gilles, A., & Cohen, K. B. (2011). Amazon Mechanical Turk: Gold mine or coal mine? Computational Linguistics, 37 (2), 413420.
Holden, C. J., Dennie, T., & Hicks, A. D. (2013). Assessing the reliability of the M5–120 on Amazon's Mechanical Turk. Computers in Human Behavior, 29, 17491754. doi:10.1016/j.chb.2013.02.020
Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399425.
International Labour Organization. (2011). Statistical update on employment in the informal economy. Geneva, Switzerland: ILO Department of Statistics.
Ipeirotis, P. G. (2010). Demographics of Mechanical Turk (Technical Report CeDER-10–01). New York: New York University.
Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice.
Lautenschlager, G. J., & Flaherty, V. L. (1990). Computer administration of questions: More desirable or more social desirability? Journal of Applied Psychology, 75, 310314.
Lelkes, Y., Krosnick, J. A., Marx, D. M., Judd, C. M., & Park, B. (2012). Complete anonymity compromises accuracy of self-reports. Journal of Experimental Social Psychology, 48, 12911299.
Maynard, D. C., & Ferdman, B. M. (2009). The marginalized workforce: How I-O psychology can make a difference. The Industrial–Organizational Psychologist, 46 (4), 2529.
Ong, A. D., & Weiss, D. J. (2000). The impact of anonymity on responses to sensitive questions. Journal of Applied Social Psychology, 30, 16911708.
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5 (5), 411419.
Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46 (4), 10231031.
Rand, D. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172179.
Ross, J., Zaldivar, A., Irani, L., & Tomlinson, B. (2010). Who are the Turkers? Worker demographics in Amazon Mechanical Turk. Proceedings from CHI ’10: The 28th ACM Conference on Human Factors in Computing Systems. Atlanta, GA.
Simons, D. J., & Chabris, C. F. (2012). Common (mis)beliefs about memory: A replication and comparison of telephone and Mechanical Turk survey methods. PLoS ONE, 7 (12), 15. doi:10.1371/journal.pone.0051876
Van Zant, A. B., & Kray, L. J. (2014). “I can't lie to your face”: Minimal face-to-face interaction promotes honesty. Journal of Experimental Social Psychology, 55, 234238.
Wolfson, S. N., & Bartkus, J. R. (2013). An assessment of experiments run on Amazon's Mechanical Turk. Mustang Journal of Business Ethics, 5, 119129.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Industrial and Organizational Psychology
  • ISSN: 1754-9426
  • EISSN: 1754-9434
  • URL: /core/journals/industrial-and-organizational-psychology
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Altmetric attention score

Full text views

Total number of HTML views: 5
Total number of PDF views: 163 *
Loading metrics...

Abstract views

Total abstract views: 610 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 18th June 2018. This data will be updated every 24 hours.