Skip to main content
×
Home
    • Aa
    • Aa

Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk

  • Adam J. Berinsky (a1), Gregory A. Huber (a2) and Gabriel S. Lenz (a3)
Abstract

We examine the trade-offs associated with using Amazon.com's Mechanical Turk (MTurk) interface for subject recruitment. We first describe MTurk and its promise as a vehicle for performing low-cost and easy-to-field experiments. We then assess the internal and external validity of experiments performed using MTurk, employing a framework that can be used to evaluate other subject pools. We first investigate the characteristics of samples drawn from the MTurk population. We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples—the modal sample in published experimental political science—but less representative than subjects in Internet-based panels or national probability samples. Finally, we replicate important published experimental work using MTurk samples.

Copyright
Corresponding author
e-mail: berinsky@mit.edu (corresponding author)
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

Charles C. Ballew II, and Alexander Todorov . 2007. Predicting political elections from rapid and unreflective face judgments. Proceedings of the National Academy of Sciences of the United States of America 104: 17948–53.

Cindy D. Kam , and Elizabeth N. Simas 2010. Risk orientations and policy frames. Journal of Politics 72: 381–96.

Cindy D. Kam , Jennifer R. Wilking , and Elizabeth J. Zechmeister 2007. Beyond the ‘narrow data base’: Another convenience sample for experimental research. Political Behavior 29: 415–40.

C. Lawson , Gabriel S. Lenz , Mike Myers , and Andy Baker . 2010. Looking like a winner: Candidate appearance and electoral success in new democracies. World Politics 62: 561–93.

C. Lawson , Gabriel S. Lenz , Mike Myers , and Andy Baker . 2010. Looking like a winner: Candidate appearance and electoral success in new democracies. World Politics 62: 561–93.

M. T. Orne 1962. On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist 17: 776–83.

M. T. Orne 1962. On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist 17: 776–83.

Kenneth A. Rasinski 1989. The effect of question wording on public support for government spending. Public Opinion Quarterly 53: 388–94.

Kazuhisa Takemura . 1994. Influence of elaboration on the framing of decision. Journal of Psychology 128: 33–9.

Kazuhisa Takemura . 1994. Influence of elaboration on the framing of decision. Journal of Psychology 128: 33–9.

Amos Tversky , and Daniel Kahneman . 1981. The framing of decisions and the psychology of choice. Science 211: 453–8.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax
Type Description Title
WORD
Supplementary Materials

Berinsky et al. supplementary material
Appendix

 Word (891 KB)
891 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 26 *
Loading metrics...

Abstract views

Total abstract views: 126 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 22nd March 2017. This data will be updated every 24 hours.