Skip to main content Accessibility help
Hostname: page-component-768ffcd9cc-s8fcc Total loading time: 0.413 Render date: 2022-12-03T08:54:46.687Z Has data issue: true Feature Flags: { "useRatesEcommerce": false } hasContentIssue true

On the Use of Crowdsourcing Labor Markets in Research

Published online by Cambridge University Press:  13 June 2016


Crowdsourcing platforms offer a source of inexpensive data for research. At their fingertips, researchers have a round-the-clock workforce to fill out surveys, participate in experiments, and content-analyze text, among other tasks that generate social science data and help support research. Thanks to its low cost and convenience, crowdlabor has quickly and uncritically become a mainstream tool in our discipline. While such platforms have been evaluated on their aptness to generate high-quality data, surprisingly little has been said about the economic or political implications of their usage. Among other aspects, this article problematizes the “state of legal exception” in which crowdlabor markets operate, their tendency to rely on a pool of economically vulnerable workers and the asymmetrical employment relations they create. Rather than offer an easy solution, I aim to open up a conversation about this unique set of challenges in order to acknowledge and address them.

Copyright © American Political Science Association 2016 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)


Adda, Gilles, Mariani, Joseph J., Besacier, Laurent, and Gelas, Hadrien. 2013. “Economic and Ethical Background of Crowdsourcing for Speech.” In Crowdsourcing for Speech Processing, ed.Eskénazi, xine, Levow, Gina-Anne et al. . Chichester, UK: John Wiley & Sons, Ltd.Google Scholar
Ahler, Douglas J. 2014. “Self-Fulfilling Misperceptions of Public Polarization.” Journal of Politics 76(3): 607–20.CrossRefGoogle Scholar
Arceneaux, Kevin. 2012. “Cognitive Biases and the Strength of Political Arguments.” American Journal of Political Science 56(2): 271–85.CrossRefGoogle Scholar
Aytes, Ayhan. 2013. “Return of the Crowds: Mechacnical Turk and Neoliberal States of Exception.” In Digital Labor: The Internet as Playground and Factory, ed. Scholz, Trebor. New York: Routledge.Google Scholar
Benkler, Yochai. 2006. The Wealth of Networks: How Social Production Transforms Markets and Freedoms. New Haven, CT: Yale University Press.Google Scholar
Benoit, Kenneth, Conway, Drew, Lauderdale, Benjamin, Laver, Michael and Mikhaylov, Slava. In press. “Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data.” American Political Science Review.
Berinsky, Adam J., Huber, Gregory A., and Lenz, Gabriel S.. 2012. “Evaluating Online Labor Markets for Experimental Research: Amazon. Com’s Mechanical Turk.” Political Analysis 20(3): 351–68.CrossRefGoogle Scholar
Brooks, Sarah M. 2014. “Insecure Democracy: Risk and Political Participation in Brazil.” Journal of Politics 76(4): 972–85.CrossRefGoogle Scholar
Burgoon, Brian and Dekker, Fabian. 2010. “Flexible Employment, Economic Insecurity and Social Policy Preferences in Europe.” Journal of European Social Policy 20(2): 126–41.CrossRefGoogle Scholar
Busarovs, Aleksejs. 2013. “Ethical Aspects of Crowdsourcing, or Is It a Modern Form of Exploitation.” International Journal of Economics & Business Administration I(1): 314.Google Scholar
Cherry, Miriam A. 2009. “Working for (Virtually) Minimum Wage: Applying the Fair Labor Standards Act in Cyberspace.” Alabama Law Review 60(5): 1077–110.Google Scholar
Clark, Andrew and Postel-Vinay, Fabien. 2009. “Job Security and Job Protection.” Oxford Economic Papers 61(2): 207–39.CrossRefGoogle Scholar
Collins, Jane L. and Mayer, Victoria. 2010. Both Hands Tied: Welfare Reform and the Race to the Bottom in the Low-Wage Labor Market. Chicago, IL: University of Chicago Press.CrossRefGoogle Scholar
Conway, Drew. 2013. “Methods for Collecting Large-Scale Non-Expert Text Coding.” SSRN Scholarly Paper ID 2260437 . Rochester, NY: Social Science Research Network.Google Scholar
Crawford, Jarret T. and Pilanski, Jane M.. 2014. “Political Intolerance, Right and Left.” Political Psychology 35(6): 841–51.CrossRefGoogle Scholar
Crump, Matthew J. C., McDonnell, John V., and Gureckis, Todd M.. 2013. “Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research.” PLoS ONE 8(3): e57410.CrossRefGoogle Scholar
Dickert, Neal and Grady, Christine. 1999. “What’s the Price of a Research Subject? Approaches to Payment for Research Participation.” New England Journal of Medicine 341(3): 198203.CrossRefGoogle Scholar
Doherty, David. 2013. “To Whom Do People Think Representatives Should Respond: Their District or the Country?” Public Opinion Quarterly 77(1): 237–55.CrossRefGoogle Scholar
Doherty, David, Dowling, Conor M., and Miller, Michael G.. 2014. “Does Time Heal All Wounds? Sex Scandals, Tax Evasion, and the Passage of Time.” PS: Political Science & Politics 47(2): 357–66.Google Scholar
Dow, Steven, Kulkarni, Anand, Bunge, Brie, Nguyen, Truc, Klemmer, Scott, and Hartmann, Björn. 2011. “Shepherding the Crowd: Managing and Providing Feedback to Crowd Workers.” In CHI ’11 Extended Abstracts on Human Factors in Computing Systems, 1669–74. CHI EA ’11. New York: ACM.Google Scholar
Fausey, Caitlin M. and Matlock, Teenie. 2011. “Can Grammar Win Elections?” Political Psychology 32(4): 563–74.CrossRefGoogle Scholar
Felstiner, Alek. 2011. “Working the Crowd: Employment and Labor Law in the Crowdsourcing Industry.” Berkeley Journal of Employment and Labor Law 32(1): 143203.Google Scholar
Fort, Karën, Adda, Gilles, and Bretonnel Cohen, K.. 2011. “Amazon Mechanical Turk: Gold Mine or Coal Mine?” Computational Linguistics 37(2): 413–20.CrossRefGoogle Scholar
Glynn, Adam N. 2013. “What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment.” Public Opinion Quarterly 77(S1): 159–72.CrossRefGoogle Scholar
Graber, Mark A. and Graber, Abraham. 2013. “Internet-based Crowdsourcing and Research Ethics: The Case for IRB Review. Journal of Medical Ethics 39(2): 115–18.CrossRefGoogle ScholarPubMed
Grimmer, Justin, Messing, Solomon, and Westwood, Sean J.. 2012. “How Words and Money Cultivate a Personal Vote: The Effect of Legislator Credit Claiming on Constituent Credit Allocation.” American Political Science Review 106(4): 703–19.CrossRefGoogle Scholar
Grimmer, Justin and Stewart, Brandon M.. 2013. “Automatic Content Analysis Methods for Political Texts.” Political Analysis 21(3): 267297.CrossRefGoogle Scholar
Guess, Andrew M. 2014. “Measure for Measure: An Experimental Test of Online Political Media Exposure.” Political Analysis 25(1): 5975.Google Scholar
Häusermann, Silja and Schwander, Hanna. 2012. “Varieties of Dualization: Identifying Insiders and Outsiders across Regimes,” in Emmenegger, Patrick, Häusermann, Silja, Palier, Bruno, Seeleib-Kaiser, Martin (eds.). The Age of Dualization. The Changing Face of Inequality in Deindustrializing Societies. Oxford and New York: Oxford University Press.Google Scholar
Hayes, Danny, Lawless, Jennifer L., and Baitinger, Gail. 2014. “Who Cares What They Wear? Media, Gender, and the Influence of Candidate Appearance.” Social Science Quarterly 95(5): 1194–212.CrossRefGoogle Scholar
Hardt, Michael and Negri, Antonio. 2000. Empire. Cambridge, MA: Harvard University Press.Google Scholar
Healy, Andrew and Lenz, Gabriel S.. 2014. “Substituting the End for the Whole: Why Voters Respond Primarily to the Election-Year Economy.” American Journal of Political Science 58(1): 3147.CrossRefGoogle Scholar
Hersh, Eitan D. and Schaffner, Brian F.. 2013. “Targeted Campaign Appeals and the Value of Ambiguity.” Journal of Politics 75(2): 520–34.CrossRefGoogle Scholar
Hopkins, Daniel J. 2014. “The Upside of Accents: Language, Inter-Group Difference, and Attitudes toward Immigration.” British Journal of Political Science 45(3): 531–57.CrossRefGoogle Scholar
Horton, John. 2010. “Why People Participate on Mechanical Turk, Now as a Mosaic Plot.” CrowdFlower. (accessed March 20, 2016).
Horton, John J., Rand, David G., and Zeckhauser, Richard J.. 2011. “The Online Laboratory: Conducting Experiments in a Real Labor Market.” Experimental Economics 14(3): 399425.CrossRefGoogle Scholar
Howe, Jeff. 2006. “The Rise of Crowdsourcing.” Wired 14(6). Retrieved from
Huff, Connor and Tingley, Dustin. 2014. “Evaluating the Demographic Characteristics and Political Preferences of MTurk Survey Respondents.” Working paper, Harvard University.
Ipeirotis, Panagiotis G. 2010. “Demographics of Mechanical Turk.” NYU Working Paper No. CEDER-10-01. Available at SSRN,
Kalleberg, Arne L., Reskin, Barbara F. and Hudson, Ken. 2000. “Bad Jobs in America: Standard and Nonstandard Employment Relations and Job Quality in the United States.” American Sociological Review 65(2): 256–78.CrossRefGoogle Scholar
Kam, Cindy D., Wilking, Jennifer R., and Zechmeister, Elizabeth J.. 2007. “Beyond the ‘Narrow Data Base’: Another Convenience Sample for Experimental Research.” Political Behavior 29(4): 415–40.CrossRefGoogle Scholar
King, Desmond and Rueda, David. 2008. “Cheap Labor: The New Politics of ‘Bread and Roses’ in Industrial Democracies.” Perspectives on Politics 6(2): 279–97.CrossRefGoogle Scholar
Kriner, Douglas L. and Shen, Francis X.. 2012. “How Citizens Respond to Combat Casualties: The Differential Impact of Local Casualties on Support for the War in Afghanistan.” Public Opinion Quarterly 76(4): 761–70.CrossRefGoogle Scholar
Lease, Matthew, Hullman, Jessica, Bigham, Jeffrey P. et al. . 2013. “Mechanical Turk Is Not Anonymous.” SSRN paper. Available online
Leeper, Thomas. 2013. “Crowdsourcing with R and the MTurk API.” Political Methodologist 20(2): 27.Google Scholar
Le, John. 2010. “Amazon Mechanical Turk Survey.” Crowd Flower. (accessed March 20, 2016).
Lorey, Isabel. 2015. State of Insecurity: Government of the Precarious. London: Verso.Google Scholar
Marvit, Mshe Z. 2014. “How Crowdworkers Became the Ghosts in the Digital Machine.” The Nation, February 5.
Mason, Winter and Suri, Siddharth. 2012. “Conducting Behavioral Research on Amazon’s Mechanical Turk.” Behavioral Research Methods 44(1): 123.CrossRefGoogle Scholar
McRobbie, Angela. 2011. “Reflections on Feminism, Immaterial Labour and the Post-Fordist Regime.” New Formations 70(1): 6076.CrossRefGoogle Scholar
Montgomery, Jacob M. and Cutler, Josh. 2013. “Computerized Adaptive Testing for Public Opinion Surveys.” Political Analysis 21(2): 172–92.CrossRefGoogle Scholar
Nelson, Barbara J. 1984. “Women’s Poverty and Women’s Citizenship: Some Political Consequences of Economic Marginality.” Signs: Journal of Women in Culture and Society 10(2): 209–32.CrossRefGoogle Scholar
Ong, Aihwa. 2006. Neoliberalism as Exception: Mutations in Citizenship and Sovereignty. Durkam, NC: Duke University Press.CrossRefGoogle Scholar
Paolacci, Gabriele, Chandler, Jesse and Ipeirotis, Panagiotis G.. 2010. “Running Experiments on Amazon Mechanical Turk.” Judgment and Decision Making 5(5): 411–19.Google Scholar
Perlin, Ross. 2012. Intern Nation: How to Earn Nothing and Learn Little in the Brave New Economy. London: Verso.Google Scholar
Ross, Andrew. 2013. “In Search of the Lost Paycheck.” In Digital Labor: T he Internet as Playground and Factory, ed. Scholz, Trebor. New York: Routledge.Google Scholar
Ross, Joel, Irani, Lilly, Silberman, M. Six, Zaldivar, Andrew and Tomlinson, Bill. 2010. “Who Are the Crowd workers? Shifting Demographics in Mechanical Turk.” In CHI ’10 Extended Abstracts on Human Factors in Computing Systems, 2863–72. CHI EA ’10. New York: ACM.Google Scholar
Rueda, David. 2007. Social Democracy Insider Out: Partisanship & Labor Market Policy in Industrialized Democracies. Oxford: Oxford University Press.CrossRefGoogle Scholar
Ruder, Alex I. 2014. “Institutional Design and the Attribution of Presidential Control: Insulating the President from Blame.” Quarterly Journal of Political Science 9(3): 301–35.CrossRefGoogle Scholar
Schmidt, Florian A. 2013a. “For a Few Dollars More—Class Action against Crowdsourcing.” Peer-Reviewed Journal About# BWPWAP, (Digital Aesthetics Research Center, Aarhus University in collaboration with transmediale 2013).
Schmidt, Florian A. 2013b. “The Good, The Bad and the Ugly: Why Crowdsourcing Needs Ethics.” In Proceedings of the Third International Conference on Cloud and Green Computing (CGC), 531–35.
Schneider, Nathan. 2015. “Intellectual Piecework.” Chronicle of Higher Education, February 16.
Scholz, Trebor. 2013. Digital Labor: The Internet as Playground and Factory. New York: Routledge.Google Scholar
Silberman, M., Ross, Joel, Irani, Lilly, and Tomlinson, Bill. 2010. “Sellers’ Problems in Human Computation Markets.” In Proceedings of the ACM SIGKDD Workshop on Human Computation 18–21. (accessed March 20, 2016).
Strolovitch, Dara Z. 2013. “Of Mancessions and Hecoveries: Race, Gender, and the Political Construction of Economic Crises and Recoveries.” Perspectives on Politics 11(1): 167–76.CrossRefGoogle Scholar
Surowiecki, James. 2004. The Wisdom of Crowds: Why the Many are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. New York: Doubleday.Google Scholar
Tappscott, Don and Williams, Anthony D.. 2006. Wikinomics: How Mass Collaboration Changes Everything. New York: Portfolio.Google Scholar
Thibodeau, Paul, Peebles, Matthew M., Grodner, Daniel J., and Durgin, Frank H.. 2013. “The Wished-For Always Wins Until the Winner Was Inevitable All Along: Motivated Reasoning and Belief Bias Regulate Emotion during Elections.” Political Psychology 36(4): 431–48.CrossRefGoogle Scholar
Tomz, Michael R. and Weeks, Jessica LP. 2013. “Public Opinion and the Democratic Peace.” American Political Science Review 107(4): 849–65.CrossRefGoogle Scholar
U.S. Department of Health and Human Services. 2013. “Considerations and Recommendations Concerning Internet Research and Human Subjects Research Regulations, with Revisions.” Available online
Vosko, Leah F., Martha, MacDonald, and Iain, Campbell. 2009. Gender and the Contours of Precarious Employment. New York: Routledge University Press.Google Scholar
Wolfson, Stephen M. and Lease, Matthew. 2011. “Look before You Leap: Legal Pitfalls of Crowdsourcing.” Proceedings of the American Society for Information Science and Technology 48(1): 110.CrossRefGoogle Scholar
Zittrain, Jonathan. 2008. “Ubiquitous Human Computing.” Philosophical Transactions Royal Society A 366(1881): 3813–21.CrossRefGoogle Scholar
Cited by

Save article to Kindle

To save this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the or variations. ‘’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

On the Use of Crowdsourcing Labor Markets in Research
Available formats

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

On the Use of Crowdsourcing Labor Markets in Research
Available formats

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

On the Use of Crowdsourcing Labor Markets in Research
Available formats

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *