Skip to main content

On the Use of Crowdsourcing Labor Markets in Research

  • Dani M. Marinova

Crowdsourcing platforms offer a source of inexpensive data for research. At their fingertips, researchers have a round-the-clock workforce to fill out surveys, participate in experiments, and content-analyze text, among other tasks that generate social science data and help support research. Thanks to its low cost and convenience, crowdlabor has quickly and uncritically become a mainstream tool in our discipline. While such platforms have been evaluated on their aptness to generate high-quality data, surprisingly little has been said about the economic or political implications of their usage. Among other aspects, this article problematizes the “state of legal exception” in which crowdlabor markets operate, their tendency to rely on a pool of economically vulnerable workers and the asymmetrical employment relations they create. Rather than offer an easy solution, I aim to open up a conversation about this unique set of challenges in order to acknowledge and address them.

Hide All
Adda, Gilles, Mariani, Joseph J., Besacier, Laurent, and Gelas, Hadrien. 2013. “Economic and Ethical Background of Crowdsourcing for Speech.” In Crowdsourcing for Speech Processing, ed.Eskénazi, xine, Levow, Gina-Anne et al. . Chichester, UK: John Wiley & Sons, Ltd.
Ahler, Douglas J. 2014. “Self-Fulfilling Misperceptions of Public Polarization.” Journal of Politics 76(3): 607–20.
Arceneaux, Kevin. 2012. “Cognitive Biases and the Strength of Political Arguments.” American Journal of Political Science 56(2): 271–85.
Aytes, Ayhan. 2013. “Return of the Crowds: Mechacnical Turk and Neoliberal States of Exception.” In Digital Labor: The Internet as Playground and Factory, ed. Scholz, Trebor. New York: Routledge.
Benkler, Yochai. 2006. The Wealth of Networks: How Social Production Transforms Markets and Freedoms. New Haven, CT: Yale University Press.
Benoit, Kenneth, Conway, Drew, Lauderdale, Benjamin, Laver, Michael and Mikhaylov, Slava. In press. “Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data.” American Political Science Review.
Berinsky, Adam J., Huber, Gregory A., and Lenz, Gabriel S.. 2012. “Evaluating Online Labor Markets for Experimental Research: Amazon. Com’s Mechanical Turk.” Political Analysis 20(3): 351–68.
Brooks, Sarah M. 2014. “Insecure Democracy: Risk and Political Participation in Brazil.” Journal of Politics 76(4): 972–85.
Burgoon, Brian and Dekker, Fabian. 2010. “Flexible Employment, Economic Insecurity and Social Policy Preferences in Europe.” Journal of European Social Policy 20(2): 126–41.
Busarovs, Aleksejs. 2013. “Ethical Aspects of Crowdsourcing, or Is It a Modern Form of Exploitation.” International Journal of Economics & Business Administration I(1): 314.
Cherry, Miriam A. 2009. “Working for (Virtually) Minimum Wage: Applying the Fair Labor Standards Act in Cyberspace.” Alabama Law Review 60(5): 1077–110.
Clark, Andrew and Postel-Vinay, Fabien. 2009. “Job Security and Job Protection.” Oxford Economic Papers 61(2): 207–39.
Collins, Jane L. and Mayer, Victoria. 2010. Both Hands Tied: Welfare Reform and the Race to the Bottom in the Low-Wage Labor Market. Chicago, IL: University of Chicago Press.
Conway, Drew. 2013. “Methods for Collecting Large-Scale Non-Expert Text Coding.” SSRN Scholarly Paper ID 2260437 . Rochester, NY: Social Science Research Network.
Crawford, Jarret T. and Pilanski, Jane M.. 2014. “Political Intolerance, Right and Left.” Political Psychology 35(6): 841–51.
Crump, Matthew J. C., McDonnell, John V., and Gureckis, Todd M.. 2013. “Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research.” PLoS ONE 8(3): e57410.
Dickert, Neal and Grady, Christine. 1999. “What’s the Price of a Research Subject? Approaches to Payment for Research Participation.” New England Journal of Medicine 341(3): 198203.
Doherty, David. 2013. “To Whom Do People Think Representatives Should Respond: Their District or the Country?” Public Opinion Quarterly 77(1): 237–55.
Doherty, David, Dowling, Conor M., and Miller, Michael G.. 2014. “Does Time Heal All Wounds? Sex Scandals, Tax Evasion, and the Passage of Time.” PS: Political Science & Politics 47(2): 357–66.
Dow, Steven, Kulkarni, Anand, Bunge, Brie, Nguyen, Truc, Klemmer, Scott, and Hartmann, Björn. 2011. “Shepherding the Crowd: Managing and Providing Feedback to Crowd Workers.” In CHI ’11 Extended Abstracts on Human Factors in Computing Systems, 1669–74. CHI EA ’11. New York: ACM.
Fausey, Caitlin M. and Matlock, Teenie. 2011. “Can Grammar Win Elections?” Political Psychology 32(4): 563–74.
Felstiner, Alek. 2011. “Working the Crowd: Employment and Labor Law in the Crowdsourcing Industry.” Berkeley Journal of Employment and Labor Law 32(1): 143203.
Fort, Karën, Adda, Gilles, and Bretonnel Cohen, K.. 2011. “Amazon Mechanical Turk: Gold Mine or Coal Mine?” Computational Linguistics 37(2): 413–20.
Glynn, Adam N. 2013. “What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment.” Public Opinion Quarterly 77(S1): 159–72.
Graber, Mark A. and Graber, Abraham. 2013. “Internet-based Crowdsourcing and Research Ethics: The Case for IRB Review. Journal of Medical Ethics 39(2): 115–18.
Grimmer, Justin, Messing, Solomon, and Westwood, Sean J.. 2012. “How Words and Money Cultivate a Personal Vote: The Effect of Legislator Credit Claiming on Constituent Credit Allocation.” American Political Science Review 106(4): 703–19.
Grimmer, Justin and Stewart, Brandon M.. 2013. “Automatic Content Analysis Methods for Political Texts.” Political Analysis 21(3): 267297.
Guess, Andrew M. 2014. “Measure for Measure: An Experimental Test of Online Political Media Exposure.” Political Analysis 25(1): 5975.
Häusermann, Silja and Schwander, Hanna. 2012. “Varieties of Dualization: Identifying Insiders and Outsiders across Regimes,” in Emmenegger, Patrick, Häusermann, Silja, Palier, Bruno, Seeleib-Kaiser, Martin (eds.). The Age of Dualization. The Changing Face of Inequality in Deindustrializing Societies. Oxford and New York: Oxford University Press.
Hayes, Danny, Lawless, Jennifer L., and Baitinger, Gail. 2014. “Who Cares What They Wear? Media, Gender, and the Influence of Candidate Appearance.” Social Science Quarterly 95(5): 1194–212.
Hardt, Michael and Negri, Antonio. 2000. Empire. Cambridge, MA: Harvard University Press.
Healy, Andrew and Lenz, Gabriel S.. 2014. “Substituting the End for the Whole: Why Voters Respond Primarily to the Election-Year Economy.” American Journal of Political Science 58(1): 3147.
Hersh, Eitan D. and Schaffner, Brian F.. 2013. “Targeted Campaign Appeals and the Value of Ambiguity.” Journal of Politics 75(2): 520–34.
Hopkins, Daniel J. 2014. “The Upside of Accents: Language, Inter-Group Difference, and Attitudes toward Immigration.” British Journal of Political Science 45(3): 531–57.
Horton, John. 2010. “Why People Participate on Mechanical Turk, Now as a Mosaic Plot.” CrowdFlower. (accessed March 20, 2016).
Horton, John J., Rand, David G., and Zeckhauser, Richard J.. 2011. “The Online Laboratory: Conducting Experiments in a Real Labor Market.” Experimental Economics 14(3): 399425.
Howe, Jeff. 2006. “The Rise of Crowdsourcing.” Wired 14(6). Retrieved from
Huff, Connor and Tingley, Dustin. 2014. “Evaluating the Demographic Characteristics and Political Preferences of MTurk Survey Respondents.” Working paper, Harvard University.
Ipeirotis, Panagiotis G. 2010. “Demographics of Mechanical Turk.” NYU Working Paper No. CEDER-10-01. Available at SSRN,
Kalleberg, Arne L., Reskin, Barbara F. and Hudson, Ken. 2000. “Bad Jobs in America: Standard and Nonstandard Employment Relations and Job Quality in the United States.” American Sociological Review 65(2): 256–78.
Kam, Cindy D., Wilking, Jennifer R., and Zechmeister, Elizabeth J.. 2007. “Beyond the ‘Narrow Data Base’: Another Convenience Sample for Experimental Research.” Political Behavior 29(4): 415–40.
King, Desmond and Rueda, David. 2008. “Cheap Labor: The New Politics of ‘Bread and Roses’ in Industrial Democracies.” Perspectives on Politics 6(2): 279–97.
Kriner, Douglas L. and Shen, Francis X.. 2012. “How Citizens Respond to Combat Casualties: The Differential Impact of Local Casualties on Support for the War in Afghanistan.” Public Opinion Quarterly 76(4): 761–70.
Lease, Matthew, Hullman, Jessica, Bigham, Jeffrey P. et al. . 2013. “Mechanical Turk Is Not Anonymous.” SSRN paper. Available online
Leeper, Thomas. 2013. “Crowdsourcing with R and the MTurk API.” Political Methodologist 20(2): 27.
Le, John. 2010. “Amazon Mechanical Turk Survey.” Crowd Flower. (accessed March 20, 2016).
Lorey, Isabel. 2015. State of Insecurity: Government of the Precarious. London: Verso.
Marvit, Mshe Z. 2014. “How Crowdworkers Became the Ghosts in the Digital Machine.” The Nation, February 5.
Mason, Winter and Suri, Siddharth. 2012. “Conducting Behavioral Research on Amazon’s Mechanical Turk.” Behavioral Research Methods 44(1): 123.
McRobbie, Angela. 2011. “Reflections on Feminism, Immaterial Labour and the Post-Fordist Regime.” New Formations 70(1): 6076.
Montgomery, Jacob M. and Cutler, Josh. 2013. “Computerized Adaptive Testing for Public Opinion Surveys.” Political Analysis 21(2): 172–92.
Nelson, Barbara J. 1984. “Women’s Poverty and Women’s Citizenship: Some Political Consequences of Economic Marginality.” Signs: Journal of Women in Culture and Society 10(2): 209–32.
Ong, Aihwa. 2006. Neoliberalism as Exception: Mutations in Citizenship and Sovereignty. Durkam, NC: Duke University Press.
Paolacci, Gabriele, Chandler, Jesse and Ipeirotis, Panagiotis G.. 2010. “Running Experiments on Amazon Mechanical Turk.” Judgment and Decision Making 5(5): 411–19.
Perlin, Ross. 2012. Intern Nation: How to Earn Nothing and Learn Little in the Brave New Economy. London: Verso.
Ross, Andrew. 2013. “In Search of the Lost Paycheck.” In Digital Labor: T he Internet as Playground and Factory, ed. Scholz, Trebor. New York: Routledge.
Ross, Joel, Irani, Lilly, Silberman, M. Six, Zaldivar, Andrew and Tomlinson, Bill. 2010. “Who Are the Crowd workers? Shifting Demographics in Mechanical Turk.” In CHI ’10 Extended Abstracts on Human Factors in Computing Systems, 2863–72. CHI EA ’10. New York: ACM.
Rueda, David. 2007. Social Democracy Insider Out: Partisanship & Labor Market Policy in Industrialized Democracies. Oxford: Oxford University Press.
Ruder, Alex I. 2014. “Institutional Design and the Attribution of Presidential Control: Insulating the President from Blame.” Quarterly Journal of Political Science 9(3): 301–35.
Schmidt, Florian A. 2013a. “For a Few Dollars More—Class Action against Crowdsourcing.” Peer-Reviewed Journal About# BWPWAP, (Digital Aesthetics Research Center, Aarhus University in collaboration with transmediale 2013).
Schmidt, Florian A. 2013b. “The Good, The Bad and the Ugly: Why Crowdsourcing Needs Ethics.” In Proceedings of the Third International Conference on Cloud and Green Computing (CGC), 531–35.
Schneider, Nathan. 2015. “Intellectual Piecework.” Chronicle of Higher Education, February 16.
Scholz, Trebor. 2013. Digital Labor: The Internet as Playground and Factory. New York: Routledge.
Silberman, M., Ross, Joel, Irani, Lilly, and Tomlinson, Bill. 2010. “Sellers’ Problems in Human Computation Markets.” In Proceedings of the ACM SIGKDD Workshop on Human Computation 18–21. (accessed March 20, 2016).
Strolovitch, Dara Z. 2013. “Of Mancessions and Hecoveries: Race, Gender, and the Political Construction of Economic Crises and Recoveries.” Perspectives on Politics 11(1): 167–76.
Surowiecki, James. 2004. The Wisdom of Crowds: Why the Many are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. New York: Doubleday.
Tappscott, Don and Williams, Anthony D.. 2006. Wikinomics: How Mass Collaboration Changes Everything. New York: Portfolio.
Thibodeau, Paul, Peebles, Matthew M., Grodner, Daniel J., and Durgin, Frank H.. 2013. “The Wished-For Always Wins Until the Winner Was Inevitable All Along: Motivated Reasoning and Belief Bias Regulate Emotion during Elections.” Political Psychology 36(4): 431–48.
Tomz, Michael R. and Weeks, Jessica LP. 2013. “Public Opinion and the Democratic Peace.” American Political Science Review 107(4): 849–65.
U.S. Department of Health and Human Services. 2013. “Considerations and Recommendations Concerning Internet Research and Human Subjects Research Regulations, with Revisions.” Available online
Vosko, Leah F., Martha, MacDonald, and Iain, Campbell. 2009. Gender and the Contours of Precarious Employment. New York: Routledge University Press.
Wolfson, Stephen M. and Lease, Matthew. 2011. “Look before You Leap: Legal Pitfalls of Crowdsourcing.” Proceedings of the American Society for Information Science and Technology 48(1): 110.
Zittrain, Jonathan. 2008. “Ubiquitous Human Computing.” Philosophical Transactions Royal Society A 366(1881): 3813–21.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Perspectives on Politics
  • ISSN: 1537-5927
  • EISSN: 1541-0986
  • URL: /core/journals/perspectives-on-politics
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Altmetric attention score

Full text views

Total number of HTML views: 12
Total number of PDF views: 90 *
Loading metrics...

Abstract views

Total abstract views: 409 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 22nd May 2018. This data will be updated every 24 hours.