Skip to main content
×
×
Home

Recruiting large online samples in the United States and India: Facebook, Mechanical Turk, and Qualtrics

  • Taylor C. Boas (a1), Dino P. Christenson (a1) and David M. Glick (a1)
Abstract

This article examines online recruitment via Facebook, Mechanical Turk (MTurk), and Qualtrics panels in India and the United States. It compares over 7300 respondents—1000 or more from each source and country—to nationally representative benchmarks in terms of demographics, political attitudes and knowledge, cooperation, and experimental replication. In the United States, MTurk offers the cheapest and fastest recruitment, Qualtrics is most demographically and politically representative, and Facebook facilitates targeted sampling. The India samples look much less like the population, though Facebook offers broad geographical coverage. We find online convenience samples often provide valid inferences into how partisanship moderates treatment effects. Yet they are typically unrepresentative on such political variables, which has implications for the external validity of sample average treatment effects.

Copyright
Corresponding author
*Corresponding author. Email: tboas@bu.edu
References
Hide All
Antin, J Shaw, A (2012) Social Desirability Bias and Self-Reports of Motivation: A Study of Amazon Mechanical Turk in the US and India. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2925–2934. ACM.
Antoun, C, Zhang, C, Conrad, FG Schober, MF (2016) Comparisons of Online Recruitment Strategies for Convenience Samples: Craigslist, Google AdWords, Facebook, and Amazon’s Mechanical Turk. Field Methods 28(3), 231246.
Arceneaux, K (2012) Cognitive Biases and the Strength of Political Arguments. American Journal of Political Science 56(2), 271285.
Ausderan, J (2014) How Naming and Shaming Affects Human Rights Perceptions in the Shamed Country. Journal of Peace Research 51(1), 8195.
Azam, M, Chin, A Prakash, N (2013) The Returns to English-Language Skills in India. Economic Development and Cultural Change 61(2), 335367.
Benjamini, Y Hochberg, Y (1995) Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological) 57(1), 289300.
Berinsky, AJ, Huber, GA Lenz, GS (2012) Evaluating Online Labor Markets for Experimental Research: Amazon.com’s Mechanical Turk. Political Analysis 20(3), 351368.
Berinsky, AJ, Margolis, MF Sances, MW (2014) Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys. American Journal of Political Science 58(3), 739753.
Boas, TC (2016) Pastors for Pinochet: Authoritarian Stereotypes and Voting for Evangelicals in Chile. Journal of Experimental Political Science 3(2), 197205.
Charnysh, V, Lucas, C Singh, P (2015) The Ties That Bind: National Identity Salience and Pro-Social Behavior Toward the Ethnic Other. Comparative Political Studies 48(3), 267300.
Christenson, DP Glick, DM (2013) Crowdsourcing Panel Studies and Real-Time Experiments in MTurk. The Political Methodologist 20(2), 2733.
Christenson, DP Glick, DM (2015a) Chief Justice Roberts’s Health Care Decision Disrobed: The Microfoundations of the Supreme Court’s Legitimacy. American Journal of Political Science 59(2), 403418.
Christenson, DP Glick, DM (2015b) Issue-Specific Opinion Change: The Supreme Court and Health Care Reform. Public Opinion Quarterly 79(4), 881905.
Clifford, S, Jewell, RM Waggoner, PD (2015) Are Samples Drawn from Mechanical Turk Valid for Research on Political Ideology? Research & Politics 2(4), 19.
Dietrich, S Winters, MS (2015) Foreign Aid and Government Legitimacy. Journal of Experimental Political Science 2(2), 164171.
Erlich, A, Jung, DF, Long, JD McIntosh, C (2017) The Double-Edged Sword of Mobilizing Citizens Via Mobile Phone in Developing Countries. San Diego, CA. Manuscript, McGill University/Emory University/University of Washington/University of California.
Gay, C, Hochschild, J White, A (2016) Americans’ Belief in Linked Fate: Does the Measure Capture the Concept? The Journal of Race, Ethnicity, and Politics 1(1), 117144.
Grimmer, J, Messing, S Westwood, SJ (2012) How Words and Money Cultivate a Personal Vote: The Effect of Legislator Credit Claiming on Constituent Credit Allocation. American Political Science Review 106(4), 703719.
Haselswerdt, J Bartels, BL (2015) Public Opinion, Policy Tools, and the Status Quo Evidence from a Survey Experiment. Political Research Quarterly 68(3), 607621.
Huber, GA, Hill, SJ Lenz, GS (2012) Sources of Bias in Retrospective Decision Making: Experimental Evidence on Voters Limitations in Controlling Incumbents. American Political Science Review 106(4), 720741.
Huff, C Tingley, D (2015) ‘Who are these People?’ Evaluating the Demographic Characteristics and Political Preferences of MTurk Survey Respondents. Research & Politics 2(3), 112.
Kapur, D, Sircar, N Vaishnav, M (2014) All in the surname. Times of India, March 23. Available at https://bit.ly/2vaNenQ, accessed 26 July 2018.
Kriner, DL Shen, FX (2016) Conscription, Inequality, and Partisan Support for War. Journal of Conflict Resolution 60(8), 14191445.
Krupnikov, Y Levine, AS (2014) Cross-Sample Comparisons and External Validity. Journal of Experimental Political Science 1(1), 5980.
Leeper, TJ Mullinix, KJ (2015) What If You Had Done Things Differently? Testing the Generalizability of Framing Effects with Parallel Experiments. Manuscript, Aarhus University, Aarhus, Denmark/Northwestern University, Evanston, IL, https://dl.dropboxusercontent.com/u/414906/ParallelExperimentsOnFraming.pdf, accessed 6 July 2015.
Levay, KE, Freese, J Druckman, JN (2016) The Demographic and Political Composition of Mechanical Turk Samples. SAGE Open 6(1), 117.
Litman, L, Robinson, J Rosenzweig, C (2014) The Relationship Between Motivation, Monetary Compensation, and Data Quality Among US-and India-based Workers on Mechanical Turk. Behavior Research Methods 47(2), 519528.
Marder, J Fritz, M (2015) The Internet’s Hidden Science Factory. PBS NewsHour, http://www.pbs.org/newshour/updates/inside-amazons-hidden-science-factory/, accessed 26 July 2018.
Mullinix, KJ, Leeper, TJ, Druckman, JN Freese, J (2015) The Generalizability of Survey Experiments. Journal of Experimental Political Science 2(2), 109138.
Necka, EA, Cacioppo, S, Norman, GJ Cacioppo, JT (2016) Measuring the Prevalence of Problematic Respondent Behaviors Among MTurk, Campus, and Community Participants. PloS One 11(6), 119.
Paolacci, G, Chandler, J Ipeirotis, PG (2010) Running Experiments on Amazon Mechanical Turk. Judgment and Decision Making 5(5), 411419.
Peer, E, Vosgerau, J Acquisti, A (2014) Reputation as a Sufficient Condition for Data Quality on Amazon Mechanical Turk. Behavior Research Methods 46(4), 10231031.
Reed, MN Kapur, D (2015) The Love for Sons and Appropriate Attire. The Hindu, January 26. Available at https://bit.ly/2JXjDnb, accessed 26 July 2018.
Samuels, D Zucco, C (2013) Using Facebook as a Subject Recruitment Tool for Survey-Experimental Research. Working Paper, Social Science Research Network, https://ssrn.com/abstract=2101458, accessed 26 July 2018.
Samuels, D Zucco, C (2014) The Power of Partisanship in Brazil: Evidence from Survey Experiments. American Journal of Political Science 58(1), 212225.
Santoso, LP, Stein, R Stevenson, R (2016) Survey Experiments with Google Consumer Surveys: Promise and Pitfalls for Academic Research in Social Science. Political Analysis 24(3), 356373.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Science Research and Methods
  • ISSN: 2049-8470
  • EISSN: 2049-8489
  • URL: /core/journals/political-science-research-and-methods
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
Type Description Title
PDF
Supplementary materials

Boas et al. supplementary material
Appendix

 PDF (3.4 MB)
3.4 MB
UNKNOWN
Supplementary materials

Boas et al. Dataset
Dataset

 Unknown

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 2
Total number of PDF views: 25 *
Loading metrics...

Abstract views

Total abstract views: 133 *
Loading metrics...

* Views captured on Cambridge Core between 8th August 2018 - 18th August 2018. This data will be updated every 24 hours.