Skip to main content

Investigator Characteristics and Respondent Behavior in Online Surveys

  • Ariel White (a1), Anton Strezhnev (a2), Christopher Lucas (a2), Dominika Kruszewska (a2) and Connor Huff (a2)...

Prior research demonstrates that responses to surveys can vary depending on the race, gender, or ethnicity of the investigator asking the question. We build upon this research by empirically testing how information about researcher identity in online surveys affects subject responses. We do so by conducting an experiment on Amazon’s Mechanical Turk in which we vary the name of the researcher in the advertisement for the experiment and on the informed consent page in order to cue different racial and gender identities. We fail to reject the null hypothesis that there is no difference in how respondents answer questions when assigned to a putatively black/white or male/female researcher.

Hide All
Adida, Claire L., Ferree, Karen E., Posner, Daniel N., and Robinson, Amanda Lea. 2016. “Who’s Asking? Interviewer Coethnicity Effects in African Survey Data.” Comparative Political Studies 49 (12): 16301660.
Benjamini, Yoav and Yekutieli, Daniel. 2001. “The Control of the False Discovery Rate in Multiple Testing Under Dependency.” Annals of Statistics 29 (4): 11651188.
Benjamini, Yoav and Yekutieli, Daniel. 2005. “False Discovery Rate–Adjusted Multiple Confidence Intervals for Selected Parameters.” Journal of the American Statistical Association 100 (469): 7181.
Benjamini, Yoav and Hochberg, Yosef. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society. Series B (Methodological) 57 (1): 289300.
Berinsky, Adam J., Margolis, Michele F., and Sances, Michael W.. 2014. “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys.” American Journal of Political Science 58 (3): 739753.
Berrens, Robert P., Bohara, Alok K., Jenkins-Smith, Hank, Silva, Carol, and Weimer, David L.. 2003. “The Advent of Internet Surveys for Political Research: A Comparison of Telephone and Internet Samples.” Political Analysis 11 (1): 122.
Bertrand, Marianne and Mullainathan, Sendhil. 2004. “Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination.” American Economic Review 94 (4): 9911013.
Bush, Sarah and Prather, Lauren. 2017. “How Electronic Devices in Face-to-Face Interviews Change Survey Behavior: Evidence From a Developing Country.” (, accessed December 29, 2017.
Butler, Daniel M. and Broockman, David E.. 2011. “Do Politicians Racially Discriminate Against Constituents? A Field Experiment on State Legislators.” American Journal of Political Science 55 (3): 463477.
Butler, Daniel M. and Homola, Jonathan. 2017. “An Empirical Justification for the Use of Racially Distinctive Names to Signal Race in Experiments.” Political Analysis 25 (1): 122130.
Chandler, Jesse, Mueller, Pam, and Paolacci, Gabriele. 2014. “Nonnaïveté Among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers.” Behavior Research Methods 46 (1): 112130.
Charnysh, Volha, Lucas, Christopher, and Singh, Prerna. 2014. “The Ties That Bind: National Identity Salience and Pro-Social Behavior Toward the Ethnic Other.” Comparative Political Studies 48 (3): 267300.
Clifford, Scott, Jewell, Ryan M., and Waggoner, Philip D.. 2015. “Are Samples Drawn from Mechanical Turk Valid for Research on Political Ideology?Research & Politics 2 (4): 19.
Cotter, Patrick R., Cohen, Jeffrey, and Coulter, Philip B.. 1982. “Race-of-Interviewer Effects in Telephone Interviews.” Public Opinion Quarterly 46 (2): 278284.
Davis, Darren W. 1997. “The Direction of Race of Interviewer Effects Among African-Americans: Donning the Black Mask.” American Journal of Political Science 41 (1): 309322.
Davis, Darren W. and Silver, Brian D.. 2003. “Stereotype Threat and Race of Interviewer Effects in a Survey on Political Knowledge.” American Journal of Political Science 47 (1): 3345.
Edelman, Benjamin, Luca, Michael, and Svirsky, Dan. 2017. “Racial Discrimination in the Sharing Economy: Evidence from a Field Experiment.” American Economic Journal: Applied Economics 9 (2): 122.
Einstein, Katherine and Glick, David M.. 2017. “Does Race Affect Access to Government Services? An Experiment Exploring Street-Level Bureaucrats and Access to Public Housing.” American Journal of Political Science 61 (1): 100116.
Fryer, Roland G. Jr., and Levitt, Steven J.. 2004. “The Causes and Consequences of Distinctively Black Names.” The Quarterly Journal of Economics 119 (3): 767806.
Grewal, Ini and Ritchie, Jane. 2006. “Ethnic and Language Matching of the Researcher and the Research Group During Design, Fieldwork and Analysis.” In Health and Social Research in Multiethnic Societies, ed. Nazroo, James Y.. Oxon: Routledge, 6581.
Groppe, David M., Urbach, Thomas P., and Kutas, Marta. 2011. “Mass Univariate Analysis of Event-Related Brain Potentials/Fields II: Simulation Studies.” Psychophysiology 48 (12): 17261737.
Hatchett, Shirley and Schuman, Howard. 1975. “White Respondents and Race-of-Interviewer Effects.” The Public Opinion Quarterly 39 (4): 523528.
Healy, Andrew and Lenz, Gabriel S.. 2014. “Substituting the End for the Whole: Why Voters Respond Primarily to the Election-Year Economy.” American Journal of Political Science 58 (1): 3147.
Huddy, Leonie, Billig, Joshua, Bracciodieta, John, Moynihan, Patrick J., and Pugliani, Patricia. 1997. “The Effect of Interviewer Gender on the Survey Response.” Political Behavior 19 (3): 197220.
Huff, Connor and Tingley, Dustin. 2015. “Who Are These People? Evaluating the Demographic Characteristics and Political Preferences of MTurk Survey Respondents.” Research & Politics 2 (3): 112.
Kinder, Donald R. and Sanders, Lynn M.. 1996. Divided by Color: Racial Politics and Democratic Ideals. Chicago: University of Chicago Press.
Kriner, Douglas L. and Shen, Francis X.. 2013. “Reassessing American Casualty Sensitivity: The Mediating Influence of Inequality.” Journal of Conflict Resolution 58 (7): 11741201.
Krupnikov, Yanna and Levine, Adam Seth. 2014. “Cross-Sample Comparisons and External Validity.” Journal of Experimental Political Science 1 (1): 5980.
Leeper, Thomas. 2013. “Crowdsourcing with R and the MTurk API.” The Political Methodologist 20: 27.
Leeper, Thomas J. 2015. MTurkR: Access to Amazon Mechanical Turk Requester API via R. R package version (, accessed December 29, 2017.
Leeper, Thomas and Thorson, Emily. 2015. “Minimal Sponsorship-Induced Bias in Web Survey Data.” (, accessed December 29, 2017.
Levay, Kevin E., Freese, Jeremy, and Druckman, James N.. 2016. “The Demographic and Political Composition of Mechanical Turk Samples.” SAGE Open 6 (1): 117.
Mullinix, Kevin J., Leeper, Thomas J., Druckman, James N., and Freese, Jeremy. 2015. “The Generalizability of Survey Experiments.” Journal of Experimental Political Science 2 (2): 109138.
Mummolo, Jonathan and Peterson, Erik. 2017. “Demand Effects in Survey Experiments: An Empirical Assessment.” (, accessed December 29, 2017.
Reese, Stephen D., Danielson, Wayne A., Shoemaker, Pamela J., Chang, Tsan-Kuo, and Hsu, Huei-Ling. 1986. “Ethnicity-of-Interviewer Effects Among Mexican-Americans and Anglos.” Public Opinion Quarterly 50 (4): 563572.
Schaeffer, Nora Cate, Dykema, Jennifer, and Maynard, Douglas W.. 2010. “Interviewers and Interviewing.” In Handbook of Survey Research, eds. Marsden, Peter V. and Wright, James D.. Bingley: Emerald, 437470.
Survey Research Center. 2010. Guidelines for Best Practice in Cross-Cultural Surveys. Ann Arbor, MI: Survey Research Center, Institute for Social Research, University of Michigan. (, accessed December 29, 2017.
Tomz, Michael and Weeks, Jessica. 2013. “Public Opinion and the Democratic Peace.” American Political Science Review 107 (4): 849865.
Wallace, Geoffrey P. R. 2013. “International Law and Public Attitudes Toward Torture: An Experimental Study.” International Organization 67 (1): 105140.
White, Ariel R., Nathan, Noah L. and Faller, Julie K.. 2015. “What Do I Need to Vote? Bureaucratic Discretion and Discrimination by Local Election Officials.” American Political Science Review 109 (1): 129142.
Word, David L., Coleman, Charles D., Nunziata, Robert and Kominski, Robert. 2008. “Demographic Aspects of Surnames from Census 2000.” (, accessed December 29, 2017.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Experimental Political Science
  • ISSN: 2052-2630
  • EISSN: 2052-2649
  • URL: /core/journals/journal-of-experimental-political-science
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Type Description Title
Supplementary materials

White et al. Dataset

Supplementary materials

White et al. supplementary material
Online Appendix

 PDF (275 KB)
275 KB


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed