Hostname: page-component-8448b6f56d-42gr6 Total loading time: 0 Render date: 2024-04-23T23:27:40.626Z Has data issue: false hasContentIssue false
  • English
  • Français

The Statistical Analysis of Misreporting on Sensitive Survey Questions

Published online by Cambridge University Press:  11 April 2017

Gregory Eady
Gregory Eady*
Affiliation:
Department of Political Science, University of Toronto, 100 St. George Street, Toronto, Ontario, Canada,  M5S 3G3. Email: gregory.eady@gmail.com
*
*Email: gregory.eady@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

What explains why some survey respondents answer truthfully to a sensitive survey question, while others do not? This question is central to our understanding of a wide variety of attitudes, beliefs, and behaviors, but has remained difficult to investigate empirically due to the inherent problem of distinguishing those who are telling the truth from those who are misreporting. This article proposes a solution to this problem. It develops a method to model, within a multivariate regression context, whether survey respondents provide one response to a sensitive item in a list experiment, but answer otherwise when asked to reveal that belief openly in response to a direct question. As an empirical application, the method is applied to an original large-scale list experiment to investigate whether those on the ideological left are more likely to misreport their responses to questions about prejudice than those on the right. The method is implemented for researchers as open-source software.

Type
Articles
Information
Political Analysis , Volume 25 , Issue 2 , April 2017 , pp. 241 - 259
Copyright
Copyright © The Author(s) 2017. Published by Cambridge University Press on behalf of the Society for Political Methodology. 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Author’s note: I am grateful to Andy Guess, Barbora Racevičiūtė, Charles Breton, Cliff van der Linden, Greg Kerr, Matt Hoffmann, Matto Mildenberger, Peter Aronow, Peter Loewen, Sarah Zendel, Seva Gunitsky, and Uyen Hoang for their helpful suggestions and comments. I thank Félix-Antoine Fortin for facilitating access to and providing support for Calcul Québec’s large-scale computing network. Open-source software for the method proposed in this article, misreport, is available for download for the statistical software R from the Comprehensive R Archive Network (CRAN) athttps://cran.r-project.org/package=misreport. Replication files for this article can be found in Eady (2016).

Contributing Editor: Jonathan Katz

References

Ansolabehere, Stephen, and Hersh, Eitan. 2012. Validation: What big data reveal about survey misreporting and the real electorate. Political Analysis 20(4):437–459.Google Scholar
Anzia, Sarah F., and Berry, Christopher R.. 2011. The Jackie (and Jill) Robinson effect: Why do congresswomen outperform congressmen? American Journal of Political Science 55(3):478–493.Google Scholar
Aronow, Peter M., Coppock, Alexander, Crawford, Forrest W., and Green, Donald P.. 2015. Combining list experiment and direct question estimates of sensitive behavior prevalence. Journal of Survey Statistics and Methodology 3(1):43–66.Google Scholar
Berinsky, Adam J. 1999. The two faces of public opinion. American Journal of Political Science 43(4):1209–1230.Google Scholar
Berinsky, Adam J., Margolis, Michele F., and Sances, Michael W.. 2014. Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. American Journal of Political Science 58(3):739–753.Google Scholar
Blair, Graeme, and Imai, Kosuke. 2012. Statistical analysis of list experiments. Political Analysis 20(1):47–77.Google Scholar
Blair, Graeme, Imai, Kosuke, and Lyall, Jason. 2014. Comparing and combining list and endorsement experiments: Evidence from Afghanistan. American Political Science Review 58(4):1043–1063.Google Scholar
Blair, Graeme, Imai, Kosuke, and Zhou, Yang-Yang. 2015. Design and analysis of the randomized response technique. Journal of the American Statistical Association 110(511):1304–1319.Google Scholar
Bullock, John G., Green, Donald P., and Ha, Shang E.. 2010. Yes, But what’s the mechanism? (Don’t expect an easy answer). Journal of Personality and Social Psychology 98(4):550–558.Google Scholar
Bullock, Will, Imai, Kosuke, and Shapiro, Jacob N.. 2011. Statistical analysis of endorsement experiments: Measuring support for militant groups in pakistan. Political Analysis 19(4):363–384.Google Scholar
Cameron, David. 2015. Extremism: PM speech. Prime Minister’s Office, 10 Downing Street, July 20.Google Scholar
Chait, Jonathan. 2015. Not a very P.C. thing to say: How the language police are perverting liberalism. New York (Magazine), January 27.Google Scholar
Corstange, Daniel. 2009. Sensitive questions, truthful answers? Modeling the list experiment with LISTIT. Political Analysis 17(1):45–63.Google Scholar
Coyne, Andrew. 2015. Trudeau cabinet should be based on merit, not gender. The National Post, November 2.Google Scholar
Cross, Jessica Smith. 2015. Metro interviews Stephen Harper: PM talks terrorism, Syrian refugees and “elite political correctness on steroids.” Metro, September 29.Google Scholar
Dempster, A. P., Laird, N. M., and Rubin, D. B.. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B 39(1):1–38.Google Scholar
Eady, Gregory. 2016. Replication data for: The statistical analysis of misreporting on sensitive survey questions. doi:10.7910/DVN/PZKBUX, V1 [Version], Dec 20, 2016.Google Scholar
Fox, Richard L., and Lawless, Jennifer L.. 2014. Uncovering the origins of the gender gap in political ambition. American Political Science Review 108(3):499–519.Google Scholar
Friedersdorf, Conor. 2015. Campus activits weaponize “safe space.” The Atlanic, November 10.Google Scholar
Fullerton, Andrew S., Dixon, Jeffrey C., and Borch, Casey. 2007. Bringing registration into models of vote overreporting. Public Opinion Quarterly 71(4):649–660.Google Scholar
Gingerich, Daniel W. 2010. Understanding off-the-books politics: Conducting inference on the determinants of sensitive behavior with randomized response surveys. Political Analysis 18(3):349–380.Google Scholar
Gingerich, Daniel W., Oliveros, Virginia, Corbacho, Ana, and Ruiz-Vega, Mauricio. 2016. When to protect? Using the crosswise model to integrate protected and direct responses in surveys of sensitive behavior. Political Analysis 24(2):132–156.Google Scholar
Glynn, Adam N. 2013. What can we learn with statistical truth serum? Public Opinion Quarterly 77(S1):159–172.Google Scholar
Hanmer, Michael J., and Kalkan, Kerem Ozan. 2013. Behind the curve: Clarifying the best approach to calculating predicted probabilities and marginal effects from limited dependent variable models. American Journal of Political Science 57(1):263–277.Google Scholar
Hughes, Geoffrey. 2010. Political correctness: A history of sematics and culture . Chichester, UK: Wiley-Blackwell.Google Scholar
Imai, Kosuke. 2011. Multivariate regression analysis for the item count technique. Journal of the American Statistical Association 106(494):407–416.Google Scholar
Imai, Kosuke, Park, Bethany, and Greene, Kenneth F.. 2015. Using the predicted responses from list experiments as explanatory variables in regression models. Political Analysis 23(2):180–196.Google Scholar
Inter-parliamentary Union. (2015). Women in national parliaments. http://www.ipu.org/wmn-e/world.htm, Accessed December 20, 2015.Google Scholar
Jeydel, Alana, and Taylor, Andrew J.. 2003. Are women legislators less effective? Evidence from the US house in the 103rd-105th congress. Political Research Quarterly 56(1):19–27.Google Scholar
Karp, Jeffrey A., and Brockington, David. 2005. Social desirability and response validity: A comparative analysis of overreporting voter turnout in five countries. The Journal of Politics 67(3):825–840.Google Scholar
King, Gary, Tomz, Michael, and Wittenberg, Jason. 2000. Making the most of statistical analyses: Improving interpretation and presentation. American Journal of Political Science 44(2):341–355.Google Scholar
Kuklinski, James H., Cobb, Michael D., and Gilens, Martin. 1997. Racial attitudes and the “New South.” The Journal of Politics 59(2):323–349.Google Scholar
Kuran, Timur. 1998. Private truths, public lies: The social consequences of preference falsification . Cambridge, MA: Harvard University Press.Google Scholar
Lawless, Jennifer L., and Pearson, Kathryn. 2008. The primary reason for women’s underrepresentation? Reevaluating the conventional wisdom The Journal of Politics 70(1):67–82.Google Scholar
Lukianoff, Greg, and Haidt, Jonathan. 2015. The coddling of the American mind. In The Atlanic , https://www.theatlantic.com/magazine/archive/2015/09/the-coddling-of-the-american-mind/399356/.Google Scholar
Noelle-Neumann, Elisabeth. 1993. The spiral of silence: Public opinion—our social skin . Chicago, IL: The University of Chicago Press.Google Scholar
Obama, Barack. 2015. Remarks by the President at town hall on college access and affordability. The White House: Office of the Press Secretary, September 14.Google Scholar
Schlosser, Edward.2015. I’m a liberal professor, and my liberal students terrify me. Vox, June 3.Google Scholar
Silver, Brian D., Anderson, Barbara A., and Abramson, Paul R.. 1986. Who overreports voting? American Political Science Review 80(2):613–624.Google Scholar
Starr, Terrell Jermaine. 2015. There’s a good reason protests at the University of Missouri didn’t want the media around. The Washington Post, November 11.Google Scholar
Taub, Amanda. 2015. I was a liberal adjunct professor. My liberal students didn’t scare me at all. Vox, June 5.Google Scholar
The Telegraph. 2016. Migrants to the west must obey the rules. The Telegraph, Editorial, January 9.Google Scholar
Volden, Craig, and Wiseman, Alan E.. 2011. Breaking gridlock: The determinants of health policy change in congress. Journal of Health Politics, Policy and Law 36(2):227–264.Google Scholar
Supplementary material: File

Eady supplementary material

Eady supplementary material 1

Download Eady supplementary material(File)
File 344.3 KB