Hostname: page-component-6766d58669-r8qmj Total loading time: 0 Render date: 2026-05-20T21:20:19.164Z Has data issue: false hasContentIssue false
  • English
  • Français

List Experiments with Measurement Error

Published online by Cambridge University Press:  20 May 2019

Graeme Blair ,
Winston Chou  and
Kosuke Imai Open the ORCID record for Kosuke Imai [Opens in a new window]
Graeme Blair
Affiliation:
Assistant Professor of Political Science, UCLA, USA. Email: graeme.blair@ucla.edu, URL: https://graemeblair.com
Winston Chou
Affiliation:
Ph.D. Candidate, Department of Politics, Princeton University, Princeton NJ 08544, USA. Email: wchou@princeton.edu, URL: http://princeton.edu/∼wchou
Kosuke Imai*
Affiliation:
Professor of Government and of Statistics, Harvard University, 1737 Cambridge Street, Institute for Quantitative Social Science, Cambridge MA 02138, USA. Email: Imai@Harvard.Edu, URL: https://imai.fas.harvard.edu
*
*Email: Imai@Harvard.Edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Measurement error threatens the validity of survey research, especially when studying sensitive questions. Although list experiments can help discourage deliberate misreporting, they may also suffer from nonstrategic measurement error due to flawed implementation and respondents’ inattention. Such error runs against the assumptions of the standard maximum likelihood regression (MLreg) estimator for list experiments and can result in misleading inferences, especially when the underlying sensitive trait is rare. We address this problem by providing new tools for diagnosing and mitigating measurement error in list experiments. First, we demonstrate that the nonlinear least squares regression (NLSreg) estimator proposed in Imai (2011) is robust to nonstrategic measurement error. Second, we offer a general model misspecification test to gauge the divergence of the MLreg and NLSreg estimates. Third, we show how to model measurement error directly, proposing new estimators that preserve the statistical efficiency of MLreg while improving robustness. Last, we revisit empirical studies shown to exhibit nonstrategic measurement error, and demonstrate that our tools readily diagnose and mitigate the bias. We conclude this article with a number of practical recommendations for applied researchers. The proposed methods are implemented through an open-source software package.

Keywords

auxiliary informationindirect questioningitem count techniquemisspecification testsensitive survey questionsunmatched count technique

Information

Type
Articles
Information
Political Analysis , Volume 27 , Issue 4 , October 2019 , pp. 455 - 480
Copyright
Copyright © The Author(s) 2019. Published by Cambridge University Press on behalf of the Society for Political Methodology. 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Contributing Editor: Jeff Gill

Authors’ note: All the proposed methods presented in this paper are implemented as part of the R package, list: Statistical Methods for the Item Count Technique and List Experiment, which is freely available for download at http://cran.r-project.org/package=list (Blair, Chou, and Imai 2017). The replication materials are available as Blair, Chou, and Imai (2019).

References

Ahlquist, John S. 2018. “List experiment design, non-strategic respondent error, and item count technique estimators.” Political Analysis 26:34–53.Google Scholar
Ahlquist, John S., Mayer, Kenneth R., and Jackman, Simon. 2014. “Alien abduction and voter impersonation in the 2012 U.S. General Election: Evidence from a survey list experiment.” Election Law Journal 13:460–475.Google Scholar
Aronow, Peter M., Coppock, Alexander, Crawford, Forrest W., and Green, Donald P.. 2015. “Combining list experiment and direct question estimates of sensitive behavior prevalence.” Journal of Survey Statistics and Methodology 3:43–66.Google Scholar
Blair, Graeme, and Imai, Kosuke. 2012. “Statistical analysis of list experiments.” Political Analysis 20:47–77.Google Scholar
Blair, Graeme, Imai, Kosuke, and Lyall, Jason. 2014. “Comparing and combining list and endorsement experiments: Evidence from Afghanistan.” American Journal of Political Science 58:1043–1063.Google Scholar
Blair, Graeme, Imai, Kosuke, and Zhou, Yang-Yang. 2015. “Design and analysis of randomized response technique.” Journal of the American Statistical Association 110:1304–1319.Google Scholar
Blair, Graeme, Chou, Winston, and Imai, Kosuke. 2017 list: Statistical methods for the item count technique and list experiment. Available at the Comprehensive R Archive Network (CRAN). https://CRAN.R-project.org/package=list.Google Scholar
Blair, Graeme, Chou, Winston, and Imai, Kosuke. 2019 “Replication data for: List experiments with measurement error.” https://doi.org/10.7910/DVN/L3GWNP, Harvard Dataverse.Google Scholar
Bullock, Will, Imai, Kosuke, and Shapiro, Jacob N.. 2011. “Statistical analysis of endorsement experiments: Measuring support for militant groups in Pakistan.” Political Analysis 19:363–384.Google Scholar
Carroll, Raymond J., Ruppert, David, Stefanski, Leonard A., and Crainiceanu, Ciprian M.. 2006. Measurement error in nonlinear models: A modern perspective . 2nd ed. London: Chapman & Hall.Google Scholar
Chou, Winston. 2018. Lying on surveys: Methods for list experiments with direct questioning. Technical report, Princeton University.Google Scholar
Chou, Winston, Imai, Kosuke, and Rosenfeld, Bryn. 2017. “Sensitive survey questions with auxiliary information.” Sociological Methods & Research , doi:10/1177/0049124117729711.Google Scholar
Corstange, Daniel. 2009. “Sensitive questions, truthful answers?: Modeling the list experiment with LISTIT.” Political Analysis 17:45–63.Google Scholar
Delgado, M. Kit, Wanner, Kathryn J., and McDonald, Catherine. 2016. “Adolescent cellphone use while driving: An overview of the literature and promising future directions for prevention.” Media and Communication 4:79–89.Google Scholar
Dempster, Arthur P., Laird, Nan M., and Rubin, Donald B.. 1977. “Maximum likelihood from incomplete data via the EM algorithm (with discussion).” Journal of the Royal Statistical Society, Series B, Methodological 39:1–37.Google Scholar
Gelman, Andrew, Jakulin, Aleks, Pittau, Maria Grazia, and Su, Yu-Sung. 2008. “A weakly informative default prior distribution for logistic and other regression models.” Annals of Applied Statistics 2:1360–1383.Google Scholar
Gingerich, Daniel W. 2010. “Understanding off-the-books politics: Conducting inference on the determinants of sensitive behavior with randomized response surveys.” Political Analysis 18:349–380.Google Scholar
Glynn, Adam N. 2013. “What can we learn with statistical truth serum?: Design and analysis of the list experiment.” Public Opinion Quarterly 77:159–172.Google Scholar
Hausman, Jerry A. 1978. “Specification tests in econometrics.” Econometrica 46:1251–1271.Google Scholar
Imai, Kosuke. 2011. “Multivariate regression analysis for the item count technique.” Journal of the American Statistical Association 106:407–416.Google Scholar
King, Gary, and Zeng, Langche. 2001. “Logistic regression in rare events data.” Political Analysis 9:137–163.Google Scholar
Lyall, Jason, Blair, Graeme, and Imai, Kosuke. 2013. “Explaining support for combatants during wartime: A survey experiment in Afghanistan.” American Political Science Review 107:679–705.Google Scholar
Miller, J. D.1984 The item-count/paired lists technique: An indirect method of surveying deviant behavior. PhD thesis, George Washington University.Google Scholar
Rosenfeld, Bryn, Imai, Kosuke, and Shapiro, Jacob. 2016. “An empirical validation study of popular survey methodologies for sensitive questions.” American Journal of Political Science 60:783–802.Google Scholar
Schreiber, Sven. 2008. “The Hausman test statistic can be negative, even asymptotically.” Jahrbücher für Nationalökonomie und Statistik 228:394–405.Google Scholar
Sobel, Richard. 2009. “Voter-ID Issues in Politics and Political Science: Editor’s Introduction.” PS: Political Science & Politics 42:81–85.Google Scholar
Supplementary material: PDF

Blair et al. supplementary material

Blair et al. supplementary material 1

Download Blair et al. supplementary material(PDF)
PDF 179.5 KB