Hostname: page-component-6766d58669-vgfm9 Total loading time: 0 Render date: 2026-05-18T20:49:26.243Z Has data issue: false hasContentIssue false
  • English
  • Français

Statistical Analysis of List Experiments

Published online by Cambridge University Press:  04 January 2017

Graeme Blair  and
Kosuke Imai
Graeme Blair*
Affiliation:
Department of Politics, Princeton University, Princeton, NJ 08544
Kosuke Imai*
Affiliation:
Department of Politics, Princeton University, Princeton, NJ 08544
*
e-mail: gblair@princeton.edu (corresponding author)
e-mail: kimai@princeton.edu (corresponding author)
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the 'Save PDF' action button.

The validity of empirical research often relies upon the accuracy of self-reported behavior and beliefs. Yet eliciting truthful answers in surveys is challenging, especially when studying sensitive issues such as racial prejudice, corruption, and support for militant groups. List experiments have attracted much attention recently as a potential solution to this measurement problem. Many researchers, however, have used a simple difference-in-means estimator, which prevents the efficient examination of multivariate relationships between respondents' characteristics and their responses to sensitive items. Moreover, no systematic means exists to investigate the role of underlying assumptions. We fill these gaps by developing a set of new statistical methods for list experiments. We identify the commonly invoked assumptions, propose new multivariate regression estimators, and develop methods to detect and adjust for potential violations of key assumptions. For empirical illustration, we analyze list experiments concerning racial prejudice. Open-source software is made available to implement the proposed methodology.

Information

Type
Research Article
Information
Political Analysis , Volume 20 , Issue 1 , Winter 2012 , pp. 47 - 77
Copyright
Copyright © The Author 2012. Published by Oxford University Press on behalf of the Society for Political Methodology 

Footnotes

Edited by R. Michael Alvarez

Authors' note: Financial support from the National Science Foundation (SES-0849715) is acknowledged. All the proposed methods presented in this paper are implemented as part of the R package, “list: Statistical Methods for the Item Count Technique and List Experiment,” which is freely available for download at http://cran.r-project.org/package=list (Blair and Imai 2011a). The replication archive is available as Blair and Imai (2011b), and the Supplementary Materials are posted on the Political Analysis Web site. We thank Dan Corstange for providing his computer code, which we use in our simulation study, as well as for useful comments. Detailed comments from the editor and two anonymous reviewers significantly improved the presentation of this paper. Thanks also to Kate Baldwin, Neal Beck, Will Bullock, Stephen Chaudoin, Matthew Creighton, Michael Donnelly, Adam Glynn, Wenge Guo, John Londregan, Aila Matanock, Dustin Tingley, Teppei Yamamoto, and seminar participants at New York University, the New Jersey Institute of Technology, and Princeton University for helpful discussions.

References

Andrews, D. W. K., and Soares, G. 2010. Inference for parameters defined by moment inequalities using generalized moment selection. Econometrica 78: 119–57.Google Scholar
Berinsky, A. J. 2004. Silent voices: Public opinion and political participation in America. Princeton, NJ: Princeton University Press.Google Scholar
Biemer, P., and Brown, G. 2005. Model-based estimation of drug use prevalence using item count data. Journal of Official Statistics 21: 287–308.Google Scholar
Blair, G., and Imai, K. 2011a. list: Statistical Methods for the Item Count Technique and List Experiment. Comprehensive R Archive Network (CRAN). http://CRAN.R-project.org/package=list.Google Scholar
Blair, G., and Imai, K. 2011b. Replication data for: Statistical analysis of list experiments. hdl:1902.1/17040. The Dataverse Network.Google Scholar
Bullock, W., Imai, K., and Shapiro, J. N. 2011. Statistical analysis of endorsement experiments: Measuring support for militant groups in Pakistan. Political Analysis 19: 363–84.Google Scholar
Burden, B. C. 2000. Voter turnout and the national election studies. Political Analysis 8: 389–98.Google Scholar
Chaudhuri, A., and Christofides, T. C. 2007. Item count technique in estimating the proportion of people with a sensitive feature. Journal of Statistical Planning and Inference 137: 589–93.Google Scholar
Chen, X., Dempster, A. P., and Liu, J. S. 1994. Weighted finite population sampling to maximize entropy. Biometrika 81: 457–69.Google Scholar
Corstange, D. 2009. Sensitive questions, truthful answers? Modeling the list experiment with LISTIT. Political Analysis 17(1): 45–63.CrossRefGoogle Scholar
Coutts, E., and Jann, B. 2011. Sensitive questions in online surveys: Experimental results for the randomized response technique (RRT) and the unmatched count technique (UCT). Sociological Methods & Research 40: 169–93.Google Scholar
Dalton, D. R., Wimbush, J. C., and Daily, C. M. 1994. Using the unmatched count technique (UCT) to estimate base rates for sensitive behavior. Personnel Psychology 47: 817–29.Google Scholar
Dempster, A. P., Laird, N. M., and Rubin, D. B. 1977. Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society, Series B, Methodological 39(1): 1–38.Google Scholar
Droitcour, J., Caspar, R. A., Hubbard, M. L., Parsley, T. L., Visscher, W., and Ezzati, T. M. 1991. Measurement errors in surveys. In The Item Count Technique as a method of indirect questioning: A review of its development and a case study application, eds. Biemer, P. P., Groves, R. M., Lyberg, L. E., Mathiowetz, N. A., and Sudman, S., 185–210. New York: John Wiley & Sons.Google Scholar
Ehm, W. 1991. Binomial approximation to the Poisson binomial distribution. Statistics & Probability Letters 11: 7–16.CrossRefGoogle Scholar
Flavin, P., and Keane, M. 2009. How angry am I? Let me count the ways: Question format bias in list experiments. Technical Report, Department of Political Science, University of Notre Dame.Google Scholar
Gelman, A., Jakulin, A., Pittau, M. G., and Su, Y. 2008. A weakly informative default prior distribution for logistic and other regression models. Annals of Applied Statistics 2: 1360–83.Google Scholar
Gilens, M., Sniderman, P. M., and Kuklinski, J. H. 1998. Affirmative action and the politics of realignment. British Journal of Political Science 28: 159–83.Google Scholar
Gingerich, D. W. 2010. Understanding off-the-books politics: Conducting inference on the determinants of sensitive behavior with randomized response surveys. Political Analysis 18: 349–80.Google Scholar
Glynn, A. N. 2010. What can we learn with statistical truth serum? Design and analysis of the list experiment. Technical Report, Department of Government, Harvard University.Google Scholar
Gonzalez-Ocantos, E., Kiewet de Jonge, C., Melendez, C., Osorio, J., and Nickerson, D. W. 2010. Vote buying and social desirability bias: Experimental evidence from Nicaragua. Technical Report, Department of Political Science, University of Notre Dame.Google Scholar
Holbrook, A. L., and Krosnick, J. A. 2010. Social desirability bias in voter turnout reports: Tests using the item count technique. Public Opinion Quarterly 74 (1): 37–67.Google Scholar
Holland, P. W. 1986. Statistics and causal inference (with discussion). Journal of the American Statistical Association 81: 945–60.Google Scholar
Holland, B. S., and Copenhaver, M. D. 1987. An improved sequentially rejective Bonferroni test procedure. Biometrics 43: 417–23.Google Scholar
Imai, K. 2011. Multivariate regression analysis for the item count technique. Journal of the American Statistical Association 106: 407–16.Google Scholar
Imai, K., King, G., and Stuart, E. A. 2008. Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society, Series A (Statistics in Society) 171: 481–502.Google Scholar
Janus, A. L. 2010. The influence of social desirability pressures on expressed immigration attitudes. Social Science Quarterly 91: 928–46.Google Scholar
Kane, J. G., Craig, S. C., and Wald, K. D. 2004. Religion and presidential politics in Florida: A list experiment. Social Science Quarterly 85: 281–93.Google Scholar
Kudô, A. 1963. A multivariate analogue of the one-sided test. Biometrika 50: 403–18.Google Scholar
Kuklinski, J. H., Cobb, M. D., and Gilens, M. 1997a. Racial attitudes and the “New South.” Journal of Politics 59: 323–49.Google Scholar
Kuklinski, J. H., Sniderman, P. M., Knight, K., Piazza, T., Tetlock, P. E., Lawrence, G. R., and Mellers, B. 1997b. Racial prejudice and attitudes toward affirmative action. American Journal of Political Science 41: 402–19.CrossRefGoogle Scholar
LaBrie, J. W., and Earleywine, M. 2000. Sexual risk behaviors and alcohol: Higher base rates revealed using the unmatched-count technique. Journal of Sex Research 37: 321–26.CrossRefGoogle Scholar
Manski, C. F. 2007. Identification for Prediction and Decision. Cambridge, MA: Harvard University Press.Google Scholar
Miller, J. D. 1984. A new survey technique for studying deviant behavior. PhD diss, The George Washington University.Google Scholar
Newey, W. K., and McFadden, D. 1994. Large sample estimation and hypothesis testing. In Handbook of Econometrics, ed. Engle, R. F. and McFadden, D. L., volume IV, 2111–245. Amsterdam, The Netherlands: North Holland.Google Scholar
Perlman, M. D. 1969. One-sided testing problems in multivariate analysis. The Annals of Mathematical Statistics 40: 549–67.Google Scholar
Presser, S., and Stinson, L. 1998. Data collection mode and social desirability bias in self-reported religious attendance. American Sociological Review 63: 137–45.Google Scholar
Raghavarao, D., and Federer, W. T. 1979. Block total response as an alternative to the randomized response method in surveys. Journal of the Royal Statistical Society, Series B, Methodological 41 (1): 40–5.Google Scholar
Rayburn, N. R., Earleywine, M., and Davison, G. C. 2003. An investigation of base rates of anti-gay hate crimes using the unmatched-count technique. Journal of Aggression, Maltreatment & Trauma 6: 137–52.Google Scholar
Redlawsk, D. P., Tolbert, C. J., and Franko, W. 2010. Voters, emotions, and race in 2008: Obama as the first black president. Political Research Quarterly 63: 875–89.Google Scholar
Shapiro, A. 1985. Asymptotic distribution of test statistics in the analysis of moment structures under inequality constraints. Biometrika 72: 133–44.Google Scholar
Sniderman, P. M., and Carmines, E. G. 1997. Reaching Beyond Race. Cambridge, MA: Harvard University Press.Google Scholar
Sniderman, P. M., Tetlock, P. E., and Piazza, T. 1992. Codebook for the 1991 National Race and Politics Survey. Survey Research Center, Berkeley, California. http://sda.berkeley.edu/D3/Natlrace/Doc/nrac.htm.Google Scholar
Streb, M. J., Burrell, B., Frederick, B., and Genovese, M. A. 2008. Social desirability effects and support for a female american president. Public Opinion Quarterly 72(1): 76–89.Google Scholar
Tourangeau, R., and Yan, T. 2007. Sensitive questions in surveys. Psychological Bulletin 133: 859–83.Google Scholar
Tsuchiya, T. 2005. Domain estimators for the item count technique. Survey Methodology 31(1): 41–51.Google Scholar
Tsuchiya, T., Hirai, Y., and Ono, S. 2007. A study of the properties of the item count technique. Public Opinon Quarterly 71: 253–72.Google Scholar
Warner, S. L. 1965. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association 60(309): 63–9.Google Scholar
Wimbush, J. C., and Dalton, D. R. 1997. Base rate for employee theft: Convergence of multiple methods. Journal of Applied Psychology 82: 756–63.Google Scholar
Wolak, F. A. 1991. The local nature of hypothesis tests involving inequality constraints in nonlinear models. Econometrica 59: 981–95.Google Scholar
Zaller, J. 2002. The statistical power of election studies to detect media exposure effects in political campaigns. Election Studies 21: 297–329.Google Scholar
Supplementary material: PDF

Blair and Imai supplementary material

List Experiments

Download Blair and Imai supplementary material(PDF)
PDF 198.4 KB