Skip to main content Accessibility help
×
Home
Hostname: page-component-8bbf57454-k9dpc Total loading time: 0.249 Render date: 2022-01-24T18:34:11.204Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": true, "newCiteModal": false, "newCitedByModal": true, "newEcommerce": true, "newUsageEvents": true }

Statistical Analysis of List Experiments

Published online by Cambridge University Press:  04 January 2017

Graeme Blair*
Affiliation:
Department of Politics, Princeton University, Princeton, NJ 08544
Kosuke Imai*
Affiliation:
Department of Politics, Princeton University, Princeton, NJ 08544
*
e-mail: gblair@princeton.edu (corresponding author)
e-mail: kimai@princeton.edu (corresponding author)
Rights & Permissions[Opens in a new window]

Abstract

HTML view is not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

The validity of empirical research often relies upon the accuracy of self-reported behavior and beliefs. Yet eliciting truthful answers in surveys is challenging, especially when studying sensitive issues such as racial prejudice, corruption, and support for militant groups. List experiments have attracted much attention recently as a potential solution to this measurement problem. Many researchers, however, have used a simple difference-in-means estimator, which prevents the efficient examination of multivariate relationships between respondents' characteristics and their responses to sensitive items. Moreover, no systematic means exists to investigate the role of underlying assumptions. We fill these gaps by developing a set of new statistical methods for list experiments. We identify the commonly invoked assumptions, propose new multivariate regression estimators, and develop methods to detect and adjust for potential violations of key assumptions. For empirical illustration, we analyze list experiments concerning racial prejudice. Open-source software is made available to implement the proposed methodology.

Type
Research Article
Copyright
Copyright © The Author 2012. Published by Oxford University Press on behalf of the Society for Political Methodology 

Footnotes

Edited by R. Michael Alvarez

Authors' note: Financial support from the National Science Foundation (SES-0849715) is acknowledged. All the proposed methods presented in this paper are implemented as part of the R package, “list: Statistical Methods for the Item Count Technique and List Experiment,” which is freely available for download at http://cran.r-project.org/package=list (Blair and Imai 2011a). The replication archive is available as Blair and Imai (2011b), and the Supplementary Materials are posted on the Political Analysis Web site. We thank Dan Corstange for providing his computer code, which we use in our simulation study, as well as for useful comments. Detailed comments from the editor and two anonymous reviewers significantly improved the presentation of this paper. Thanks also to Kate Baldwin, Neal Beck, Will Bullock, Stephen Chaudoin, Matthew Creighton, Michael Donnelly, Adam Glynn, Wenge Guo, John Londregan, Aila Matanock, Dustin Tingley, Teppei Yamamoto, and seminar participants at New York University, the New Jersey Institute of Technology, and Princeton University for helpful discussions.

References

Andrews, D. W. K., and Soares, G. 2010. Inference for parameters defined by moment inequalities using generalized moment selection. Econometrica 78: 119–57.Google Scholar
Berinsky, A. J. 2004. Silent voices: Public opinion and political participation in America. Princeton, NJ: Princeton University Press.Google Scholar
Biemer, P., and Brown, G. 2005. Model-based estimation of drug use prevalence using item count data. Journal of Official Statistics 21: 287308.Google Scholar
Blair, G., and Imai, K. 2011a. list: Statistical Methods for the Item Count Technique and List Experiment. Comprehensive R Archive Network (CRAN). http://CRAN.R-project.org/package=list.Google Scholar
Blair, G., and Imai, K. 2011b. Replication data for: Statistical analysis of list experiments. hdl:1902.1/17040. The Dataverse Network.Google Scholar
Bullock, W., Imai, K., and Shapiro, J. N. 2011. Statistical analysis of endorsement experiments: Measuring support for militant groups in Pakistan. Political Analysis 19: 363–84.CrossRefGoogle Scholar
Burden, B. C. 2000. Voter turnout and the national election studies. Political Analysis 8: 389–98.CrossRefGoogle Scholar
Chaudhuri, A., and Christofides, T. C. 2007. Item count technique in estimating the proportion of people with a sensitive feature. Journal of Statistical Planning and Inference 137: 589–93.CrossRefGoogle Scholar
Chen, X., Dempster, A. P., and Liu, J. S. 1994. Weighted finite population sampling to maximize entropy. Biometrika 81: 457–69.CrossRefGoogle Scholar
Corstange, D. 2009. Sensitive questions, truthful answers? Modeling the list experiment with LISTIT. Political Analysis 17(1): 4563.CrossRefGoogle Scholar
Coutts, E., and Jann, B. 2011. Sensitive questions in online surveys: Experimental results for the randomized response technique (RRT) and the unmatched count technique (UCT). Sociological Methods & Research 40: 169–93.CrossRefGoogle Scholar
Dalton, D. R., Wimbush, J. C., and Daily, C. M. 1994. Using the unmatched count technique (UCT) to estimate base rates for sensitive behavior. Personnel Psychology 47: 817–29.CrossRefGoogle Scholar
Dempster, A. P., Laird, N. M., and Rubin, D. B. 1977. Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society, Series B, Methodological 39(1): 138.Google Scholar
Droitcour, J., Caspar, R. A., Hubbard, M. L., Parsley, T. L., Visscher, W., and Ezzati, T. M. 1991. Measurement errors in surveys. In The Item Count Technique as a method of indirect questioning: A review of its development and a case study application, eds. Biemer, P. P., Groves, R. M., Lyberg, L. E., Mathiowetz, N. A., and Sudman, S., 185210. New York: John Wiley & Sons.Google Scholar
Ehm, W. 1991. Binomial approximation to the Poisson binomial distribution. Statistics & Probability Letters 11: 716.CrossRefGoogle Scholar
Flavin, P., and Keane, M. 2009. How angry am I? Let me count the ways: Question format bias in list experiments. Technical Report, Department of Political Science, University of Notre Dame.Google Scholar
Gelman, A., Jakulin, A., Pittau, M. G., and Su, Y. 2008. A weakly informative default prior distribution for logistic and other regression models. Annals of Applied Statistics 2: 1360–83.CrossRefGoogle Scholar
Gilens, M., Sniderman, P. M., and Kuklinski, J. H. 1998. Affirmative action and the politics of realignment. British Journal of Political Science 28: 159–83.CrossRefGoogle Scholar
Gingerich, D. W. 2010. Understanding off-the-books politics: Conducting inference on the determinants of sensitive behavior with randomized response surveys. Political Analysis 18: 349–80.CrossRefGoogle Scholar
Glynn, A. N. 2010. What can we learn with statistical truth serum? Design and analysis of the list experiment. Technical Report, Department of Government, Harvard University.Google Scholar
Gonzalez-Ocantos, E., Kiewet de Jonge, C., Melendez, C., Osorio, J., and Nickerson, D. W. 2010. Vote buying and social desirability bias: Experimental evidence from Nicaragua. Technical Report, Department of Political Science, University of Notre Dame.Google Scholar
Holbrook, A. L., and Krosnick, J. A. 2010. Social desirability bias in voter turnout reports: Tests using the item count technique. Public Opinion Quarterly 74 (1): 3767.CrossRefGoogle Scholar
Holland, P. W. 1986. Statistics and causal inference (with discussion). Journal of the American Statistical Association 81: 945–60.Google Scholar
Holland, B. S., and Copenhaver, M. D. 1987. An improved sequentially rejective Bonferroni test procedure. Biometrics 43: 417–23.CrossRefGoogle Scholar
Imai, K. 2011. Multivariate regression analysis for the item count technique. Journal of the American Statistical Association 106: 407–16.CrossRefGoogle Scholar
Imai, K., King, G., and Stuart, E. A. 2008. Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society, Series A (Statistics in Society) 171: 481502.CrossRefGoogle Scholar
Janus, A. L. 2010. The influence of social desirability pressures on expressed immigration attitudes. Social Science Quarterly 91: 928–46.CrossRefGoogle Scholar
Kane, J. G., Craig, S. C., and Wald, K. D. 2004. Religion and presidential politics in Florida: A list experiment. Social Science Quarterly 85: 281–93.CrossRefGoogle Scholar
Kudô, A. 1963. A multivariate analogue of the one-sided test. Biometrika 50: 403–18.CrossRefGoogle Scholar
Kuklinski, J. H., Cobb, M. D., and Gilens, M. 1997a. Racial attitudes and the “New South.” Journal of Politics 59: 323–49.CrossRefGoogle Scholar
Kuklinski, J. H., Sniderman, P. M., Knight, K., Piazza, T., Tetlock, P. E., Lawrence, G. R., and Mellers, B. 1997b. Racial prejudice and attitudes toward affirmative action. American Journal of Political Science 41: 402–19.CrossRefGoogle Scholar
LaBrie, J. W., and Earleywine, M. 2000. Sexual risk behaviors and alcohol: Higher base rates revealed using the unmatched-count technique. Journal of Sex Research 37: 321–26.CrossRefGoogle Scholar
Manski, C. F. 2007. Identification for Prediction and Decision. Cambridge, MA: Harvard University Press.Google Scholar
Miller, J. D. 1984. A new survey technique for studying deviant behavior. PhD diss, The George Washington University.Google Scholar
Newey, W. K., and McFadden, D. 1994. Large sample estimation and hypothesis testing. In Handbook of Econometrics, ed. Engle, R. F. and McFadden, D. L., volume IV, 2111–245. Amsterdam, The Netherlands: North Holland.Google Scholar
Perlman, M. D. 1969. One-sided testing problems in multivariate analysis. The Annals of Mathematical Statistics 40: 549–67.CrossRefGoogle Scholar
Presser, S., and Stinson, L. 1998. Data collection mode and social desirability bias in self-reported religious attendance. American Sociological Review 63: 137–45.CrossRefGoogle Scholar
Raghavarao, D., and Federer, W. T. 1979. Block total response as an alternative to the randomized response method in surveys. Journal of the Royal Statistical Society, Series B, Methodological 41 (1): 40–5.Google Scholar
Rayburn, N. R., Earleywine, M., and Davison, G. C. 2003. An investigation of base rates of anti-gay hate crimes using the unmatched-count technique. Journal of Aggression, Maltreatment & Trauma 6: 137–52.CrossRefGoogle Scholar
Redlawsk, D. P., Tolbert, C. J., and Franko, W. 2010. Voters, emotions, and race in 2008: Obama as the first black president. Political Research Quarterly 63: 875–89.CrossRefGoogle Scholar
Shapiro, A. 1985. Asymptotic distribution of test statistics in the analysis of moment structures under inequality constraints. Biometrika 72: 133–44.CrossRefGoogle Scholar
Sniderman, P. M., and Carmines, E. G. 1997. Reaching Beyond Race. Cambridge, MA: Harvard University Press.Google Scholar
Sniderman, P. M., Tetlock, P. E., and Piazza, T. 1992. Codebook for the 1991 National Race and Politics Survey. Survey Research Center, Berkeley, California. http://sda.berkeley.edu/D3/Natlrace/Doc/nrac.htm.Google Scholar
Streb, M. J., Burrell, B., Frederick, B., and Genovese, M. A. 2008. Social desirability effects and support for a female american president. Public Opinion Quarterly 72(1): 7689.CrossRefGoogle Scholar
Tourangeau, R., and Yan, T. 2007. Sensitive questions in surveys. Psychological Bulletin 133: 859–83.CrossRefGoogle ScholarPubMed
Tsuchiya, T. 2005. Domain estimators for the item count technique. Survey Methodology 31(1): 4151.Google Scholar
Tsuchiya, T., Hirai, Y., and Ono, S. 2007. A study of the properties of the item count technique. Public Opinon Quarterly 71: 253–72.Google Scholar
Warner, S. L. 1965. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association 60(309): 63–9.CrossRefGoogle ScholarPubMed
Wimbush, J. C., and Dalton, D. R. 1997. Base rate for employee theft: Convergence of multiple methods. Journal of Applied Psychology 82: 756–63.CrossRefGoogle Scholar
Wolak, F. A. 1991. The local nature of hypothesis tests involving inequality constraints in nonlinear models. Econometrica 59: 981–95.CrossRefGoogle Scholar
Zaller, J. 2002. The statistical power of election studies to detect media exposure effects in political campaigns. Election Studies 21: 297329.CrossRefGoogle Scholar
Supplementary material: PDF

Blair and Imai supplementary material

List Experiments

Download Blair and Imai supplementary material(PDF)
PDF 198 KB
You have Access
263
Cited by

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Statistical Analysis of List Experiments
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Statistical Analysis of List Experiments
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Statistical Analysis of List Experiments
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *