Skip to main content Accessibility help
×
Home
Hostname: page-component-684899dbb8-mhx7p Total loading time: 0.321 Render date: 2022-05-20T21:55:28.023Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "useNewApi": true }

Relaxing the No Liars Assumption in List Experiment Analyses

Published online by Cambridge University Press:  10 May 2019

Yimeng Li*
Affiliation:
Graduate Student, Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, CA 91106, USA. Email: yimeng.li@caltech.edu

Abstract

The analysis of list experiments depends on two assumptions, known as “no design effect” and “no liars”. The no liars assumption is strong and may fail in many list experiments. I relax the no liars assumption in this paper, and develop a method to provide bounds for the prevalence of sensitive behaviors or attitudes under a weaker behavioral assumption about respondents’ truthfulness toward the sensitive item. I apply the method to a list experiment on the anti-immigration attitudes of California residents and on a broad set of existing list experiment datasets. The prevalence of different items and the correlation structure among items on the list jointly determine the width of the bound estimates. In particular, the bounds tend to be narrower when the list consists of items of the same category, such as multiple groups or organizations, different corporate activities, and various considerations for politician decision-making. My paper illustrates when the full power of the no liars assumption is most needed to pin down the prevalence of the sensitive behavior or attitude, and facilitates estimation of the prevalence robust to violations of the no liars assumption for many list experiment applications.

Type
Articles
Copyright
Copyright © The Author(s) 2019. Published by Cambridge University Press on behalf of the Society for Political Methodology. 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Author’s note: I thank Ines Levin for collecting and sharing with me the data I use in Section 3.1. Previous versions of this research were presented as a poster at the 34th Annual Meeting of the Society for Political Methodology (Polmeth 2017), and in a paper session at the 2018 Annual Meeting of the Midwest Political Science Association (MPSA 2018). I thank R. Michael Alvarez, Jonathan N. Katz, Seo-young Silvia Kim, Ines Levin, Lucas Núñez, Alejandro Robinson-Cortés, Robert Sherman, Matthew Shum, and participants at my poster and paper presentations at Polmeth 2017 and MPSA 2018 for discussions and comments. All errors are my own. Replication data are available in Li (2018) and an R function to implement the proposed bounds is available at https://github.com/lymolympic/list_relaxed_liars.

Contributing Editor: Jeff Gill

References

Ahlquist, J. S. 2018. “List Experiment Design, Non-Strategic Respondent Error, and Item Count Technique Estimators.” Political Analysis 26(1):3453.10.1017/pan.2017.31CrossRefGoogle Scholar
Ahlquist, J. S., Mayer, K. R., and Jackman, S.. 2014. “Alien Abduction and Voter Impersonation in the 2012 U.S. General Election: Evidence from a Survey List Experiment.” Election Law Journal: Rules, Politics, and Policy 13(4):460475.10.1089/elj.2013.0231CrossRefGoogle Scholar
Alvarez, R. M., Atkeson, L. R., Levin, I., and Li, Y.. 2019. “Paying Attention to Inattentive Survey Respondents.” Political Analysis 27(2):145162.10.1017/pan.2018.57CrossRefGoogle Scholar
Aronow, P. M., Coppock, A., Crawford, F. W., and Green, D. P.. 2015. “Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence.” Journal of Survey Statistics and Methodology 3(1):4366.10.1093/jssam/smu023CrossRefGoogle ScholarPubMed
Berinsky, A. J., Margolis, M. F., and Sances, M. W.. 2014. “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys.” American Journal of Political Science 58(3):739753.10.1111/ajps.12081CrossRefGoogle Scholar
Blair, G., and Imai, K.. 2012. “Statistical Analysis of List Experiments.” Political Analysis 20(1):4777.CrossRefGoogle Scholar
Blair, G., Imai, K., and Lyall, J.. 2014. “Comparing and Combining List and Endorsement Experiments: Evidence from Afghanistan.” American Journal of Political Science 58(4):10431063.10.1111/ajps.12086CrossRefGoogle Scholar
Coffman, K. B., Coffman, L. C., and Marzilli Ericson, K. M.. 2017. “The Size of the LGBT Population and the Magnitude of Antigay Sentiment Are Substantially Underestimated.” Management Science 63(10):31683186.CrossRefGoogle Scholar
Coppock, A. 2017. “Did Shy Trump Supporters Bias the 2016 Polls? Evidence from a Nationally-Representative List Experiment.” Statistics, Politics and Policy 8(1):2940.CrossRefGoogle Scholar
Corstange, D. 2009. “Sensitive Questions, Truthful Answers? Modeling the List Experiment with LISTIT.” Political Analysis 17(1):4563.CrossRefGoogle Scholar
Eady, G. 2017. “The Statistical Analysis of Misreporting on Sensitive Survey Questions.” Political Analysis 25(2):241259.10.1017/pan.2017.8CrossRefGoogle Scholar
Frye, T., Gehlbach, S., Marquardt, K. L., and Reuter, O. J.. 2016. “Is Putin’s Popularity Real? Post-Soviet Affairs 33(1):115.CrossRefGoogle Scholar
Glynn, A. N.2010. “What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment.” Unpublished manuscript.Google Scholar
Glynn, A. N. 2013. “What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment.” Public Opinion Quarterly 77(S1):159172.CrossRefGoogle Scholar
González-Ocantos, E. et al. . 2012. “Vote Buying and Social Desirability Bias: Experimental Evidence from Nicaragua.” American Journal of Political Science 56(1):202217.CrossRefGoogle Scholar
González-Ocantos, E., Kiewiet de Jonge, C., and Nickerson, D. W.. 2015. “Legitimacy Buying: The Dynamics of Clientelism in the Face of Legitimacy Challenges.” Comparative Political Studies 48(9):11271158.10.1177/0010414015574882CrossRefGoogle Scholar
Heerwig, J. A., and McCabe, B. J.. 2009. “Education and Social Desirability Bias: The Case of a Black Presidential Candidate.” Social Science Quarterly 90(3):674686.CrossRefGoogle Scholar
Holbrook, A. L., and Krosnick, J. A.. 2010. “Social Desirability Bias in Voter Turnout Reports: Tests Using the Item Count Technique.” Public Opinion Quarterly 74(1):3767.CrossRefGoogle Scholar
Imai, K. 2011. “Multivariate Regression Analysis for the Item Count Technique.” Journal of the American Statistical Association 106(494):407416.CrossRefGoogle Scholar
Imai, K., Park, B., and Greene, K. F.. 2015. “Using the Predicted Responses from List Experiments as Explanatory Variables in Regression Models.” Political Analysis 23(2):180196.10.1093/pan/mpu017CrossRefGoogle Scholar
Imbens, G. W., and Manski, C. F.. 2004. “Confidence Intervals for Partially Identified Parameters.” Econometrica 72(6):18451857.10.1111/j.1468-0262.2004.00555.xCrossRefGoogle Scholar
Kane, J. G., Craig, S. C., and Wald, K. D.. 2004. “Religion and Presidential Politics in Florida: A List Experiment.” Social Science Quarterly 85(2):281293.10.1111/j.0038-4941.2004.08502004.xCrossRefGoogle Scholar
Kiewiet de Jonge, C. P. 2015. “Who Lies About Electoral Gifts? Experimental Evidence from Latin America.” Public Opinion Quarterly 79(3):710739.CrossRefGoogle Scholar
Kiewiet de Jonge, C. P., and Nickerson, D. W.. 2014. “Artificial Inflation or Deflation? Assessing the Item Count Technique in Comparative Surveys.” Political Behavior 36(3):659682.10.1007/s11109-013-9249-xCrossRefGoogle Scholar
Köszegi, B. 2006. “Ego Utility, Overconfidence, and Task Choice.” Journal of the European Economic Association 4(4):673707.CrossRefGoogle Scholar
Kramon, E., and Weghorst, K. R.. 2012. “Measuring Sensitive Attitudes in Developing Countries: Lessons from Implementing the List Experiment.” Newsletter of the APSA Experimental Section 3(2):1424.Google Scholar
Kuklinski, J. H., Cobb, M. D., and Gilens, M.. 1997. “Racial Attitudes and the ‘New South’.” The Journal of Politics 59(2):323349.CrossRefGoogle Scholar
Lax, J. R., Phillips, J. H., and Stollwerk, A. F.. 2016. “Are Survey Respondents Lying about Their Support for Same-Sex Marriage? Lessons from a List Experiment.” Public Opinion Quarterly 80(2):510533.CrossRefGoogle ScholarPubMed
Li, Y.2018. “Replication Data for: Relaxing the No Liars Assumption in List Experiment Analyses.” https://doi.org/10.7910/DVN/DFRSKZ, Harvard Dataverse, V1.CrossRefGoogle Scholar
Malesky, E. J., Gueorguiev, D. D., and Jensen, N. M.. 2015. “Monopoly Money: Foreign Investment and Bribery in Vietnam, a Survey Experiment.” American Journal of Political Science 59(2):419439.10.1111/ajps.12126CrossRefGoogle Scholar
Meng, T., Pan, J., and Yang, P.. 2014. “Conditional Receptivity to Citizen Participation: Evidence From a Survey Experiment in China.” Comparative Political Studies 50(4):399433.CrossRefGoogle Scholar
Oppenheimer, D. M., Meyvis, T., and Davidenko, N.. 2009. “Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power.” Journal of Experimental Social Psychology 45(4):867872.CrossRefGoogle Scholar
Redlawsk, D. P., Tolbert, C. J., and Franko, W.. 2010. “Voters, Emotions, and Race in 2008: Obama as the First Black President.” Political Research Quarterly 63(4):875889.10.1177/1065912910373554CrossRefGoogle Scholar
Rosenfeld, B., Imai, K., and Shapiro, J. N.. 2016. “An Empirical Validation Study of Popular Survey Methodologies for Sensitive Questions.” American Journal of Political Science 60(3):783802.CrossRefGoogle Scholar
Stoye, J. 2009. “More on Confidence Intervals for Partially Identified Parameters.” Econometrica 77(4):12991315.Google Scholar
Streb, M. J., Burrell, B., Frederick, B., and Genovese, M. A.. 2008. “Social Desirability Effects and Support for a Female American President.” Public Opinion Quarterly 72(1):7689.CrossRefGoogle Scholar
Supplementary material: File

Li supplementary material

Li supplementary material 1

Download Li supplementary material(File)
File 888 KB
1
Cited by

Save article to Kindle

To save this article to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Relaxing the No Liars Assumption in List Experiment Analyses
Available formats
×

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

Relaxing the No Liars Assumption in List Experiment Analyses
Available formats
×

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

Relaxing the No Liars Assumption in List Experiment Analyses
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *