Skip to main content Accessibility help
Hostname: page-component-56f9d74cfd-mtzzk Total loading time: 0.434 Render date: 2022-06-27T12:45:51.934Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "useNewApi": true }

Eliciting Beliefs as Distributions in Online Surveys

Published online by Cambridge University Press:  04 February 2021

Lucas Leemann*
Department of Political Science, University of Zürich, Zürich, Switzerland. Email:
Lukas F. Stoetzer
Humboldt University of Berlin, Cluster of Excellence SCRIPTS, Berlin, Germany. Email:
Richard Traunmüller
School of Social Sciences, University of Mannheim, Mannheim, Germany. Email:
Corresponding author Lucas Leemann


Citizens’ beliefs about uncertain events are fundamental variables in many areas of political science. While beliefs are often conceptualized in the form of distributions, obtaining reliable measures in terms of full probability densities is a difficult task. In this letter, we ask if there is an effective way of eliciting beliefs as distributions in the context of online surveys. Relying on experimental evidence, we evaluate the performance of five different elicitation methods designed to capture citizens’ uncertain expectations. Our results suggest that an elicitation method originally proposed by Manski (2009) performs well. It measures average citizens’ subjective belief distributions reliably and is easily implemented in the context of regular (online) surveys. We expect that a wider use of this method will lead to considerable improvements in the study of citizens’ expectations and beliefs.

© The Author(s) 2021. Published by Cambridge University Press on behalf of the Society for Political Methodology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)


Edited by Daniel Hopkins


Alvarez, R. M., and Brehm, J.. 1997. “Are Americans Ambivalent Towards Racial Policies?American Journal of Political Science 41:345374.CrossRefGoogle Scholar
Bartels, L. M. 2002. “Beyond the Running Tally: Partisan Bias in Political Perceptions.” Political Behavior 24(2):117150.CrossRefGoogle Scholar
Berinsky, A. J. 2017. “Rumors and Health Care Reform: Experiments in Political Misinformation.” British Journal of Political Science 47(2):241262.CrossRefGoogle Scholar
Berinsky, A. J, Huber, G. A., and Lenz, G. S.. 2012. “Evaluating Online Labor Markets for Experimental Research: Amazon. com’s Mechanical Turk.” Political Analysis 20(3):351368.CrossRefGoogle Scholar
Berinsky, A. J, Margolis, M. F., and Sances, M. W.. 2014. “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-administered Surveys.” American Journal of Political Science 58(3):739753.CrossRefGoogle Scholar
Bullock, J. G. 2009. “Partisan Bias and the Bayesian Ideal in the Study of Public Opinion.” The Journal of Politics 71(3):11091124.CrossRefGoogle Scholar
Coppock, A. 2018. “Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach.” Political Science Research and Methods 7:116.Google Scholar
Cosmides, L., and Tooby, J.. 1996. “Are Humans Good Intuitive Statisticians after all? Rethinking Some Conclusions from the Literature on Judgment under Uncertainty.” Cognition 58(1):173.CrossRefGoogle Scholar
Delavande, A., and Rohwedder, S.. 2008. “Eliciting Subjective Probabilities in Internet Surveys.” Public Opinions Quarterly 72(5):866891.CrossRefGoogle ScholarPubMed
Garthwaite, P. H., Kadane, J. B., and O’Hagan, A.. 2005. “Statistical Methods for Eliciting Probability Distributions.” Journal of the American Statistical Association 100(470):680701.CrossRefGoogle Scholar
Gerber, A., and Green, D.. 1999. “Misperceptions About Perceptual Bias.” Annual Review of Political Science 2(1):189210.CrossRefGoogle Scholar
Gill, J., and Freeman, J. R.. 2013. “Dynamic Elirefd Priors for Updating Covert Networks.” Network Science 1(1):6894.CrossRefGoogle Scholar
Gill, J., and Walker, L. D.. 2005. “Elirefd Priors for Bayesian Model Specifications in Political Science Research.” The Journal of Politics 67(3):841872.CrossRefGoogle Scholar
Goldstein, D. G., and Rothschild, D.. 2014. “Lay Understanding of Probability Distributions.” Judgment & Decision Making 9(1):114.Google Scholar
Kuru, O., Pasek, J., and Traugott, M. W.. 2017. “Motivated Reasoning in the Perceived Credibility of Public Opinion Polls.” Public Opinion Quarterly 81(2):422446.CrossRefGoogle Scholar
Kynn, M. 2008. “The ‘Heuristics and Biases’ Bias in Expert Elicitation.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 171(1):239264.Google Scholar
Lebo, M. J., and Cassino, D.. 2007. “The Aggregated Consequences of Motivated Reasoning and the Dynamics of Partisan Presidential Approval.” Political Psychology 28(6):719746.CrossRefGoogle Scholar
Leemann, L., Traunmueller, R., and Stoetzer, L. F.. 2020. “Replication Data for: Eliciting Beliefs as Distributions in Online Surveys.”, Harvard Dataverse, V1, UNF:6:AJOSTeuI2rt9XmpP7/2rlg== [fileUNF].Google Scholar
Leiter, D., Murr, A., Ramirez, E. R., and Stegmaier, M.. 2018. “Social Networks and Citizen Election Forecasting: The More Friends the Better.” International Journal of Forecasting 34(2):235248.CrossRefGoogle Scholar
Madson, G. J., and Hillygus, D. S.. 2019. “All the Best Polls Agree with Me: Bias in Evaluations of Political Polling.” Political Behavior 5:118.Google Scholar
Manski, C. F. 2009. Identification for Prediction and Decision. Cambridge, MA: Harvard University Press.Google Scholar
Mason, W, and Suri, S.. 2012. “Conducting Behavioral Research on Amazon’s Mechanical Turk.” Behavior Research Methods 44(1):123.CrossRefGoogle ScholarPubMed
Morris, D. E., Oakley, J. E., and Crowe, J. A.. 2014. “A Web-Based Tool for Eliciting Probability Distributions from Experts.” Environmental Modelling & Software 52:14.CrossRefGoogle Scholar
Mullinix, K. J., Leeper, T. J., Druckman, J. N., and Freese, J.. 2015. “The Generalizability of Survey Experiments.” Journal of Experimental Political Science 2(2):109138.CrossRefGoogle Scholar
Murr, A. 2011. “‘Wisdom of Crowds’? A Decentralised Election Forecasting Model that Uses Citizens’ Local Expectations.” Electoral Studies 30(4):771783.CrossRefGoogle Scholar
O’Hagan, A. et al. 2006. Uncertain Judgements: Eliciting Experts’ Probabilities. Boca Raton, FL: John Wiley & Sons.CrossRefGoogle Scholar
Peterson, C., and Miller, A.. 1964. “Mode, Median, and Mean as Optimal Strategies.” Journal of Experimental Psychology 68(4):363.CrossRefGoogle ScholarPubMed
Savage, L. J. 1971. “Elicitation of Personal Probabilities and Expectations.” Journal of the American Statistical Association 66(336):783801.CrossRefGoogle Scholar
Schlag, K. H., Tremewan, J., and Van der Weele, J. J.. 2015. “A Penny for Your Thoughts: A Survey of Methods for Eliciting Beliefs.” Experimental Economics 18(3):457490.CrossRefGoogle Scholar
Spetzler, C. S., and Stael von Holstein, C.-A. S.. 1975. “Exceptional Paper—Probability Encoding in Decision Analysis.” Management Science 22(3):340358.CrossRefGoogle Scholar
Thomas, K. A., and Clifford, S.. 2017. “Validity and Mechanical Turk: An Assessment of Exclusion Methods and Interactive Experiments.” Computers in Human Behavior 77:184197.CrossRefGoogle Scholar
Tversky, A., and Kahneman, D.. 1971. “Belief in the Law of Small Numbers.” Psychological Bulletin 76(2):105.CrossRefGoogle Scholar
Tversky, A., and Kahneman, D.. 1973. “Availability: A Heuristic for Judging Frequency and Probability.” Cognitive Psychology 5(2):207232.CrossRefGoogle Scholar
Tversky, A., and Kahneman, D. l.. 1974. “Judgment under Uncertainty: Heuristics and Biases.” Science 185(4157):11241131.CrossRefGoogle ScholarPubMed
Wallsten, T. S., and Budescu, D. V.. 1983. “State of the Art—Encoding Subjective Probabilities: A Psychological and Psychometric Review.” Management Science 29(2):151173.CrossRefGoogle Scholar
Zaller, J., and Feldman, S.. 1992. “A Simple Theory of the Survey Response: Answering Questions and Revealing Preferences.” American Journal of Political Science 36:579616.CrossRefGoogle Scholar
Supplementary material: PDF

Leemann et al. supplementary material

Leemann et al. supplementary material

Download Leemann et al. supplementary material(PDF)
PDF 593 KB
Cited by

Save article to Kindle

To save this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the or variations. ‘’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Eliciting Beliefs as Distributions in Online Surveys
Available formats

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

Eliciting Beliefs as Distributions in Online Surveys
Available formats

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

Eliciting Beliefs as Distributions in Online Surveys
Available formats

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *