Skip to main content

Behavioral Consequences of Probabilistic Precision: Experimental Evidence from National Security Professionals

  • Jeffrey A. Friedman, Jennifer S. Lerner and Richard Zeckhauser

National security is one of many fields where experts make vague probability assessments when evaluating high-stakes decisions. This practice has always been controversial, and it is often justified on the grounds that making probability assessments too precise could bias analysts or decision makers. Yet these claims have rarely been submitted to rigorous testing. In this paper, we specify behavioral concerns about probabilistic precision into falsifiable hypotheses which we evaluate through survey experiments involving national security professionals. Contrary to conventional wisdom, we find that decision makers responding to quantitative probability assessments are less willing to support risky actions and more receptive to gathering additional information. Yet we also find that when respondents estimate probabilities themselves, quantification magnifies overconfidence, particularly among low-performing assessors. These results hone wide-ranging concerns about probabilistic precision into a specific and previously undocumented bias that training may be able to correct.

Hide All
Barnes Alan. 2015. Making Intelligence Analysis More Intelligent. Intelligence and National Security 31 (1):327–44.
Berinsky Adam J., Huber Gregory A., and Lenz Gabriel S.. 2012. Evaluating Online Labor Markets for Experimental Research:'s Mechanical Turk. Political Analysis 20 (3):351–68.
Betts Richard K. 2006. Enemies of Intelligence: Knowledge and Power in American National Security. New York: Columbia University Press.
Beyerchen Alan. 1992/93. Clausewitz, Nonlinearity, and the Unpredictability of War. International Security 17 (3):5990.
Budescu David V., Por Han-Hui, Broomell Stephen B., and Smithson Michael. 2014. The Interpretation of IPCC Probabilistic Statements Around the World. Nature Climate Change 4: 508–12.
Cokely Edward T., Galesic Mirta, Schulz Eric, Ghazal Saima, and Garcia-Retamero Rocio. 2012. Measuring Risk Literacy: The Berlin Numeracy Test. Judgment and Decision Making 7 (1):2547.
Dhami Mandeep K. 2013. Understanding and Communicating Uncertainty in Intelligence Analysis. Report Prepared for Her Majesty's Government. London, UK.
Dhami Mandeep K., Mandel David R., Mellers Barbara A., and Tetlock Philip E.. 2015. Improving Intelligence Analysis with Decision Science. Perspectives on Psychological Science 10 (6):753–57.
Ellsberg Daniel. 1961. Risk, Ambiguity, and the Savage Axioms. Quarterly Journal of Economics 75 (4):643–69.
Fingar Thomas. 2011. Reducing Uncertainty: Intelligence and National Security. Stanford, CA: Stanford Security Studies.
Friedman Jeffrey A., and Zeckhauser Richard. 2015. Handling and Mishandling Estimative Probability. Intelligence and National Security 30 (1):7799.
Gigerenzer Gerd. 2002. Calculated Risks: How to Know When Numbers Deceive You. New York: Simon and Schuster.
Hafner-Burton Emilie M., Haggard Stephan, Lake David A., and Victor David G.. 2017. The Behavioral Revolution and the Study of International Relations. International Organization 71 (S1):S1S31.
Heuer Richards J. Jr. 1999. Psychology of Intelligence Analysis. Washington, DC: Center for the Study of Intelligence.
Huff Connor, and Tingley Dustin. 2015. “Who Are These People?” Evaluating the Demographic Characteristics and Political Preferences of MTurk Survey Respondents. Research and Politics 2 (3):112.
Hyde Susan D. 2015. Experiments in International Relations: Lab, Survey, and Field. Annual Review of Political Science 18 (1):403–24.
Jervis Robert. 1997. System Effects: Complexity in Political and Social Life. Princeton, NJ: Princeton University Press.
Jervis Robert. 2010. Why Intelligence Fails: Lessons from the Iranian Revolution and the Iraq War. Ithaca, NY: Cornell University Press.
Johnson Dominic D.P. 2004. Overconfidence and War: The Havoc and Glory of Positive Illusions. Cambridge, MA: Harvard University Press.
Johnston Rob. 2005. Analytic Culture in the US Intelligence Community. Washington, DC: Center for the Study of Intelligence.
Kahneman Daniel, and Renshon Jonathan. 2007. Why Hawks Win. Foreign Policy 158:3438.
Kent Sherman. 1964. Words of Estimative Probability. Studies in Intelligence 8 (4):4965.
Kertzer Joshua D., and Brutger Ryan. 2016. Decomposing Audience Costs: Bringing the Audience Back into Audience Cost Theory. American Journal of Political Science 60 (1):234–49.
Lanir Zvi, and Kahneman Daniel. 2006. An Experiment in Decision Analysis in Israel in 1975. Studies in Intelligence 50 (4). Available at <>.
Levy Jack S. 2013. Psychology and Foreign Policy Decision-Making. In The Oxford Handbook of Political Psychology, 2nd ed., edited by Huddy Leonie, Sears David O., and Levy Jack S., 301–33. Oxford, UK: Oxford University Press.
Lowenthal Mark M. 2006. Intelligence: From Secrets to Policy, 3rd ed. Washington, DC: CQ Press.
Mandel David R., and Barnes Alan. 2014. Accuracy of Forecasts in Strategic Intelligence. Proceedings of the National Academy of Sciences 111 (30):10984–89.
McChrystal Stanley. 2009. COMISAF Initial Assessment. Kabul, Afghanistan: Headquarters, International Security Assistance Force.
Mellers Barbara, Ungar Lyle, Baron Jonathan, Ramos Jaime, Gurcay Burcu, Fincher Katrina, Scott Sydney E., Moore Don, Atanasov Pavel, Swift Samuel A., Murray Terry, Stone Eric, and Tetlock Philip E.. 2014. Psychological Strategies for Winning a Geopolitical Forecasting Tournament. Psychological Science 25 (5):1106–15.
Mintz Alex, and Geva Nehemia, eds. 1997. Decisionmaking on War and Peace: The Cognitive-Rational Debate. Boulder, CO: Lynne Rienner.
Mosteller Frederick, and Youtz Cleo. 1990. Quantifying Probabilistic Expressions. Statistical Science 5 (1):212.
Piercey M. David. 2009. Motivated Reasoning and Verbal vs. Numerical Probability Assessment: Evidence from an Accounting Context. Organizational Behavior and Human Decision Processes 108 (2):330–41.
Press Daryl G., Sagan Scott D., and Valentino Benjamin A.. 2013. Atomic Aversion: Experimental Evidence on Taboos, Traditions, and the Non-Use of Nuclear Weapons. American Political Science Review 107 (1):188206.
Rapport Aaron. 2015. Waging War, Planning Peace: US Noncombat Operations and Major Wars. Ithaca, NY: Cornell University Press.
Renshon Jonathan. 2015. Losing Face and Sinking Costs: Experimental Evidence on the Judgment of Political and Military Leaders. International Organization 69 (3):659–95.
Rovner Joshua. 2011. Fixing the Facts: National Security and the Politics of Intelligence. Ithaca, NY: Cornell University Press.
Savage Leonard J. 1954. The Foundations of Statistics. New York: Wiley.
Shapiro Jacob N., and Cohen Dara Kay. 2007. Color Blind: Lessons from the Failed Homeland Security Advisory System. International Security 32 (2):121—54.
Sunstein Cass R. 2014. Valuing Life: Humanizing the Regulatory State. Chicago, IL: University of Chicago Press.
Tetlock Philip E. 2005. Expert Political Judgment. Princeton, NJ: Princeton University Press.
Tetlock Philip E., and Gardner Dan. 2015. Superforecasting: The Art and Science of Prediction. New York: Crown.
Tillers Peter, and Gottfried Jonathan. 2006. “Case Comment—United States v. Copeland.Law, Probability, and Risk 5 (2):135–57.
Tomz Michael R., and Weeks Jessica P.. 2013. Public Opinion and the Democratic Peace. American Political Science Review 107 (4):849–65.
US Army. 1997. Field Manual 101-5: Staff Organization and Operations. Washington, DC: Department of the Army.
US Army. 2009. Field Manual 5-0: The Operations Process. Washington, DC: Department of the Army.
Wallsten Thomas. 1990. Costs and Benefits of Vague Information. In Insights in Decision Making, edited by Hogarth Robin M., 2843. Chicago, IL: University of Chicago Press.
Williamson Vanessa. 2016. On the Ethics of Crowdsourced Research. PS: Political Science and Politics 49 (1):7781.
Zimmer Alf C. 1984. A Model for the Interpretation of Verbal Predictions. International Journal of Man-Machine Studies 20 (1):121–34.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

International Organization
  • ISSN: 0020-8183
  • EISSN: 1531-5088
  • URL: /core/journals/international-organization
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
Type Description Title
Supplementary materials

Friedman et al supplementary material
Friedman et al supplementary material 1

 Unknown (938 KB)
938 KB


Altmetric attention score

Full text views

Total number of HTML views: 17
Total number of PDF views: 217 *
Loading metrics...

Abstract views

Total abstract views: 1225 *
Loading metrics...

* Views captured on Cambridge Core between 25th September 2017 - 19th February 2018. This data will be updated every 24 hours.