Skip to main content Accessibility help
×
Home

Behavioral Consequences of Probabilistic Precision: Experimental Evidence from National Security Professionals

  • Jeffrey A. Friedman, Jennifer S. Lerner and Richard Zeckhauser

Abstract

National security is one of many fields where experts make vague probability assessments when evaluating high-stakes decisions. This practice has always been controversial, and it is often justified on the grounds that making probability assessments too precise could bias analysts or decision makers. Yet these claims have rarely been submitted to rigorous testing. In this paper, we specify behavioral concerns about probabilistic precision into falsifiable hypotheses which we evaluate through survey experiments involving national security professionals. Contrary to conventional wisdom, we find that decision makers responding to quantitative probability assessments are less willing to support risky actions and more receptive to gathering additional information. Yet we also find that when respondents estimate probabilities themselves, quantification magnifies overconfidence, particularly among low-performing assessors. These results hone wide-ranging concerns about probabilistic precision into a specific and previously undocumented bias that training may be able to correct.

Copyright

References

Hide All
Barnes, Alan. 2015. Making Intelligence Analysis More Intelligent. Intelligence and National Security 31 (1):327–44.
Berinsky, Adam J., Huber, Gregory A., and Lenz, Gabriel S.. 2012. Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk. Political Analysis 20 (3):351–68.
Betts, Richard K. 2006. Enemies of Intelligence: Knowledge and Power in American National Security. New York: Columbia University Press.
Beyerchen, Alan. 1992/93. Clausewitz, Nonlinearity, and the Unpredictability of War. International Security 17 (3):5990.
Budescu, David V., Por, Han-Hui, Broomell, Stephen B., and Smithson, Michael. 2014. The Interpretation of IPCC Probabilistic Statements Around the World. Nature Climate Change 4: 508–12.
Cokely, Edward T., Galesic, Mirta, Schulz, Eric, Ghazal, Saima, and Garcia-Retamero, Rocio. 2012. Measuring Risk Literacy: The Berlin Numeracy Test. Judgment and Decision Making 7 (1):2547.
Dhami, Mandeep K. 2013. Understanding and Communicating Uncertainty in Intelligence Analysis. Report Prepared for Her Majesty's Government. London, UK.
Dhami, Mandeep K., Mandel, David R., Mellers, Barbara A., and Tetlock, Philip E.. 2015. Improving Intelligence Analysis with Decision Science. Perspectives on Psychological Science 10 (6):753–57.
Ellsberg, Daniel. 1961. Risk, Ambiguity, and the Savage Axioms. Quarterly Journal of Economics 75 (4):643–69.
Fingar, Thomas. 2011. Reducing Uncertainty: Intelligence and National Security. Stanford, CA: Stanford Security Studies.
Friedman, Jeffrey A., and Zeckhauser, Richard. 2015. Handling and Mishandling Estimative Probability. Intelligence and National Security 30 (1):7799.
Gigerenzer, Gerd. 2002. Calculated Risks: How to Know When Numbers Deceive You. New York: Simon and Schuster.
Hafner-Burton, Emilie M., Haggard, Stephan, Lake, David A., and Victor, David G.. 2017. The Behavioral Revolution and the Study of International Relations. International Organization 71 (S1):S1S31.
Heuer, Richards J. Jr. 1999. Psychology of Intelligence Analysis. Washington, DC: Center for the Study of Intelligence.
Huff, Connor, and Tingley, Dustin. 2015. “Who Are These People?” Evaluating the Demographic Characteristics and Political Preferences of MTurk Survey Respondents. Research and Politics 2 (3):112.
Hyde, Susan D. 2015. Experiments in International Relations: Lab, Survey, and Field. Annual Review of Political Science 18 (1):403–24.
Jervis, Robert. 1997. System Effects: Complexity in Political and Social Life. Princeton, NJ: Princeton University Press.
Jervis, Robert. 2010. Why Intelligence Fails: Lessons from the Iranian Revolution and the Iraq War. Ithaca, NY: Cornell University Press.
Johnson, Dominic D.P. 2004. Overconfidence and War: The Havoc and Glory of Positive Illusions. Cambridge, MA: Harvard University Press.
Johnston, Rob. 2005. Analytic Culture in the US Intelligence Community. Washington, DC: Center for the Study of Intelligence.
Kahneman, Daniel, and Renshon, Jonathan. 2007. Why Hawks Win. Foreign Policy 158:3438.
Kent, Sherman. 1964. Words of Estimative Probability. Studies in Intelligence 8 (4):4965.
Kertzer, Joshua D., and Brutger, Ryan. 2016. Decomposing Audience Costs: Bringing the Audience Back into Audience Cost Theory. American Journal of Political Science 60 (1):234–49.
Lanir, Zvi, and Kahneman, Daniel. 2006. An Experiment in Decision Analysis in Israel in 1975. Studies in Intelligence 50 (4). Available at <https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/csi-studies/studies/vol50no4/an-experiment-in-decision-analysis-in-israel-in-1975.html>.
Levy, Jack S. 2013. Psychology and Foreign Policy Decision-Making. In The Oxford Handbook of Political Psychology, 2nd ed., edited by Huddy, Leonie, Sears, David O., and Levy, Jack S., 301–33. Oxford, UK: Oxford University Press.
Lowenthal, Mark M. 2006. Intelligence: From Secrets to Policy, 3rd ed. Washington, DC: CQ Press.
Mandel, David R., and Barnes, Alan. 2014. Accuracy of Forecasts in Strategic Intelligence. Proceedings of the National Academy of Sciences 111 (30):10984–89.
McChrystal, Stanley. 2009. COMISAF Initial Assessment. Kabul, Afghanistan: Headquarters, International Security Assistance Force.
Mellers, Barbara, Ungar, Lyle, Baron, Jonathan, Ramos, Jaime, Gurcay, Burcu, Fincher, Katrina, Scott, Sydney E., Moore, Don, Atanasov, Pavel, Swift, Samuel A., Murray, Terry, Stone, Eric, and Tetlock, Philip E.. 2014. Psychological Strategies for Winning a Geopolitical Forecasting Tournament. Psychological Science 25 (5):1106–15.
Mintz, Alex, and Geva, Nehemia, eds. 1997. Decisionmaking on War and Peace: The Cognitive-Rational Debate. Boulder, CO: Lynne Rienner.
Mosteller, Frederick, and Youtz, Cleo. 1990. Quantifying Probabilistic Expressions. Statistical Science 5 (1):212.
Piercey, M. David. 2009. Motivated Reasoning and Verbal vs. Numerical Probability Assessment: Evidence from an Accounting Context. Organizational Behavior and Human Decision Processes 108 (2):330–41.
Press, Daryl G., Sagan, Scott D., and Valentino, Benjamin A.. 2013. Atomic Aversion: Experimental Evidence on Taboos, Traditions, and the Non-Use of Nuclear Weapons. American Political Science Review 107 (1):188206.
Rapport, Aaron. 2015. Waging War, Planning Peace: US Noncombat Operations and Major Wars. Ithaca, NY: Cornell University Press.
Renshon, Jonathan. 2015. Losing Face and Sinking Costs: Experimental Evidence on the Judgment of Political and Military Leaders. International Organization 69 (3):659–95.
Rovner, Joshua. 2011. Fixing the Facts: National Security and the Politics of Intelligence. Ithaca, NY: Cornell University Press.
Savage, Leonard J. 1954. The Foundations of Statistics. New York: Wiley.
Shapiro, Jacob N., and Cohen, Dara Kay. 2007. Color Blind: Lessons from the Failed Homeland Security Advisory System. International Security 32 (2):121—54.
Sunstein, Cass R. 2014. Valuing Life: Humanizing the Regulatory State. Chicago, IL: University of Chicago Press.
Tetlock, Philip E. 2005. Expert Political Judgment. Princeton, NJ: Princeton University Press.
Tetlock, Philip E., and Gardner, Dan. 2015. Superforecasting: The Art and Science of Prediction. New York: Crown.
Tillers, Peter, and Gottfried, Jonathan. 2006. “Case Comment—United States v. Copeland.Law, Probability, and Risk 5 (2):135–57.
Tomz, Michael R., and Weeks, Jessica P.. 2013. Public Opinion and the Democratic Peace. American Political Science Review 107 (4):849–65.
US Army. 1997. Field Manual 101-5: Staff Organization and Operations. Washington, DC: Department of the Army.
US Army. 2009. Field Manual 5-0: The Operations Process. Washington, DC: Department of the Army.
Wallsten, Thomas. 1990. Costs and Benefits of Vague Information. In Insights in Decision Making, edited by Hogarth, Robin M., 2843. Chicago, IL: University of Chicago Press.
Williamson, Vanessa. 2016. On the Ethics of Crowdsourced Research. PS: Political Science and Politics 49 (1):7781.
Zimmer, Alf C. 1984. A Model for the Interpretation of Verbal Predictions. International Journal of Man-Machine Studies 20 (1):121–34.
Type Description Title
UNKNOWN
Supplementary materials

Friedman et al supplementary material
Friedman et al supplementary material 1

 Unknown (938 KB)
938 KB

Behavioral Consequences of Probabilistic Precision: Experimental Evidence from National Security Professionals

  • Jeffrey A. Friedman, Jennifer S. Lerner and Richard Zeckhauser

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed