Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-848d4c4894-hfldf Total loading time: 0 Render date: 2024-05-08T00:27:03.410Z Has data issue: false hasContentIssue false

8 - Eliciting Probabilities from Experts

Published online by Cambridge University Press:  05 June 2012

By
Stephen C. Hora
Edited by
Ward Edwards ,
Ralph F. Miles Jr.  and
Detlof von Winterfeldt
Stephen C. Hora
Affiliation:
College of Business and Economics, University of Hawaii-Hilo
Ralph F. Miles Jr.
Affiliation:
California Institute of Technology
Detlof von Winterfeldt
Affiliation:
University of Southern California
Get access

Summary

ABSTRACT. Decision models sometimes require the use of expert judgments to quantify uncertainties. Modes and methods for eliciting probabilities are presented in this chapter. Criteria for selecting issues and experts are discussed and protocols for acquiring these judgments derived from research and practice are described. These protocols are designed to eliminate problems that sometimes arise when using experts for uncertainty quantification. A range of considerations for organizing a probability elicitation is presented as a guide for developing a protocol for a given situation. The identification and avoidance of psychological biases arising from the application of heuristics is discussed as important background for probability elicitation. Probability training is presented as one method for reducing these biases. Measures of the goodness of assessed probabilities and probability distributions such as calibration measures and proper scores are given. Methods for completing a distribution from assessment fractiles are shown.

The ability of decision analysis to deal with significant uncertainties is one of its most attractive features. Probabilities or probability distributions provide the primary means for integrating uncertainties into a decision model. Often times, however, the acquisition of probabilistic information is, itself, a challenge. One method of developing probabilistic input for a decision model is to engage experts who have special knowledge about the likelihood of values or events in question. In order that the judgments of these experts be integrated with other types of information in the decision analysis, they should be codified as probabilities.

Type
Chapter
Information
Advances in Decision Analysis
From Foundations to Applications
 , pp. 129 - 153
Publisher: Cambridge University Press
Print publication year: 2007

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Alpert, M. & Raiffa, H. (1982). A progress report on the training of probability assessors in judgment under uncertainty: Heuristics and biases. In Kahneman, D., Slovic, P., and Tversky, A., (Eds.) Judgment under uncertainty: Heuristics and biases, Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Bar-Hillel, M. (2001). Subjective probability judgments, in Smelser, N. J. & Baltes, D. B. (Eds.) International Encyclopedia of the Social & Behavioral Sciences, Amsterdam: Elsevier Science Ltd. 15248–15251.Google Scholar
Bar-Hillel, M. & Neter, E. (1993). How alike is it versus how likely is it: A disjunction fallacy in stereotype judgments. Journal of Personality and Social Psychology 65, 1119–1132.CrossRefGoogle Scholar
Bonano, E. J., Hora S. C., Keeney, R. L., and von Winterfeldt, D. (1990). Elicitation and Use of Expert Judgment in Performance Assessment for High-Level Radioactive Waste Repositories. NUREG/CR-5411. Washington, DC: U.S. Nuclear Regulatory Commission.
Brenner, L. A., Koehler, D. J., Liberman, V. (1996). Overconfidence in probability and frequency judgments: A critical examination. Organizational Behavior and Human Decision Processes, 65, 212–219.CrossRefGoogle Scholar
Christensen-Szalanski, J. & Bushyhead, J. (1981). Physicians' Use of Probabilistic Information in a Real Clinical Setting. Journal of Experimental Psychology: Human Perception and Performance, 7, 928–935.Google Scholar
Clemen, R. T., Fischer, G. W., & Winkler, R. L. (2000). Assessing dependence: Some experimental results. Management Science, 46, 1100–1115.CrossRefGoogle Scholar
Clemen, R. T. & Winkler, R. L. (1985). limits for the precision and value of information from dependent sources. Operations Research, 33, 427–442.CrossRefGoogle Scholar
Clemen, R. T. & Reilly, T. (1999). Correlations and copulas for decision and risk analysis. Management Science, 45, 208–224.CrossRefGoogle Scholar
Cooke, R. M. (1991). Experts in Uncertainty, Oxford: Oxford University Press.Google Scholar
Cover, T. M. & Thomas, J. A. (1991). Elements of information theory. New York: Wiley-Interscience.CrossRefGoogle Scholar
Finetti, B. (1974). Theory of Probability, Vol. 1. New York: John Wiley and Sons.Google Scholar
de Finetti, B. (1977). Probabilities of probabilities: a real problem or a misunderstanding? in Aykac, A. and Brumat, C., (Eds.) New Developments in the Application of Bayesian Methods, Amsterdam: North Holland Publishing Company.Google Scholar
Druzdel, M. J. (1989). Verbal Uncertainty Expressions: Literature Review, Technical Report CMU-EPP-1990–02–02. Pittsburgh: Department of Engineering, Carnegie Mellon University.
Fischoff, B. & MacGregor, D. (1982). Subjective Confidence in Forecasts. Journal of Forecasting, 1, 155–72.CrossRefGoogle Scholar
Fischhoff, B., Slovic, P., & Lichtenstein, S. (1978). Fault trees: Sensibility of estimated failure probabilities to problem representation. Journal of Experimental Psychology: Human Perception and Performance, 4, 330–344.Google Scholar
Fox, C. R. & Clemen, R. T. (2005). Subjective probability assessment in decision analysis: Partition dependence and bias toward the ignorance prior. Management Science, 51, 1417–1432.CrossRefGoogle Scholar
Fox, C. R. & Tversky, A. (1998). A belief-based account of decision under uncertainty, Management Science, 44, 879–95.CrossRefGoogle Scholar
Gigerenzer, G. (1991). How to make cognitive illusions to disappear: Beyond heuristics and biases. In Stroebe, W. & Hewstone, M. (Ed.), European Review of Social Psychology, vol. 2. Chichester: Wiley.Google Scholar
Gigerenzer, G., Hoffrage, U., & Kleinbolting, H. (1991). Probabilistic mental models: A Brunswikian theory of confidence. Psychological Review, 98, 506–528.CrossRefGoogle Scholar
Gilovich, T., Griffin, D., and Kahneman, D. (Eds.). (2002). Heuristics and biases: The psychology of intuitive judgment.Cambridge, UK: Cambridge University Press.CrossRef
Haim, E. (1982). Characterization and Construction of Proper Scoring Rules. Unpublished doctoral dissertation, University of California, Berkeley.Google Scholar
Harper, F. T., Hora, S. C., Young, M. L.Miller, L. A.Lui, C. H.McKay, M. D.Helton, J. C., Goossens, L. H. J.Cooke, R. M.Pasler-Sauer, J.Kraan, B. & Jones, J. A. (1994). Probability Accident Consequence Uncertainty Analysis, Vols. 1–3, (NUREG/CR-6244, EUR 15855 EN)Brussels: USNRC and CEC DGⅫ Ⅻ.Google Scholar
Helton, J. C. & Burmaster, D. E., Eds. (1996). Special issue on treatment of aleatory and epistemic uncertainty. Reliability Engineering & System Safety, 54, 2–3.Google Scholar
Hora, S. C. (2004). Probability Judgments for Continuous Quantities: Linear Combinations and Calibration. Management Science, 50, 597–604.CrossRefGoogle Scholar
Hora, S., Dodd, N. G., & Hora, J. (1993). The Use of Decomposition in Probability Assessments of Continuous Variables. The Journal of Behavioral Decision Making, 6, 133–147.CrossRefGoogle Scholar
Hora, S. C., Hora, J. A., & Dodd, N. G. (1992). Assessment of Probability Distributions for Continuous Random Variables: A Comparison of the Bisection and Fixed Value Methods. Organizational Behavior and Human Decision Processes, 51, 133–155.CrossRefGoogle Scholar
Hora, S. C. & Iman, R. L. (1989). Expert Opinion in Risk Analysis: The NUREG-1150 Experience. Nuclear Science and Engineering, 102, 323–331.CrossRefGoogle Scholar
Hora, S. C. & Jensen, M. (2002). Expert Judgement Elicitation, Stockholm:. Swedish Radiation Protection Authority.Google Scholar
Juslin, P. (1994). The overconfidence phenomenon as a consequence of informal experimenter guided selection of almanac items. Organizational Behavior and Human Decision Processes, 57, 226–246.CrossRefGoogle Scholar
Kahneman, D., Slovic, P., & Tversky, A. (Eds.) (1982). Judgment under uncertainty: Heuristics and biases. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Keeney, R., & Winterfeldt, D. (1991). Eliciting probabilities from experts in complex technical problems. IEEE Transactions on Engineering Management, 38, 191–201.CrossRefGoogle Scholar
Kent, S. (1964). Words of Estimated Probability, in Steury, D. P., ed., Sherman Kent and the Board of National Estimates: Collected Essays. Washington, DC: Center for the Study of Intelligence, 1994.Google Scholar
Keren, G. (1988). On the ability of monitoring nonvertical perceptions and uncertainty knowledge: some calibration studies. Acta Psychologica, 67, 95–119.CrossRefGoogle Scholar
Klayman, J., Soll, J. B., Gonzalez-Vallejo, C., Barlas, S. (1999). Overconfidence: It depends on how, whom you ask. Organizational Behavior and Human Decision Processes, 79, 216–2247.CrossRefGoogle Scholar
Knight, F. H. (1921). Risk, Uncertainty, and Profit. Boston: Houghton Mifflin Company.Google Scholar
Lichtenstein, S., & Fischhoff, B. (1977). Do those who know more also know more about how much they know?Organizational Behavior and Human Performance, 20, 159–183.CrossRefGoogle Scholar
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982). Calibration of probabilities: The state of the art to 1980, in Kahneman, D., Slovic, P., and Tversky, A., (Eds.) Judgment under uncertainty: Heuristics and biases, Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Matheson, J. E. & Winkler, R. L. (1976). Scoring Rules for Continuous Probability Distributions, Management Science, 22, 1087–1096.CrossRefGoogle Scholar
Merkhofer, M. W. (1987). Quantifying judgmental uncertainty: Methodology, experiences, and insights. IEEE Transactions on Systems, Man, and Cybernetics, 17, 741–752.CrossRefGoogle Scholar
Morgan, M. G., Henrion, M. (1992). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis, New York: Cambridge University Press.Google Scholar
Murphy, A. H. (1973). A New Vector Partition of the Probability Score. Journal of Applied Meteorology, 12 595–6.2.0.CO;2>CrossRefGoogle Scholar
Murphy, A. H. & Winkler, R. L. (1977). Reliability of Subjective Probability Forecasts of Precipitation and Temperature. Applied Statistics, 26, 41–47.CrossRefGoogle Scholar
Murphy, A. H. (1972). Scalar and Vector Partitions of the Probability Score (Part I) Two-state situation. Journal of Applied Meteorology, 11, 273–82.2.0.CO;2>CrossRefGoogle Scholar
Ortiz, N. R., Wheeler, T. A., Breeding, R. J., Hora, S., Meyer, M. A., & Keeney, R. L. (1991). The Use of Expert Judgment in the NUREG-1150. Nuclear Engineering and Design, 126, 313–331.CrossRefGoogle Scholar
Raiffa, H. & Schlaifer, R. (1961). Applied Statistical Decision Theory, Cambridge: MIT Press.Google Scholar
Ravinder, H. V., Kleinmuntz, D. N., & Dyer, J. S. (1988). The reliability of subjective probability assessments obtained through decomposition. Management Science, 34, 186–199.CrossRefGoogle Scholar
Savage, L. J. (1954). The Foundations of Statistics. New York: John Wiley and Sons (second edition, 1972, New York: Dover).Google Scholar
Savage, L. J. (1971). The elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66, 783–801.CrossRefGoogle Scholar
Skyrms, B. (1980). “Higher order degrees of belief.” In “Prospects for Pragmatism: Essays in Honor of F. P. Ramsey,” Mellor, D. H. (Ed.), Cambridge University Press, Cambridge, pp. 109–137.Google Scholar
Sloman, S., Rottenstreich, Y., Wisniewski, E., Hadjichristidis, C., & Fox, C. R. (2004). Typical versus atypical unpacking and superadditive judgment. Journal of Experimental Psychology, 30, 573–582.Google ScholarPubMed
Spetzler, C. S. & Holstein, Stael C. A. S. (1975). Probability Encoding in Decision Analysis. Management Science, 22, 340–358.CrossRefGoogle Scholar
Stael von Holstein, C. A. S. & Matheson, J. E. (1979). A manual for encoding probability distributions. Menlo Park, CA: SRI International.Google Scholar
Stael von Holstein, C. A. S. (1972). Probabilistic forecasting: An experiment related to the stock market. Organizational Behavior and Human Performance, 8, 139–158.CrossRefGoogle Scholar
Trauth, K. M., Hora, S. C. & Guzowski, R. V. (1994). A formal expert judgment procedure for performance assessments of the waste isolation pilot plant, SAND93–2450. Albuquerque: Sandia National Laboratories.CrossRefGoogle Scholar
Tversky, A. & Kahneman, D. (1973). Availability a Heuristic for Judging Probability, Cognitive Psychology 5, 207–232. and in an abbreviated form in D. Kahneman, P. Slovic, and A. Tversky, (Eds.) Judgment under uncertainty: Heuristics and biases. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Tversky, A. & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science 185, 1124–1131 and in D. Kahneman, P. Slovic, and A. Tversky, (Eds.) Judgment under uncertainty: Heuristics and biases. Cambridge: Cambridge University Press.CrossRefGoogle ScholarPubMed
Tversky, A. & Koehler, D. J. (1994). Support theory: A nonextensional representation of subjective probability. Psychological Review, 101, 547–567.CrossRefGoogle Scholar
U.S. Nuclear Regulatory Commission (1975). Reactor safety study: WASH-1400, NUREG-751014. Washington: U.S. Nuclear Regulatory Commission.
Winkler, R. L. (1971). Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66, 675–685.CrossRefGoogle Scholar
Winkler, R. L. (1996). Scoring rules and the evaluation of probabilities. Test, 5, 1–60.CrossRefGoogle Scholar
Winman, A., Hansson, P., & Juslin, P. (2004). Subjective probability intervals: How to cure overconfidence by interval evaluation. Journal of Experimental Psychology: Learning Memory and Cognition, 30, 1167–1175.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×