Book contents
- Frontmatter
- Contents
- List of Contributors
- Preface
- 1 Introduction
- PART I HISTORY AND FOUNDATIONS OF DECISION ANALYSIS
- PART II STRUCTURING DECISION PROBLEMS
- PART III PROBABILITIES AND BAYES NETS
- 8 Eliciting Probabilities from Experts
- 9 Aggregating Probability Distributions
- 10 Model Building with Belief Networks and Influence Diagrams
- 11 A Bayesian Approach to Learning Causal Networks
- PART IV UTILITIES
- PART V RISK ANALYSIS
- PART VI DECISION ANALYSIS IN A BEHAVIORAL AND ORGANIZATIONAL CONTEXT
- PART VII APPLICATIONS OF DECISION ANALYSIS
- Index
- References
8 - Eliciting Probabilities from Experts
Published online by Cambridge University Press: 05 June 2012
Book contents
- Frontmatter
- Contents
- List of Contributors
- Preface
- 1 Introduction
- PART I HISTORY AND FOUNDATIONS OF DECISION ANALYSIS
- PART II STRUCTURING DECISION PROBLEMS
- PART III PROBABILITIES AND BAYES NETS
- 8 Eliciting Probabilities from Experts
- 9 Aggregating Probability Distributions
- 10 Model Building with Belief Networks and Influence Diagrams
- 11 A Bayesian Approach to Learning Causal Networks
- PART IV UTILITIES
- PART V RISK ANALYSIS
- PART VI DECISION ANALYSIS IN A BEHAVIORAL AND ORGANIZATIONAL CONTEXT
- PART VII APPLICATIONS OF DECISION ANALYSIS
- Index
- References
Summary
ABSTRACT. Decision models sometimes require the use of expert judgments to quantify uncertainties. Modes and methods for eliciting probabilities are presented in this chapter. Criteria for selecting issues and experts are discussed and protocols for acquiring these judgments derived from research and practice are described. These protocols are designed to eliminate problems that sometimes arise when using experts for uncertainty quantification. A range of considerations for organizing a probability elicitation is presented as a guide for developing a protocol for a given situation. The identification and avoidance of psychological biases arising from the application of heuristics is discussed as important background for probability elicitation. Probability training is presented as one method for reducing these biases. Measures of the goodness of assessed probabilities and probability distributions such as calibration measures and proper scores are given. Methods for completing a distribution from assessment fractiles are shown.
The ability of decision analysis to deal with significant uncertainties is one of its most attractive features. Probabilities or probability distributions provide the primary means for integrating uncertainties into a decision model. Often times, however, the acquisition of probabilistic information is, itself, a challenge. One method of developing probabilistic input for a decision model is to engage experts who have special knowledge about the likelihood of values or events in question. In order that the judgments of these experts be integrated with other types of information in the decision analysis, they should be codified as probabilities.
Information
- Type
- Chapter
- Information
- Advances in Decision AnalysisFrom Foundations to Applications, pp. 129 - 153Publisher: Cambridge University PressPrint publication year: 2007
References
Alpert, M. & Raiffa, H. (1982). A progress report on the training of probability assessors in judgment under uncertainty: Heuristics and biases. In , , and , (Eds.) Judgment under uncertainty: Heuristics and biases, Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Bar-Hillel, M. (2001). Subjective probability judgments, in & (Eds.) International Encyclopedia of the Social & Behavioral Sciences, Amsterdam: Elsevier Science Ltd. 15248–15251.Google Scholar
& (1993). How alike is it versus how likely is it: A disjunction fallacy in stereotype judgments. Journal of Personality and Social Psychology 65, 1119–1132.CrossRefGoogle Scholar
Bonano, E. J., Hora S. C., Keeney, R. L., and von Winterfeldt, D. (1990). Elicitation and Use of Expert Judgment in Performance Assessment for High-Level Radioactive Waste Repositories. NUREG/CR-5411. Washington, DC: U.S. Nuclear Regulatory Commission.
, , (1996). Overconfidence in probability and frequency judgments: A critical examination. Organizational Behavior and Human Decision Processes, 65, 212–219.CrossRefGoogle Scholar
& (1981). Physicians' Use of Probabilistic Information in a Real Clinical Setting. Journal of Experimental Psychology: Human Perception and Performance, 7, 928–935.Google Scholar
, , & (2000). Assessing dependence: Some experimental results. Management Science, 46, 1100–1115.CrossRefGoogle Scholar
& (1985). limits for the precision and value of information from dependent sources. Operations Research, 33, 427–442.CrossRefGoogle Scholar
& (1999). Correlations and copulas for decision and risk analysis. Management Science, 45, 208–224.CrossRefGoogle Scholar
de Finetti, B. (1977). Probabilities of probabilities: a real problem or a misunderstanding? in and , (Eds.) New Developments in the Application of Bayesian Methods, Amsterdam: North Holland Publishing Company.Google Scholar
Druzdel, M. J. (1989). Verbal Uncertainty Expressions: Literature Review, Technical Report CMU-EPP-1990–02–02. Pittsburgh: Department of Engineering, Carnegie Mellon University.
& (1982). Subjective Confidence in Forecasts. Journal of Forecasting, 1, 155–72.CrossRefGoogle Scholar
, , & (1978). Fault trees: Sensibility of estimated failure probabilities to problem representation. Journal of Experimental Psychology: Human Perception and Performance, 4, 330–344.Google Scholar
& (2005). Subjective probability assessment in decision analysis: Partition dependence and bias toward the ignorance prior. Management Science, 51, 1417–1432.CrossRefGoogle Scholar
& (1998). A belief-based account of decision under uncertainty, Management Science, 44, 879–95.CrossRefGoogle Scholar
Gigerenzer, G. (1991). How to make cognitive illusions to disappear: Beyond heuristics and biases. In & (Ed.), European Review of Social Psychology, vol. 2. Chichester: Wiley.Google Scholar
, , & (1991). Probabilistic mental models: A Brunswikian theory of confidence. Psychological Review, 98, 506–528.CrossRefGoogle Scholar
, , and (Eds.). (2002). Heuristics and biases: The psychology of intuitive judgment.Cambridge, UK: Cambridge University Press.CrossRef
(1982). Characterization and Construction of Proper Scoring Rules. Unpublished doctoral dissertation, University of California, Berkeley.Google Scholar
, , , & (1994). Probability Accident Consequence Uncertainty Analysis, Vols. 1–3, (NUREG/CR-6244, EUR 15855 EN)Brussels: USNRC and CEC DGⅫ Ⅻ.Google Scholar
& , Eds. (1996). Special issue on treatment of aleatory and epistemic uncertainty. Reliability Engineering & System Safety, 54, 2–3.Google Scholar
(2004). Probability Judgments for Continuous Quantities: Linear Combinations and Calibration. Management Science, 50, 597–604.CrossRefGoogle Scholar
, , & (1993). The Use of Decomposition in Probability Assessments of Continuous Variables. The Journal of Behavioral Decision Making, 6, 133–147.CrossRefGoogle Scholar
, , & (1992). Assessment of Probability Distributions for Continuous Random Variables: A Comparison of the Bisection and Fixed Value Methods. Organizational Behavior and Human Decision Processes, 51, 133–155.CrossRefGoogle Scholar
& (1989). Expert Opinion in Risk Analysis: The NUREG-1150 Experience. Nuclear Science and Engineering, 102, 323–331.CrossRefGoogle Scholar
& (2002). Expert Judgement Elicitation, Stockholm:. Swedish Radiation Protection Authority.Google Scholar
(1994). The overconfidence phenomenon as a consequence of informal experimenter guided selection of almanac items. Organizational Behavior and Human Decision Processes, 57, 226–246.CrossRefGoogle Scholar
, , & (Eds.) (1982). Judgment under uncertainty: Heuristics and biases. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
, & (1991). Eliciting probabilities from experts in complex technical problems. IEEE Transactions on Engineering Management, 38, 191–201.CrossRefGoogle Scholar
Kent, S. (1964). Words of Estimated Probability, in , ed., Sherman Kent and the Board of National Estimates: Collected Essays. Washington, DC: Center for the Study of Intelligence, 1994.Google Scholar
(1988). On the ability of monitoring nonvertical perceptions and uncertainty knowledge: some calibration studies. Acta Psychologica, 67, 95–119.CrossRefGoogle Scholar
, , , (1999). Overconfidence: It depends on how, whom you ask. Organizational Behavior and Human Decision Processes, 79, 216–2247.CrossRefGoogle Scholar
, & (1977). Do those who know more also know more about how much they know?Organizational Behavior and Human Performance, 20, 159–183.CrossRefGoogle Scholar
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982). Calibration of probabilities: The state of the art to 1980, in , , and , (Eds.) Judgment under uncertainty: Heuristics and biases, Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
& (1976). Scoring Rules for Continuous Probability Distributions, Management Science, 22, 1087–1096.CrossRefGoogle Scholar
(1987). Quantifying judgmental uncertainty: Methodology, experiences, and insights. IEEE Transactions on Systems, Man, and Cybernetics, 17, 741–752.CrossRefGoogle Scholar
, . (1992). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis, New York: Cambridge University Press.Google Scholar
(1973). A New Vector Partition of the Probability Score. Journal of Applied Meteorology, 12 595–6.2.0.CO;2>CrossRefGoogle Scholar
& (1977). Reliability of Subjective Probability Forecasts of Precipitation and Temperature. Applied Statistics, 26, 41–47.CrossRefGoogle Scholar
(1972). Scalar and Vector Partitions of the Probability Score (Part I) Two-state situation. Journal of Applied Meteorology, 11, 273–82.2.0.CO;2>CrossRefGoogle Scholar
, , , , , & (1991). The Use of Expert Judgment in the NUREG-1150. Nuclear Engineering and Design, 126, 313–331.CrossRefGoogle Scholar
, , & (1988). The reliability of subjective probability assessments obtained through decomposition. Management Science, 34, 186–199.CrossRefGoogle Scholar
(1954). The Foundations of Statistics. New York: John Wiley and Sons (second edition, 1972, New York: Dover).Google Scholar
(1971). The elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66, 783–801.CrossRefGoogle Scholar
Skyrms, B. (1980). “Higher order degrees of belief.” In “Prospects for Pragmatism: Essays in Honor of F. P. Ramsey,” (Ed.), Cambridge University Press, Cambridge, pp. 109–137.Google Scholar
, , , , & (2004). Typical versus atypical unpacking and superadditive judgment. Journal of Experimental Psychology, 30, 573–582.Google ScholarPubMed
& (1975). Probability Encoding in Decision Analysis. Management Science, 22, 340–358.CrossRefGoogle Scholar
& (1979). A manual for encoding probability distributions. Menlo Park, CA: SRI International.Google Scholar
(1972). Probabilistic forecasting: An experiment related to the stock market. Organizational Behavior and Human Performance, 8, 139–158.CrossRefGoogle Scholar
, & (1994). A formal expert judgment procedure for performance assessments of the waste isolation pilot plant, SAND93–2450. Albuquerque: Sandia National Laboratories.CrossRefGoogle Scholar
& (1973). Availability a Heuristic for Judging Probability, Cognitive Psychology 5, 207–232. and in an abbreviated form in D. Kahneman, P. Slovic, and A. Tversky, (Eds.) Judgment under uncertainty: Heuristics and biases. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
& (1974). Judgment under uncertainty: Heuristics and biases. Science 185, 1124–1131 and in D. Kahneman, P. Slovic, and A. Tversky, (Eds.) Judgment under uncertainty: Heuristics and biases. Cambridge: Cambridge University Press.CrossRefGoogle ScholarPubMed
& (1994). Support theory: A nonextensional representation of subjective probability. Psychological Review, 101, 547–567.CrossRefGoogle Scholar
U.S. Nuclear Regulatory Commission (1975). Reactor safety study: WASH-1400, NUREG-751014. Washington: U.S. Nuclear Regulatory Commission.
(1971). Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66, 675–685.CrossRefGoogle Scholar
, , & (2004). Subjective probability intervals: How to cure overconfidence by interval evaluation. Journal of Experimental Psychology: Learning Memory and Cognition, 30, 1167–1175.Google Scholar
Accessibility standard: Unknown
Accessibility compliance for the PDF of this book is currently unknown
and may be updated in the future.
- 49
- Cited by