Book contents
- Frontmatter
- Contents
- List of Contributors
- Preface
- 1 Introduction
- PART I HISTORY AND FOUNDATIONS OF DECISION ANALYSIS
- PART II STRUCTURING DECISION PROBLEMS
- PART III PROBABILITIES AND BAYES NETS
- 8 Eliciting Probabilities from Experts
- 9 Aggregating Probability Distributions
- 10 Model Building with Belief Networks and Influence Diagrams
- 11 A Bayesian Approach to Learning Causal Networks
- PART IV UTILITIES
- PART V RISK ANALYSIS
- PART VI DECISION ANALYSIS IN A BEHAVIORAL AND ORGANIZATIONAL CONTEXT
- PART VII APPLICATIONS OF DECISION ANALYSIS
- Index
- References
9 - Aggregating Probability Distributions
Published online by Cambridge University Press: 05 June 2012
Book contents
- Frontmatter
- Contents
- List of Contributors
- Preface
- 1 Introduction
- PART I HISTORY AND FOUNDATIONS OF DECISION ANALYSIS
- PART II STRUCTURING DECISION PROBLEMS
- PART III PROBABILITIES AND BAYES NETS
- 8 Eliciting Probabilities from Experts
- 9 Aggregating Probability Distributions
- 10 Model Building with Belief Networks and Influence Diagrams
- 11 A Bayesian Approach to Learning Causal Networks
- PART IV UTILITIES
- PART V RISK ANALYSIS
- PART VI DECISION ANALYSIS IN A BEHAVIORAL AND ORGANIZATIONAL CONTEXT
- PART VII APPLICATIONS OF DECISION ANALYSIS
- Index
- References
Summary
ABSTRACT. This chapter is concerned with the aggregation of probability distributions in decision and risk analysis. Experts often provide valuable information regarding important uncertainties in decision and risk analyses because of the limited availability of hard data to use in those analyses. Multiple experts are often consulted in order to obtain as much information as possible, leading to the problem of how to combine or aggregate their information. Information may also be obtained from other sources such as forecasting techniques or scientific models. Because uncertainties are typically represented in terms of probability distributions, we consider expert and other information in terms of probability distributions. We discuss a variety of models that lead to specific combination methods. The output of these methods is a combined probability distribution, which can be viewed as representing a summary of the current state of information regarding the uncertainty of interest. After presenting the models and methods, we discuss empirical evidence on the performance of the methods. In the conclusion, we highlight important conceptual and practical issues to be considered when designing a combination process for use in practice.
Introduction
Expert judgments can provide useful information for forecasting, making decisions, and assessing risks. Such judgments have been used informally for many years. In recent years, the use of formal methods to combine expert judgments has become increasingly commonplace. Cooke (1991) reviews many of the developments over the years as attempts have been made to use expert judgments in various settings.
Information
- Type
- Chapter
- Information
- Advances in Decision AnalysisFrom Foundations to Applications, pp. 154 - 176Publisher: Cambridge University PressPrint publication year: 2007
References
Armstrong, J. S. (2001a). Combining forecasts. In (Ed.), Principles of forecasting: A handbook for researchers and practitioners. Norwell, MA: Kluwer, pp. 417–439.CrossRefGoogle Scholar
(Ed.). (2001b). Principles of forecasting: A handbook for researchers and practitioners. Norwell, MA: Kluwer.CrossRefGoogle Scholar
, , and (1975). The use of the decomposition principle in making judgments. Organizational Behavior and Human Performance, 14, 257–263.CrossRefGoogle Scholar
(1979). Normal Bayesian dialogues. Journal of the American Statistical Association, 74, 837–846.CrossRefGoogle Scholar
, and (1969). The combination of forecasts. Operational Research Quarterly, 20, 451–468.CrossRefGoogle Scholar
, , , and (1989). Seismic hazard characterization of 69 nuclear sites east of the Rocky Mountains. Vol. 1–8, NUREG/CR 5250, UCID-21517.
(1975). A Bayesian approach to the linear combination of forecasts. Operational Research Quarterly, 26, 325–329.CrossRefGoogle Scholar
(1988). Combining forecasts. European Journal of Operational Research, 33, 223–229.CrossRefGoogle Scholar
, and (1991). Interaction of judgmental and statistical forecasting methods: Issues and analysis. Management Science, 37, 501–518.CrossRefGoogle Scholar
, , and (1994). Testing for unreliable estimators and insignificant forecasts in combined forecasts. Journal of Forecasting, 13, 611–624.CrossRefGoogle Scholar
, and (1993). Some approximations useful to the use of dependent information sources. Reliability Engineering and System Safety, 42, 67–86.CrossRefGoogle Scholar
(1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5, 559–583.CrossRefGoogle Scholar
, , and (2000). Assessing dependence: Some experimental results. Management Science, 46, 1100–1115.CrossRefGoogle Scholar
Clemen, R. T., Jones, S. K., and Winkler, R. L. (1996). Aggregating forecasts: An empirical evaluation of some Bayesian methods. In , , and (Eds.), Bayesian statistics and econometrics: Essays in honor of Arnold Zellner. New York: Wiley, pp. 3–13.Google Scholar
, and (1999). Correlations and copulas for decision and risk analysis. Management Science, 45, 208–224.CrossRefGoogle Scholar
, and (1985). Limits for the precision and value of information from dependent sources. Operations Research, 33, 427–442.CrossRefGoogle Scholar
Clemen, R. T., and Winkler, R. L. (1987). Calibrating and combining precipitation probability forecasts. In (Ed.), Probability and Bayesian statistics. New York: Plenum, pp. 97–110.CrossRefGoogle Scholar
, and (1990). Unanimity and compromise among probability forecasters. Management Science, 36, 767–779.CrossRefGoogle Scholar
, and (1993). Aggregating point estimates: A flexible modeling approach. Management Science, 39, 501–515.CrossRefGoogle Scholar
, and (1999). Combining probability distributions from experts in risk analysis. Risk Analysis, 19, 187–203.CrossRefGoogle Scholar
(1991). Experts in uncertainty: Opinion and subjective probability in science. New York: Oxford University Press.Google Scholar
, , and (1991). Advances in probability distributions with given marginals: Beyond the copulas. Dordrecht, The Netherlands: Kluwer.CrossRefGoogle Scholar
(1992). Some compelling intuitions about group consensus decisions, theoretical and empirical research, and interpersonal aggregation phenomena: Selected examples, 1950–1990. Organizational Behavior and Human Decision Processes, 52, 3–38.CrossRefGoogle Scholar
, , and (1989). Clinical versus actuarial judgment. Science, 243, 1668–1673.CrossRefGoogle ScholarPubMed
de Finetti, B. (1937). La prévision: Ses lois logiques, ses sources subjectives. Annales de l'Institut Henri Poincaré, 7, 1–68. Translated in 1980 by H. E. Kyburg, Jr., Foresight. Its logical laws, its subjective sources. In and (Eds.), Studies in subjective probability (2nd ed.). Huntington, New York: Robert E. Krieger, pp. 53–118.Google Scholar
, , and (2002). Assessment of cost uncertainties for large technology projects: A methodology and an application. Interfaces, 32 (Jul/Aug), 52–66.CrossRefGoogle Scholar
EPRI (1986). Seismic hazard methodology for the central and eastern United States. Vol. 1: Methodology. NP-4/26. Palo Alto, CA: Electric Power Research Institute.
, and (1989). Subjective vs. objective combining of forecasts: An experiment. Journal of Forecasting, 8, 331–341.CrossRefGoogle Scholar
(1981). Consensus of opinion. European Journal of Operational Research, 7, 332–340.CrossRefGoogle Scholar
French, S. (1985). Group consensus probability distributions: A critical survey. In , , , and (Eds.), Bayesian statistics 2. Amsterdam: North-Holland, pp. 183–197.Google Scholar
, , and (2005). Statistical methods for eliciting prior distributions. Journal of the American Statistical Association, 100, 680–700.CrossRefGoogle Scholar
, , and (1995). Modeling expert opinion rising as a partial probabilistic specification. Journal of the American Statistical Association, 90, 598–604.CrossRefGoogle Scholar
(1984). Pooling operators with the marginalization property. Canadian Journal of Statistics, 12, 153–163.CrossRefGoogle Scholar
, and (1990). Allocating the weights in the linear opinion pool. Journal of Forecasting, 9, 53–73.CrossRefGoogle Scholar
, and (1985). Modeling expert judgments for Bayesian updating. Annals of Statistics, 13, 1198–1212.CrossRefGoogle Scholar
, and (1986). Combining probability distributions: A critique and annotated bibliography. Statistical Science, 1, 114–148.CrossRefGoogle Scholar
, and (1997). Proper analysis of the accuracy of group judgments. Psychological Bulletin, 121, 149–167.CrossRefGoogle Scholar
, and (1982). Assessment of a prior distribution for the correlation coefficient in a bivariate normal distribution. Journal of the Royal Statistical Society, Series A, 145, 237–249.CrossRefGoogle Scholar
(1972). Action selection and likelihood estimation by individuals and groups. Organizational Behavior and Human Performance, 7, 121–141.CrossRefGoogle Scholar
, and (1999). The expected value of information and the probability of surprise. Risk Analysis, 19, 135–152.CrossRefGoogle Scholar
Hastie, R. (1986). Experimental evidence on group accuracy. In and (Eds.), Information pooling and group decision making. Greenwich, CT: JAI Press, pp. 129–157.Google Scholar
(1982). Group vs. individual performance: Are N+1 heads better than one?Psychological Bulletin, 91, 517–539.CrossRefGoogle Scholar
(1992). Acquisition of expert judgment: Examples from risk assessment. Journal of Energy Engineering, 118, 136–148.CrossRefGoogle Scholar
(2004). Probability judgments for continuous quantitites: Linear combinations and calibration. Management Science, 50, 597–604.CrossRefGoogle Scholar
, , and (1993). The use of decomposition in probability assessments on continuous variables. Journal of Behavioral Decision Making, 6, 133–147.CrossRefGoogle Scholar
, and (1989). Expert opinion in risk analysis: The NUREG-1150 methodology. Nuclear Science and Engineering, 102, 323–331.CrossRefGoogle Scholar
, and (1996). Copula models for aggregating expert opinions. Operations Research, 44, 444–457.CrossRefGoogle Scholar
, and (1972). Subjective probability: A judgment of representativeness. Cognitive Psychology, 3, 430–454.CrossRefGoogle Scholar
, and (1989). On the uses of expert judgment on complex technical problems. IEEE Transactions on Engineering Management, 36, 83–86.CrossRefGoogle Scholar
, and (1991). Eliciting probabilities from experts in complex technical problems. IEEE Transactions on Engineering Management, 38, 191–201.CrossRefGoogle Scholar
, and (2001). Analysis of colorectal cancer screening regimens. Health Care Management Science, 4, 257–267.Google Scholar
, and (2006). Intuitions about combining opinions: Misappreciation of the averaging principle. Management Science, 52, 111–127.CrossRefGoogle Scholar
, , and (1986). The accuracy of combining judgmental and statistical forecasts. Management Science, 32, 1521–1532.CrossRefGoogle Scholar
(2005). Bayesian nonparametric combination of multivariate expert forecasts. Ph.D. dissertation, Duke University.Google Scholar
(1983). Reconciliation of probability distributions. Operations Research, 31, 866–880.CrossRefGoogle Scholar
Lindley, D. V. (1985). Reconciliation of discrete probability distributions. In , , , and (Eds.), Bayesian statistics 2. Amsterdam: North-Holland, pp. 375–390.Google Scholar
, , and (1998). Combining expert judgment by hierarchical modeling: An application to physician staffing. Management Science, 44, 149–161.CrossRefGoogle Scholar
, , and (1988). Structuring knowledge retrieval: An analysis of decomposing quantitative judgments. Organizational Behavior and Human Decision Processes, 42, 303–323.CrossRefGoogle Scholar
(1987). Quantifying judgmental uncertainty: Methodology, experience, and insights. IEEE Transactions on Systems, Man, and Cybernetics, 17, 741–752.CrossRefGoogle Scholar
, , and (2005). Analysis of correlated expert judgments from extended pairwise comparisons. Decision Analysis, 2, 17–29.CrossRefGoogle Scholar
, and (1990). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge, MA: Cambridge University Press.CrossRefGoogle Scholar
, and (1995). Subjective judgments by climate experts. Envirionmental Science and Technology, 29, 468–476.Google ScholarPubMed
(1977). Combining expert judgments: A Bayesian approach. Management Science, 23, 679–693.CrossRefGoogle Scholar
(1983). An axiomatic approach to expert resolution. Management Science, 29, 24–32.CrossRefGoogle Scholar
, , and (1987). A critique of current practice for the use of expert opinions in probabilistic risk assessment. Reliability Engineering and System Safety, 20, 63–85.CrossRefGoogle Scholar
, and (1974). Experience with forecasting univariate time series and the combination of forecasts. Journal of the Royal Statistical Society, Series A, 137, 131–149.CrossRefGoogle Scholar
, and (1992). Expert judgment in risk analysis and management: Process, context, and pitfalls. Risk Analysis, 12, 83–93.CrossRefGoogle ScholarPubMed
, , and (1988). The reliability of subjective probabilities obtained through decomposition. Management Science, 34, 186–199.CrossRefGoogle Scholar
(1994). Improving the accuracy of group judgment; A process intervention combining group facilitation, social judgment analysis, and information technology. Organizational Behavior and Human Decision Processes, 58, 246–270.CrossRefGoogle Scholar
(1979). Improving the quality of group judgment: Social judgment analysis and the Delphi technique. Organizational Behavior and Human Performance, 24, 73–92.CrossRefGoogle Scholar
, , and (1990). Combining forecasts: Operational adjustments to theoretically optimal rules. Management Science, 36, 1044–1056.CrossRefGoogle Scholar
(1978). Assessing probability with multiple individuals: Group interaction versus mathematical aggregation. Report No. 78–3, Social Science Research Institute, University of Southern California.Google Scholar
(1994). Improved framework for uncertainty analysis: Accounting for unsuspected errors. Risk Analysis, 14, 441–447.CrossRefGoogle Scholar
, , , and (1994). Quantifying the credibility of energy projections from trends in past data: The U.S. energy sector. Energy Policy, 22, 119–130.CrossRefGoogle Scholar
(1989). An examination of group process in judgmental forecasting. International Journal of Forecasting, 5, 171–178.CrossRefGoogle Scholar
, and (1975). Probability encoding in decision analysis. Management Science, 22, 340–358.CrossRefGoogle Scholar
(1972). Probabilistic forecasting: An experiment related to the stock market. Organizational Behavior and Human Performance, 8, 139–158.CrossRefGoogle Scholar
West, M. (1988). Modelling expert opinion. In , , , and (Eds.), Bayesian statistics, 3, Amsterdam: New Holland, pp. 493–508.Google Scholar
(1992). Modelling agent forecast distributions. Journal of the Royal Statistical Society, Series B, 54, 553–567.Google Scholar
, and (1992). Modelling probabilistic agent opinion. Journal of the Royal Statistical Society, Series B, 54, 285–299.Google Scholar
(1968). The consensus of subjective probability distributions. Management Science, 15, 361–375.CrossRefGoogle Scholar
(1981). Combining probability distributions from dependent information sources. Management Science, 27, 479–488.CrossRefGoogle Scholar
, and (1992). Sensitivity of weights in combining forecasts. Operations Research, 40, 609–614.CrossRefGoogle Scholar
, and (2004). Multiple experts vs. multiple methods: Combining correlation assessments. Decision Analysis, 1, 167–176.CrossRefGoogle Scholar
, and (1983). The combination of forecasts. Journal of the Royal Statistical Society, Series A, 146, 150–157.CrossRefGoogle Scholar
, and (1993). Evaluating and combining physicians' probabilities of survival in an intensive care unit. Management Science, 39, 1526–1543.CrossRefGoogle Scholar
, , , , , and (1995). An assessment of the risk of chronic lung injury attributable to long-term ozone exposure. Operations Research, 43, 19–28.CrossRefGoogle Scholar
, , and (1988). The consistency, coherence, and calibration of holistic, decomposed, and recomposed judgmental probability forecasts. Journal of Forecasting, 7, 185–199.CrossRefGoogle Scholar
, , and (1990). Uncertainties in system analysis: Probabilistic versus nonprobabilistic theories. Reliability Engineering and System Safety, 30, 163–181.CrossRefGoogle Scholar
Accessibility standard: Unknown
Why this information is here
This section outlines the accessibility features of this content - including support for screen readers, full keyboard navigation and high-contrast display options. This may not be relevant for you.Accessibility Information
Accessibility compliance for the PDF of this chapter is currently unknown
and may be updated in the future.
- 82
- Cited by