The combination of human forecasters’ subjective probability estimates usually improves upon the estimates provided by individual forecasters. In order to combine the probability estimates in a Bayes optimal way, prior work proposed a normative Bayesian fusion model that models the estimates with a beta distribution conditioned on their truth value. However, this model assumes conditionally independent probability estimates, although estimates provided by different forecasters are usually correlated. Here, we introduce a Bayesian model for combining subjective probability estimates that explicitly considers their correlation. We assume that an estimate provided by a forecaster for a given query depends on both the forecaster’s skill and the query’s difficulty. The correlation between probability estimates provided by different forecasters is assumed to be caused by the queries that make the forecasters provide similar estimates, for example, correct and highly confident estimates for very easy queries. Our model represents the probability estimates with a beta distribution conditioned on their truth value. It explicitly models the forecasters’ skills and the queries’ difficulties with skill parameters specific for each forecaster and difficulty parameters specific for each query. In this way, it can model the correlations between probability estimates and consider it when combining the estimates. Evaluations on a data set consisting of the subjective probability estimates of 85 human forecasters for 180 queries show improved fusion performance in terms of Brier score compared to related Bayesian fusion models. In particular, it outperforms independent fusion models that suffer from overconfidence.

Combining experts’ subjective probability estimates is a fundamental task with broad applicability in domains ranging from finance to public health. However, it is still an open question how to combine such estimates optimally. Since the beta distribution is a common choice for modeling uncertainty about probabilities, here we propose a family of normative Bayesian models for aggregating probability estimates based on beta distributions. We systematically derive and compare different variants, including hierarchical and non-hierarchical as well as asymmetric and symmetric beta fusion models. Using these models, we show how the beta calibration function naturally arises in this normative framework and how it is related to the widely used Linear-in-Log-Odds calibration function. For evaluation, we provide the new Knowledge Test Confidence data set consisting of subjective probability estimates of 85 forecasters on 180 queries. On this and another data set, we show that the hierarchical symmetric beta fusion model performs best of all beta fusion models and outperforms related Bayesian fusion models in terms of mean absolute error.