In the preceding sections we have frequently employed the saddle-point approximation (Gaussian approximation) to problems which are nonlinear in the parameters. Two requirements have been more or less tacitly assumed in doing so: the first assumption is that the true posterior resembles relatively closely a multivariate Gaussian. This is frequently the case, but of course not necessarily generally so. The second assumption is that the limits of integration and hence the support of the parameters is (—∞, ∞). This latter assumption can be relaxed to the requirement that the parameter support and position and width of the posterior are such that extending the limits of integration causes negligible approximation error to the integral. This assumption is, however, frequently not justified. Since analytic integration of a multivariate Gaussian within finite limits is not possible, the only solution to the problem is numerical integration. This does not sound like a complicated task, but we shall see in the sequel that things are different in many dimensions and frequently people discussing the topic are misled by a dimension fallacy.

The evaluation of multidimensional integrals of arbitrary posterior distributions, which would of course also include multimodal posteriors, can become rather complicated. For pedagogical reasons, we shall pursue a much simpler problem and limit the discussion to problems where a Gaussian approximation exists and discuss how to account by numerical integration for approximation errors introduced by differences between the true shape of the posterior and its Gaussian approximation and by finite limits of integration.

In previous chapters we have applied the Bayesian and the frequentist approach to some basic problems. The key differences of the two approaches are summarized in Table 20.1. The last two rows deserve additional remarks, which will be given in the following sections.

Prior knowledge is prior data

In scientific problems the present data are certainly not the only information that is known for the problem under consideration. Usually, there exists a wealth of knowledge in the form of previous experimental data and theoretical facts, such as positivity constraints, sum rules, asymptotic behaviour. A scientist is never in the situation that only the current data count. The Bayesian approach allows us to exploit all this information consistently. It has been criticized that priors are a subjective element of the theory. This is not really correct, as the Bayesian approach is internally consistent and deterministic. The only part that could be described as subjective is the knowledge that goes into the prior probabilities. But this degree of subjectivity is actually the foundation of all science. It is the expertise that exists in the respective discipline. The generation of experimental data is based on the same subjectivity, as it is generally motivated by prior knowledge in the form of previous data or theoretical models.

A great part of the prior knowledge is a summary of a conglomeration of previous measurements, with completely different meanings and sources of statistical errors.

The problem of estimating values of a function from a given set of data [yi, xi} is a generalization of the well-known and much simpler problem of interpolation. In interpolation we infer the values of a function f(x) which takes on the values yi = f(xi) at the pivotal points {xi} at arguments x between the pivots, xk ≤ x ≤ xk+1 ∀k. The interpolation problem becomes a function estimation problem if the data yi which are regarded as samples from the function f(x) at argument xi are deteriorated by noise. In this case we must abandon the requirement yi = f(xi) and require the function to pass through the given data {xi, yi} in some sensible optimal way. We distinguish two categories of function estimation. In the first category the function f(x) is a member of the class of functions f(x∣θ) parametrized by a set of parameters θ. The simplest of these curves is a straight line passing through the origin of the coordinate system whose single parameter is the slope. Parametric function estimation amounts in this case to the determination of the single parameter ‘slope’. This kind of function estimation is formally identical to the previously treated parameter estimation and regression. In this chapter we shall therefore only deal with the second category, nonparametric function estimation.