This chapter provides a brief presentation of the philosophical and math¬ematical foundations of Bayesian inference. The connections to classical statistical inference are also briefly discussed.

Philosophy of Bayesian inference

The purpose of Bayesian inference (Bernardo and Smith, 1994; Gelman et al., 2004) is to provide a mathematical machinery that can be used for modeling systems, where the uncertainties of the system are taken into account and the decisions are made according to rational principles. The tools of this machinery are the probability distributions and the rules of probability calculus.

If we compare the so-called frequentist philosophy of statistical analysis to Bayesian inference the difference is that in Bayesian inference the prob¬ability of an event does not mean the proportion of the event in an infinite number of trials, but the uncertainty of the event in a single trial. Because models in Bayesian inference are formulated in terms of probability dis¬tributions, the probability axioms and computation rules of the probability theory (see, e.g., Shiryaev, 1996) also apply in Bayesian inference.

Connection to maximum likelihood estimation

Consider a situation where we know the conditional distribution p(yk | θ) of conditionally independent random variables (measurements) y1:T = {y1,…, yT}, but the parameter θ ∊ ℝd is unknown.