Introduction

The additive models of Chapters 8 and 11 have many attractive features. The joint effect of all the predictor variables upon the response is expressed as a sum of individual effects. These individual effects show how the expected response varies as any single predictor varies with the others held fixed at arbitrary values; because of the additivity, the effect of one predictor does not depend on the values at which the others are fixed. Thus, the individual component functions can be plotted separately to visualize the effect of each predictor, and these functions – taken together – allow us to understand the joint effects of all the predictors upon the expected response. If, for example, we wish to find conditions under which the expected response is maximized, then we need only maximize separately each of the component functions of the additive model. In summary, it is extremely convenient whenever an additive model provides an accurate summary of the data.

However, there are no guarantees that an additive model will provide a satisfactory fit in any given situation. Nonadditivity means that, as one predictor is varied, the effect on the expected response depends on the fixed values of the other predictors. A deviation from additivity is called an interaction.

Absolutely right, James Clerk, as always. But which calculus of probabilities? At present, we have two, totally different, calculi: Classical and Quantum. For discussion of the relation between the two calculi, see Subsection 461M.

Please see my thanks to Feynman, Gruska, Isham, Nielsen and Chuang, Penrose, Preskill, …, in Appendix D.

In computer jargon, this chapter is a ‘several-pass’ account: we begin with vague ideas, and tighten these up in stages.

No previous knowledge of Quantum Theory is assumed. Except for a brief discussion of the Uncertainty Principle, we deal exclusively with comparatively easy finite-dimensional situations. We do behave with somewhat greater freedom in our use of Linear Algebra.

I ought to stress the following: Quantum Computing is by no means the most wonderful part of Quantum Theory. If Nature had not chosen to exploit in the design of spin-½ particles the fact that (as we shall see later) ‘SU(2) is the double cover of SO(3)’, then there would be no Chemistry, no Biology, no you!. To a large extent, we are using Quantum Computing as the simplest context in which to learn some Quantum Probability.

My hope is that by the time you finish this chapter, you will be better equipped to read critically the huge amount of ‘popular Science’ (some very good, a lot not so good) on the Uncertainty Principle, entanglement, etc, and that, more importantly, you will be persuaded to learn more from the real experts.

Probability and Statistics used to be married; then they separated; then they got divorced; now they hardly ever see each other. (That's a mischievous first sentence, but there is more than an element of truth in it, what with the subjects' having different journals and with Theoretical Probability's sadly being regarded by many – both mathematicians and statisticians – as merely a branch of Analysis.) In part, this book is a move towards much-needed reconciliation. It is written at a difficult time when Statistics is being profoundly changed by the computer revolution, Probability much less so.

This book has many unusual features, as you will see later and as you may suspect already! Above all, it is meant to be fun. I hope that in the Probability you will enjoy the challenge of some ‘almost paradoxical’ things and the important applications to Genetics, to Bayesian filtering, to Poisson processes, etc. The real-world importance of Statistics is self-evident, and I hope to convey to you that subject's very considerable intrinsic interest, something too often underrated by mathematicians. The challenges of Statistics are somewhat different in kind from those in Probability, but are just as substantial.

(a) For whom is the book written? There are different answers to this question. The book derives from courses given at Bath and at Cambridge, and an explanation of the situation at those universities identifies two of the possible (and quite different) types of reader.

In this chapter, we look briefly at three further topics from Probability: Conditional Expectations (CEs); Martingales; Poisson Processes. The first of these leads directly on to the second. The third is somewhat different.

First, we look at the conditional expectation E(X|A) of a Random Variable X given that event A occurs, and at the decomposition

We used these ideas in solving the ‘average wait for patterns’ problems in Chapter 1.

We then examine one of the central breakthroughs in Probability, the definition (due in its most general form largely to Kolmogorov) of E(X|Y), the Random Variable which is the conditional expectation of X given the Random Variable Y. This is a rather subtle concept, for which we shall later see several motivating examples. To get the idea: if X and Y are the scores on two successive throws of a fair die, then E(x + Y|Y) = 3½ + Y. Here, E(X|Y) = E(X) = 3½because, since X is independent of Y, knowing the value of Y does not change our view of the expectation of X. Of course, E(Y|Y) = Y because ‘if we are given Y, then we know what Y is’, something which needs considerable clarification!

Oa. Exercise. What is E(Y|X + Y) for this example? (We return to this shortly, but common sense should allow you to write down the answer even without knowing how things are defined.)

Please, do read the Preface first!

Note. Capitalized ‘Probability’ and ‘Statistics’ (when they do not start a sentence) here refer to the subjects. Thus, Probability is the study of probabilities, and Statistics that of statistics. (Later, we shall also use ‘Statistics’ as opposed to ‘statistics’ in a different, technical, way.) ‘Maths’ is English for ‘Math’, ‘Mathematics’ made friendly.

Conditioning: an intuitive first look

One of the main aims of this book is to teach you to ‘condition’: to use conditional probabilities and conditional expectations effectively. Conditioning is an extremely powerful and versatile technique.

In this section and (more especially) the next, I want you to give your intuition free rein, using common sense and not worrying over much about rigour. All of the ideas in these sections will appear later in a more formal setting. Indeed, because fudging issues is not an aid to clear understanding, we shall study things with a level of precision very rare for books at this level; but that is for later. For now, we use common sense, and though we would later regard some of the things in these first two sections as ‘too vague’, ‘non-rigorous’, etc, I think it is right to begin the book as I do. Intuition is much more important than rigour, though we do need to know how to back up intuition with rigour.

We start again ‘from scratch’, developing Probability in the only order that logic allows.

This chapter is important, but not too exciting (except in discussion of what is perhaps the most brilliant piece of magical trickery in Maths): important in that it covers the Addition Rule; not too exciting in the main because it does not go beyond the Addition Rule to the Multiplication Rules involving conditioning and independence. Without those Multiplication Rules, we can assign actual probabilities to events only in an extremely limited set of cases.

As mentioned at Discussion 25B, we are concerned with constructing a Mathematical Model for a Real-World Experiment: we make no attempt to define probability in the real world because such an attempt is doomed. In the mathematical model, probability ℙ is just a function which assigns numbers to events in a manner consistent with the Addition Rule (and with conditional-probability considerations) – and nothing more. And events are just (technically, ‘measurable’) subsets of a certain set, always denoted by Ω. This set is called the sample space and it represents the set of all possible outcomes ω of our experiment. In Statistics, the experiment has been performed, and we have some information – in many cases, all information – about the actual outcome ωact, the particular point of Ω actually ‘realized’.

I have deferred the topic of joint and conditional pdfs for as long as possible. Since we are now equipped to appreciate at least some of its usefulness, its study is now much more interesting than it would have been earlier. The fundamental ‘F’ and ‘t’ distributions of Statistics are introduced in this chapter. Bayesian Statistics is extended to cover multi-parameter situations, and we see how it may be made effective by Gibbs sampling.

One of the reasons for my deferring the topic of joint pdfs is that it can look a bit complicated at first sight. It isn't really. Amongst the theory as it applies to Probability and Statistics, only the ‘Jacobian’ Theorem 249C has any substance; and you can, if you like, take that for granted. (Gibbs sampling certainly has substance too!) What is a little offputting is the notation. Wherever possible, I simplify the appearance of things by using vector notation.

Note. We defer the study of the most important joint pdf, that of the multivariate normal distribution, until Chapter 8.

We start with the simple situation for joint pmfs, which proves a valuable guide to that for joint pdfs.

ORIENTATION. Sums of IID RVs are of central importance in both Probability and Statistics. Pmfs and pdfs are not that easy to use: for example it is already rather tricky to find the pdf of the sum of n IID RVs each with the uniform U [0,1] distribution. To handle sums of IID RVs effectively, we need new methods; and generating functions provide these.

The main result in this chapter is the miraculous Central Limit Theorem (CLT) which says that the sum of a large number of IID RVs each with ANY distribution of finite variance has approximately a normal distribution. The normal distribution is therefore very special in this sense: we discover why. (We shall see later that the normal distribution is very special in several other ways, too.) There are at least three nice methods of proving the CLT, the first by generating functions, the second by using the fact that one can find the sequence of sums of IID Variables within a ‘Brownian-motion’ process, and the third (not quite so general) by using the maximum-entropy property of the normal distribution. We take a heuristic look at the first of these. Section I.8 of Volume 1 of Rogers & Williams [199] describes the second method: it requires much more background material. For hints on the maximum-entropy method, see 198H below. Historical note: The first case of the CLT was proved by de Moivre in 1706!

Some results in this chapter are too difficult to prove in full at this level; but precise statements (and, sometimes, sketched proofs) of the results are given.

These days, there are so many brilliant young (and youngish) probabilists and statisticians around that I am not going to single any out. Tributes to people who helped develop the fields are more ‘classical’. But, of course, many of the books mentioned are modern.

The Bibliography is fairly extensive, though still a small fraction of what it could/should have been. For everyone, there are many books in the Bibliography of great interest. And for those determined to strive, to seek, to find, but not to yield, the Bibliography contains – for inspiration – some books and papers (several of which were mentioned in the main text) which are at a significantly more advanced level than this book.

Probability

In the early days, Probability Theory was developed by Cardano, Galileo, Pascal, de Moivre, two Bernoullis, Bayes, Laplace, Gauss, Markov, Tchebychev. In the 20th century, the Measure Theory of Borel and Lebesgue allowed Borel, Wiener, Kolmogorov [140], Doob [66] and others to put Probability on a rigorous basis and to extend its scope dramatically. Lévy [151, 152] provided much of the intuition and inspiration.

Volume 1 of Feller's book [75], a little more challenging than this present one, will always hold a special place in the hearts of probabilists. It is quite challenging, and should perhaps be read after, or in conjunction with, this one. Pitman [184] is a good route-in as prerequisite for the Probability in this book, should you need one.