In the past it may have been true that Stephen Senn's (2003) analogy was right. Paraphrasing his words for the sake of moderate language: scientists regarded statistics as the one-night stand: the quick fix, avoiding long-term entanglement. This analogy is no longer apt. Statistical procedures now drive many if not most areas of current astrophysics and cosmology. In particular the currently understood nature of our Universe is a product of statistical analysis of large and combined data sets. Here we briefly describe the scene in three areas dominating definition of the current model of the Universe and its history. The three areas inextricably tie together the shape and content of the Universe and the formation of structure and galaxies, leading to life as we know it. While these sketches are not reviews, we show by cross-referencing how frequently our preceding discussions play in to current research in cosmology.

The galaxy universe

The story of galaxy formation since 1990 is based on two premises. Firstly, it was widely accepted that the matter content in the Universe is primarily cold and dark – CDM prevails. The recognition of dark matter was slow, despite Zwicky (1937) demonstrating its existence via the cosmic virial theorem. The measurements of rotation curves of spiral galaxies (e.g. Rubin et al., 1980) convinced us.

Peter Scheuer started this. In 1977 hewalked into JVW's office in the Cavendish Lab and quietly asked for advice on what further material should be taught to the new intake of Radio Astronomy graduate students (that year including the hapless CRJ). JVW, wrestling with simple Chi-square testing at the time, blurted out ‘They know nothing about practical statistics …’. Peter left thoughtfully. A day later he returned. ‘Good news! The Management Board has decided that the students are going to have a course on practical statistics.’ Can I sit in, JVW asked innocently. ‘Better news! The Management Board has decided that you're going to teach it …’.

So, for us, began the notion of practical statistics. A subject that began with gambling is not an arcane academic pursuit, but it is certainly subtle as well. It is fitting that Peter Scheuer was involved at the beginning of this (lengthy) project; his style of science exemplified both subtlety and pragmatism. We hope that we can convey something of both. If an echo of Peter's booming laugh is sometimes heard in these pages, it is because we both learned from him that a useful answer is often much easier – and certainly much more entertaining – than you at first think.

After the initial course, the material for this book grew out of various further courses, journal articles and the abundant personal experience that results from understanding just a little of any field of knowledge that counts Gauss and Laplace amongst its originators.

Teaching is highly educational for teachers. Teaching from the first edition revealed to us how much students enjoyed Monte Carlo methods, and the ability with such methods to test and to check every derivation, test, procedure or result in the book. Thus, a change in the second edition is to introduce Monte Carlo as early as possible (Chapter 2). Teaching also revealed to us areas in which we assumed too much (and too little). We have therefore aimed for some smoothing of learning gradients where slope changes have appeared to be too sudden. Chapters 6 and 7 substantially amplify our previous treatments of Bayesian hypothesis testing/modelling, and include much more on model choice and Markov chain Monte Carlo (MCMC) analysis. Our previous chapter on 2D (sky distribution) analysis has been significantly revised. We have added a final chapter sketching the application of statistics to some current areas of astrophysics and cosmology, including galaxy formation and large-scale structure, weak gravitational lensing, and the cosmological microwave background (CMB) radiation.

We received very helpful comments from anonymous referees whom CUP consulted about our proposals for the second edition. These reviewers requested that we keep the book (a) practical and (b) concise and – small, or ‘backpackable’, as one of them put it. We have additional colleagues to thank either for further discussions, finding errata or because we just plain missed them from our first edition list: Matthew Colless, Jim Condon, Mike Disney, Alan Heavens, Martin Hendry, Jim Moran, Douglas Scott, Robert Smith and Malte Tewes.

One of the attractive features of the Bayesian method is that it offers a principled way of making choices between models. In classical statistics, we may fit to a model, say by least squares, and then use the resulting χ2 statistic to decide if we should reject the model. We would do this if the deviations from the model are unlikely to have occurred by chance. However, it is not clear what to do if the deviations are likely to have occurred, and it is even less clear what to do if several models are available. For example, if a model is in fact correct, the significance level derived from a χ2 test (or, indeed, any significance test) will be uniformly distributed between zero and one (Exercise 7.1).

The problem with model choice by χ2 (or any similar classical method) is that these methods do not answer the question we wish to ask. For a model H and data D, a significance level derived from a minimum χ2 tells us about the conditional probability, prob(D | H).

In contrast to previous chapters, we now consider data transformation, how to transform data in order to produce improved outcomes in either extracting or enhancing signal.

There are many observations consisting of sequential data, such as intensity as a function of position as a radio telescope is scanned across the sky or as signal varies across a row on a CCD detector, single-slit spectra, time-measurements of intensity (or any other property). What sort of issues might concern us?

1. (i) trend-finding; can we predict the future behaviour of data?

2. (ii) baseline detection and/or assessment, so that signal on this baseline can be analysed;

3. (iii) signal detection, identification, for example, of a spectral line or source in sequential data for which the noise may be comparable in magnitude to the signal;

4. (iv) filtering to improve signal-to-noise ratio;

5. (v) quantifying the noise;

6. (vi) period-finding; searching the data for periodicities;

7. (vii) correlation of time series to find correlated signal between antenna pairs or to find spectral lines;

8. (viii) modelling; many astronomical systems give us our data convolved with some more or less known instrumental function, and we need to take this into account to get back to the true data.

Science is about decision. Building instruments, collecting data, reducing data, compiling catalogues, classifying, doing theory – all of these are tools, techniques or aspects which are necessary. But we are not doing science unless we are deciding something; only decision counts. Is this hypothesis or theory correct? If not, why not? Are these data self-consistent or consistent with other data? Adequate to answer the question posed? What further experiments do they suggest?

We decide by comparing. We compare by describing properties of an object or sample, because lists of numbers or images do not present us with immediate results enabling us to decide anything. Is the faint smudge on an image a star or a galaxy? We characterize its shape, crudely perhaps, by a property, say the full-width half-maximum, the FWHM, which we compare with the FWHM of the point-spread function. We have represented a data set, the image of the object, by a statistic, and in so doing we reach a decision.

Statistics are there for decision and because we know a background against which to take a decision.

In embarking on statistics we are entering a vast area, enormously developed for the Gaussian distribution in particular. This is classical territory; historically, statistics were developed because the approach now called Bayesian had fallen out of favour. Hence, direct probabilistic inferences were superseded by the indirect and conceptually different route, going through statistics and intimately linked to hypothesis testing. The use of statistics is not particularly easy. The alternatives to Bayes' methods are subtle and not very obvious; they are also associated with some fairly formidable mathematical machinery. We will avoid this, presenting only results and showing the use of statistics, while trying to make clear the conceptual foundations.

Statistics

Statistics are designed to summarize, reduce or describe data. The formal definition of a statistic is that it is some function of the data alone. For a set of data X1, X2, …, some examples of statistics might be the average, the maximum value or the average of the cosines. Statistics are therefore combinations of finite amounts of data. In the following discussion, and indeed throughout, we try to distinguish particular fixed values of the data, and functions of the data alone, by upper case (except for Greek letters). Possible values, being variables, we will denote in the usual algebraic spirit by lower case.

It is often the case that we need to do sample comparison: we have someone else's data to compare with ours; or someone else's model to compare with our data; or even our data to compare with our model. We need to make the comparison and to decide something. We are doing hypothesis testing – are our data consistent with a model, with somebody else's data? In searching for correlations as we were in Chapter 4, we were hypothesis testing; in the model-fitting of Chapter 6 we are involved in data modelling and parameter estimation.

A frequentist point of view might be to consider the entire science of statistical inference as hypothesis testing followed by parameter estimation. However, if experiments were properly designed, the Bayesian approach would be right: it answers the sample-comparison questions we wished to pose in the first place, namely what is the probability, given the data, that a particular model is right? Or: what is the probability, given two sets of data, that they agree? The two-stage process should be unecessary at best.

When we make a set of measurements, it is instinct to try to correlate the observations with other results. One or more motives may be involved in this instinct. For instance we might wish (a) to check that other observers' measurements are reasonable, (b) to check that our measurements are reasonable, (c) to test a hypothesis, perhaps one for which the observations were explicitly made, or (d) in the absence of any hypothesis, any knowledge or anything better to do with the data, to find if they are correlated with other results in the hope of discovering some new and universal truth.

The fishing trip

Take the last point first. Suppose that we have plotted something against something, on a fishing expedition of this type. There are grave dangers on this expedition, and we must ask ourselves the following questions.

1. Does the eye see much correlation? If not, calculation of a formal correlation statistic is probably a waste of time.

2. Could the apparent correlation be due to selection effects? Consider, for instance, the beautiful correlation in Figure 4.1, in which Sandage (1972) plotted radio luminosities of sources in the 3CR catalogue as a function of distance modulus. […]

The distribution of objects on the celestial sphere, or on an imaged patch of this sphere, has ever been a major preoccupation of astronomers. Avoiding here the science of image processing, province of thousands of books and papers, we consider some of the common statistical approaches used to quantify sky distributions in order to permit contact with theory. Before we turn to the adopted statistical weaponry of galaxy distribution, we discuss some general statistics applicable to the spherical surface.

Statistics on a spherical surface

The distribution of objects on the celestial sphere is the distribution of directions of a set of unit vectors. Many other 3D spaces face similar issues of distribution, such as the Poincaré sphere with unit vectors indicating the state of polarization of radiation. Geophysical topics (orientation of paeleomagnetism, for instance) motivate much analysis.

Thus, this is a thriving sub-field of statistics and there is an excellent handbook (Fisher et al., 1987). The emphasis is on statistical modelling and a variety of distributions is available.

Whether He does or not, the concepts of probability are important in astronomy for two reasons.

1. Astronomical measurements are subject to random measurement error, perhaps more so than most physical sciences because of our inability to re-run experiments and our perpetual wish to observe at the extreme limit of instrumental capability. We have to express these errors as precisely and usefully as we can. Thus, when we say ‘an interval of 10-6 units, centred on the measured mass of the Moon, has a 95 per cent chance of containing the true value’, it is a much more quantitative statement than ‘the mass of the Moon is 1 ± 10-6 units’. The second statement really only means anything because of some unspoken assumption about the distribution of errors. Knowing the error distribution allows us to assign a probability, or measure of confidence, to the answer.

2. The inability to do experiments on our subject matter leads us to draw conclusions by contrasting properties of controlled samples. These samples are often small and subject to uncertainty in the same way that a Gallup poll is subject to ‘sampling error’. In astronomy we draw conclusions such as: ‘the distributions of luminosity in X-ray-selected Type I and Type II objects differ at the 95 per cent level of significance.’ Very often the strength of this conclusion is dominated by the number of objects in the sample and is virtually unaffected by observational error.

Of the vast literature, we point to some works which we have found useful, enlightening or just plain entertaining. We bin these into six types (somewhat arbitrarily as there is much overlap): popular, the basic text, the rigorous text, the data analysis manual, the texts considering statistical packages, and the statistics treatments of specialist interest to astronomers.

1. The classic popular books have legendary titles: How to Lie with Statistics (Huff, 1973), Facts from Figures (Moroney, 1965), Statistics in Action (Sprent, 1977) and Statistics without Tears (Rowntree, 1981). They are all fun. To this list we can now add The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century (Salsburg, 2002), an entertaining exposition of the development of modern statistics; Struck by Lightning: the Curious World of Probabilities (Rosenthal, 2006); Making Sense of Statistics: A Non-mathematical Approach (Wood, 2003), and Dicing with Death: Chance, Risk and Health (Senn, 2003). This latter is a devastatingly blunt, funny and erudite exposition of the importance and application of statistics in decision processes which may affect the lives of millions. As a popular book it is heavy-going in parts; but for scientists, budding or mature, it is a rewarding read.

2. Textbooks come in types (a) and (b), both of which cover similar material for the first two-thirds of each book. They start with descriptive or summarizing statistics (mean, standard deviation), the distributions of these statistics, and move to the concept of probability and hence statistical inference and hypothesis testing, including correlation of two variables. […]