In the past it may have been true that Stephen Senn's (2003) analogy was right. Paraphrasing his words for the sake of moderate language: scientists regarded statistics as the one-night stand: the quick fix, avoiding long-term entanglement. This analogy is no longer apt. Statistical procedures now drive many if not most areas of current astrophysics and cosmology. In particular the currently understood nature of our Universe is a product of statistical analysis of large and combined data sets. Here we briefly describe the scene in three areas dominating definition of the current model of the Universe and its history. The three areas inextricably tie together the shape and content of the Universe and the formation of structure and galaxies, leading to life as we know it. While these sketches are not reviews, we show by cross-referencing how frequently our preceding discussions play in to current research in cosmology.

The galaxy universe

The story of galaxy formation since 1990 is based on two premises. Firstly, it was widely accepted that the matter content in the Universe is primarily cold and dark – CDM prevails. The recognition of dark matter was slow, despite Zwicky (1937) demonstrating its existence via the cosmic virial theorem. The measurements of rotation curves of spiral galaxies (e.g. Rubin et al., 1980) convinced us.

Peter Scheuer started this. In 1977 hewalked into JVW's office in the Cavendish Lab and quietly asked for advice on what further material should be taught to the new intake of Radio Astronomy graduate students (that year including the hapless CRJ). JVW, wrestling with simple Chi-square testing at the time, blurted out ‘They know nothing about practical statistics …’. Peter left thoughtfully. A day later he returned. ‘Good news! The Management Board has decided that the students are going to have a course on practical statistics.’ Can I sit in, JVW asked innocently. ‘Better news! The Management Board has decided that you're going to teach it …’.

So, for us, began the notion of practical statistics. A subject that began with gambling is not an arcane academic pursuit, but it is certainly subtle as well. It is fitting that Peter Scheuer was involved at the beginning of this (lengthy) project; his style of science exemplified both subtlety and pragmatism. We hope that we can convey something of both. If an echo of Peter's booming laugh is sometimes heard in these pages, it is because we both learned from him that a useful answer is often much easier – and certainly much more entertaining – than you at first think.

After the initial course, the material for this book grew out of various further courses, journal articles and the abundant personal experience that results from understanding just a little of any field of knowledge that counts Gauss and Laplace amongst its originators.

Teaching is highly educational for teachers. Teaching from the first edition revealed to us how much students enjoyed Monte Carlo methods, and the ability with such methods to test and to check every derivation, test, procedure or result in the book. Thus, a change in the second edition is to introduce Monte Carlo as early as possible (Chapter 2). Teaching also revealed to us areas in which we assumed too much (and too little). We have therefore aimed for some smoothing of learning gradients where slope changes have appeared to be too sudden. Chapters 6 and 7 substantially amplify our previous treatments of Bayesian hypothesis testing/modelling, and include much more on model choice and Markov chain Monte Carlo (MCMC) analysis. Our previous chapter on 2D (sky distribution) analysis has been significantly revised. We have added a final chapter sketching the application of statistics to some current areas of astrophysics and cosmology, including galaxy formation and large-scale structure, weak gravitational lensing, and the cosmological microwave background (CMB) radiation.

We received very helpful comments from anonymous referees whom CUP consulted about our proposals for the second edition. These reviewers requested that we keep the book (a) practical and (b) concise and – small, or ‘backpackable’, as one of them put it. We have additional colleagues to thank either for further discussions, finding errata or because we just plain missed them from our first edition list: Matthew Colless, Jim Condon, Mike Disney, Alan Heavens, Martin Hendry, Jim Moran, Douglas Scott, Robert Smith and Malte Tewes.

One of the attractive features of the Bayesian method is that it offers a principled way of making choices between models. In classical statistics, we may fit to a model, say by least squares, and then use the resulting χ2 statistic to decide if we should reject the model. We would do this if the deviations from the model are unlikely to have occurred by chance. However, it is not clear what to do if the deviations are likely to have occurred, and it is even less clear what to do if several models are available. For example, if a model is in fact correct, the significance level derived from a χ2 test (or, indeed, any significance test) will be uniformly distributed between zero and one (Exercise 7.1).

The problem with model choice by χ2 (or any similar classical method) is that these methods do not answer the question we wish to ask. For a model H and data D, a significance level derived from a minimum χ2 tells us about the conditional probability, prob(D | H).

In contrast to previous chapters, we now consider data transformation, how to transform data in order to produce improved outcomes in either extracting or enhancing signal.

There are many observations consisting of sequential data, such as intensity as a function of position as a radio telescope is scanned across the sky or as signal varies across a row on a CCD detector, single-slit spectra, time-measurements of intensity (or any other property). What sort of issues might concern us?

1. (i) trend-finding; can we predict the future behaviour of data?

2. (ii) baseline detection and/or assessment, so that signal on this baseline can be analysed;

3. (iii) signal detection, identification, for example, of a spectral line or source in sequential data for which the noise may be comparable in magnitude to the signal;

4. (iv) filtering to improve signal-to-noise ratio;

5. (v) quantifying the noise;

6. (vi) period-finding; searching the data for periodicities;

7. (vii) correlation of time series to find correlated signal between antenna pairs or to find spectral lines;

8. (viii) modelling; many astronomical systems give us our data convolved with some more or less known instrumental function, and we need to take this into account to get back to the true data.

Science is about decision. Building instruments, collecting data, reducing data, compiling catalogues, classifying, doing theory – all of these are tools, techniques or aspects which are necessary. But we are not doing science unless we are deciding something; only decision counts. Is this hypothesis or theory correct? If not, why not? Are these data self-consistent or consistent with other data? Adequate to answer the question posed? What further experiments do they suggest?

We decide by comparing. We compare by describing properties of an object or sample, because lists of numbers or images do not present us with immediate results enabling us to decide anything. Is the faint smudge on an image a star or a galaxy? We characterize its shape, crudely perhaps, by a property, say the full-width half-maximum, the FWHM, which we compare with the FWHM of the point-spread function. We have represented a data set, the image of the object, by a statistic, and in so doing we reach a decision.

Statistics are there for decision and because we know a background against which to take a decision.