Milgrom has proposed that the appearance of discrepancies between the Newtonian dynamical mass and the directly observable mass in astronomical systems could be due to a breakdown of Newtonian dynamics in the limit of low accelerations rather than the presence of unseen matter. Milgrom's hypothesis, modified Newtonian dynamics or MOND, has been remarkably successful in explaining systematic properties of spiral and elliptical galaxies and predicting in detail the observed rotation curves of spiral galaxies with only one additional parameter—a critical acceleration which is on the order of the cosmologically interesting value of CH〪. Here I review the empirical successes of this idea and discuss its possible extention to cosmology and structure formation.

Introduction

Modified Newtonian dynamics (MOND) is an ad hoc modification of Newton's law of gravity or inertia proposed by Milgrom (1983) as an alternative to cosmic dark matter. The motivation for this and other such proposals is obvious: So long as the only evidence for dark matter is its global gravitational effect, then its presumed exitance is not independent of the assumed form of the law of gravity or inertia on astronomical scales. In other words, either the universe contains large quantities of unseen matter, or gravity (or the response of particles to gravity) is not generally the same as it appears to be in the solar system.

A close scrutiny of the microlensing results towards the Magellanic clouds reveals that the stars within the Magellanic clouds are major contributors as lenses, and the contribution of MACHOs to dark matter is 0 to 5%. The principal results which lead to this conclusion are the following:

1. (i) Out of the ∼17 events detected so far towards the Magellanic Clouds, the lens location has been securely determined for one binary-lens event through its caustic-crossing timescale. In this case, the lens was found to be within the Magellanic Clouds. Although less certain, lens locations have been determined for three other events and in each of these three events, the lens is most likely within the Magellanic clouds.

2. (ii) If most of the lenses are MACHOs in the Galactic halo, the timescales would imply that the MACHOs in the direction of the LMC have masses of the order of 0.5 M⊙, and the MACHOs in the direction of the SMC have masses of the order of 2 to 3 M⊙. This is inconsistent with even the most flattened model of the Galaxy. If, on the other hand, they are caused by stars within the Magellanic Clouds, the masses of the stars are of the order of 0.2 M⊙ for both the LMC as well as the SMC.

3. (iii) If 50% of the lenses are in binary systems similar to the stars in the solar neighborhood, ∼10% of the events are expected to show binary characteristics.

4. […]

There are now two cosmological constant problems: (i) why the vacuum energy is so small and (ii) why it comes to dominate at about the epoch of galaxy formation. Anthropic selection appears to be the only approach that can naturally resolve both problems. This approach presents some challenges to particle physics models.

The problems

Until recently, there was only one cosmological constant problem and hardly any solutions. Now, within the scope of a few years, we have made progress on both accounts. We now have two cosmological constant problems (CCPs) and a number of proposed solutions. In this talk I am going to review the situation, focusing mainly on the anthropic approach and on its implications for particle physics models. I realize that the anthropic approach has a low approval rating among physicists. But I think its bad reputation is largely undeserved. When properly used, this approach is quantitative and has no mystical overtones that are often attributed to it. Moreover, at present this appears to be the only approach that can solve both CCPs. I will also comment on other approaches to the problems.

The cosmological constant is (up to a factor) the vacuum energy density, ρv.

For physicists, recent developments in astrophysics and cosmology present exciting challenges. We are conducting “experiments” in energy regimes some of which will be probed by accelerators in the near future, and others which are inevitably the subject of more speculative theoretical investigations. Dark matter is an area where we have hope of making discoveries both with accelerator experiments and dedicated searches. Inflation and dark energy lie in regimes where presently our only hope for a fundamental understanding lies in string theory.

Introduction

It is a truism that the development of astronomy, astrophysics, cosmology relies on our understanding of the relevant laws of physics. It is thus no surprise that my astronomy colleagues tend to know more classical mechanics, electricity and magnetism, atomic and nuclear physics than my colleagues in particle theory.

As we consider many of the questions which we now face in cosmology, we must confront the fact that we simply do not know the relevant laws of nature. The public often asks us “What came before the Big Bang?” We usually think of this as requiring understanding of physics at the Planck scale. But at present we can't even come close. Ignorance sets in slightly above nucleosynthesis, and becomes severe by the time we reach the weak scale. Some of the questions which trouble us will be settled by experiment over the next decades; some require new theoretical developments. Needless to say, it is possible that much will remain obscure for a long time.

Recent measurements of temperature fluctuations in the cosmic microwave background (CMB) indicate that the Universe is flat and that large-scale structure grew via gravitational infall from primordial adiabatic perturbations. Both of these observations seem to indicate that we are on the right track with inflation. But what is the new physics responsible for inflation? This question can be answered with observations of the polarization of the CMB. Inflation predicts robustly the existence of a stochastic background of cosmological gravitational waves with an amplitude proportional to the square of the energy scale of inflation. This gravitational-wave background induces a unique signature in the polarization of the CMB. If inflation took place at an energy scale much smaller than that of grand unification, then the signal will be too small to be detectable. However, if inflation had something to do with grand unification or Planckscale physics, then the signal is conceivably detectable in the optimistic case by the Planck satellite, or if not, then by a dedicated post-Planck CMB polarization experiment. Realistic developments in detector technology as well as a proper scan strategy could produce such a post-Planck experiment that would improve on Planck's sensitivity to the gravitational-wave background by several orders of magnitude in a decade timescale.

The simplest models for the formation of large-scale structure are reviewed. On the assumption that the dark matter is cold and collisionless, LSS data are able to measure the total amount of matter, together with the baryon fraction and the spectral index of primordial fluctuations. There are degeneracies between these parameters, but these are broken by the addition of extra information such as CMB fluctuation data. The CDM models are confronted with recent data, especially the 2dF Galaxy Redshift Survey, which was the first to measure more than 100,000 redshifts. The 2dFGRS power spectrum is measured to ≲ 10% accuracy for k > 0.02 h Mpc–1, and is well fitted by a CDM model with Ωmh = 0.20 ± 0.03 and a baryon fraction of 0.15 ± 0.07. In combination with CMB data, a flat universe with Ωm ⋍ 0.3 is strongly favored. In order to use LSS data in this way, an understanding of galaxy bias is required. A recent approach to bias, known as the ‘halo model’ allows important insights into this phenomenon, and gives a calculation of the extent to which bias can depend on scale.

Structure formation in the CDM model

The origin and formation of large-scale structure in cosmology is a key problem that has generated much work over the years. Out of all the models that have been proposed, this talk concentrates on the simplest: gravitational instability of small initial density fluctuations.

In contrast to previous chapters, we now consider data transformation, how to transform data in order to produce better statistics, either to extract signal or to enhance signal.

There are many observations consisting of sequential data, such as intensity as a function of position as a radio telescope is scanned across the sky or as signal varies across a row on a CCD detector, single-slit spectra, time-measurements of intensity (or any other property). What sort of issues might concern us?

1. baseline detection and/or assessment, so that signal on this baseline can be analysed;

2. signal detection, identification for example of a spectral line or source in sequential data for which the noise may be comparable in magnitude to the signal;

3. filtering to improve signal-to-noise ratio;

4. quantifying the noise;

5. period-finding; searching the data for periodicities;

6. trend-finding; can we predict the future behaviour of subsequent data?

7. correlation of time series to find correlated signal between antenna pairs or to find spectral lines;

8. modelling; many astronomical systems give us our data convolved with some more-or-less known instrumental function, and we need to take this into account to get back to the true data.

The distinctive aspect of these types of analysis is that the feature of interest only emerges after a transformation.

‘Detection’ is one of the commonest words in the practising astronomers' vocabulary. It is the preliminary to much else that happens in astronomy, whether it means locating a spectral line, a faint star or a gamma-ray burst. Indeed of its wide range of meanings, here we take the location, and confident measurement, of some sort of feature in a fixed region of an image or spectrum.

When a detection is obvious to even the most sceptical referee, statistical questions usually do not arise in the first instance. The parameters that result from such a detection have a signal-to-noise ratio so high that the detection finds its way into the literature as fact. However, elusive objects or features at the limit of detectability tend to become the focus of interest in any branch of astronomy. Then, the notion of detection (and non-detection) requires careful examination and definition.

Non-detections are especially important because they define how representative any catalogue of objects may be. This set of non-detections can represent vital information in deducing the properties of a population of objects; if something is never detected, that too is a fact, and can be exploited statistically. Every observation potentially contains information.

Science is about decision. Building instruments, collecting data, reducing data, compiling catalogues, classifying, doing theory – all of these are tools, techniques or aspects which are necessary. But we are not doing science unless we are deciding something; only decision counts. Is this hypothesis or theory correct? If not, why not? Are these data self-consistent or consistent with other data? Adequate to answer the question posed? What further experiments do they suggest?

We decide by comparing. We compare by describing properties of an object or sample, because lists of numbers or images do not present us with immediate results enabling us to decide anything. Is the faint smudge on an image a star or a galaxy? We characterize its shape, crudely perhaps, by a property, say the full-width half-maximum, the FWHM, which we compare with the FWHM of the point-spread function. We have represented a dataset, the image of the object, by a statistic, and in so doing we reach a decision.

Statistics are there for decision and because we know a background against which to take a decision. To this end, every measurement we make, and every parameter or value we derive, requires an error estimate, a measure of range (expressed in terms of probability) that encompasses our belief of the true value of the parameter. We are taught this by our masters in the course of interminable undergrad lab experiments.

Whether He does or not, the concepts of probability are important in astronomy for two reasons.

1. Astronomical measurements are subject to random measurement error, perhaps more so than most physical sciences because of our inability to rerun experiments and our perpetual wish to observe at the extreme limit of instrumental capability. We have to express these errors as precisely and usefully as we can. Thus when we say ‘an interval of 10−6 units, centred on the measured mass of the Moon, has a 95 per cent chance of containing the true value’, it is a much more quantitative statement than ‘the mass of the Moon is 1±10−6 units’. The second statement really only means anything because of some unspoken assumption about the distribution of errors. Knowing the error distribution allows us to assign a probability, or measure of confidence, to the answer.

2. The inability to do experiments on our subject matter leads us to draw conclusions by contrasting properties of controlled samples. These samples are often small and subject to uncertainty in the same way that a Gallup poll is subject to ‘sampling error’. In astronomy we draw conclusions such as ‘the distributions of luminosity in X-ray-selected Type I and Type II objects differ at the 95 per cent level of significance’. Very often the strength of this conclusion is dominated by the number of objects in the sample and is virtually unaffected by observational error.

In embarking on statistics we are entering a vast area, enormously developed for the Gaussian distribution in particular. This is classical territory; historically, statistics were developed because the approach now called Bayesian had fallen out of favour. Hence direct probabilistic inferences were superseded by the indirect and conceptually different route, going through statistics and intimately linked to hypothesis testing. The use of statistics is not particularly easy. The alternatives to Bayesian methods are subtle and not very obvious; they are also associated with some fairly formidable mathematical machinery. We will avoid this, presenting only results and showing the use of statistics, while trying to make clear the conceptual foundations.

Statistics

Statistics are designed to summarize, reduce or describe data. The formal definition of a statistic is that it is some function of the data alone. For a set of data X1, X2, …, some examples of statistics might be the average, the maximum value or the average of the cosines. Statistics are therefore combinations of finite amounts of data. In the following discussion, and indeed throughout, we try to distinguish particular fixed values of the data, and functions of the data alone, by upper case (except for Greek letters). Possible values, being variables, we will denote in the usual algebraic spirit by lower case.

The summarizing aspect of statistics is exemplified by those describing (1) location and (2) spread or scatter.

It is often the case that we need to do sample comparison: we have someone else's data to compare with ours; or someone else's model to compare with our data; or even our data to compare with our model. We need to make the comparison and to decide something. We are doing hypothesis testing – are our data consistent with a model, with somebody else's data? In searching for correlations as we were in Chapter 4, we were hypothesis testing; in the model fitting of Chapter 6 we are involved in data modelling and parameter estimation.

Classical methods of hypothesis testing may be either parametric or non-parametric, distribution-free as it is sometimes called. Bayesian methods necessarily involve a known distribution. We have described the concepts of Bayesian versus frequentist and parametric versus non-parametric in the introductory Chapters 1 and 2. Table 5.1 summarizes these apparent dichotomies and indicates appropriate usage.

That non-parametric Bayesian tests do not exist appears self-evident, as the key Bayesian feature is the probability of a particular model in the face of the data. However, it is not quite this clear-cut, and there has been consideration of non-parametric methods in a Bayesian context (Gull & Fielden 1986). If we understand the data so that we can model its collection process, then the Bayesian route beckons (see Chapter 2 and its examples).

The distribution of objects on the celestial sphere, or on an imaged patch of this sphere, has ever been a major preoccupation of astronomers. Avoiding here the science of image processing, the province of thousands of books and papers, we consider some of the common statistical approaches used to quantify sky distributions in order to permit contact with theory. Before we turn to the adopted statistical weaponry of galaxy distribution, we discuss some general statistics applicable to the spherical surface.

Statistics on a spherical surface

Abstractly, the distribution of objects on the celestial sphere is simply the distribution of directions of a set of unit vectors. In this respect, other three-dimensional spaces may be of interest, like the Poincaré sphere with unit vectors indicating the state of polarization of radiation.

This is a thriving subfield of statistics and there is an excellent hand-book (Fisher, Lewis & Embleton 1987). Much of the motivation comes from geophysical topics (orientation of palaeomagnetism, for instance) but many other ‘spaces’ are of interest. The emphasis is on statistical modelling and a variety of distributions is available. The Fisher distribution, one of the most popular, plays a similar role in spherical statistics to that played by the Gaussian in ordinary statistics.

Peter Scheuer started this. In 1977 he walked into JVW's office in the Cavendish Lab and quietly asked for advice on what further material should be taught to the new intake of Radio Astronomy graduate students (that year including the hapless CRJ). JVW, wrestling with simple chi-square testing at the time, blurted out ‘They know nothing about practical statistics’. Peter left thoughtfully. A day later he returned. ‘Good news! The Management Board has decided that the students are going to have a course on practical statistics.’ Can I sit in, JVW asked innocently. ‘Better news! The Management Board has decided that you're going to teach it’.

So, for us, began the notion of practical statistics. A subject that began with gambling is not an arcane academic pursuit, but it is certainly subtle as well. It is fitting that Peter Scheuer was involved at the beginning of this (lengthy) project; his style of science exemplified both subtlety and pragmatism. We hope that we can convey something of both. If an echo of Peter's booming laugh is sometimes heard in these pages, it is because we both learned from him that a useful answer is often much easier – and certainly much more entertaining – than you at first think.

After the initial course, the material for this book grew out of various further courses, journal articles, and the abundant personal experience that results from understanding just a little of any field of knowledge that counts Gauss and Laplace amongst its originators.