The explicit recognition of uncertainty is central to the statistical sciences. Notions such as prior information, probability models, likelihood, standard errors and confidence limits are all intended to formalize uncertainty and thereby make allowance for it. In simple situations, the uncertainty of an estimate may be gauged by analytical calculation based on an assumed probability model for the available data. But in more complicated problems this approach can be tedious and difficult, and its results are potentially misleading if inappropriate assumptions or simplifications have been made.

For illustration, consider Table 1.1, which is taken from a larger tabulation (Table 7.4) of the numbers of AIDS reports in England and Wales from mid-1983 to the end of 1992. Reports are cross-classified by diagnosis period and length of reporting delay, in three-month intervals. A blank in the table corresponds to an unknown (as yet unreported) entry. The problem was to predict the states of the epidemic in 1991 and 1992, which depend heavily on the values missing at the bottom right of the table.

The data support the assumption that the reporting delay does not depend on the diagnosis period. In this case a simple model is that the number of reports in row j and column k of the table has a Poisson distribution with mean µjk = exp(αj + βk).

Introduction

In Chapter 6 we showed how the basic bootstrap methods of earlier chapters extend to linear regression. The broad aim of this chapter is to extend the discussion further, to various forms of nonlinear regression models — especially generalized linear models and survival models — and to nonparametric regression, where the form of the mean response is not fully specified.

A particular feature of linear regression is the possibility of error-based resampling, when responses are expressible as means plus homoscedastic errors. This is particularly useful when our objective is prediction. For generalized linear models, especially for discrete data, responses cannot be described in terms of additive errors. Section 7.2 describes ways of generalizing error-based resampling for such models. The corresponding development for survival data is given in Section 7.3. Section 7.4 looks briefly at nonlinear regression with additive error, mainly to illustrate the useful contribution that resampling methods can make to analysis of such models. There is often a need to estimate the potential accuracy of predictions based on regression models, and Section 6.4 contained a general discussion of resampling methods for this. In Section 7.5 we focus on one type of application, the estimation of misclassification rates when a binary response y corresponds to a classification.

Not all relationships between a response y and covariates x can be readily modelled in terms of a parametric mean function of known form.

Introduction

In previous chapters our models have involved variables independent at some level, and we have been able to identify independent components that can be simulated. Where a model can be fitted and residuals of some sort identified, the same ideas can be applied in the more complex problems discussed in this chapter. Where that model is parametric, parametric simulation can in principle be used to obtain resamples, though Markov chain Monte Carlo techniques may be needed in practice. But in nonparametric situations the dependence may be so complex, or our knowledge of it so limited, that neither of these approaches is feasible. Of course some assumption of repeatedness within the data is essential, or it is impossible to proceed. But the repeatability may not be at the level of individual observations, but of groups of them, and there is typically dependence between as well as within groups. This leads to the idea of constructing bootstrap data by taking blocks of some sort from the original observations. The area is in rapid development, so we avoid a detailed mathematical exposition, and merely sketch key aspects of the main ideas. In Section 8.2 we describe some of the resampling schemes proposed for time series. Section 8.3 outlines some ideas useful in resampling point processes.

Introduction

In the previous chapter we laid out the basic elements of resampling or bootstrap methods, in the context of the analysis of a single homogeneous sample of data. This chapter deals with how those ideas are extended to some more complex situations, and then turns to uses for variations and elaborations of simple bootstrap schemes.

In Section 3.2 we describe how to construct resampling algorithms for several independent samples, and then in Section 3.3 we discuss briefly the use of partial modelling, either qualitative or semiparametric, a topic explored more fully in the later chapters on regression models (Chapters 6 and 7). Section 3.4 examines when it is worthwhile to modify the statistic by using a smoothed empirical distribution function. In Sections 3.5 and 3.6 we turn to situations where data are censored or missing and therefore are incomplete. One relatively simple situation where the standard bootstrap must be modified to succeed is finite population sampling, which we consider in Section 3.7. In Section 3.8 we deal with simple situations of hierarchical variation. Section 3.9 is an account of nested bootstrapping, where we outline how to overcome some of the shortcomings of a single bootstrap calculation by a further level of simulation. Section 3.10 describes bootstrap diagnostics, which are concerned with the assessment of sensitivity of resampling analysis to individual observations, as well as the use of bootstrap output to suggest modifications to the calculations.

A great deal of hyperbole has been devoted to neural networks, both in their first wave around 1960 (Widrow & Hoff, 1960; Rosenblatt, 1962) and in their renaissance from about 1985 (chiefly inspired by Rumelhart & McClelland, 1986), but the ideas of biological relevance seem to us to have detracted from the essence of what is being discussed, and are certainly not relevant to practical applications in pattern recognition. Because ‘neural networks’ has become a popular subject, it has collected many techniques which are only loosely related and were not originally biologically motivated. In this chapter we will discuss the core area of feed-forward or ‘back-propagation’ neural networks, which can be seen as extensions of the ideas of the perceptron (Section 3.6). From this connection, these networks are also known as multi-layer perceptrons.

A formal definition of a feed-forward network is given in the glossary. Informally, they have units which have one-way connections to other units, and the units can be labelled from inputs (low numbers) to outputs (high numbers) so that each unit is only connected to units with higher numbers. The units can always be arranged in layers so that connections go from one layer to a later layer. This is best seen graphically; see Figure 5.1.

Pattern recognition has a long and respectable history within engineering, especially for military applications, but the cost of the hardware both to acquire the data (signals and images) and to compute the answers made it for many years a rather specialist subject. Hardware advances have made the concerns of pattern recognition of much wider applicability. In essence it covers the following problem:

There are many examples from everyday life:

Many of these are currently performed by human experts, but it is increasingly becoming feasible to design automated systems to replace the expert and either perform better (as in credit scoring) or ‘clone’ the expert (as in aids to medical diagnosis).

Neural networks have arisen from analogies with models of the way that humans might approach pattern recognition tasks, although they have developed a long way from the biological roots. Great claims have been made for these procedures, and although few of these claims have withstood careful scrutiny, neural network methods have had great impact on pattern recognition practice.

The use of tree-based methods for classification is relatively unfamiliar in both statistics and pattern recognition, yet they are widely used in some applications such as botany (Figure 7.1) and medical diagnosis as being extremely easy to comprehend (and hence have confidence in).

The automatic construction of decision trees dates from work in the social sciences by Morgan & Sonquist (1963) and Morgan & Messenger (1973). (Later work such as Doyle, 1973, and Doyle & Fenwick, 1975, commented on the pitfalls of such automated procedures.) In statistics Breiman et al. (1984) had a seminal influence both in bringing the work to the attention of statisticians and in proposing new algorithms for constructing trees. At around the same time decision tree induction was beginning to be used in the field of machine learning, which we review in Section 7.4, and in engineering (for example, Sethi & Sarvarayudu, 1982).

The terminology of trees is graphic, although conventionally trees such as Figure 7.2 are shown growing down the page. The root is the top node, and examples are passed down the tree, with decisions being made at each node until a terminal node or leaf is reached. Each non-terminal node contains a question on which a split is based. Each leaf contains the label of a classification. A subtree of T is a tree with root a node of T; it is a rooted subtree if its root is the root of T.