The supervised methods considered so far have learnt both the structure of the probability distributions and the numerical values from the training set, or in the case of parametric methods, imposed a conventional structure for convenience. Other methods incorporate non-numerical ‘real-world’ knowledge about the subject domain into the structure of the probability distributions. Such knowledge is often about causal relationships, or perhaps the lack of causality as expressed by conditional independence.

These ideas have been most explored within the field of expert systems. This is a loosely defined area, and definitions vary:

Unsupervised methods are used when no classes are defined a priori, or when they are but the data are to be used to confirm that these are suitable classes. Examples of the latter type are quite common in biology, where species are often defined by physical characteristics, and datasets of biochemical measurements become available. The interesting question is then whether the physical and biochemical measurements define the same classification. A variant of this occurs with our Leptograpsus crabs data. There the division into species was based on colour, and the interesting question is whether this is supported by morphological differences. Our analyses hitherto have been to find supporting morphological differences, but this begs the question of whether there might be even more striking differences unrelated to colour.

Unsupervised methods are generally designed for visualization, either 0 to show views of the data which indicate groups, or to show affinities between the examples by displaying similar examples close together. Dendrograms are a one-dimensional display of similarity, with the height of the join indicating (dis)similarity. For example, Figure 9.1 shows a dendrogram of the Cushing's syndrome data. Each pair is joined in the tree, and the height at which they are joined is an indication of their dissimilarity. This plot shows clearly that one point (labelled u) is very different from the rest, and does tend to group the diseases together, imperfectly. However, this is two-dimensional data, and the data can be plotted as in Figure 1.2 on page 11.

This book is primarily about pattern recognition, which covers a wide range of activities from many walks of life. It is something which we humans are particularly good at; we receive data from our senses and are often able, immediately and without conscious effort, to identify the source of the data. For example, many of us can

Science, technology and business has brought to us many similar tasks, including

In this chapter we consider the problem of what features should be included when designing our classifier. We should make clear at the outset that this is an impossible problem; there may be no substitute for trying them all and seeing how well the resulting classifier works. However, this may be computationally impracticable, and unless a large test set is available it may be impossible to avoid selection effects, of choosing the best of a large class of classifiers on that particular test set and not for the population.

To illustrate the difficulty, consider a battery of diagnostic tests T1, …, Tm for a fairly rare disease, which perhaps around 5% of all patients tested actually have. Suppose test T1 correctly picks up 99% of the real cases and has a very low false positive rate. However, there is a rare special form of the disease that T1 cannot detect, but T2 can, yet T2 is inaccurate on the normal disease form. If we test the diagnostic tests one at a time, we will never even think of including T2, yet T1 and T2 together may give a nearly perfect classifier by declaring a patient diseased if T1 is positive or T1 is negative and T2 is positive. This illustrates that considering features one at a time may not be sufficient.

Our aim in this chapter is to indicate single features which are likely to have good discriminatory power (feature selection) or linear combinations of features with the same aim (feature extraction).