In this chapter we recapitulate the beginnings of probability theory. The reader to whom this subject is completely new may wish first to consult a more leisurely introduction, such as McColl (1997).

Sample spaces

There are different schools on the meaning of probability. For example, it is argued that a statement such as ‘The Scottish National Party has a probability of 1/5 of winning the election’ is meaningless because the experiment ‘have an election’ cannot be repeated to order. The way out has proved to be an axiomatic approach, originated by Kolmogorov (see Figure 9.1) in 1933, in which all participants, though begging to differ on some matters of interpretation, can nevertheless agree on the consequences of the rules (see e.g. Kolmogorov, 1956b). His work included a rigorous definition of conditional expectation, a crucial and fruitful concept in current work in many areas and applications of probability.

Sample spaces and events

Model 9.1 We begin with the idea that, corresponding to an experiment E, there is a set S, the sample space, consisting of all possible outcomes. In the present context an event A is a set of outcomes, that is A ⊆ S. Then it is a matter of definition that, if E is performed with outcome a, the event A occurs if and only if a ∈ A.

Often, but not always, the outcomes are conveniently represented by numbers, as illustrated in examples below.

In Chapters 1 and 2 we have classified plane isometries, discovered some important principles of how they combine, and made a first application to patterns whose symmetry group is either the dihedral group D2n or its rotation subgroup Cn. Before investigating plane patterns it is a logical and useful step to classify the 1-dimensional, or braid, patterns, be aware of their symmetries, and get a little practice in both recognizing and creating them.

Definition 3.1 We say ν is a translation vector of pattern F if Tν is a translation symmetry. Then a braid (band, frieze) pattern is a pattern in the plane, all of whose translation vectors are parallel. In particular, a and −a are parallel. We will usually call this parallel direction horizontal, and the perpendicular direction vertical. Other names used are longitudinal and transverse, respectively. A symmetry group of a braid is sometimes called a line group.

As noted in Section 1.1, we are investigating patterns which are discrete: they do not have translation or other symmetries which move the pattern by arbitrarily small amounts. Thus, amongst the collection of all translation symmetries of the pattern there is a translation Ta of least but not zero magnitude. Of course it is not unique, for example T−a has the same magnitude |a| as Ta. We rephrase an observation from the preliminary discussion of braids preceding Figure 2.7. It may be derived more formally from Theorem 3.3.

An artificial neural network, or just net, may be thought of firstly in pattern recognition terms, say converting an input vector of pixel values to a character they purport to represent. More generally, a permissible input vector is mapped to the correct output, by a process in some way analogous to the neural operation of the brain (Figure 18.1). In Section 18.1 we work our way up from Rosenblatt's Perceptron, with its rigorously proven limitations, to multilayer nets which in principle can mimic any input–output function. The idea is that a net will generalise from suitable input–output examples by setting free parameters called weights.

In Section 18.2 the nets are mainly self-organising, in that they construct their own categories of classification. We include learning vector quantisation and the topologically based Kohonen method. Related nets give an alternative view of Principal Component Analysis. In Section 18.3 Shannon's extension of entropy to the continuous case opens up the criterion of Linsker (1988) that neural network weights should be chosen to maximise mutual information between input and output. We include a 3D image processing example due to Becker and Hinton (1992). Then the further Shannon theory of rate distortion is applied to vector quantization and the LBG quantiser.

In Section 18.4 we begin with the Hough Transform and its widening possibilities for finding arbitrary shapes in an image. We end with the related idea of tomography, rebuilding an image from projections.

In this chapter we introduce the basic idea of entropy, quantifying an amount of information, and in its light we consider some important methods of encoding a sequence of symbols. We shall be thinking of these as text, but they also apply to a byte sequence representing pixel values of a digital image. In the next chapter we shall develop information theory to take account of noise, both visual and otherwise. Here we focus on ‘noiseless encoding’ in preparation for that later step. However, before leaving this chapter we take time to examine an alternative approach to quantifying information, which has resulted in the important idea of Minimum Description Length as a new principle in choosing hypotheses and models.

The idea of entropy

Shannon (1948), the acknowledged inventor of information theory, considered that a basis for his theory already existed in papers of Niquist (1924) and Hartley (1928). The latter had argued that the logarithm function was the most natural function for measuring information. For example, as Shannon notes, adding one relay to a group doubles the number of possible states, but adds one to the base 2 log of this number. Thus information might be measured as the number of bits, or binary digits bi = 0, 1, required to express an integer in binary form: bm … b1b0 = Σibi2i. For example, 34 = 100010 takes six bits. Shannon proposed a 5-component model of a communication system, reproduced in Figure 12.1.

Both wavelets and fractals (even just fractal dimension) have seen an explosion of applications in recent years. This text will point mainly to the vision side, but at the end of this chapter we give references that indicate something of the range. The story begins with Section 16.1 which is about fractals; this points to the value of the scaling theme, thereafter explored through wavelets and multiresolution. The present chapter concentrates on wavelets with the most structure: the Haar and Daubechies types. In the next this is relaxed for application to B-splines, then to surface wavelets.

Nature, fractals and compression

The potential to compress an image has to do with its degree of redundancy, and so comes broadly under image analysis. Here, more specifically, we are interested in the redundancy that may come from self-similarity of different parts of the image, rather than from more general correlation. This idea arose essentially from Mandelbrot's observations about the nature of natural images. For example, one part may be approximately congruent to another at a different scale, as in Figure 16.1. The basic way to turn this into a method of compression goes back to Barnsley and Sloan (1988) and is the subject of Section 16.1.3. For subsequent exploitation see Section 16.1.4 plus Barnsley (1992), Barnsley and Hurd (1994), Peitgen et al. (1992), Fisher (1995) and Lu (1997).

The purpose of this chapter is to provide a modest introduction to the huge and important topics of sampling and inference, which will serve our purpose in succeeding chapters. This is not a stand-alone chapter, indeed it provides many illustrations of the significance of early sections on probability, just as they in turn utilise the preceding linear algebra/matrix results. So what is the present chapter about? The short answer, which will be amplified section by section, is the interpretation of data, having in mind ultimately the interpretation of pixel values in computer images.

We begin with the idea of a sample, a sequence of determinations X1, …, Xn of a random variable X. We seek statistics, i.e. functions f(X1, …, Xn), to help answer questions such as (a) given that a distribution is of a certain type: Poisson, exponential, normal, …, how can we estimate the distribution parameters and with what certainty, (b) given a sample, what is the underlying distribution, again with what certainty? Sections 11.2, 11.3 and 11.4 utilise the methods of Section 11.1.

In Section 11.2 we introduce the Bayesian approach, distinct from the Bayesian Theorem, but ultimately based upon it. The idea is to improve on an imprecise model of a situation or process by utilising every piece of data that can be gathered. The section concludes with the Bayes pattern classsifer, a first step in object/pattern recognition.

The Fourier Transform is a wonderful way of splitting a function into helpful parts, possibly modifying those parts and putting them back together again. One gains insight and/or the power to change things in a desired direction. Here we are particularly interested in its value for interpreting and restoring digital image data. Although the story of the Fourier Transform really begins with the so-called continuous case, where the definitions are integrals, our main concern is with the discrete version, which in any case is what is generally used when implementing even the continuous transform. We come to the continuous case second.

We begin in Section 14.1 with basic definitions and tools rooted in simple yet powerful properties of the complex numbers. We introduce filtering and the Convolution Theorem, two reasons for the wide use of the transform. Redundancy in the DFT is utilised to arrive at the Fast Fourier Transform (FFT), which reduces the complexity of calculation from O(N2) to O(N log2N), another reason for the DFT's ubiquity.

The short Section 14.2 introduces the Continuous Fourier Transform and its tool the Dirac delta function, concluding with the highly desirable properties listed in Table 14.2. In Section 14.3 we explore connections between the three types of Fourier Transform: Fourier series, the continuous transform, and the DFT; noting that for a finite interval appropriately sampled the DFT is a good approximation to the continuous version.

The B-spline has long been important in computer graphics for representing curves and surfaces, but it was recently realised that the recursive subdivision method of construction could be used to formulate B-splines in wavelet terms; this led to excellent new applications (see Stollnitz et al., 1996) in which curves and surfaces could be analysed or modified at any chosen scale, from local to global. In addition to proving results, we provide some exemplification of these things.

By introducing an equivalent definition of B-splines as an m-fold convolution of boxes we bring out an intimate connection with the Fourier transform. This in turn provides an alternative derivation of such formulae as the Cox–de Boor relations.

The last two sections are designated as appendices, optional follow-up to the main treatment. In Section 17.4 we derive wavelet-identifying formulae that hold for arbitrary size of control polygon, whilst Section 17.5 addresses mathematical aspects of the natural generalisation from curve to surface wavelets achievable by the subdivision system of Loop (1987). Multiresolution and editing examples are exhibited.

Splines from boxes

We begin with Bézier splines, to give background and to introduce some of the ideas behind splines, including that of convexity in Section 17.1.2. Moving on to the even more useful B-splines in Section 17.1.3, we present their definition as a convolution of box functions, excellent for the coming wavelet formulation.