Markov chains are the simplest mathematical models for random phenomena evolving in time. Their simple structure makes it possible to say a great deal about their behaviour. At the same time, the class of Markov chains is rich enough to serve in many applications. This makes Markov chains the first and most important examples of random processes. Indeed, the whole of the mathematical study of random processes can be regarded as a generalization in one way or another of the theory of Markov chains.

This book is an account of the elementary theory of Markov chains, with applications. It was conceived as a text for advanced undergraduates or master's level students, and is developed from a course taught to undergraduates for several years. There are no strict prerequisites but it is envisaged that the reader will have taken a course in elementary probability. In particular, measure theory is not a prerequisite.

The first half of the book is based on lecture notes for the undergraduate course. Illustrative examples introduce many of the key ideas. Careful proofs are given throughout. There is a selection of exercises, which forms the basis of classwork done by the students, and which has been tested over several years. Chapter 1 deals with the theory of discrete-time Markov chains, and is the basis of all that follows. You must begin here. The material is quite straightforward and the ideas introduced permeate the whole book.

In the first three chapters we have given an account of the elementary theory of Markov chains. This already covers a great many applications, but is just the beginning of the theory of Markov processes. The further theory inevitably involves more sophisticated techniques which, although having their own interest, can obscure the overall structure. On the other hand, the overall structure is, to a large extent, already present in the elementary theory. We therefore thought it worth while to discuss some features of the further theory in the context of simple Markov chains, namely, martingales, potential theory, electrical networks and Brownian motion. The idea is that the Markov chain case serves as a guiding metaphor for more complicated processes. So the reader familiar with Markov chains may find this chapter helpful alongside more general higher-level texts. At the same time, further insight is gained into Markov chains themselves.

Martingales

A martingale is a process whose average value remains constant in a particular strong sense, which we shall make precise shortly. This is a sort of balancing property. Often, the identification of martingales is a crucial step in understanding the evolution of a stochastic process.

Applications of Markov chains arise in many different areas. Some have already appeared to illustrate the theory, from games of chance to the evolution of populations, from calculating the fair price for a random reward to calculating the probability that an absent-minded professor is caught without an umbrella. In a real-world problem involving random processes you should always look for Markov chains. They are often easy to spot. Once a Markov chain is identified, there is a qualitative theory which limits the sorts of behaviour that can occur – we know, for example, that every state is either recurrent or transient. There are also good computational methods – for hitting probabilities and expected rewards, and for long-run behaviour via invariant distributions.

In this chapter we shall look at five areas of application in detail: biological models, queueing models, resource management models, Markov decision processes and Markov chain Monte Carlo. In each case our aim is to provide an introduction rather than a systematic account or survey of the field. References to books for further reading are given in each section.

Markov chains in biology

Randomness is often an appropriate model for systems of high complexity, such as are often found in biology. We have already illustrated some aspects of the theory by simple models with a biological interpretation. See Example 1.1.5 (virus), Exercise 1.1.6 (octopus), Example 1.3.4 (birth-and-death chain) and Exercise 2.5.1 (bacteria).

The material on continuous-time Markov chains is divided between this chapter and the next. The theory takes some time to set up, but once up and running it follows a very similar pattern to the discrete-time case. To emphasise this we have put the setting-up in this chapter and the rest in the next. If you wish, you can begin with Chapter 3, provided you take certain basic properties on trust, which are reviewed in Section 3.1. The first three sections of Chapter 2 fill in some necessary background information and are independent of each other. Section 2.4 on the Poisson process and Section 2.5 on birth processes provide a gentle warm-up for general continuous-time Markov chains. These processes are simple and particularly important examples of continuous-time chains. Sections 2.6–2.8, especially 2.8, deal with the heart of the continuous-time theory. There is an irreducible level of difficulty at this point, so we advise that Sections 2.7 and 2.8 are read selectively at first. Some examples of more general processes are given in Section 2.9. As in Chapter 1 the exercises form an important part of the text.

Q-matrices and their exponentials

In this section we shall discuss some of the basic properties of Q-matrices and explain their connection with continuous-time Markov chains.

Introduction

This chapter is concerned with the initial processing of analogue and digital signal sources prior to transmission over a public switched telephone network (PSTN). A major topic is pulse code modulation (PCM), a technique for encoding analogue signals and transmitting them over a digital link. (The name was originally chosen by analogy with other pulse modulation techniques such as pulse amplitude modulation (PAM) and pulse width modulation (PWM).) PCM involves sampling the analogue waveform at an appropriate rate, encoding the samples in digital (normally binary) form, and then transmitting the coded samples using a suitable digital waveform (which is also often binary but may well be ternary, quaternary or other). At the receiver the digital waveform is decoded, and the original message signal is reconstructed from the sample values. The complete process is illustrated in Fig. 7.1, where the transmitted digital signal has been shown as a baseband binary waveform for simplicity.

The most common application of PCM is perhaps in telephony, although the analogue message signal can originate from a wide range of sources other than a telephone handset: telemetry, radio, video, and so on. Similar techniques are also used for digital audio recording.

Later sections of the chapter introduce alternatives to standard PCM for the digital transmission of analogue signals, and also discuss the need for modems when using digital sources such as facsimile or computer terminals. Finally, some aspects of interfacing to an integrated services digital network ISDN) are introduced.

Many of the concepts discussed in this book can be illustrated by simple computer simulations. Commercial packages are available for some of them, or simple programs can be written in programming languages such as ‘Basic’ or ‘Pascal’. However, for many topics in digital systems a spreadsheet provides a particularly easy and illuminating demonstration of important techniques.

The following are examples of how spreadsheets can be used to illustrate some of the topics in the text. The details will depend upon the particular spreadsheet package used: it is assumed that readers are already familiar with the package to which they have access. Two different spreadsheet packages are used in the examples below. The zero forcing equaliser is illustrated in ‘Excel’ on an ‘Apple’ ‘Macintosh’ computer, while the other two examples are shown implemented on ‘SuperCalc4’.

The reader is encouraged to use these examples as the starting point for experimentation with different parameters and configurations. Other examples of the use of shreadsheets for modelling in engineering will be found in Bissell and Chapman (1989).

A zero forcing equaliser (Section 4.3.4)

Fig. C1 shows a shreadsheet to simulate the zero forcing equaliser of Fig. 4.13. The input (column A) shifts into the first delay stage (column B) then into the second delay stage (column C). The three coefficients (– 0.266,0.866 and 0.204) have been entered in locations E3, E4 and E5 respectively.

In this book we try to give a representative (but not comprehensive) treatment of the digital transmission of signals. Our main aim has been to render the material truly accessible to second or third year undergraduate students, practising engineers requiring updating, or graduate physicists or mathematicians beginning work in the digital transmission sector of the telecommunications industry. This has led to a book whose important features are:

Engineering is a pragmatic activity, and its models and theory primarily a means to an end. As with other engineering disciplines, much of telecommunications is driven by practicalities: the design of line codes (Chapter 6), or the synchronous digital hierarchy (Chapter 8), for example, owe little to any complicated theoretical analysis of digital telecommunications! Yet even such pragmatic activities take place against a background of constraints which telecommunications engineers sooner or later translate into highly abstract models involving bandwidth, spectra, noise density, probability distributions, error rates, and so on. To present these vitally important ideas – in a limited number of contexts, but in sufficient detail to be properly understood by the reader – is the main aim of this book. Thus timeand frequency-domain modelling tools form one constant theme (whether as part of the theory of pulse shaping and signal detection in Chapter 4, or as a background to the niceties of optical receiver design in Chapter 9); the constant battle against noise and the drive to minimise errors is another.