Unpredictability and non-determinism are all around us. The future behaviour of any system—from an elementary particle to a complex organism—may follow a number of possible paths. Some of these paths may be more likely than others, but none is absolutely certain. Such unpredictable behaviour, and the phenomena that cause it, are usually described as ‘random’. Whether randomness is in the nature of reality, or is the result of imperfect knowledge, is a philosophical question which need not concern us here. More important is to learn how to deal with randomness, how to quantify it and take it into account, so as to be able to plan and make rational choices in the face of uncertainty.

The theory of probabilities was developed with this object in view. Its domain of applications, which was originally confined mainly to various games of chance, now extends over most scientific and engineering disciplines.

This chapter is intended as a self-contained introduction; it describes all the concepts and results of probability theory that will be used in the rest of the book. Examples and exercises are included. However, it is impossible to provide a thorough coverage of a major branch of mathematics in one chapter. The reader is therefore assumed to have encountered at least some of this material already.

The designers and users of complex systems have an interest in knowing how those systems behave under different conditions. This is true in all engineering domains, from transport and manufacturing to computing and communications. It is necessary to have a clear understanding, both qualitative and quantitative, of the factors that influence the performance and reliability of a system. Such understanding may be obtained by constructing and analysing mathematical models. The purpose of this book is to provide the necessary background, methods and techniques.

A model is inevitably an approximation of reality: a number of simplifying assumptions are usually made. However, that need not diminish the value of the insights that can be gained. A mathematical model can capture all the essential features of a system, display underlying trends and provide quantitative relations between input parameters and performance characteristics. Moreover, analysis is cheap, whereas experimentation is expensive. A few simple calculations carried out on the back of an envelope can often yield as much information as hours of observations or simulations.

The systems in which we are interested are subjected to demands of random character. The processes that take place in response to those demands are also random. Accordingly, the modelling tools that are needed to study such systems come from the domains of probability theory, stochastic processes and queueing theory.

Some of the most important applications of probabilistic modelling techniques are in the area of distributed systems. The term ‘distributed’ means, in this context, that various tasks that are somehow related can be carried out by different servers which may or may not be in different geographical locations. Such a broad definition covers a great variety of applications, in the areas of manufacturing, transport, computing and communications. To study the behaviour of a distributed system, one normally needs a model involving a number of service centres, with jobs arriving and circulating among them according to some random or deterministic routeing pattern. This leads in a natural way to the concept of a network of queues.

A queueing network can be thought of as a connected directed graph whose nodes represent service centres. The arcs between those nodes indicate one-step moves that jobs may make from service centre to service centre (the existence of an arc from nodei to nodej does not necessarily imply one from j to i). Each node has its own queue, served according to some scheduling strategy. Jobs may be of different types and may follow different routes through the network. An arc without origin leading into a node (or one without destination leading out of a node) indicates that jobs arrive into that node from outside (or depart from it and leave the network). Figure 4.1 shows a five-node network, with external arrivals into nodes 1 and 2, and external departures from nodes 1 and 5. At this level of abstraction, only the connectivity of the nodes is specified; nothing is said about their internal structure, nor about the demands that jobs place on them.

At a certain level of abstraction, computing and communication systems as well as banking, manufacturing and transport systems, can be described in terms of ‘jobs’ and ‘servers’, i.e. requests for service and devices that provide service. The jobs may be computing tasks, input/output commands, telephone calls, data packets. The servers may be processors, storage devices, communication channels, software modules. A model aimed at evaluating and predicting the performance of such a system has to capture the following essential aspects of its behaviour:

1. (a) The pattern of demand, i.e. the the manner in which jobs arrive into the system and the nature of services that they require.

2. (b) The competition for service, i.e. the effect of admission, queueing and routing policies on performance.

3. This chapter is devoted to (a). It introduces tools and results that are used when modelling the arrivals and services of jobs.

Renewal processes

Consider a phenomenon which takes place first at time 0 and thereafter keeps occurring, at random intervals, ad infinitum. Denote the consecutive instants of occurrence by Tn (n = 0,1,…; T0 0), and let Sn Tn — Tn-1 (n = 1,2,…) be the intervals between them. Assume that the random variables Sn are independent and identically distributed.

Markov chains are the simplest mathematical models for random phenomena evolving in time. Their simple structure makes it possible to say a great deal about their behaviour. At the same time, the class of Markov chains is rich enough to serve in many applications. This makes Markov chains the first and most important examples of random processes. Indeed, the whole of the mathematical study of random processes can be regarded as a generalization in one way or another of the theory of Markov chains.

This book is an account of the elementary theory of Markov chains, with applications. It was conceived as a text for advanced undergraduates or master's level students, and is developed from a course taught to undergraduates for several years. There are no strict prerequisites but it is envisaged that the reader will have taken a course in elementary probability. In particular, measure theory is not a prerequisite.

The first half of the book is based on lecture notes for the undergraduate course. Illustrative examples introduce many of the key ideas. Careful proofs are given throughout. There is a selection of exercises, which forms the basis of classwork done by the students, and which has been tested over several years. Chapter 1 deals with the theory of discrete-time Markov chains, and is the basis of all that follows. You must begin here. The material is quite straightforward and the ideas introduced permeate the whole book.

In the first three chapters we have given an account of the elementary theory of Markov chains. This already covers a great many applications, but is just the beginning of the theory of Markov processes. The further theory inevitably involves more sophisticated techniques which, although having their own interest, can obscure the overall structure. On the other hand, the overall structure is, to a large extent, already present in the elementary theory. We therefore thought it worth while to discuss some features of the further theory in the context of simple Markov chains, namely, martingales, potential theory, electrical networks and Brownian motion. The idea is that the Markov chain case serves as a guiding metaphor for more complicated processes. So the reader familiar with Markov chains may find this chapter helpful alongside more general higher-level texts. At the same time, further insight is gained into Markov chains themselves.

Martingales

A martingale is a process whose average value remains constant in a particular strong sense, which we shall make precise shortly. This is a sort of balancing property. Often, the identification of martingales is a crucial step in understanding the evolution of a stochastic process.

Applications of Markov chains arise in many different areas. Some have already appeared to illustrate the theory, from games of chance to the evolution of populations, from calculating the fair price for a random reward to calculating the probability that an absent-minded professor is caught without an umbrella. In a real-world problem involving random processes you should always look for Markov chains. They are often easy to spot. Once a Markov chain is identified, there is a qualitative theory which limits the sorts of behaviour that can occur – we know, for example, that every state is either recurrent or transient. There are also good computational methods – for hitting probabilities and expected rewards, and for long-run behaviour via invariant distributions.

In this chapter we shall look at five areas of application in detail: biological models, queueing models, resource management models, Markov decision processes and Markov chain Monte Carlo. In each case our aim is to provide an introduction rather than a systematic account or survey of the field. References to books for further reading are given in each section.

Markov chains in biology

Randomness is often an appropriate model for systems of high complexity, such as are often found in biology. We have already illustrated some aspects of the theory by simple models with a biological interpretation. See Example 1.1.5 (virus), Exercise 1.1.6 (octopus), Example 1.3.4 (birth-and-death chain) and Exercise 2.5.1 (bacteria).