Although the MCMC approach to simulation, described in the previous chapter, is highly useful, let us note two drawbacks of the method:

1. (A) The main theoretical basis for the MCMC method is Theorem 5.2, which guarantees that the distribution μ(n) at time n of an irreducible and aperiodic Markov chain started in a fixed state converges to the stationary distribution π as n → ∞. But this does not imply that μ(n) ever becomes equal to π, only that it comes very close. As a matter of fact, in the vast majority of examples, we have μ(n) ≠ π for all n (see, e.g., Problem 2.3). Hence, no matter how large n is taken to be in the MCMC algorithm, there will still be some discrepancy between the distribution of the output and the desired distribution π.

2. (B) In order to make the error due to (A) small, we need to figure out how large n needs to be taken in order to make sure that the discrepancy between μ(n) and π (measured in the total variation distance dTV(μ(n), π)) is smaller than some given ε > 0. In many situations, it has turned out to be very difficult to obtain upper bounds on how large n needs to be taken, that are small enough to be of any practical use.

Problem (A) above is in itself not a particularly serious obstacle. In most situations, we can tolerate a small error in the distribution of the output, as long as we have an idea about how small it is.

Recall, from the beginning of Chapter 8, the problems (A) and (B) with the MCMC method. In that chapter, we saw one approach to solving these problems, namely to prove that an MCMC chain converges sufficiently quickly to its equilibrium distribution.

In the early 1990's, some ideas about a radically different approach began to emerge. The breakthrough came in a 1996 paper by Jim Propp and David Wilson [PW], both working at MIT at that time, who presented a refinement of the MCMC method, yielding an algorithm which simultaneously solves problems (A) and (B) above, by

1. (A*) producing an output whose distribution is exactly the equilibrium distribution π, and

2. (B*) determining automatically when to stop, thus removing the need to compute any Markov chain convergence rates beforehand.

This algorithm has become known as the Propp–Wilson algorithm, and is the main topic of this chapter. The main feature distinguishing the Propp–Wilson algorithm from ordinary MCMC algorithms is that it involves running not only one Markov chain, but several copies of it, with different initial values. Another feature which is important (we shall soon see why) is that the chains are not run from time 0 and onwards, but rather from some time in the (possibly distant) past, and up to time 0.

Due to property (A*) above, the Propp–Wilson algorithm is sometimes said to be an exact, or perfect simulation algorithm.

We go on with a more specific description of the algorithm.

A drawback of the Propp–Wilson algorithm introduced in the previous two chapters is the need to reuse old random numbers: Recall that Markov chains are started at times -N1, -N2, … (where N1 < N2 < …) and so on until j is large enough so that starting from time -Nj gives coalescence at time 0. A crucial ingredient in the algorithm is that when the Markov chains start at time -Ni, the same random numbers as in previous runs should be used from time -Ni-1 and onwards. The typical implementation of the algorithm is therefore to store all new random numbers, and to read them again when needed in later runs. This may of course be costly in terms of computer memory, and the worst-case scenario is that one suddenly is forced to abort a simulation when the computer has run out of memory.

Various approaches to coping with this problem have been tried. For instance, some practitioners of the algorithm have circumvented the need for storage of random numbers by certain manipulations of (the seeds of) the random number generator. Such manipulations may, however, lead to all kinds of unexpected and unpleasant problems, and we therefore advise the reader to avoid them.

There have also been various attempts to modify the Propp–Wilson algorithm in such a way that each random number only needs to be used once. For instance, one could modify the algorithm by using new random variables each time that old ones are supposed to be used.

A key matter in many (most?) practical applications of Markov theory is the ability to simulate Markov chains on a computer. This chapter deals with how that can be done.

We begin by stating a lie:

This is a lie for at least two reasons:

1. (A) The numbers U0, U1, … obtained from random number generators are not uniformly distributed on [0, 1]. Typically, they have a finite binary (or decimal) expansion, and are therefore rational. In contrast, it can be shown that a random variable which (truly) is uniformly distributed on [0, 1] (or in fact any continuous random variable) is irrational with probability 1.

2. (B) U0, U1, … are not even random! Rather, they are obtained by some deterministic procedure. For this reason, random number generators are sometimes (and more accurately) called pseudo-random number generators.

The most important of these objections is (B), because (A) tends not to be a very big problem when the number of binary or decimal digits is reasonably large (say, 32 bits). Over the decades, a lot of effort has been put into constructing (pseudo-)random number generators whose output is as indistinguishable as possible from a true i.i.d. sequence of uniform [0, 1] random variables.

Combinatorics is the branch of mathematics which deals with finite objects or sets, and the ways in which these can be combined. Basic objects that often arise in combinatorics are, e.g., graphs and permutations. Much of combinatorics deals with the following sort of problem:

Given some set S, what is the number of elements of S?

Let us give a few examples of such counting problems; the reader will probably be able to think of several interesting variations of these.

Example 9.1 What is the number of permutations r = (r1, …, rq) of the set {1, …, q} with the property that no two numbers that differ by exactly 1 are adjacent in the permutation?

Example 9.2 Imagine a chessboard, and a set of 32 domino tiles, such that one tile is exactly large enough to cover two adjacent squares of the chessboard. In how many different ways can the 32 tiles be arranged so as to cover the entire chessboard?

Example 9.3 Given a graph G = (V, E), in how many ways can we pick a subset W of the vertex set V, with the property that no two vertices in W are adjacent in G? In other words, how many different feasible configurations exist for the hard-core model (see Example 7.1) on G?

Example 9.4 Given an integer q and a graph G = (V, E), how many different q-colorings (Example 7.3) are there for G?

In this chapter, we are interested in algorithms for solving counting problems.

Markov theory is a huge subject (much bigger than indicated by these notes), and consequently there are many books written on it. Three books that have influenced the present text are the ones by Brémaud [B], Grimmett & Stirzaker [GS], and (the somewhat more advanced book by) Durrett [Du]. Another nice introduction to the topic is the book by Norris [N]. Some of my Swedish compatriots will perhaps prefer to consult the texts by Rydén & Lindgren [RL] and Enger & Grandell [EG]. The reader can find plenty of additional material (more general theory, as well as other directions for applications) in any of these references.

Still on the Markov theory side (Chapters 2–6) of this text, there are two particular topics that I would warmly recommend for further study to anyone with a taste for mathematical elegance and the power and simplicity of probabilistic arguments: The first one is the coupling method, which was used to prove Theorems 5.2 and 8.1, and which also underlies the algorithms in Chapters 10–12; see the books by Lindvall [L] and by Thorisson [T]. The second topic is the relation between reversible Markov chains and electrical networks, which is delightfully treated in the book by Doyle & Snell [DSn]. Häggström [H] gives a short introduction in Swedish.

Another goldmine for the ambitious student is the collection of papers edited by Snell [Sn], where many exciting topics in probability, several of which concern Markov chains and/or randomized algorithms, are presented on a level accessible to advanced undergraduates.