Markov Chains and Stochastic Stability is one of those rare instances of a young book that has become a classic. In understanding why the community has come to regard the book as a classic, it should be noted that all the key ingredients are present. Firstly, the material that is covered is both interesting mathematically and central to a number of important applications domains. Secondly, the core mathematical content is nontrivial and had been in constant evolution over the years and decades prior to the first edition's publication; key papers were scattered across the literature and had been published in widely diverse journals. So, there was an obvious need for a thoughtful and well-organized book on the topic. Thirdly, and most important, the topic attracted two authors who were research experts in the area and endowed with remarkable skill in communicating complex ideas to specialists and applications-focused users alike, and who also exhibited superb taste in deciding which key ideas and approaches to emphasize.

When the first edition of the book was published in 1993, Markov chains already had a long tradition as mathematical models for stochastically evolving dynamical systems arising in the physical sciences, economics, and engineering, largely centered on discrete state space formulations. A great deal of theory had been developed related to Markov chain theory, both in discrete state space and general state space. However, the general state space theory had grown to include multiple (and somewhat divergent) mathematical strands, having much to do with the fact that there are several natural (but different) ways that one can choose to generalize the fundamental countable state concept of irreducibility to general state space.

A new edition of Meyn & Tweedie – what for?

The majority of topics covered in this book are well established. Ancient topics such as the Doeblin decomposition and even more modern concepts such as f-regularity are mature and not likely to see much improvement. Why then is there a need for a new edition?

Publication of this book in the Cambridge Mathematical Library is a way to honor my friend and colleague Richard Tweedie. The memorial article [103] contains a survey of his contributions to applied probability and statistics and an announcement of the initiation of the Tweedie New Researcher Award Fund. Royalties from the book will go to Catherine Tweedie and help to support the memorial fund.

Richard would be very pleased to know that our book will be placed on the shelves next to classics in the mathematical literature such as Hardy, Littlewood, and Pólya's Inequalities and Zygmund's Trigonometric Series, as well as more modern classics such as Katznelson's An Introduction to Harmonic Analysis and Rogers and Williams' Diffusions, Markov Processes and Martingales.

Other reasons for this new edition are less personal.

Motivation for topics in the book has grown along with growth in computer power since the book was last printed in March of 1996. The need for more efficient simulation algorithms for complex Markovian models, or algorithms for computation of optimal policies for controlled Markov models, has opened new directions for research on Markov chains [29, 113, 10, 245, 27, 267]. It has been exciting to see new applications to diverse topics including optimization, statistics, and economics.

In Part II we developed the ideas of stability largely in terms of recurrence structures. Our concern was with the way in which the chain returned to the “center” of the space, how sure we could be that this would happen, and whether it might happen in a finite mean time.

Part III is devoted to the perhaps even more important, and certainly deeper, concepts of the chain “settling down”, or converging, to a stable or stationary regime.

In our heuristic introduction to the various possible ideas of stability in Section 1.3, such convergence was presented as a fundamental idea, related in the dynamical systems and deterministic contexts to asymptotic stability. We noted briefly, in (10.4) in Chapter 10, that the existence of a finite invariant measure was a necessary condition for such a stationary regime to exist as a limit. In Chapter 12 we explored in much greater detail the way in which convergence of Pn to a limit, on topological spaces, leads to the existence of invariant measures.

In this chapter we begin a systematic approach to this question from the other side. Given the existence of π, when do the n-step transition probabilities converge in a suitable way to π?

We will prove that for positive recurrent ψ-irreducible chains, such limiting behavior takes place with no topological assumptions, and moreover the limits are achieved in a much stronger way than under the tightness assumptions in the topological context. The Aperiodic Ergodic Theorem, which unifies the various definitions of positivity, summarizes this asymptotic theory.

The structure of Markov chains is essentially probabilistic, as we have described it so far. In examining the stability properties of Markov chains, the context we shall most frequently use is also a probabilistic one: in Part II, stability properties such as recurrence or regularity will be defined as certain return to sets of positive ψ-measure, or as finite mean return times to petite sets, and so forth.

Yet for many chains, there is more structure than simply a σ-field and a probability kernel available, and the expectation is that any topological structure of the space will play a strong role in defining the behavior of the chain. In particular, we are used thinking of specific classes of sets in ℝn as having intuitively reasonable properties. When there is a topology, compact sets are thought of in some sense as manageable sets, having the same sort of properties as a finite set on a countable space; and so we could well expect “stable” chains to spend the bulk of their time in compact sets. Indeed, we would expect compact sets to have the sort of characteristics we have identified, and will identify, for small or petite sets.

Conversely, open sets are “non-negligible” in some sense, and if the chain is irreducible we might expect it at least to visit all open sets with positive probability. This indeed forms one alternative definition of “irreducibility”.

In this, the first chapter in which we explicitly introduce topological considerations, we will have, as our two main motivations, the desire to link the concept of ψ-irreducibility with that of open set irreducibility and the desire to identify compact sets as petite.

As with all stochastic processes, there are two directions from which to approach the formal definition of a Markov chain.

The first is via the process itself, by constructing (perhaps by heuristic arguments at first, as in the descriptions in Chapter 2) the sample path behavior and the dynamics of movement in time through the state space on which the chain lives. In some of our examples, such as models for queueing processes or models for controlled stochastic systems, this is the approach taken. From this structural definition of a Markov chain, we can then proceed to define the probability laws governing the evolution of the chain.

The second approach is via those very probability laws. We define them to have the structure appropriate to a Markov chain, and then we must show that there is indeed a process, properly defined, which is described by the probability laws initially constructed. In effect, this is what we have done with the forward recurrence time chain in Section 2.4.1.

From a practitioner's viewpoint there may be little difference between the approaches. In many books on stochastic processes, such as Çinlar [59] or Karlin and Taylor [194], the two approaches are used, as they usually can be, almost interchangeably; and advanced monographs such as Nummelin [303] also often assume some of the foundational aspects touched on here to be well understood.

Since one of our goals in this book is to provide a guide to modern general space Markov chain theory and methods for practitioners, we give in this chapter only a sketch of the full mathematical construction which provides the underpinning of Markov chain theory.

Despite our best efforts, we understand that the scope of this book inevitably leads to the potential for confusion in readers new to the subject, especially in view of the variety of approaches to stability which we have given, the many related and perhaps (until frequently used) forgettable versions of the “Foster-Lyapunov” drift criteria, and the sometimes widely separated conditions on the various models which are introduced throughout the book.

At the risk of repetition, we therefore gather together in this Appendix several discussions which we hope will assist in giving both the big picture, and a detailed illustration of how the structural results developed in this book may be applied in different contexts.

We first give a succinct series of equivalences between and implications of the various classifications we have defined, as a quick “mud map” to where we have been. In particular, this should help to differentiate between those stability conditions which are “almost” the same.

Secondly, we list together the drift conditions, in slightly abbreviated form, together with references to their introduction and the key theorems which prove that they are indeed criteria for different forms of stability and instability. As a guide to their usage we then review the analysis of one specific model (the scalar threshold autoregression, or SETAR model).

This model incorporates a number of sub-models (specifically, random walks and scalar linear models) which we have already analyzed individually: thus, although not the most complex model available, the SETAR model serves to illustrate many of the technical steps needed to convert elegant theory into practical use in a number of fields of application.