In Part II we developed the ideas of stability largely in terms of recurrence structures. Our concern was with the way in which the chain returned to the “center” of the space, how sure we could be that this would happen, and whether it might happen in a finite mean time.

Part III is devoted to the perhaps even more important, and certainly deeper, concepts of the chain “settling down”, or converging, to a stable or stationary regime.

In our heuristic introduction to the various possible ideas of stability in Section 1.3, such convergence was presented as a fundamental idea, related in the dynamical systems and deterministic contexts to asymptotic stability. We noted briefly, in (10.4) in Chapter 10, that the existence of a finite invariant measure was a necessary condition for such a stationary regime to exist as a limit. In Chapter 12 we explored in much greater detail the way in which convergence of Pn to a limit, on topological spaces, leads to the existence of invariant measures.

In this chapter we begin a systematic approach to this question from the other side. Given the existence of π, when do the n-step transition probabilities converge in a suitable way to π?

We will prove that for positive recurrent ψ-irreducible chains, such limiting behavior takes place with no topological assumptions, and moreover the limits are achieved in a much stronger way than under the tightness assumptions in the topological context. The Aperiodic Ergodic Theorem, which unifies the various definitions of positivity, summarizes this asymptotic theory.

The structure of Markov chains is essentially probabilistic, as we have described it so far. In examining the stability properties of Markov chains, the context we shall most frequently use is also a probabilistic one: in Part II, stability properties such as recurrence or regularity will be defined as certain return to sets of positive ψ-measure, or as finite mean return times to petite sets, and so forth.

Yet for many chains, there is more structure than simply a σ-field and a probability kernel available, and the expectation is that any topological structure of the space will play a strong role in defining the behavior of the chain. In particular, we are used thinking of specific classes of sets in ℝn as having intuitively reasonable properties. When there is a topology, compact sets are thought of in some sense as manageable sets, having the same sort of properties as a finite set on a countable space; and so we could well expect “stable” chains to spend the bulk of their time in compact sets. Indeed, we would expect compact sets to have the sort of characteristics we have identified, and will identify, for small or petite sets.

Conversely, open sets are “non-negligible” in some sense, and if the chain is irreducible we might expect it at least to visit all open sets with positive probability. This indeed forms one alternative definition of “irreducibility”.

In this, the first chapter in which we explicitly introduce topological considerations, we will have, as our two main motivations, the desire to link the concept of ψ-irreducibility with that of open set irreducibility and the desire to identify compact sets as petite.

As with all stochastic processes, there are two directions from which to approach the formal definition of a Markov chain.

The first is via the process itself, by constructing (perhaps by heuristic arguments at first, as in the descriptions in Chapter 2) the sample path behavior and the dynamics of movement in time through the state space on which the chain lives. In some of our examples, such as models for queueing processes or models for controlled stochastic systems, this is the approach taken. From this structural definition of a Markov chain, we can then proceed to define the probability laws governing the evolution of the chain.

The second approach is via those very probability laws. We define them to have the structure appropriate to a Markov chain, and then we must show that there is indeed a process, properly defined, which is described by the probability laws initially constructed. In effect, this is what we have done with the forward recurrence time chain in Section 2.4.1.

From a practitioner's viewpoint there may be little difference between the approaches. In many books on stochastic processes, such as Çinlar [59] or Karlin and Taylor [194], the two approaches are used, as they usually can be, almost interchangeably; and advanced monographs such as Nummelin [303] also often assume some of the foundational aspects touched on here to be well understood.

Since one of our goals in this book is to provide a guide to modern general space Markov chain theory and methods for practitioners, we give in this chapter only a sketch of the full mathematical construction which provides the underpinning of Markov chain theory.

Despite our best efforts, we understand that the scope of this book inevitably leads to the potential for confusion in readers new to the subject, especially in view of the variety of approaches to stability which we have given, the many related and perhaps (until frequently used) forgettable versions of the “Foster-Lyapunov” drift criteria, and the sometimes widely separated conditions on the various models which are introduced throughout the book.

At the risk of repetition, we therefore gather together in this Appendix several discussions which we hope will assist in giving both the big picture, and a detailed illustration of how the structural results developed in this book may be applied in different contexts.

We first give a succinct series of equivalences between and implications of the various classifications we have defined, as a quick “mud map” to where we have been. In particular, this should help to differentiate between those stability conditions which are “almost” the same.

Secondly, we list together the drift conditions, in slightly abbreviated form, together with references to their introduction and the key theorems which prove that they are indeed criteria for different forms of stability and instability. As a guide to their usage we then review the analysis of one specific model (the scalar threshold autoregression, or SETAR model).

This model incorporates a number of sub-models (specifically, random walks and scalar linear models) which we have already analyzed individually: thus, although not the most complex model available, the SETAR model serves to illustrate many of the technical steps needed to convert elegant theory into practical use in a number of fields of application.

The results presented in this book have been written in the desire that practitioners will use them. We have tried therefore to illustrate the use of the theory in a systematic and accessible way, and so this book concentrates not only on the theory of general space Markov chains, but on the application of that theory in considerable detail.

We will apply the results which we develop across a range of specific applications: typically, after developing a theoretical construct, we apply it to models of increasing complexity in the areas of systems and control theory, both linear and nonlinear, both scalar and vector valued; traditional “applied probability” or operations research models, such as random walks, storage and queueing models, and other regenerative schemes; and models which are in both domains, such as classical and recent time series models.

These are not given merely as “examples” of the theory: in many cases, the application is difficult and deep of itself, whilst applications across such a diversity of areas have often driven the definition of general properties and the links between them. Our goal has been to develop the analysis of applications on a step-by-step basis as the theory becomes richer throughout the book.

To motivate the general concepts, then, and to introduce the various areas of application, we leave until Chapter 3 the normal and necessary foundations of the subject, and first introduce a cross-section of the models for which we shall be developing those foundations.

In this chapter we introduce the culminating form of the geometric ergodicity theorem, and show that such convergence can be viewed as geometric convergence of an operator norm; simultaneously, we show that the classical concept of uniform (or strong) ergodicity, where the convergence in (13.4) is bounded independently of the starting point, becomes a special case of this operator norm convergence.

We also take up a number of other consequences of the geometric ergodicity properties proven in Chapter 15, and give a range of examples of this behavior. For a number of models, including random walk, time series and state space models of many kinds, these examples have been held back to this point precisely because the strong form of ergodicity we now make available is met as the norm, rather than as the exception. This is apparent in many of the calculations where we verified the ergodic drift conditions (V2) or (V3): often we showed in these verifications that the stronger form (V4) actually held, so that unwittingly we had proved V-uniform or geometric ergodicity when we merely looked for conditions for ergodicity.