Introduction

The Kalman-Bucy filter and its many variations are widely used throughout science and engineering for estimating the state of time varying systems. Applications to computer vision in particular are described in. A typical filter contains a model of the system dynamics and of the measurement process. It is given a sequence of measurements obtained over an extended time, and produces from this sequence an estimate of the probability density function for the system state, conditional on the measurements. If the filter is optimal, then the estimated density is the exact conditional density. The best known optimal filter is the Kalman-Bucy filter.

In sections 1.1, 1.2 and 1.3 of this chapter, we briefly introduce basic notions and some results borrowed from the theory of discrete time homogeneous countable Markov chains (MC).

In section 1.4, some well known examples of MCs are given, for which a complete classification can be obtained by elementary methods: simple probabilistic arguments in 1.4.1, explicit solution of recurrent equations in 1.4.2, generating functions in 1.4.3.

It is not our intention to devote a detailed section to the fundamentals of probability theory, which are presented in a plethora of excellent text-books. Thus, we only introduce in fact the minimal basic notions and notation useful for our purpose.

• The events are the subsets of some abstract set Ω, which belong to Σ, the σ-algebra defined on Ω.

• The couple (Ω, Σ) is a measurable space and the sets belonging to Σ are

Σ-measurable sets.

• The triple (Ω, Σ, µ), where µ, is a positive measure defined on Σ, is a measure space. A probability space is a measure space of total measure 1, i.e. µ(Σ) = 1, and in this case most of the time we shall write (Ω, Σ, P).

• A Σ-measurable real-valued function f with domain Ω is called a random variable. More generally a random element ϕ with values in a measurable space (X, B) is a measurable mapping of (Ω, Σ, P) into (X, B). For X = RN or ZN, B being the σ-algebra of Borel sets, we shall speak of random vectors.

Introduction

This book differs essentially from the existing monographs on countable Markov chains. It intends to be, on the one hand, much more constructive than books similar to, for example Chung's [Chu67] and, on the other hand, much less constructive than some elementary monographs on queueing theory, where the emphasis is mainly put on the derivation of explicit expressions. The method of generating functions, which is to be sure the most constructive approach, is not included, since the dimension of the problems it can solve is small (in general ≤ 2). Our book could equally be called Constructive use of Lyapounov functions method. Here the term constructive is taken in the sense close to the one widely accepted in constructive mathematical physics. One can say that the objects considered have a sufficiently rich structure to be concrete, although the results may not always be explicit enough, as commonly understood. Semantically, it is permissible to say that our methods are more qualitative constructive than quantitative constructive.

The main goal of the book is to provide methods allowing a complete classification (necessary and sufficient conditions) or, in other words, allowing us to say when a Markov chain is ergodic, null recurrent or transient. Moreover, it turns out that, without doing much additional work, it is possible to study the stability (continuity or even analyticity) with respect to parameters, the rate of convergence to equilibrium, …, etc. by using the same Lyapounov functions.

Stochastic processes and filtrations

Let (Y, ℳ, P) be a probability space and denote points in Y by q. A Borel measurable function η: Y → Rd is called an Rd-valued random variable on (Y,ℳ). The expectation Eη of η with respect to the probability measure P is defined by Eη = ∫yηdP. The Rd-valued random variable η on (Y,ℳ, P) induces a probability measure v on Rd defined by v(B) = P(η ∈ B), for B ∈ Bd, where Bd is the Borel σ-algebra on Rd; v is called the distribution of η.

A family ℳt, t ≥ 0, of σ-algebras satisfying ℳt1 ⊆ ℳt2 for t1 ≤ t2 and ℳt ⊆ ℳ for all t ≥ 0 is called a filtration on (Y, ℳ). If for all t ≥ 0, ℳt includes all sets of P-measure zero, the filtration is called complete on (Y, ℳ, P). If ℳ't is the smallest σ-algebra containing ℳt and all sets of P-measure zero, then ℳ't is called the completion of ℳt.

A measurable map ζ: [0, ∞) × Y →,Rd is called an Rd-valued stochastic process on (Y,ℳ, P). We frequently suppress the variable q ∈ Y and write ζ(t) for ζ(t, q). The stochastic process ζ(t) on (Y, ℳ, P) induces a probability measure v on B([0, ∞), Rd), the space of measurable functions from [0, ∞) to Rd with the sup-norm topology, defined by v(B) = P(ζ(·) ∈ B), for Borel sets B in B([0, ∞), Rd). The measure v is called the distribution of ζ(·).

In the first three sections of this chapter, we recall a number of standard results from functional analysis and elliptic PDE theory. The majority of these results are stated without proof; the reader is referred to the notes at the end of the chapter.

The spectral theory of compact operators

Let B be a Banach space and let A be a linear operator defined on a dense subspace D ⊂ B and taking values in B. The operator A is called closed if its graph {(x, Ax); x ∈ D) is a closed subset of B × B. It is called closable if it can be extended to a closed operator. Clearly, A is closeable if and only if, whenever xn → 0 and Axn converges, then, in fact, Axn → 0. It follows from the closed graph theorem that a closed operator defined on a Banach space B is bounded. Thus, if A is closed and unbounded, its domain of definition D must necessarily be a proper subspace of B. The resolvent set ρ(A) of a closed densely defined operator A is defined as the collection of complex numbers λ for which λ – A is a bijection of D onto B. If λ ∈ ρ(A), then (λ − A)−1 is a closed operator from B into B and it thus follows from the closed graph theorem that (λ − A)−1 is in fact a bounded operator. The resolvent set can be shown to be an open subset of ℂ.