The Markov property and its immediate consequences

Typically, the subject of Markov chains represents a logical continuation from a basic course of probability. We will study a class of random processes describing a wide variety of systems of theoretical and practical interest (and sometimes simply amusing). The fact that deep insight into the subject is possible without using sophisticated mathematical tools may also be an explanation of why Markov chains are popular in so many different disciplines which are seemingly remote from pure mathematics.

The basic model for the first half of the book will be a system which changes state in discrete time, according to some random mechanism. The collection of states is called a state space and throughout the whole book will be assumed finite or countable; we will denote it by I. Each i ∈ I is called a state; our system will always be in one of these states. Sometimes we will know what state the system occupies and sometimes only that the system is in state i with some probability.