Introduction

The counting processes {N(t); t > 0} described in Section 2.1.1 have the property that N(t) changes at discrete instants of time, but is defined for all real t > 0. The Markov chains to be discussed in this chapter are stochastic processes defined only at integer values of time, n = 0,1,…. At each integer time n ≥ 0, there is an integer-valued random variable (rv) Xn , called the state at time n, and the process is the family of rv s {Xn; n ≥ 0}. We refer to these processes as integer-time processes. An integer-time process {Xn; n ≥ 0} can also be viewed as a process {X(t); t ≥ 0} defined for all real t by taking X(t) = Xn for n ≤ t ≤ n + 1, but since changes occur only at integer times, it is usually simpler to view the process only at those integer times.

In general, for Markov chains, the set of possible values for each rv Xn is a countable set S. If S is countably infinite, it is usually taken to be S = {0,1,2,…}, whereas if S is finite, it is usually taken to be S = {1,…, M}. In this chapter (except for Theorems 4.2.8 and 4.2.9), we restrict attention to the case in which S is finite, i.e., processes whose sample functions are sequences of integers, each between 1 and M. There is no special significance to using integer labels for states, and no compelling reason to include 0 for the countably infinite case and not for the finite case. For the countably infinite case, the most common applications come from queueing theory, where the state often represents the number of waiting customers, which might be zero. For the finite case, we often use vectors and matrices, where positive integer labels simplify the notation. In some examples, it will be more convenient to use more illustrative labels for states.