1. Introduction
Let S be a finite or countably infinite state space, and for
$1\le n \le \infty$
, let
$P_n = (P_n(x, y) \colon x, y \in S)$
be a one-step transition matrix indexed by S. Suppose that for
$1\le n \le \infty$
,
$P_n$
possesses a unique stationary distribution
$\pi_n$
. If, for each
$x, y \in S$
,
\begin{equation} P_n(x, y) \to P_\infty(x, y) \end{equation}
as
$n \to \infty$
, this raises the natural question of when
\begin{equation} \pi_n(y) \to \pi_\infty(y) \end{equation}
as
$n \to \infty$
, for each
$y \in S$
. When
$P_n$
is irreducible and aperiodic for
$1\le n \le \infty$
,
\begin{equation} P_n^m(x, y) \to \pi_n(y) \end{equation}
as
$ m \to \infty $
, and (2) and (3) assert that for each
$ x, y \in S $
, the iterated limits commute, so that
\begin{equation} \lim_{m \to \infty} \lim_{n \to \infty} P_n^m(x, y) = \lim_{n \to \infty} \lim_{m \to \infty} P_n^m(x, y),\end{equation}
We refer to the interchangeability of the limits in m and n associated with (4) as the weak interchange property and to (2) as the stationary distribution convergence property. The weak interchange property implies that whether we proceed vertically and then horizontally in the diagram below, versus horizontally and then vertically, we get the same result.
But this raises the question of the dotted line, diagonal convergence. In particular, when is it true, in the presence of (1), (2), and (3), that whenever
$ m_n \to \infty $
we have that
\begin{equation} P_n^{m_n}(x, y) \to \pi_\infty(y) \end{equation}
as
$n\to \infty$
, for
$x, y \in S$
?
When (5) holds for every sequence
$ m_n \to \infty $
, this is equivalent, in the presence of (1), (2), and (3), to requiring that
\begin{equation} \sup_{m\ge 0} \sum_{y \in S} |P_n^m(x, y) - P^m_\infty(x, y)| \to 0 \end{equation}
as
$n\to\infty$
. We refer to (6) as the strong interchange property, since it implies that the marginals of the Markov chain driven by
$ P_n $
can be well approximated by those of
$ P_\infty $
at all time scales, namely short, long, and intermediate scales. (The weak interchange property guarantees only that the marginals of the chain
$ (X_n\colon{n \geq 0} )$
under
$ P_n $
are well approximated for short scales – fixed m – and long scales – in equilibrium).) Note that the strong interchange property ensures, for example, that all lag-m covariances between
$f(X_j)$
and
$f(X_{j+m})$
computed under
$P_n$
are uniformly (in j and m) close to those computed under
$P_\infty$
, so that the mixing structure can be suitably approximated (at least when f is bounded).
Perhaps surprisingly, the weak interchange property implies diagonal convergence and the strong interchange property in this setting of countable state spaces; see Corollary 1. However, in Section 4 we provide a counter-example when
$ S = [0, 1] $
in which the convergence in (1), (2), and (3) is replaced by weak convergence. While the weak interchange and strong interchange properties are generally not equivalent in continuous state space, we prove that they are equivalent when (1), (2), and (3) are generalized to chains exhibiting total variation convergence (Proposition 1), and when weak convergence is supplemented by a suitable Lipschitz inclusion condition on the transition kernel of the chain (Proposition 2). When Proposition 2 is applied to the delay sequence of the
$\mathrm{GI}/\mathrm{G}/1$
queue, we obtain a generalization of a result of [Reference Breuer4]; see Example 1.
The weak interchange problem has been studied in the setting of diffusion approximations for queues [Reference Braverman and Dai3, Reference Budhiraja and Lee5, Reference Dai, Glynn and Xu6, Reference Garmarnik and Zeevi8, Reference Gurvich11, Reference Lee, Ward and Ye15, Reference Ye and Yao22, Reference Ye and Yao23]. This issue also arises in the study of continuity of stochastic models as a function of their ‘input sequences’; see, for example, [Reference Kennedy13, Reference Whitt19, Reference Whitt20]. [Reference Karr12] provides conditions guaranteeing that a sequence of continuous state space Markov chains exhibits the weak interchange property.
As noted above, conditions (1)–(3) ensure that the Markov chain associated with
$ P_n $
is well approximated at both short and long time scales. When S is countably infinite, stationary distribution convergence implies diagonal convergence. We might hope that virtually all performance measures for the chain evolving under
$ P_n $
can be approximated by the corresponding performance measure under
$ P_\infty $
. This turns out to be true in the countable state setting; see Theorem 2 and Proposition 3.
We conclude by noting that the questions discussed in this paper are also highly relevant to the extensive literature on numerical truncation methods for Markov chains. Early work includes [Reference Golub and Seneta9, Reference Golub and Seneta10, Reference Seneta17, Reference Seneta18, Reference Wolf21], in which conditions are developed that ensure that the weak interchange property holds for a sequence
$ P_n $
of transition matrix approximations to
$ P_\infty $
. This is relevant to numerical algorithms intended to compute the stationary distribution of the limiting infinite state transition matrix
$ P_\infty $
. Similarly, we might wish to use the finite-dimensional matrix
$ P_n^m $
as an approximation to
$ P_\infty^m $
, or to compute expectations for
$ P_\infty $
involving first-transition analysis by solving corresponding finite-dimensional linear systems involving
$ P_n $
. The results in the current paper make clear that any truncation method that preserves stationary distribution convergence is guaranteed to also consistently approximate all these other performance measures associated with
$ P_\infty $
. This is especially useful in view of the fact that the great majority of the existing literature on state space truncation has focused on the question of stationary distribution convergence.
The paper is organized as follows. Section 2 develops a general theorem (Theorem 1) that provides conditions under which the weak interchange property implies the strong interchange property, and then specializes the result first to the total variation convergence setting (Proposition 1) and then to the weak convergence setting (Proposition 2). Section 3 shows that when the state space is countably infinite, the stationary distribution convergence property implies convergence of all the expectations that are computable via first transition analysis. Section 4 provides a discussion of a counter-example demonstrating that the weak interchange property does not imply the strong interchange property in general, when the Markov chain has a continuous state space. Finally, Section 5 extends the theory to the Markov jump process settings.
2. The weak and strong interchange properties
Let
$ X = (X_m\colon{m \geq 0} )$
be an S-valued Markov chain. (We do not assume in this section that S is finite or countably infinite, unless otherwise stated.) For
$ 1 \leq n \leq \infty $
and
$ x \in S $
, let
$ P_{n,x}({\cdot}) $
(
$ \mathbb{E}_{n,x}({\cdot}) $
) be the probability (expectation) on the path-space
$ \Omega = S^\infty $
under which
$ X_0 = x $
and
$P_{n,x}(X_{m+1} \in \mathrm{d}z \mid X_m = y) = P_n(y, \mathrm{d}z)$
for
$ y, z \in S $
and
$ m \geq 0 $
, where
$ P_n = ( P_n(y, \mathrm{d}z)\colon y,z \in S ) $
is a transition kernel on S.
Given a finite signed measure
$ \nu $
on S, and a set
$ \mathcal{F} $
of (suitably measurable) bounded real-valued functions, put
$\| \nu \|_{\mathcal{F}} \triangleq \sup_{f \in \mathcal{F}} |\int_S f(x)\,\nu({\mathrm{d}} x)|$
.
We note that, in general,
$ \| \cdot \|_{\mathcal{F}} $
is not a norm, since
$ \| \nu \|_{\mathcal{F}} = 0 $
may not imply that
$ \nu = 0 $
. When
$ \mathcal{F}_w $
is the set of functions f for which
$ |f(x)| \leq w(x) $
for
$ x \in S $
(with w positive), we denote
$ \| \nu \|_{\mathcal{F}_w} $
as
$ \| \nu \|_w $
.
Remark 1. The quantity
$ \|\cdot\|_w $
is the w-weighted total variation norm. When
$ w = e $
with
$ e(x) = 1 $
for
$ x \in S $
, this corresponds to the (standard) total variation norm. Given a sequence
$ (\eta_n\colon{1\le n \le \infty}) $
of probabilities for which
$ \|\eta_n - \eta_\infty\|_e \to 0 $
as
$ n \to \infty $
, we write
$ \eta_n \overset{\mathrm{tv}}{\to} \eta_\infty $
as
$ n \to \infty $
.
We say that
$\mathcal{F}$
is universally bounded if
$\kappa(\mathcal{F}) \triangleq \sup\{|f(x)|\colon x\in S, f\in \mathcal{F}\}\lt\infty$
. Given a transition kernel Q on S, we say that Q has the
$\mathcal{F}$
inclusion property if
$ Qf \in \mathcal{F} $
whenever
$ f \in \mathcal{F} $
, where
$(Qf)(x) = \int_S f(y)\,Q(x, {\mathrm{d}} y)$
for
$ x \in S $
.
Remark 2. Note that every transition kernel Q has the
$ \mathcal{F}_e $
inclusion property, since
$ |f(x)| \leq 1 $
for
$ x \in S $
always implies that
$ |(Qf)(x)| \leq 1 $
for
$ x \in S $
.
Theorem 1. Suppose
$\mathcal{F}$
is universally bounded. For
$ 1 \leq n \leq \infty $
, suppose that
$ P_n $
possesses a unique stationary distribution
$ \pi_n $
and has the
$ \mathcal{F} $
inclusion property. Assume that, for each
$ x \in S $
:
-
(i)
$ \| P_{n,x}(X_1 \in \cdot) - P_{\infty,x}(X_1 \in \cdot) \|_{\mathcal{F}} \to 0 $
as
$ n \to \infty $
; -
(ii)
$ \| P_{\infty,x}(X_m \in \cdot) - \pi_\infty({\cdot}) \|_{\mathcal{F}} \to 0 $
as
$ m \to \infty $
; -
(iii)
$ \| \pi_n - \pi_\infty \|_{\mathcal{F}} \to 0 $
as
$ n \to \infty $
.
Then, for each
$ x \in S $
,
\begin{equation} \sup_{m \geq 0} \| P_{n,x}(X_m \in \cdot) - P_{\infty,x}(X_m \in \cdot) \|_{\mathcal{F}} \to 0 \end{equation}
as
$n \to \infty$
.
We note that (7) implies that if
$ m_n \to \infty $
,
$ \| P_{n,x}(X_{m_n} \in \cdot) - \pi_\infty({\cdot}) \|_{\mathcal{F}} \to 0 $
, so that it implies diagonal convergence. An interesting aspect of the theorem is that it does not assume that
$ \| P_{n,x}(X_m \in \cdot) - \pi_n({\cdot}) \|_{\mathcal{F}} \to 0 $
as
$ m \to \infty $
.
Proof of Theorem 1. We first note that if
$f\in\mathcal{F}$
then, for
$m\ge 2$
,
\begin{align} & |\mathbb{E}_{n,x}f(X_m) - \mathbb{E}_{\infty,x}f(X_m)| \nonumber \\ & \qquad = \bigg|\int_S P_{\infty,x}(X_{m-1} \in {\mathrm{d}} y)[(P_{\infty} f)(y) - (P_n f)(y)] \nonumber \\ & \qquad\quad + \int_S P_{\infty,x}(X_{m-1} \in {\mathrm{d}} y)(P_n f)(y) - \int_S P_{n,x}(X_{m-1} \in {\mathrm{d}} y)(P_n f)(y)\bigg| \nonumber \\ & \qquad \leq \int_S P_{\infty,x}(X_{m-1} \in {\mathrm{d}} y) \|P_{\infty,y}(X_{1}\in \cdot) - P_{n,y}(X_{1}, \cdot)\|_{\mathcal{F}} \nonumber \\ & \qquad\quad + \|P_{\infty,x}(X_{m-1}\in \cdot) - P_{n,x}(X_{m-1}\in \cdot)\|_{\mathcal{F}}, \end{align}
where we use the fact that
$P_n f\in \mathcal{F}$
for the inequality. Observe that
\begin{equation*} \| P_{\infty,y}(X_1\in \cdot) - P_{n,y}(X_1\in \cdot) \|_{\mathcal{F}} \le 2 \kappa(\mathcal{F}), \end{equation*}
and hence (i), in combination with the bounded convergence theorem, shows that the first term on the right-hand side of (8) converges to 0. Induction in m therefore establishes that (i) implies that
$ \| P_{\infty,x}(X_m\in \cdot) - P_{n,x}(X_m\in \cdot) \|_{\mathcal{F}}\rightarrow 0 $
as
$n\to\infty$
.
We next observe that for
$ 1 \leq n \leq \infty $
and
$ m \geq 0 $
,
\begin{align} \| P_{n,x}(X_{m+1} \in \cdot) - \pi_n({\cdot}) \|_{\mathcal{F}} & = \sup_{f\in\mathcal{F}}\bigg|\int_Sf(y)(P_{n,x}(X_{m+1}\in{\mathrm{d}} y) - \pi_n({\mathrm{d}} y))\bigg| \nonumber \\ & = \sup_{f\in\mathcal{F}}\bigg|\int_S(P_{n}f)(y)(P_{n,x}(X_m\in{\mathrm{d}} y) - \pi_n({\mathrm{d}} y))\bigg| \nonumber \\ & = \sup_{P_nf:f\in\mathcal{F}}\bigg|\int_Sg(y)(P_{n,x}(X_m\in{\mathrm{d}} y) - \pi_n({\mathrm{d}} y))\bigg| \nonumber \\ & \le \sup_{g\in\mathcal{F}}\bigg|\int_Sg(y)(P_{n,x}(X_m\in{\mathrm{d}} y) - \pi_n({\mathrm{d}} y))\bigg| \nonumber \\ & = \|P_{n,x}(X_m \in \cdot) - \pi_n({\cdot}) \|_{\mathcal{F}}, \end{align}
so
$(\| P_{n,x}(X_m \in \cdot) - \pi_n({\cdot}) \|_{\mathcal{F}}\colon m\ge 0)$
is a non-increasing sequence. (We used the stationarity of
$ \pi_n $
in the second equality and the
$ \mathcal{F} $
-inclusion property for the inequality.)
For
$ \varepsilon \gt 0 $
, property (ii) implies the existence of
$t= t(\varepsilon) $
for which
\begin{equation*} \| P_{\infty,x}(X_{t} \in \cdot) - \pi_\infty({\cdot}) \|_{\mathcal{F}} \lt \varepsilon / 2. \end{equation*}
Since
$ \| \cdot \|_{\mathcal{F}} $
satisfies the triangle inequality,
\begin{align} & \quad \| P_{n,x}(X_m \in \cdot) - P_{\infty,x}(X_m \in \cdot) \|_{\mathcal{F}} \nonumber \\ & \quad \le \max_{0 \leq j \lt t} \| P_{n,x}(X_j \in \cdot) - P_{\infty,x}(X_j \in \cdot) \|_{\mathcal{F}} + \sup_{j \geq t} \| P_{n,x}(X_j \in \cdot) - P_{\infty,x}(X_j \in \cdot) \|_{\mathcal{F}} \nonumber \\ & \quad \leq \max_{0 \leq j \lt t} \| P_{n,x}(X_j \in \cdot) - P_{\infty,x}(X_j \in \cdot) \|_{\mathcal{F}} + \sup_{j \geq t} \| P_{n,x}(X_j \in \cdot) - \pi_n({\cdot}) \|_{\mathcal{F}} \nonumber \\ & \qquad + \| \pi_n({\cdot})- \pi_\infty({\cdot}) \|_{\mathcal{F}} + \sup_{j \geq t} \| P_{\infty,x}(X_j \in \cdot) - \pi_\infty({\cdot}) \|_{\mathcal{F}}. \nonumber \\ & \quad \leq \max_{0 \leq j \lt t} \| P_{n,x}(X_j \in \cdot) - P_{\infty,x}(X_j \in \cdot) \|_{\mathcal{F}} + \| P_{n,x}(X_t \in \cdot) - \pi_n({\cdot}) \|_{\mathcal{F}} \nonumber \\ & \qquad + \| \pi_n - \pi_\infty \|_{\mathcal{F}} + \| P_{\infty,x}(X_t \in \cdot) - \pi_\infty({\cdot}) \|_{\mathcal{F}} \nonumber \\ & \quad = \max_{0 \leq j \lt t} \| P_{n,x}(X_j \in \cdot) - P_{\infty,x}(X_j \in \cdot) \|_{\mathcal{F}} \nonumber \\ & \qquad + \| (P_{n,x}(X_t \in \cdot) - P_{\infty,x}(X_t \in \cdot)) - (\pi_n({\cdot}) - \pi_\infty({\cdot})) \nonumber \\ & \qquad + P_{\infty,x}(X_t\in \cdot)-\pi_\infty ({\cdot}) \|_{\mathcal{F}} + \| \pi_n - \pi_\infty \|_{\mathcal{F}} + \| P_{\infty,x}(X_t \in \cdot) - \pi_\infty({\cdot}) \|_{\mathcal{F}} \nonumber \\ & \quad \leq 2\max_{0 \leq j \le t} \| P_{n,x}(X_j \in \cdot) - P_{\infty,x}(X_j \in \cdot) \|_{\mathcal{F}} + 2 \| \pi_n - \pi_\infty \|_{\mathcal{F}} + 2 \|P_{\infty,x}(X_t\in\cdot) - \pi_\infty({\cdot})\|_{\mathcal{F}} \nonumber \\ & \quad \le 2 \max_{0 \leq j \le t} \| P_{n,x}(X_j \in \cdot) - P_{\infty,x}(X_j \in \cdot) \|_{\mathcal{F}} + 2 \| \pi_n - \pi_\infty \|_{\mathcal{F}} + \varepsilon. \end{align}
Here, we used (9) for the third inequality. Hence, (10) implies that
\begin{align*} \sup_{m \geq 0} \| P_{n,x}(X_m \in \cdot) &- P_{\infty,x}(X_m \in \cdot) \|_{\mathcal{F}} \\ & \quad \leq 2\max_{0 \leq j \le t} \| P_{n,x}(X_j \in \cdot) - P_{\infty,x}(X_j \in \cdot) \|_{\mathcal{F}} + 2 \| \pi_n - \pi_\infty \|_{\mathcal{F}} + \varepsilon. \end{align*}
We now send
$ n \to \infty $
using (i) and (iii), and then send
$ \varepsilon \downarrow 0 $
, yielding the desired conclusion.
We now apply this theorem to obtain diagonal convergence when we have convergence in total variation.
Proposition 1. Assume that, for
$ 1 \leq n \leq \infty $
,
$ P_n $
has a unique stationary distribution
$ \pi_n $
, and suppose that, for each
$ x \in S $
,
-
(i)
$ P_n(x, \cdot) \overset{\mathrm{tv}}{\to} P_\infty(x, \cdot) $
as
$ n \to \infty $
; -
(ii)
$ P_{\infty,x}(X_m \in \cdot) \overset{\mathrm{tv}}{\to} \pi_\infty({\cdot}) $
as
$ m \to \infty $
; -
(iii)
$ \pi_n \overset{\mathrm{tv}}{\to} \pi_\infty $
as
$ n \to \infty $
.
Then, for
$ x \in S $
,
$ \sup_{m \geq 0} \| P_{n,x}(X_m \in \cdot) - P_{\infty,x}(X_m \in \cdot) \|_e \to 0 $
as
$n \to \infty$
.
Proof. We apply Theorem 1 with
$ \mathcal{F} = \mathcal{F}_e $
and note that
$\mathcal{F}_e$
is universally bounded.
Corollary 1. Suppose that S is finite or countably infinite. For
$1 \leq n \leq \infty$
, assume that the transition matrix
$P_n = (P_n(x,y)\colon{x,y \in S})$
has a unique stationary distribution
$\pi_n = (\pi_n(x)\colon{x \in S})$
. If, for each
$x, y \in S$
,
-
(i)
$P_n(x,y) \to P_\infty(x,y)$
as
$n \to \infty$
; -
(ii)
$P_\infty^m(x,y) \to \pi_\infty(y)$
as
$m \to \infty$
; -
(iii)
$\pi_n(y) \to \pi_\infty(y)$
as
$n \to \infty$
,
then
$\sup_{m \geq 0} \sum_{y \in S} | P_n^m(x,y) - P_\infty^m(x,y)| \to 0$
as
$n \to \infty$
.
The corollary follows immediately from Scheffé’s lemma (see, for example, [Reference Billingsley2, p. 246]), since it shows that pointwise convergence of
$\nu$
-densities implies
$L^1$
convergence. So, when S is discrete, the weak interchange property implies diagonal convergence.
Remark 3. In the truncation context for countably infinite chains, it is standard that the nth truncation corresponds to a finite state space
$S_n$
, where
$\phi \neq S_1 \subseteq S_2 \subseteq \cdots$
and
$\cup_{n=1}^{\infty} S_n = S$
, and
$(P_n(x,y)\colon x,y \in S_n)$
is a transition matrix with a single closed communicating class (so that
$(\pi_n(x)\colon x \in S_n)$
exists uniquely). To apply the corollary to the truncation setting, we extend
$(P_n(x,y)\colon x,y \in S_n)$
and
$(\pi_n(x)\colon x \in S_n)$
to S by choosing an arbitrary
$z \in S_1$
and putting
$P_n(x,z) = 1$
for
$x \notin S_n$
and
$\pi_n(x) = 0$
for
$x \notin S_n$
. This extension makes
$S_n$
absorbing for
$P_n$
. We conclude that whenever a truncation algorithm exhibits stationary distribution convergence, then
$P_\infty^m(x,y)$
may be approximated by
$P_n^m(x,y)$
across all time values at m for n sufficiently large and x, y fixed, so that the truncation approximates well all the marginals of the Markov chain under
$P_\infty$
at all time scales. (We note that without such a result, we would need to worry about the range of m over which good approximations hold.)
We now extend Corollary 1 to the more general weighted total variation norm
$\|\cdot\|_w$
when S is discrete.
Corollary 2. Suppose that S is finite or countably infinite. If the conditions of Corollary 1 hold,
$P_\infty$
is irreducible and
\begin{equation} \sum_{x \in S} w(x) \pi_n(x) \to \sum_{x \in S} w(x) \pi_\infty(x) \end{equation}
as
$n \to \infty$
, then
$ \sup_{m \geq 0} \sum_{y \in S} w(y) | P_n^m(x,y) - P_\infty^m(x,y)| \to 0 $
as
$n \to \infty$
.
Proof. We observe that the stationarity of
$\pi_n$
implies that
\begin{equation*} \sum_{x \in S} \pi_n(x) P_n^m(x,y) = \pi_n(y) \end{equation*}
for
$y \in S$
and
$m, n \geq 1$
. So,
$\pi_n(x) P_n^m(x,y) \leq \pi_n(y)$
. The irreducibility of
$P_\infty$
implies that
$\pi_n(x) \gt 0$
for
$n\ge n_0$
with
$n_0$
sufficiently large. Hence,
\begin{equation} P_n^m(x,y) \leq \frac{\pi_n(y)}{\pi_n(x)} \end{equation}
for
$x, y \in S$
and
$n\ge n_0$
. Since
$\pi_n \overset{\mathrm{tv}}{\to} \pi_\infty$
and (11) holds, w is uniformly integrable with respect to
$(\pi_n\colon n \geq 1)$
(see [Reference Billingsley2]), so that
$ \sup_{n_0 \leq j \leq \infty} \sum_{y} w(y) \pi_j(y) \mathbf{1}(w(y) \geq b) \to 0 $
as
$r \to \infty$
. Hence, in view of (12),
\begin{align*} \sum_{y \in S} w(y) | P_n^m(x,y) - P_\infty^m(x,y)| & \le b\max_{j \geq 0} \| P_{n, x}(X_j({\cdot})) - P_{\infty, x}(X_j({\cdot})) \|_e \\ & \quad + 2 \sup_{n_0\leq j\leq \infty} \sum_{y \in S} w(y) \pi_j(y) \mathbf{1}(w(y) \gt b) / \min_{n_0\leq j\leq \infty} \pi_j(x). \end{align*}
Sending first
$n \to \infty$
(and applying Corollary 1) and then
$b \to \infty$
yields the corollary.
We conclude this section with a version of Proposition 1 that involves weak convergence rather than total variation convergence. Assume now that S is a metric space with metric d. For
$r \gt0$
, let
$\mathrm{Lip}_\mathrm{b}(r)$
be the family of functions f such that
$|f| \leq e$
and
\begin{equation} |f(x) - f(y)| \leq r \,d(x, y) \end{equation}
for
$x, y \in S$
, so that f has Lipschitz constant at most r. Note that
$\mathrm{Lip}_\mathrm{b}(r)$
is a universally bounded family of functions. We now assume that for
$1 \leq n \leq \infty$
,
$P_n$
has the
$\mathrm{Lip}_\mathrm{b}(1)$
inclusion property. In particular, this holds if the Markov chain X can be represented as a random iterated function system under
$P_n$
, in which, conditional on
$X_0 = x$
,
\begin{equation*}X_m = (\varphi_n(m) \circ \varphi_{n}(m-1) \circ \ldots \circ \varphi_n(1))(x),\end{equation*}
where
$(\varphi_n(i) \colon i \geq 1)$
is a sequence of independent and identically distributed (i.i.d.) random functions mapping S into S such that
${\mathbb{E}} d(\varphi_n(1)(x), \varphi_n(1)(y)) \leq r\,d(x, y)$
for
$x, y \in S$
, where
$0\le r\le 1$
. To verify the inclusion property, note that for
$f \in \mathrm{Lip}_\mathrm{b}(1)$
,
\begin{equation*} |(P_n f)(x) - (P_n f)(y)| \leq \mathbb{E}|f(\varphi_n(1)(x)) - f(\varphi_n(1)(y))| \leq \mathbb{E} |d(\varphi_n(1)(x), \varphi_n(1)(y))| \leq r\,d(x, y)\end{equation*}
for
$x,y\in S$
, so that
$P_n f \in \mathrm{Lip}_\mathrm{b}(r) \subseteq \mathrm{Lip}_\mathrm{b}(1)$
.
Remark 4. Suppose that
$\mathrm{Lip}(r)$
is the family of functions satisfying (13) (but with no requirement that
$|f|\le e$
). It is worth noting that
$\|\eta_1-\eta_2\|_{\mathrm{Lip}(1)}$
is then exactly the Wasserstein 1-distance
$W_1(\eta_1,\eta_2)$
.
Let
$\Rightarrow$
denote weak convergence in S.
Proposition 2. Suppose that S is a separable metric space with metric d, and assume that for
$1 \leq n \leq \infty$
,
$P_n$
has the
$\mathrm{Lip}_\mathrm{b}(1)$
inclusion property and possesses a unique stationary distribution
$\pi_n$
. Assume further that, for each
$x \in S$
,
-
(i)
$P_n(x, \cdot) \Rightarrow P_\infty(x, \cdot)$
as
$n \to \infty$
; -
(ii)
$P_{\infty, x}(X_m\in \cdot) \Rightarrow \pi_\infty({\cdot})$
as
$m \to \infty$
; -
(iii)
$\pi_n \Rightarrow \pi_\infty$
as
$n \to \infty$
.
Then,
\begin{equation} \sup_{m \geq 0} \| P_{n, x}(X_m\in\cdot) - P_{\infty, x}(X_m\in \cdot) \|_{\mathrm{Lip}_\mathrm{b}(1)} \to 0 \quad \text{as } n \to \infty. \end{equation}
Proof. We will apply Theorem 1 with
$\mathcal{F} = \mathrm{Lip}_\mathrm{b}(1)$
. All that is then needed to finish the proof is to establish that weak convergence implies convergence in
$\|\cdot\|_{\mathrm{Lip}_\mathrm{b}(1)}$
. To this end, if
$\eta_n\Rightarrow \eta_\infty$
as
$n\rightarrow\infty$
, the Skorohod representation theorem ensures that there exists
$(\Lambda_n\colon1\le n\le \infty)$
such that
$\Lambda_n$
has distribution
$\eta_n$
for
$1\le n\le \infty$
and
$\Lambda_n\rightarrow\Lambda_\infty$
almost surely as
$n\rightarrow\infty$
. Then, for
$f\in \mathrm{Lip}_\mathrm{b}(1)$
,
\begin{equation*} \bigg|\int_S (\eta_n({\mathrm{d}} x) - \eta_\infty({\mathrm{d}} x))f(x)\bigg| = |{\mathbb{E}}(f(\Lambda_n) - f(\Lambda_\infty))| \le |{\mathbb{E}}(d(\Lambda_n,\Lambda_\infty)\land 2)| \end{equation*}
uniformly in f (where
$a\land b \triangleq \min(a,b)$
. We used the boundedness and Lipschitzness of f for the inequality. Use of the bounded convergence theorem then proves that
$\|\eta_n-\eta_\infty\|_{\mathrm{Lip}_\mathrm{b}(1)} \to 0$
as
$n\to\infty$
.
Example 1. Suppose that
$(X_j\colon j\geq 0)$
is a Markov chain describing the delay sequence for the single-server
$\mathrm{GI}/\mathrm{G}/1$
queue, so that, under
$P_n$
,
$ (P_nf)(x) = \mathbb{E}f([x + Z_n]^+) $
for some random variable
$Z_n$
, where
$[y]^{+} \triangleq \max(y, 0)$
. If
$f \in \mathrm{Lip}_\mathrm{b}(1)$
, then
\begin{equation*} |(P_nf)(x) - (P_nf)(y)| \leq \mathbb{E}[|[x + Z_n]^+ - [y + Z_n]^+|]\leq |x - y|, \end{equation*}
so
$P_nf \in \mathrm{Lip}_\mathrm{b}(1)$
, and hence
$P_n$
has the
$\mathrm{Lip}_\mathrm{b}(1)$
contraction property. If
$Z_n \nearrow Z_\infty$
as
$n \to \infty$
with
$\mathbb{E} Z_\infty \lt 0$
, then it is well known that
$\pi_n$
exists for
$1 \leq n \leq \infty$
[Reference Kiefer and Wolfowitz14, Reference Lindley16], and conditions (i) and (ii) of Proposition 2 are easily verified.
For condition (iii), let
$((Z_n(i)\colon n \geq 1)\colon i \geq 1)$
be an i.i.d. sequence of copies of
$\{Z_n \colon n \geq 1\}$
and put
$S_n(k) = \sum_{j=1}^k Z_n(j)$
. Then
$ \pi_n({\cdot}) = P(\max_{k \geq 0} S_n(k) \in\cdot) $
(see [Reference Kiefer and Wolfowitz14]), and
$\max_{k \geq 0} S_n(k) = \max_{0 \leq k \leq L_n} S_n(k)$
, where
$L_n = \max \{k \geq 0 \colon S_n(k) \gt 0 \} \lt \infty$
. Since
$Z_n(j) \leq Z_\infty (j)$
,
$L_n \leq L_\infty$
. So,
\begin{equation*} \max_{k \geq 0} S_n(k) = \max_{0 \leq k \leq L_\infty} S_n(k) \nearrow \max_{0 \leq k \leq L_\infty} S_\infty(k), \end{equation*}
so that
$\pi_n \Rightarrow \pi_\infty$
as
$n \to \infty$
. Consequently, (14) holds for this model, so that diagonal convergence follows in particular. This result both complements earlier findings for the
$\mathrm{M}/\mathrm{G}/1$
queue [Reference Breuer4] and strengthens the conclusions reached there (via application of Proposition 2).
Example 2. Suppose that
$(X_j\colon j\geq0)$
is a contractive Markov chain on
$S\subseteq\mathbb{R}^d$
, for which
$P_n$
can be represented as
$P_n(x, \cdot) = P(\varphi_n(x)\in \cdot)$
, where
$(\varphi_n\colon n\geq1)$
is a sequence of continuous random mappings
$\varphi_n\colon S\to S$
defined on a common probability space for which:
-
(a)
$\mathbb{E} | \varphi_n(x) - \varphi_n(y) | \leq r \|x - y\|$
; -
(b)
$\mathbb{E} |\varphi_n(x) - x| {\lt}\infty$
; -
(c)
$\varphi_n\to \varphi_\infty$
uniformly on compact sets as
$n \to \infty$
; -
(d)
$\mathbb{E} |\varphi_n(x) - \varphi_\infty(x)| \to 0$
for
$n \ge 1$
for all
$x, y \in S$
, where
$r \lt 1$
.
As noted earlier, (b) implies that
$P_n$
has the
$\mathrm{Lip}_\mathrm{b}(1)$
inclusion property for
$1\le n \le \infty$
. Hence, Proposition 2 is applicable, once we verify (i)–(iii).
Assumption (c) ensures that (i) holds. For (ii), let
$\{(\varphi_n(k)\colon n\ge 1)\colon k \geq 1\}$
be a sequence of independent copies of
$(\varphi_n \colon n \geq 1)$
.
For
$x \in S$
, let
$\beta_n(k, x) = (\varphi_n(1) \circ \varphi_n(2) \circ \cdots \circ \varphi_n(k))(x)$
, and note that
$P_{n,x}(X_k \in \cdot) = \mathbb P(\beta_n(k, x) \in \cdot)$
.
Under our contraction condition (a), it is known that there exists
$\beta_n(\infty)$
such that
\begin{equation} \mathbb{E}|\beta_n(\infty)-\beta_n(k, x)| \leq \frac{r^k}{1 - r}\mathbb{E}|\varphi_n(x)-x|, \end{equation}
so that
$ P_{n,x}(X_k \in \cdot) \Rightarrow \pi_n({\cdot}) = \mathbb P(\beta_n(\infty) \in \cdot) $
as
$k\rightarrow\infty$
, validating (ii) for
$n=\infty$
; see, for example, [Reference Diaconis and Freedman7, Section 5] for the bound (15).
For (iii), we start by noting that (c) ensures that
\begin{equation*} \beta_n(k, x) = \varphi(1,\varphi_n(k,x))\to {\varphi(1,\varphi_\infty(k,x)) }= \beta_\infty(k,x) \end{equation*}
as
$n \to \infty$
. Similarly, we conclude that
$\beta_n(k, x) \to \beta_{\infty}(k,x)$
as
$n\to\infty$
for
$k \ge 3$
. For
$k \ge 1$
and
$\varepsilon \gt 0$
, we now write
\begin{multline} \mathbb P\big(|\beta_n(\infty) - \beta_n(k, x)| \gt \varepsilon\big) \\ \leq \mathbb P\big(|\beta_n(\infty)-\beta_n(k, x)| + |\beta_{\infty}(\infty)-\beta_\infty(k, x)| \gt \varepsilon/2\big) + \mathbb P\big(|\beta_n(k, x) - \beta_{\infty}(k, x)| \gt \varepsilon / 2\big). \end{multline}
For the first term on the right-hand side, we use (15) and Markov’s inequality to obtain
\begin{align} & \mathbb P\big(|\beta_n(\infty)-\beta_n(k, x)| + |\beta_{\infty}(k,x)-\beta_\infty(\infty)| \gt \varepsilon/2\big) \nonumber \\ & \qquad \le \frac{2}{\varepsilon}\mathbb{E}|\beta_n(\infty)-\beta_n(k, x)| + |\beta_{\infty}(\infty)-\beta_\infty(k, x)| \nonumber \\ & \qquad \le \frac{2}{\varepsilon}\frac{r^k}{1-r}\big(\mathbb{E}|\varphi_n(x) - x| + |\varphi_\infty(x) - x|\big). \end{align}
We now let
$n \to \infty$
in (16), using the upper bound (17) and (d), thereby yielding
\begin{equation*} \overline{\lim}_{n \to \infty} \mathbb P\big(|\beta_n(\infty) - \beta_\infty(\infty)| \gt \varepsilon\big) \leq \frac{4 r^k}{\varepsilon(1 - r)} \mathbb{E}|\varphi_\infty(x) - x|. \end{equation*}
Since k was arbitrary, we may now send
$k \to \infty$
, use (b), and conclude that
$\beta_n(\infty) \overset{\mathrm{p}}{\to} \beta_{\infty}(\infty)$
as
$n \to \infty$
, yielding (iii).
3. Convergence of first transition expectations
In this section, we focus exclusively on the case where S is finite or countably infinite. Our goal is to show that when a sequence
$(P_n\colon 1 \leq n \leq \infty)$
is such that
$P_n \to P_\infty$
as
$n \to \infty$
, and
$\pi_n \to \pi_\infty$
, then it is guaranteed that expected hitting times, expected infinite horizon discounted rewards, and many other associated expectations computed for
$P_n$
converge to those associated with
$P_\infty$
. As noted in the introduction, stationary distribution convergence then allows us to approximate these
$P_n$
expectations by those associated with the limiting model
$P_\infty$
or, equivalently, in computing
$P_\infty$
expectations we can use the
$P_n$
expectations as approximations (as in the truncation setting; see Remark 3).
Given
$\emptyset \neq C \subseteq S$
, let
$T = \inf\{n \geq 0 \colon X_n \in C^\mathrm{c}\}$
be the first hitting time of
$C^\mathrm{c}$
. (Note that
$C^\mathrm{c}$
may be empty, in which case
$T = \infty$
.) For
$r \colon S \to \mathbb{R}_+$
and
$\alpha \colon S \rightarrow [0, \infty)$
, let
\begin{equation} u_n^*(x) = \mathbb{E}_{n, x}\Bigg[\sum_{j=0}^{T}\exp\Bigg({-}\sum_{k=0}^{j-1}\alpha(X_k)\Bigg)r(X_j)\Bigg]\end{equation}
and note that
$u_n^* = ( u_n^*(x) \colon x \in S )$
is the minimal non-negative solution of
\begin{equation} u_n(x) = r(x) + \sum_{y \in C^\mathrm{c}} G_n(x, y) r(y) + \sum_{y \in C} G_n(x, y) u_n(y) \end{equation}
for
$x \in C$
, where
$G_n = (G_n(x, y) \colon x, y \in S )$
has entries given by
$G_n(x, y)= \exp(-\alpha(x))P_n(x, y)$
for
$x, y \in S$
. We say that
$u_n^*(x)$
is a first transition expectation, since the linear system (19) can be derived from (18) by conditioning on the first transition state value
$X_1$
.
If we set
$\alpha(y) = 0$
for
$y \in S$
and
$r = e$
, then
$u_n^*(x)$
is the mean hitting time of
$C^\mathrm{c}$
starting from x. On the other hand, if
$\alpha(y) = \alpha \gt 0$
and
$C^\mathrm{c} = \emptyset$
, then
$u_n^*(x)$
is the expected infinite horizon discounted reward starting from x. If
$ r(y) = 0 $
for
$ y \in C $
, then
$ u_n^*(x) $
is the
$\alpha$
-discounted reward collected when the Markov chain hits
$ C^\mathrm{c} $
, where
$\alpha({\cdot})$
represents a state-dependent risk-free rate (as occurs in many finance settings, in which
$ C^\mathrm{c} $
is the exercise region for an option).
Theorem 2. Suppose that S is finite or countably infinite, and that
$r \in \mathcal{F}_e$
. Assume that
$P_\infty = ( P_\infty(x, y) \colon x, y \in S )$
is irreducible and that
$P_n = ( P_n(x, y) \colon x, y \in S ) $
has a unique stationary distribution
$\pi_n$
for
$1 \leq n \leq \infty$
. If, for each
$ x, y \in S $
,
-
(i)
$ P_n(x, y) \to P_\infty(x, y) $
as
$ n \to \infty $
; -
(ii)
$ \pi_n(x) \to \pi_\infty(x) $
as
$ n \to \infty $
,
then, for each
$ x \in S $
,
$ u_n^*(x) \to u_\infty^*(x) $
as
$ n \to \infty $
.
Proof. We first note that condition (i) ensures that
$P_n(x, \cdot) \overset{\text{tv}}{\to} P_\infty(x, \cdot)$
as
$n \to \infty$
. The proof of Theorem 1 can be easily modified to ensure that, for
$ m \geq 1 $
,
\begin{equation*} \| P_{n, x} ( (X_0, X_1, \ldots, X_m) \in \cdot ) - P_{\infty, x} ( (X_0, X_1, \ldots, X_m) \in \cdot ) \|_e \to 0 \end{equation*}
as
$ n \to \infty $
, from which it easily follows that
\begin{equation} P_{n,x}\Bigg(\sum_{j=0}^{T}\exp\Bigg({-}\sum_{k=0}^{j-1}\alpha(X_k)\Bigg)r(X_j) \in \cdot\Bigg) \implies P_{\infty,x}\Bigg(\sum_{j=0}^{T}\exp\Bigg({-}\sum_{k=0}^{j-1}\alpha(X_k)\Bigg)r(X_j) \in \cdot\Bigg) \end{equation}
as
$ n \to \infty $
.
Next, if
$ x \in C^\mathrm{c} $
, then
$ u_n^*(x) = r(x) $
for
$1 \leq n \leq \infty$
, so the result is trivial. If
$ x \in C $
, let
$ \tau(x) = \inf \{ n \geq 1 \colon X_n = x \} $
be the first entry time into x, and note that the strong Markov property implies that
\begin{align*} u_n^*(x) & = \mathbb{E}_{n,x}\Bigg[\sum_{j=0}^{T\land(\tau(x)-1)}\exp\Bigg({-}\sum_{k=0}^{j-1}\alpha(X_k)\Bigg)r(X_j)\Bigg] \\ & \quad + \mathbb{E}_{n,x}\Bigg[\exp\Bigg({-}\sum_{k=0}^{\tau(x)-1}\alpha(X_k)\Bigg) \mathbf{1}\{T \gt \tau(x)\} u_n^*(x) \Bigg]. \end{align*}
Note that since
$ r \in \mathcal{F}_e $
and
$ \alpha({\cdot}) $
is non-negative,
\begin{align} \sum_{j=0}^{T \land (\tau(x)-1)} \exp\Bigg({-}\sum_{k=0}^{j-1} \alpha(X_k)\Bigg) r(X_j) & \leq \tau(x),\\[-15pt]\nonumber \end{align}
\begin{align} \exp\Bigg({-}\sum_{k=0}^{\tau(x)-1} \alpha(X_k)\Bigg) \mathbf{1}\{T \gt \tau(x)\} & \leq 1. \end{align}
Because
$\pi_n(x) = (\mathbb{E}_{n,x}\tau(x))^{-1} \to \pi_\infty(x) = (\mathbb{E}_{\infty,x}\tau(x))^{-1} \gt 0$
(as a consequence of (ii) and irreducibility), (20) implies that the left-hand sides of (21) and (22) are uniformly integrable over the family of probabilities
$(P_{n,x} \colon n \geq n_0 )$
(for
$ n_0 $
sufficiently large). Hence,
\begin{align} \mathbb{E}_{n,x}\Bigg[\sum_{k=0}^{T\land(\tau(x)-1)}\exp\Bigg({-}\sum_{j=0}^{k-1}\alpha(X_j)\Bigg)r(X_k)\Bigg] & \to \mathbb{E}_{\infty,x}\Bigg[\sum_{k=0}^{T\land(\tau(x)-1)}\exp\Bigg({-}\sum_{j=0}^{k-1}\alpha(X_k)\Bigg) r(X_j)\Bigg],\\[-15pt]\nonumber \end{align}
\begin{align} \mathbb{E}_{n,x}\Bigg[\exp\Bigg({-}\sum_{j=0}^{\tau(x)-1}\alpha(X_k)\Bigg)\mathbf{1}\{T \gt \tau(x)\}\Bigg] & \to \mathbb{E}_{\infty,x}\Bigg[\exp\Bigg({-}\sum_{j=0}^{\tau(x)-1}\alpha(X_j)\Bigg) \mathbf{1}\{T \gt \tau(x)\}\Bigg] \end{align}
as
$n\rightarrow\infty$
. If
$ u_n^*(x) \lt \infty $
, then
\begin{equation} u_n^*(x) = \frac{\mathbb{E}_{n,x}\Big[\sum_{k=0}^{T\land(\tau(x)-1)}\exp\Big({-}\sum_{j=0}^{k-1}\alpha(X_k)\Big)r(X_j)\Big]} {1 - \mathbb{E}_{n,x}\Big[\exp\Big({-}\sum_{j=0}^{\tau(x)-1}\alpha(X_j)\Big)\mathbf{1}\{T \gt \tau(x)\}\Big]}. \end{equation}
It follows from (23) and (24) that if
$ u_\infty^*(x) \lt \infty $
, then
$ u_n^*(x) \to u_\infty^*(x) $
. On the other hand, if
$ u_\infty^*(x) = \infty $
, then
\begin{align*} \mathbb{E}_{\infty,x}\Bigg[\exp\Bigg({-}\sum_{j=0}^{\tau(x)-1}\alpha(X_j)\Bigg)\mathbf{1}\{T \gt \tau(x)\}\Bigg] & = 1, \\ \mathbb{E}_{\infty,x}\Bigg[\sum_{j=0}^{T\land(\tau(x)-1)}\exp\Bigg({-}\sum_{k=0}^{j-1}\alpha(X_k)\Bigg)r(X_j)\Bigg] & \gt 0, \end{align*}
so that (23), (24), and (25) again imply that
$u_n^*(x) \to \infty = u_\infty^*(x)$
.
Hence, stationary distribution convergence automatically implies convergence of all expectations covered by Theorem 2. An implication is that in developing convergent truncation schemes, it is enough to establish stationary distribution convergence in order to establish ‘universal convergence’ across all the expectations considered above.
We conclude this section with a modest extension to Theorem 2, covering the case where r is unbounded.
Proposition 3. Suppose
$ P_\infty $
is irreducible, that
$ P_n $
has a stationary distribution
$\pi_n$
for
$ 1\le n \le \infty $
, and that Theorem 2(i) and (ii) hold. If, in addition,
\begin{equation} \sum_{x \in S} \pi_n(x) w(x) \to \sum_{x \in S} \pi_\infty(x) w(x) \end{equation}
as
$n\to\infty$
, then
$ u_n^*(x) \to u_\infty^*(x) $
as
$ n \to \infty $
for each
$ r \in \mathcal{F}_w $
.
Proof. Since
$ \pi_n(x) \to \pi_\infty(x) \gt 0 $
as
$ n \to \infty $
as a consequence of (ii) and irreducibility, x must lie in
$ P_n $
’s positive recurrent closed communicating class for n sufficiently large. Applying regenerative process theory, we find that
\begin{equation*} \sum_{y\in S}\pi_n(y)w(y) = \frac{\mathbb{E}_{n,x}\big[\sum_{j=0}^{\tau(x)-1}w(X_j)\big]}{\mathbb{E}_{n,x}\tau(x)} \end{equation*}
for
$ 1 \leq n \leq \infty $
[Reference Asmussen1]. Hence, (26) implies that
\begin{equation*} \frac{\mathbb{E}_{n,x}\big[\sum_{j=0}^{\tau(x)-1}w(X_j)\big]}{\mathbb{E}_{n,x}\tau(x)} = \sum_{y\in S}\pi_n(y)w(y) \rightarrow \sum_{y\in S}\pi_\infty(y)w(y) = \frac{\mathbb{E}_{\infty,x}\big[\sum_{j=0}^{\tau(x)-1}w(X_j)\big]}{\mathbb{E}_{\infty,x}\tau(x)} \end{equation*}
as
$n\to\infty$
. Since
$ \mathbb{E}_{n,x} \tau(x) \to \mathbb{E}_{\infty,x} \tau(x) $
as
$ n \to \infty $
, we find that
\begin{equation*} \mathbb{E}_{n,x}\Bigg[\sum_{j=0}^{\tau(x)-1}w(X_j)\Bigg] \to \mathbb{E}_{\infty,x}\Bigg[\sum_{j=0}^{\tau(x)-1}w(X_j)\Bigg] \end{equation*}
as
$n\to\infty$
. In view of (20) with
$ \alpha({\cdot}) = 0 $
,
$ T = \tau(x) $
, and
$ r = w $
, we may therefore conclude that
$\sum_{j=0}^{\tau(x)-1} w(X_j) $
is uniformly integrable under
$ ( P_{n,x} \colon 1 \leq n \leq \infty ) $
. If
$ r \in \mathcal{F}_w $
, then
\begin{equation*} \sum_{k=0}^{T\land(\tau(x)-1)}\exp\Bigg({-}\sum_{j=0}^{k-1}\alpha(X_j)\Bigg)r(X_k) \leq \sum_{k=0}^{\tau(x)-1}w(X_k), \end{equation*}
so
$\sum_{k=0}^{T\land(\tau(x)-1)} \exp\big({-}\sum_{j=0}^{k-1}\alpha(X_j)\big)r(X_k)$
is also uniformly integrable under
$( P_{n,x} \colon 1 \leq $
$n \leq \infty ) $
. Utilizing (20) with T replaced by
$ T \wedge (\tau(x) - 1) $
, we conclude that
\begin{equation*} \mathbb{E}_{n,x}\Bigg[\sum_{k=0}^{T\wedge(\tau(x)-1)}\exp\Bigg({-}\sum_{j=0}^{k-1}\alpha(X_j)\Bigg)r(X_k)\Bigg] \to \mathbb{E}_{\infty,x}\Bigg[\sum_{k=0}^{T\wedge(\tau(x)-1)}\exp\Bigg({-}\sum_{j=0}^{k-1}\alpha(X_j)\Bigg)r(X_k)\Bigg] \end{equation*}
as
$n\to\infty$
. We now apply (25), after which the rest of the argument follows identically to the proof of Theorem 2.
4. A counter-example
As shown in Corollary 1, the weak interchange property automatically implies diagonal convergence when S is countably infinite. In this section, we show that this can fail to be true when S is a continuous state space.
Let
$ S = [0, 1] $
. For
$ 1 \leq n \lt \infty $
, we put
\begin{equation*} P_n(x, \cdot) = \begin{cases} \delta_{x/2}({\cdot}) & \text{if } x \gt 2^{-n}, \\ \delta_{1}({\cdot}) & \text{if } 2^{-n-1} \lt x \leq 2^{-n}, \\ \dfrac{1}{n+1}\displaystyle\sum_{j=0}^{n} \delta_{2^{-j}}({\cdot}) & \text{if } 0 \leq x \leq 2^{-n-1}. \end{cases}\end{equation*}
Also, for
$ n = \infty $
, put
$ P_\infty(x, \cdot) = \delta_{x/2}({\cdot}) $
for
$0 \leq x \leq 1 $
, and note that
$ \pi_\infty({\cdot}) = \delta_{0}({\cdot}) $
is the unique stationary distribution of
$ P_\infty $
.
Observe that, for each
$ x \gt 0 $
, once n is large enough that
$ 2^{-n} \lt x $
, the Markov chain starting from x under
$ P_n $
takes
$m_n(x) \triangleq n - \lfloor \log_2 x \rfloor$
steps to move into the interval
$(2^{-n-1}, 2^{-n}]$
, after which the chain jumps to state 1. So,
\begin{equation} P_{n,x}( X_{m_n(x)+1} \in \cdot ) = \delta_1({\cdot}). \end{equation}
Once the chain hits state 1, the chain visits the states
$ 2^{-1}, 2^{-2}, \ldots, 2^{-n} $
and repeats this visitation pattern from that point onwards.
On the other hand, if
$ x \in (2^{-n-1}, 2^{-n}] $
, the chain immediately jumps to 1, after which the chain visits
$ 2^{-1}, 2^{-2}, \ldots, 2^{-n} $
and then indefinitely repeats this pattern. If
$ x \in [0, 2^{-n-1}] $
, the chain chooses one of the states
$ 1, 2^{-1}, \ldots, 2^{-n} $
uniformly at random, after which the chain enters the periodic ‘orbit’
$ 1, 2^{-1}, \ldots, 2^{-n} $
and repeats that pattern forever. It follows that
\begin{equation*} \frac{1}{m}\sum_{j=0}^{m-1}P_{n, x}(X_j \in \cdot) \overset{\text{tv}}{\to} \pi_n({\cdot}) \triangleq \frac{1}{n+1} \sum_{j=0}^{n} \delta_{2^{-j}}({\cdot})\end{equation*}
as
$ n \to \infty $
. (In fact, X is a periodic positive recurrent Harris chain under
$ P_n $
.)
Note that
\begin{equation} P_n(x, \cdot) \overset{\text{tv}}{\rightarrow} P_\infty(x, \cdot) \end{equation}
for
$ 0 \lt x \leq 1 $
as
$ n \to \infty $
, whereas
\begin{equation} P_n(0, \cdot) \Rightarrow P_\infty(0, \cdot) \end{equation}
as
$ n \to \infty $
. Also, for
$ x \in S $
,
\begin{equation} P_{\infty,x}(X_m \in \cdot) \Rightarrow \pi_\infty({\cdot}) \end{equation}
as
$ m \to \infty $
, and
\begin{equation} \pi_n \Rightarrow \pi_\infty. \end{equation}
as
$ n \to \infty $
. Consequently, (28)–(31) imply that the Markov chain constructed here satisfies conditions (i)–(iii) of Proposition 2 (but, critically, this example does not have the Lipb(1) inclusion property), so it has the weak interchange property. However, (27) shows that for
$ x \gt 0 $
,
\begin{equation*} P_{n,x} ( X_{m_n(x)+1} \in \cdot ) = \delta_1({\cdot}) \rlap{$\quad\not$}\implies \delta_0({\cdot}) = \pi_\infty({\cdot}),\end{equation*}
and hence diagonal convergence fails in this example.
Remark 5. If we wish to modify this example so that
$ P_n $
is Feller for
$ 1 \leq n \lt \infty $
(so that
$ P_n(x_k, \cdot) \Rightarrow P_n(x_\infty, \cdot) $
whenever
$ x_k \to x_\infty $
), this is easy to do. In particular, we can modify the transition probabilities in
$ (2^{-n+1}, 2^{-n}) $
and
$ (0, 2^{-n-1}) $
to be mixtures of the endpoint transition probabilities, where the mixture weights are chosen to make the transition probabilities weakly continuous.
We conclude this section by noting that the theory of Section 2 depends critically on the fact that
$P_{\infty,x}(X_m \in \cdot) \Rightarrow \pi_\infty({\cdot})$
as
$ m \to \infty $
. This typically holds only when the chain is suitably aperiodic under
$ P_\infty $
. If
$ P_\infty $
induces a periodic Markov chain, we can expect the result
\begin{equation*}\sup_{m \geq 0} \| P_{n,x}(X_m \in \cdot) - P_{\infty, x}(X_m \in \cdot) \| \to 0\end{equation*}
to hold only when
$ P_n $
matches the periodic structure of
$ P_\infty $
. But
\begin{equation} P_n(x, \cdot) \Rightarrow P_\infty(x, \cdot) \end{equation}
is too weak a condition to assure that the periodic structure of
$ P_n $
matches that of
$ P_\infty $
. (For example, when S is countably infinite, we can achieve (32) and yet include a state
$ z \in S $
for which
$ P_n(z, z) = 1/n $
, so that
$ P_n $
is aperiodic for
$ 1 \leq n \lt \infty $
, regardless of the periodicity of
$ P_\infty $
.)
5. Extension to Markov jump processes
In this section, we briefly describe how our theory extends to Markov jump processes
$X = (X(t) \colon t \geq 0)$
with a countably infinite state space
$\mathcal{S}$
. Here,
$P_n = (P_n(x,y) \colon x,y \in \mathcal{S})$
for
$1\le n\le\infty$
is replaced by the rate matrix
$Q_n = (Q_n(x,y) \colon x,y \in \mathcal{S})$
, where
$Q_n$
satisfies the requirements:
-
•
$Q_n(x,y) \geq 0$
for
$x \neq y, x,y \in \mathcal{S}$
; -
•
$\sum_{y\neq x} Q_n(x,y) \lt\infty$
; -
•
$\sum_{y} Q_n(x,y) = 0$
for
$x,y\in\mathcal{S}$
. We further require throughout this section that
$Q_n$
describes a non-explosive jump process; see [Reference Asmussen1, Chapter II] for the definition and sufficient conditions. As in Section 2, we let
$P_{n,x}({\cdot})$
be the probability on the path-space of X under which X starts in state
$x \in \mathcal{S}$
and evolves according to
$Q_n$
.
Here is our extension of Corollary 1 to Markov jump processes.
Proposition 4. Suppose that, for
$1\le n \le \infty$
,
$Q_n$
has a unique stationary distribution
$\pi_n$
. If:
-
(i)
$Q_n(x, y) \to Q_\infty(x, y)$
as
$n \to \infty$
for each
$x, y \in \mathcal{S}$
; -
(ii)
$Q_\infty$
is irreducible; -
(iii)
$\pi_n(x) \to \pi_\infty(x)$
as
$n \to \infty$
for each
$x \in \mathcal{S}$
,
then for each
$x \in \mathcal{S}$
,
$ \sup_{t \geq 0} \| P_{n,x}(X(t) \in \cdot) - P_{\infty, x}(X(t) \in \cdot) \|_e \to 0 $
as
$n \to \infty$
.
Proof. We note that since
$Q_\infty$
is non-explosive, irreducible, and positive recurrent (since it possesses a stationary distribution),
$ P_{\infty, x}(X(t) = y) \to \pi_\infty(y) $
as
$t \to \infty$
for each
$x\in \mathcal{S}$
. Put
$\lambda_n(x) = -Q_n(x,x)$
and let
\begin{equation*} R_n(x, y) = \begin{cases} \dfrac{Q_n(x, y)}{\lambda_n(x)}, & x \neq y, \\ 0, & x = y, \end{cases} \end{equation*}
so that
$\lambda_n(x)$
is the jump rate out of state
$x \in \mathcal{S}$
and
$R_n = (R_n(x, y) \colon x, y \in \mathcal{S})$
is the transition matrix of the embedded discrete-time Markov chain under
$Q_n$
. Given this,
\begin{equation} R_n(x, y) \to R_\infty(x, y) \end{equation}
as
$n \to \infty$
and
$\lambda_n(x) \to \lambda_\infty(x)\lt\infty$
for
$x \in \mathcal{S}$
. Consequently, if
$(Y_j\colon j\ge 0)$
is the embedded discrete-time Markov chain, the proof of Theorem 1 establishes that (33) implies that
\begin{equation} P_{n,x}(Y_0 = y_0, \ldots, Y_m = y_m) \to P_{\infty,x}(Y_0 = y_0, \ldots, Y_m = y_m) \end{equation}
as
$n\to\infty$
, for each
$x \in \mathcal{S}$
and
$n \geq 1$
. Furthermore, if
$T_i$
is the time of the ith jump of X,
\begin{equation*} T_i = \tau_0 +\cdots + \tau_{i-1}, \end{equation*}
where
$\tau_i$
is the time spent in the ith state visited. Under
$Q_n$
, the
$\tau_i$
can be represented as
\begin{equation*} \tau_i = \frac{1}{\lambda_n(Y_i)}\chi_i, \end{equation*}
where the
$\chi_i$
are i.i.d. exponential random variables with mean 1. Since
$\lambda_n(x) \to \lambda_\infty(x)$
as
$n \to \infty$
, evidently
\begin{equation} P_{n,x}((\tau_0, \tau_1, \ldots, \tau_m) \in \cdot \mid Y_j, 0 \leq j \leq m) \Rightarrow P_{\infty,x}((\tau_0, \tau_1, \ldots, \tau_m) \in \cdot \mid Y_j, 0 \leq j \leq m). \end{equation}
Since
$X(t) = \sum_{m=0}^\infty Y_m \mathbf{1}(T_{m-1} \leq t \lt T_{m+1})$
, it follows from (33), (34), and (35) that
\begin{equation*} P_{n,x}(X(1) \in \cdot) \Rightarrow P_{\infty,x}(X(1) \in \cdot) \end{equation*}
as
$n \to \infty$
. We can then apply Corollary 1 to the sequence of transition matrices
\begin{equation*}(P_{n,x}(X(1) = y) \colon x, y \in \mathcal{S})\end{equation*}
to conclude that
\begin{equation} \sup_{m \geq 0} \| P_{n,x}(X(m) \in \cdot) - P_{\infty,x}(X(m) \in \cdot) \|_e \to 0 \quad \text{as } n \to \infty. \end{equation}
Since the
$\mathcal{F}_e$
inclusion property ensures that
$\| P_{n,x}(X(t) \in \cdot) - \pi_n({\cdot}) \|_e \to 0$
is non-increasing in t, we obtain the theorem from (36).
The results of Section 3 are straightforward to extend to the jump process setting. The arguments of Theorem 2 and Proposition 3 go over without change in continuous time, upon recognizing that
\begin{equation*} \pi_n(y) = \mathbb{E}_{n,x} \int_0^{\tau(x)} \mathbf{1}(X(t) = y) \, {\mathrm{d}} t \lt\infty \quad \text{and} \quad \sum_x \pi_n(x) w(x) = \frac{\mathbb{E}_{n,x} \int_0^{\tau(x)} w(X(s)) \, {\mathrm{d}} s}{\mathbb{E}_{n,x} \tau(x)}\end{equation*}
as
$n\to\infty$
, where
$\tau(x)$
is the first time at which X enters x (i.e.
$\tau(x) = \inf\{t \ge 0 \colon X(t) = x,$
$X(t-) \neq x\}$
) and
$w({\cdot})$
is a non-negative function. We obtain the following result.
Proposition 5. Suppose that
$Q_\infty$
is irreducible and that
$Q_n$
has a unique stationary distribution
$\pi_n$
for
$1\le n\le \infty$
. If
-
(i)
$Q_n(x, y) \to Q_\infty(x, y)$
, -
(ii)
$\pi_n(x) \to \pi_\infty(x)$
, -
(iii)
$\sum_{z \in \mathcal{S}} \pi_n(z) w(z) \to \sum_{z \in \mathcal{S}} \pi_\infty(z) w(z)$
as
$n \to \infty$
for each
$x, y \in \mathcal{S}$
, then
\begin{multline*} \mathbb{E}_{n,x}\int_0^T\exp\bigg({-}\int_0^s\alpha(X(u))\,{\mathrm{d}} u\bigg)r(X(s))\,{\mathrm{d}} s \\ \to \mathbb{E}_{\infty,x}\int_0^T\exp\bigg({-}\int_0^s\alpha(X(u))\,{\mathrm{d}} u\bigg)r(X(s))\,{\mathrm{d}} s \end{multline*}
for each non-negative
$r\colon\mathcal{S}\to\mathbb{R}_+$
and
$\alpha\colon\mathcal{S}\to\mathbb{R}_+$
, for which
$r \in \mathcal{F}_w$
.
We conclude that stationary distribution convergence implies the strong interchange property, and also convergence of first transition expectations, in the Markov jump process setting.
Acknowledgements
We thank Yifu Tang for helpful research assistance. We thank the referees for their insightful comments, which led us to a sharper understanding of the key role of stationary distribution convergence in the current setting.
Funding information
There are no funding bodies to thank relating to the creation of this article.
Competing interests
There were no competing interests to declare which arose during the preparation or publication process of this article.
