Labeled Markov decision processes

Labeled Markov decision processes (LMDPs) are used to model planning problems where each decision has a potentially probabilistic outcome. LMDPs augment standard Markov decision processes (Baier and Katoen, Reference Baier and Katoen2008) with state labels, allowing properties such as safety and liveness to be assigned to a sequences of states.

A path of the LMDP ${\mathcal{M}}$ is an infinite state sequence ${\unicode{x03C3}} = s_0 s_1 s_2 \cdots$ such that for all $i \ge 0$ , there exists $a_i\,\in\, A(s)$ and $s_{i}, s_{i+1}\,\in\, S$ with $P(s_i,a_i,s_{i+1}) \gt 0$ . The semantic path corresponding to ${\unicode{x03C3}}$ is given by $L({\unicode{x03C3}}) = L(s_0)L( s_1)\cdots$ , derived using the labeling function L(s). Given a path ${\unicode{x03C3}}$ , the ith state is denoted by ${\unicode{x03C3}}[i] = s_i$ . We denote the prefix by ${\unicode{x03C3}}[{:}i] = s_0 s_1\cdots s_i$ and suffix by ${\unicode{x03C3}}[i{+}1{:}] = s_{i+1} s_{i+2}\cdots$ .

LTL and limit-deterministic Büchi automata

In an LMDP ${\mathcal{M}}$ , whether a given semantic path $L({\unicode{x03C3}})$ satisfies a property such as avoiding unsafe states can be expressed using LTL. LTL can specify the change of labels along the path by connecting Boolean variables over the labels with two propositional operators, negation $(\neg)$ and conjunction $(\wedge)$ , and two temporal operators, next $(\bigcirc)$ and until $(\cup)$ .

Definition 2. The LTL formula is defined by the syntax

(1) $${\unicode{x03C6}} :: = {\rm{true}}\,\left| \alpha \right|\,{{\unicode{x03C6}} _1} \wedge {{\unicode{x03C6}} _2}\,|\neg {\unicode{x03C6}} | \bigcirc {\unicode{x03C6}} \,|\,{{\unicode{x03C6}} _1} \cup {{\unicode{x03C6}} _2},\alpha\,\in\, \Lambda.$$

Satisfaction of an LTL formula ${\unicode{x03C6}}$ on a path ${\unicode{x03C3}}$ of an MDP (denoted by ${\unicode{x03C3}} \models {\unicode{x03C6}}$ ) is defined as, $\alpha\in \Lambda$ is satisfied on ${\unicode{x03C3}}$ if $\alpha\in L({\unicode{x03C3}}[1])$ , $\bigcirc {\unicode{x03C6}}$ is satisfied on ${\unicode{x03C3}}$ if ${\unicode{x03C6}}$ is satisfied on ${\unicode{x03C3}}[1{:}]$ , ${\unicode{x03C6}}_1 \cup {\unicode{x03C6}}_2$ is satisfied on ${\unicode{x03C3}}$ if there exists i such that ${\unicode{x03C3}}[i{:}] \models {\unicode{x03C6}}_2$ and for all $j \lt i,{\mkern 1mu} {\unicode{x03C3}} [j:] \models {{\unicode{x03C6}} _1}$ .

Other propositional and temporal operators can be derived from previous operators, e.g., (or) ${\unicode{x03C6}}_1 \vee {\unicode{x03C6}}_2 := \neg(\neg {\unicode{x03C6}}_1 \wedge \neg {\unicode{x03C6}}_2)$ , (eventually) $\lozenge {\unicode{x03C6}} := {\rm true} \cup {\unicode{x03C6}}$ and (always) $ \square {\unicode{x03C6}} := \neg \lozenge \neg {\unicode{x03C6}}$ .

We can use Limit-Deterministic Büchi Automata (LDBA) to check the satisfaction of an LTL formula on a path.

A run is an infinite sequence of transitions ${\unicode{x03C1}} = (q_0,w_0,q_1), (q_1,w_1, q_2) \cdots$ such that for all $i \ge 0$ , $q_{i+1}\,\in\, {\rm\delta}(q_i, w_i)$ . The run ${\unicode{x03C1}}$ is accepted by the LDBA if it satisfies the Büchi condition, i.e., ${\rm inf}(\rm {\unicode{x03C1}}) \cap B \ne \emptyset$ , where ${\rm inf}(\rm {\unicode{x03C1}})$ denotes the set of automaton states visited by ${\unicode{x03C1}}$ infinitely many times.

A path ${\unicode{x03C3}}=s_0s_1\dots$ of an LMDP ${\mathcal{M}}$ is considered accepted by an LDBA $\mathcal{A}$ if the semantic path $L({\unicode{x03C3}})$ is the corresponding word w of an accepting run ${\unicode{x03C1}}$ after elimination of $\epsilon$ -transitions.

Product MDP

Planning problems for LTL objectives typically requires a (history-dependent) policy, which determines the current action based on all previous state visits.

Using the LDBA, we construct a product MDP that augments the MDP state space along with the automaton state space. The state of the product MDP encodes both the physical state and the progression of the LTL objective. In this manner, we “lift” the planning problem to the product MDP. Given that the state of the product MDP now encodes all the information necessary for planning, the action can be determined by the current state of the product MDP, resulting in memoryless policies. Formally, the product MDP is defined as follows:

Definition 5. A product MDP $ {\mathcal{M}}^{\times} = ( S^{\times}, A^{\times}, P^{\times},{\it s}_{\rm 0}^{\times}, B^{\times})$ of an LMDP $ {\mathcal{M}} = ( S, A, P, {\it s}_{\rm 0}, \Lambda, L)$ and an LDBA ${\mathcal{A}}= (\mathcal{Q}, \Sigma,{\rm\delta},{\it q}_{\rm 0},B)$ is defined by the set of states $S^\times = S \times {\mathcal{Q}}$ , the set of actions ${A}^\times = { {A}} \cup \{\epsilon_q|q\in {\mathcal{Q}}\}$ , the transition probability function

$${P^ \times }(\langle s,q\rangle, a,\langle s',q'\rangle ) = \left\{ {\matrix{ {P(s,a,s')} \hfill & \!\! {q' = {\rm\delta} (q,L(s)),a \,\notin\, {A^{\rm\epsilon}}} \hfill \cr 1 \hfill & {a = {{\rm\epsilon}_{q'}},q'\,\in\, {\rm\delta} (q,),s = s'} \hfill \cr 0 \hfill & {{\rm{otherwise}}} \hfill \cr } } \right.,$$

the initial state $s_0^\times=\langle s_0,q_0 \rangle$ , and the set of accepting states $B^\times=\{\langle s,q \rangle\,\in\, S^\times |q\in B\}$ . We say a path ${\unicode{x03C3}}$ satisfies the Büchi condition ${\unicode{x03C6}}_B$ if ${\rm inf}({\unicode{x03C3}})\cap B^\times\ne\emptyset$ . Here, ${\rm inf}({\unicode{x03C3}})$ denotes the set of states visited infinitely many times on ${\unicode{x03C3}}$ .

The transitions of the product MDP ${\mathcal{M}}^\times $ are derived by combining the transitions of the MDP $ {\mathcal{M}} $ and the LDBA $ \mathcal{A} $ . Specifically, the multiple $ \epsilon $ -transitions starting from the same states in the LDBA are differentiated by their respective end states q and are denoted as $ \epsilon_q$ . These $ \epsilon $ -transitions in the LDBA give rise to corresponding $ \epsilon $ -actions in the product MDP, each occurring with a probability of 1. The limit-deterministic nature of LDBAs ensures that the presence of these $\epsilon$ -actions within the product MDPs does not prevent the quantitative analysis of the MDPs for planning. In other words, any optimal policy for a product MDP induces an optimal policy for the original MDP, as formally stated below.

The case ${\rm\gamma} \lt 1$

As ${\rm\gamma} \lt 1$ , the invertibility of $(I-\Gamma_B P_\pi)$ can be shown by applying Gershgorin circle theorem (Bell, Reference Bell1965, Theorem 0) to $\Gamma_B P_\pi$ . Specifically, any eigenvalue $\lambda$ of $\Gamma_B P_\pi$ satisfies $|\lambda | \lt 1$ since each row sum of $\Gamma_B P_\pi$ is strictly less than 1. Then, the solution for the Bellman equation (10) can be uniquely determined as

(11) $$\matrix{ {\left[ {\matrix{ {{V^B}} \cr {{V^{\neg B}}} \cr } } \right]} \hfill & {\!\!\!\!\! = (1 - {{\rm\gamma} _B}){{({I_{m + n}} - {\Gamma _B}{P_\pi })}^{ - 1}}\left[ {\matrix{ {{{\mathbb I}_m}} \cr {{{\mathbb O}_n}} \cr } } \right].} \hfill \cr } $$

Proof. The solution of the Bellman equation (10) can be determined uniquely by matrix operation (11) if $(I-\Gamma_B P_\pi)$ is invertible. The invertibility is shown using the Gershgorin circle theorem (Bell, Reference Bell1965, Theorem 0), which claims the following. For a square matrix A, define the radius as $r_i:=\sum_{j\ne i}{\vert A_{ij}\vert}$ . Then, each eigenvalue of A is in at least one of the Gershgorin disks ${\mathcal D}(A_{ii},r_i):=\{z:\vert z-A_{ii}\vert\le r_i\}$ .

For the matrix $\Gamma_B P_\pi$ , at its i-th row, we have the center of the disk as $(\Gamma_B P_\pi)_{ii}=(\Gamma_B)_{ii}{(P_\pi)}_{ii}$ , and the radius as $r_i=\sum_{j\ne i}{\vert (\Gamma_B P_\pi)_{ij}\vert} ={(\Gamma_B)}_{ii}(1-{(P_\pi)}_{ii})$ . We can upper bound the disk as

(12) $$\matrix{ {{\cal D}({{({\Gamma _B}{P_\pi })}_{ii}},{r_i})} \hfill & {\!\!\!\! = \{ z:|z - {{({\Gamma _B}{P_\pi })}_{ii}}| \le {r_i}\} } \hfill \cr {} \hfill & \!\! \!\!{ \subseteq \{ z:|z| \le {{({\Gamma _B}{P_\pi })}_{ii}} + {r_i}\} } \hfill \cr {} \hfill & \!\! \!\!{ \subseteq \{ z:|z| \le {\rm\gamma} \} .} \hfill \cr } $$

Since all Gershgorin disks share the same upper bound, the union of all disks is also bounded by

(13) $$\bigcup\limits_{i\,\in\, S} {{\cal D}({{({\Gamma _B}{P_\pi })}_{ii}},{r_i})} \subseteq \{ z:|z| \le {\rm\gamma} \} .$$

The inequality ${\rm\gamma} \lt 1$ ensures that any eigenvalue $\lambda$ of $\Gamma_B P$ satisfies $|\lambda | \lt 1$ . Thus $(I-\Gamma_B P_\pi)$ is invertible and the solution can be uniquely determined by (11). The value function satisfies the Bellman equation (10), which has a unique solution, which is the value function. Thus, the proposition holds. $\square$

The case ${\rm\gamma}=1$

For ${\rm\gamma}=1$ , the matrix $(I-\Gamma_B P_\pi)$ may not be invertible, causing the Bellman equation (10) to have multiple solutions. Since the solution may not be the value function here, we use $U^{B}\in \mathbb{R}^m$ and $U^{\neg B}\in \mathbb{R}^n$ to represent a solution on states in B and $\neg B$ , respectively. In an induced MC, a path starts in an initial state, travels finite steps among the transient states, and eventually enters a BSCC. If the induced MC has only accepting BSCCs, the connection between all states in B can be captured by a new transition matrix, and the Bellman operator is contractive on the states in B. Thus, we can show the solution is unique in all the states in the following subsection. In the general case where rejecting BSCCs also exists, we introduce a sufficient condition of fixing all solutions within rejecting BSCCs to zero. We demonstrate the uniqueness of the solution under this condition first on $U^{B}$ and then on $U^{\neg B}$ in the second part of this section.

When the MC only has accepting BSCCs

This section focuses on proving that the Bellman equation (10) has a unique solution when there are no rejecting BSCCs in the MC. The result is as follows,

Proposition 2. If the MC only has accepting BSCCs (i.e., no rejecting BSCCs) and ${\rm\gamma}=1$ in the surrogate reward (3), then the Bellman equation (10) has a unique solution $[{U^B}^T, {U^{\neg B}}^T]^T= \mathbb{I}$ .

The intuition behind the proof is to capture the connection between all states B by a new transition matrix $P_\pi^B$ in Lemma 3. Then, one can use $I-{\rm\gamma}_B P_\pi^B$ is invertible to show the solutions $U_B$ is unique and furthermore, show $U^{\neg B}$ is uniquely determined by $U^{B}$ in Lemma 4.

Lemma 3. If the MC ${\mathcal{M}}_\pi=(S, P_\pi, s_{\rm 0},\Lambda, L)$ only has accepting BSCCs, one can represent the transitions between accepting states as an MC with only accepting states ${\mathcal M}_\pi^B:=(B, P_\pi^B,\mu,\Lambda, L)$ where

• B is the set of accepting states,

• $P_\pi^B$ is the transition probability matrix defined by

  (14) $$P_\pi ^B: = {P_{\pi, B \to B}} + {P_{\pi, B \to \neg B}}{(I - {P_{\pi, \neg B \to \neg B}})^{ - 1}}{P_{\pi, \neg B \to B}}.$$

• $\mu\,\in\, \Delta(B)$ is the initial distribution and determined by $s_0$ as

  (15) $$\eqalign{ & {\rm{if}}\;{\mkern 1mu} {s_0}\,\in\, B,\quad \mu (s) = \left\{ {\matrix{ 1 \hfill & {s = {s_0}} \hfill \cr 0 \hfill & {s \ne {s_0}} \hfill \cr } } \right., \cr & {\rm{if}}\;{\mkern 1mu} {s_0} \;\notin\; B,\quad \mu (s) = {({P_{init}})_{{s_0}s}} \cr} $$
  where $P_{init} := (I-P_{\pi,\neg B\to \neg B})^{-1}P_{\pi,\neg B\to B}$ is a matrix. Each element $(P_{init})_{ij}$ represents the probability of a path leaving the state $i\in {\neg B}$ and visiting state $j\in B$ without visiting any state in B between the leave and visit.

Proof. We start with constructing a transition matrix $P_\pi^B$ for the states in B whose (i, j)th element, denoted by $(P_\pi^B)_{ij}$ , is the probability of visiting jth state in B without visiting any state in B after leaving the ith state in B.

(16) $$\matrix{ {P_\pi ^B} \hfill & {\!\!\!\!: = {P_{\pi, B \to B}} + {P_{\pi, B \to \neg B}}\sum\limits_{k = 0}^\infty {P_{\pi, \neg B \to \neg B}^k} {P_{\pi, \neg B \to B}}.} \hfill \cr } $$

In (16), the matrix element $(P_{\pi,\neg B\to \neg B}^k)_{ij}$ represents the probability of a path leaving the state i and visiting state j after k steps without travelling through any states in B. However, the absence of rejecting BSCCs ensures that any path will visit a state in B in finite times with probability 1. Thus, for any $i, j\,\in\, \neg B$ , $\lim_{k\to \infty}(P_{\pi,\neg B\to \neg B}^k)_{ij}=0$ . This limit implies any eigenvalue $\lambda$ of $P_{\pi,\neg B\to \neg B}$ satisfies $|\lambda | \lt 1$ and therefore $\sum\nolimits_{k = 0}^\infty {P_{\pi, \neg B \to \neg B}^k} $ can be replaced by $(I - P_{\pi,\neg B\rightarrow \neg B})^{-1}$ in (16),

(17) $$P_\pi ^B = {P_{\pi, B \to B}} + {P_{\pi, B \to \neg B}}{(I - {P_{\pi, \neg B \to \neg B}})^{ - 1}}{P_{\pi, \neg B \to B}}.$$

Since all the elements on the right-hand side are greater or equal to zero, for any $i, j\,\in\, B$ , $(P_\pi^B)_{ij}\ge 0$ . Since there are only accepting BSCCs in the MC, given a path starting from an arbitrary state in B, the path will visit an accepting state in finite steps with probability one, ensuring that for all $i\in B$ , $\sum_{j\in S} (P_\pi^B)_{ij}=1$ . Thus, $P_\pi^B$ is a probability matrix that can be used to describe the behaviour of an MC with the state space as B only.

Lemma 4. Suppose there is no rejecting BSCC, for ${\rm\gamma} =1$ in (3), the Bellman equation (10) is equivalent to the following form,

(18) $$\matrix{ {} \hfill & {\left[ {\matrix{ {{U^B}} \cr {{U^{\neg B}}} \cr } } \right] = (1 - {{\rm\gamma} _B})\left[ {\matrix{ {{{\mathbb I}_m}} \cr {{{\mathbb O}_n}} \cr } } \right]} \hfill \cr {} \hfill & {\!\! + \left[ {\matrix{ {{{\rm\gamma} _B}{I_{m \times m}}} & {} \cr {} & {{I_{n \times n}}} \cr } } \right]\left[ {\matrix{ {P_\pi ^B} & {} \cr {{P_{\pi, \neg B \to B}}} & {{P_{\pi, \neg B \to \neg B}}} \cr } } \right]\left[ {\matrix{ {{U^B}} \cr {{U^{\neg B}}} \cr } } \right].} \hfill \cr } $$

The equation (18) implies that the solution $U^{B}$ does not rely on the rejecting states $\neg B$ . Subsequently, we leverage the fact that $U^{\neg B}$ is uniquely determined by $U^{B}$ to establish the uniqueness of the overall solution V.

Proof. We prove this lemma by showing the equivalence between $P_\pi^B U^{B}$ and $P_{\pi,B\to B}U^{B} + P_{\pi,B\to\neg B}U^{\neg B}$ . From the Bellman equation (10), we have $U^{\neg B}=P_{\pi,\neg B\to B} U^{B} + P_{\pi,\neg B\to \neg B}U^{\neg B}$

(19) $$\eqalign{ & {P_{\pi, B \to B}}{U^B} + {P_{\pi, B \to \neg B}}{U^{\neg B}} \cr & \mathop = \limits^{\unicode{x24B6}} \left( {{P_{\pi, B \to B}} + {P_{\pi, B \to \neg B}}{P_{\pi, \neg B \to B}}} \right){U^B} \cr & + {P_{\pi, B \to \neg B}}{P_{\pi, \neg B \to \neg B}}{U^{\neg B}} \cr & \mathop = \limits^{\unicode{x24B7}} \left( {{P_{\pi, B \to B}} + {P_{\pi, B \to \neg B}}{P_{\pi, \neg B \to B}}} \right){U^B} \cr & + \left( {{P_{\pi, B \to \neg B}}{P_{\pi, \neg B \to \neg B}}{P_{\pi, \neg B \to B}}} \right){U^B} \cr & + {P_{\pi, B \to \neg B}}P_{\pi, \neg B \to \neg B}^2{U^{\neg B}} \cr & \vdots \cr & \mathop = \limits^{ \unicode{x24B8}} \mathop {{\rm{lim}}}\limits_{ K \to \infty } \Bigg(\left( {{P_{\pi, B \to B}} + {P_{\pi, B \to \neg B}}\mathop \sum \limits_{k = 0}^K P_{\pi, \neg B \to \neg B}^{k}{P_{\pi, \neg B \to B}}} \right){U^B} \cr & + {P_{\pi, B \to \neg B}}P_{\pi, \neg B \to \neg B}^{K + 1}{U^{\neg B}}\Bigg) \cr & \mathop = \limits^\unicode{x24B9} P_\pi ^B{U^B} \cr} $$

where the equality Ⓐ holds as we replace $U^{\neg B}$ in the last term $P_{\pi,B\to\neg B}U^{\neg B}$ by $P_{\pi,\neg B\to B} U^{B} + P_{\pi,\neg B\to \neg B}U^{\neg B}$ . Similarily, the equalities Ⓑ and Ⓒ hold as we keep replacing $U^{\neg B}$ in the last term by $P_{\pi,\neg B\to B} U^{B} + P_{\pi,\neg B\to \neg B}U^{\neg B}$ . The equality Ⓓ holds by the definition of $P_\pi^B$ . $\square$

Lemma 4 shows the uniqueness of the solution. Directly solving equation (18) completes the proof for Proposition 2.

Proof for Proposition 2. From equation (18), we obtain the expression for the MC with only accepting states,

(20) $${U^B} = (1 - {{\rm\gamma} _B}){\mathbb I} + {{\rm\gamma} _B}P_\pi ^B{U^B}.$$

Given that all the eigenvalues of $P_\pi^B$ are within the unit disk and ${{\rm\gamma} _B} \lt 1$ , the matrix $(I-{\rm\gamma}_B P_\pi^B)$ is invertible. $U^{B}$ is uniquely determined by

(21) $${U^B} = (1 - {{\rm\gamma} _B}){(I - {{\rm\gamma} _B}P_\pi ^B)^{ - 1}}{\mathbb I}.$$

Moving to the set of rejecting states $U^{\neg B}$ , from equation (18) we have,

(22) $$\matrix{ {{U^{\neg B}}} \hfill & {\!\!\!\! = {P_{\pi, \neg B \to B}}{U^B} + {P_{\pi, \neg B \to \neg B}}{U^{\neg B}}} \hfill \cr {} \hfill & {\!\! \!\!= {{(I - {P_{\pi, \neg B \to \neg B}})}^{ - 1}}{P_{\pi, \neg B \to B}}{U^B}.} \hfill \cr } $$

Given the uniqueness of $U^{B}$ , and the invertibility of $(I-P_{\pi,\neg B\rightarrow \neg B})$ , we conclude that $U^{\neg B}$ is also unique.

Let $U^{B}=\mathbb{I}_m$ and $U^{\neg B}=\mathbb{I}_n$ , the Bellman equation (10) holds as the summation of each row of a probability matrix $P_B$ is always one,

(23) $$\matrix{ {\left[ {\matrix{ {{{\mathbb I}_m}} \cr {{{\mathbb I}_n}} \cr } } \right]} \hfill & {\!\! = (1 - {{\rm\gamma} _B})\left[ {\matrix{ {{{\mathbb I}_m}} \cr {{{\mathbb O}_n}} \cr } } \right] + \left[ {\matrix{ {{{\rm\gamma} _B}{I_{m \times m}}} & {} \cr {} & {{I_{n \times n}}} \cr } } \right]{P_\pi }\left[ {\matrix{ {{{\mathbb I}_m}} \cr {{{\mathbb I}_n}} \cr } } \right].} \hfill \cr } $$

Therefore, in the absence of rejecting BSCCs, the unique solution to the Bellman equation (10) is $\mathbb{I}$ . $\square$

Proposition 2 shows the solutions for the states inside an accepting BSCC have to be 1. All states outside the BSCC cannot be reached from a state inside the BSCC, thus the solution for states outside this BSCC is not involved in the solution for states inside the BSCC. By Lemma 4, the Bellman equation for an accepting BSCC can be rewritten into the form of (18) where $U_B$ and $U_{\neg B}$ stands for the solution for accepting states and rejecting states inside this BSCC, and vector $\mathbb{I}$ is the unique solution.

When accepting and rejecting BSCC both exist in the MC

Having established the uniqueness of solutions in the case of accepting BSCCs, we now shift our focus to the general case involving rejecting BSCCs. We state in Proposition 2 that the solutions for the states in the accepting BSCCs are unique and equal to $\mathbb{I}$ . We now demonstrate that setting the solutions for the states in rejecting BSCCs to $\mathbb{O}$ ensures the uniqueness and correctness of the solutions for all states. We partition the state space further into $\{B_A, B_T, \neg B_A, \neg B_R, \neg B_T\}$ . Here the subscripts A and R denote accepting and rejecting (unrelated to the action set A or reward function R). Specifically,

• $B_A$ is the set of accepting states in the BSCCs,

• $B_T:=B\backslash B_A$ is the set of transient accepting states,

• $\neg B_A$ is the set of rejecting states in the accepting BSCCs,

• $\neg B_R$ is the set of rejecting states in the rejecting BSCCs,

• $\neg B_T:=\neg B\backslash (\neg B_A \cup \neg B_R)$ is set of transient rejecting states.

We rewrite the Bellman equation (10) in the form of (24).

(24) $${\matrix{ {} \hfill & {\left[ {\matrix{ {{U^{{B_T}}}} \cr {{U^{{B_A}}}} \cr {{U^{\neg {B_T}}}} \cr {{U^{\neg {B_A}}}} \cr {{U^{\neg {B_R}}}} \cr } } \right] = (1 - {{\rm\gamma} _B})\left[ {\matrix{ {{{\mathbb I}_m}} \cr {{{\mathbb O}_n}} \cr } } \right]} \hfill \cr {} \hfill & \!\!\!\!\!\!{ + \left[ {\matrix{ {{{\rm\gamma} _B}{I_{m \times m}}} & {} \cr {} & {{I_{n \times n}}} \cr } } \right]} \hfill \cr {} \hfill & \!\!\!\!\!\!\!\times{\left[ {\matrix{ {{P_{\pi, {B_T} \to {B_T}}}} & {{P_{\pi, {B_T} \to {B_A}}}} & {{P_{\pi, {B_T} \to \neg {B_T}}}} & {{P_{\pi, {B_T} \to \neg {B_A}}}} & {{P_{\pi, {B_T} \to \neg {B_R}}}} \cr {} & {{P_{\pi, {B_A} \to {B_A}}}} & {} & {{P_{\pi, {B_A} \to \neg {B_A}}}} & {{P_{\pi, {B_A} \to \neg {B_R}}}} \cr {{P_{\pi, \neg {B_T} \to {B_T}}}} & {{P_{\pi, \neg {B_T} \to {B_A}}}} & {{P_{\pi, \neg {B_T} \to \neg {B_T}}}} & {{P_{\pi, \neg {B_T} \to \neg {B_A}}}} & {{P_{\pi, \neg {B_T} \to \neg {B_R}}}} \cr {} & {{P_{\pi, \neg {B_A} \to {B_A}}}} & {} & {{P_{\pi, \neg {B_A} \to \neg {B_A}}}} & {{P_{\pi, \neg {B_A} \to \neg {B_R}}}} \cr {} & {} & {} & {} & {{P_{\pi, \neg {B_R} \to \neg {B_R}}}} \cr } } \right]} \hfill \cr {} \hfill & \!\!\!\!\!\times{\left[ {\matrix{ {{U^{{B_T}}}} \cr {{U^{{B_A}}}} \cr {{U^{\neg {B_T}}}} \cr {{U^{\neg {B_A}}}} \cr {{U^{\neg {B_R}}}} \cr } } \right].} \hfill \cr }} $$

The solution for states inside BSCCs has been fixed as $[{U^{B_A}}^T, {U^{\neg B_A}}^T]^T=\mathbb{I}$ and $U^{\neg B_R}=\mathbb{O}$ . The solution $U^{B_T}$ and $U^{\neg B_T}$ for transient states remain to be shown as unique. We rewrite the Bellman equation (24) into the following form (25) where $U^{B_T}$ and $U^{\neg B_T}$ are the only variables,

(25) $$\matrix{ {} \hfill & {\left[ {\matrix{ {{U^{{B_T}}}} \cr {{U^{\neg {B_T}}}} \cr } } \right] = \left[ {\matrix{ {{{\rm\gamma} _B}{I_{{m_1} \times {m_1}}}} & {} \cr {} & {{I_{{n_1} \times {n_1}}}} \cr } } \right]} \hfill \cr {} \hfill & {\left[ {\matrix{ {{P_{\pi, {B_T} \to {B_T}}}} & {{P_{\pi, {B_T} \to \neg {B_T}}}} \cr {{P_{\pi, \neg {B_T} \to {B_T}}}} & {{P_{\pi, \neg {B_T} \to \neg {B_T}}}} \cr } } \right]\left[ {\matrix{ {{U^{{B_T}}}} \cr {{U^{\neg {B_T}}}} \cr } } \right] + \left[ {\matrix{ {{B_1}} \cr {{B_2}} \cr } } \right]} \hfill \cr } $$

here $m_1=\vert U^{B_T}\vert$ , $m_2=\vert U^{B_A}\vert$ , $n_1=\vert U^{\neg B_T}\vert$ , $n_2=\vert U^{\neg B_A}\vert$ , and

$$\eqalign{ & \left[ {\matrix{ {{B_1}} \cr {{B_2}} \cr } } \right] = (1 - {{\rm\gamma} _B})\left[ {\matrix{ {{{\mathbb I}_{{m_1}}}} \cr {{{\mathbb O}_{{n_1}}}} \cr } } \right] + \left[ {\matrix{ {{{\rm\gamma} _B}{I_{{m_1} \times {m_1}}}} & {} \cr {} & {{I_{{n_1} \times {n_1}}}} \cr } } \right] \cr & \left[ {\matrix{ {{P_{\pi, {B_T} \to {B_A}}}} & {{P_{\pi, {B_T} \to \neg {B_A}}}} \cr {{P_{\pi, \neg {B_T} \to {B_A}}}} & {{P_{\pi, \neg {B_T} \to \neg {B_A}}}} \cr } } \right]\left[ {\matrix{ {{{\mathbb I}_{{m_2}}}} \cr {{{\mathbb I}_{{n_2}}}} \cr } } \right]. \cr} $$

Lemma 5. The equation (25) has a unique solution.

We demonstrate $U^{B_T}$ does not rely on states in $\neg B_T$ and $U^{\neg B_T}$ is uniquely determined by $U^{B_T}$ . Then the uniqueness of $U^{B_T}$ can be shown first, consequently, uniqueness of $U^{\neg B_T}$ can be shown.

Proof. First we show $U^{B_T}$ is unique since its calculation can exclude $U^{\neg B}$ . By equation (25),

(26) $$\matrix{ {{U^{\neg {B_T}}}} \hfill & {\!\! \!\!= {P_{\pi, \neg {B_T} \to {B_T}}}{U^{{B_T}}} + {P_{\pi, \neg {B_T} \to \neg {B_T}}}{U^{\neg {B_T}}}} \hfill \cr\cr {} \hfill & {\!\!\!\! + \left[ {\matrix{ {{P_{\pi, \neg {B_T} \to {B_A}}}} & {{P_{\pi, \neg {B_T} \to \neg {B_A}}}} \cr } } \right]{\mathbb I}} \hfill \cr } $$

The following equalities hold as we keep expanding $U^{\neg B_T}$ ,

(27) $$\eqalign{ & {P_{\pi, {B_T} \to {B_T}}}{U^{{B_T}}} + {P_{\pi, {B_T} \to \neg {B_T}}}{U^{\neg {B_T}}} \cr & \mathop = \limits^{\unicode{x24B6}} ({P_{\pi, {B_T} \to {B_T}}} + {P_{\pi, {B_T} \to \neg {B_T}}}{P_{\pi, \neg {B_T} \to {B_T}}}){U^{{B_T}}} \cr & + {P_{\pi, {B_T} \to \neg {B_T}}}\left[ {\matrix{ {{P_{\pi, \neg {B_T} \to {B_A}}}} & {{P_{\pi, \neg {B_T} \to \neg {B_A}}}} \cr } } \right]{\mathbb I} \cr & + {P_{\pi, {B_T} \to \neg {B_T}}}{P_{\pi, \neg {B_T} \to \neg {B_T}}}{U^{\neg {B_T}}} \cr & \vdots \cr & \mathop = \limits^{\unicode{x24B7}} ({P_{\pi, {B_T} \to {B_T}}} + \cr & {P_{\pi, {B_T} \to \neg {B_T}}}\sum\limits_{k = 0}^K {P_{\pi, \neg {B_T} \to \neg {B_T}}^k} {P_{\pi, \neg {B_T} \to {B_T}}}){U^{{B_T}}} \cr & + {P_{\pi, {B_T} \to \neg {B_T}}}\sum\limits_{k = 0}^K {P_{\pi, \neg {B_T} \to \neg {B_T}}^k} \cr & \left[ {\matrix{ {{P_{\pi, \neg {B_T} \to {B_A}}}} & {{P_{\pi, \neg {B_T} \to \neg {B_A}}}} \cr } } \right]{\mathbb I} \cr & + {P_{\pi, {B_T} \to \neg {B_T}}}P_{\pi, \neg {B_T} \to \neg {B_T}}^{K + 1}{U^{\neg {B_T}}} \cr} $$

where the equality Ⓐ holds as we replace $U^{\neg B_T}$ in the last term by (26). Similarily, the equality Ⓑ hold as we keep expanding $U^{\neg B_T}$ in the last term by (26). Since $P_{\pi,\neg B_T\to\neg B_T}$ only contains the transition probabilities between the transient states, for any $i,j\in \neg B_T$ , $\lim_{K\to\infty}{(P_{\pi,\neg B_T\to\neg B_T}^K)_{ij}}=0$ . Taking $K\to \infty$ in (26) and using the fact that $(I-P_{\pi,\neg B_T\to\neg B_T})^{-1} = \sum_{k=0}^\infty{P_{\pi,\neg B_T\to\neg B_T}^k}$ , we have

(28) $$\matrix{ {} \hfill & {{P_{\pi, {B_T} \to {B_T}}}{U^{{B_T}}} + {P_{\pi, {B_T} \to \neg {B_T}}}{U^{\neg {B_T}}}} \hfill \cr = \hfill & {({P_{\pi, {B_T} \to {B_T}}} + } \hfill \cr {} \hfill & {{P_{\pi, {B_T} \to \neg {B_T}}}{{(I - {P_{\pi, \neg {B_T} \to \neg {B_T}}})}^{ - 1}}{P_{\pi, \neg {B_T} \to {B_T}}}){U^{{B_T}}}} \hfill \cr + \hfill & {{P_{\pi, {B_T} \to \neg {B_T}}}{{(I - {P_{\pi, \neg {B_T} \to \neg {B_T}}})}^{ - 1}}} \hfill \cr {} \hfill & {\left[ {\matrix{ {{P_{\pi, \neg {B_T} \to {B_A}}}} & {{P_{\pi, \neg {B_T} \to \neg {B_A}}}} \cr } } \right]{\mathbb I}.} \hfill \cr } $$

Plugging (28) into (25), we show the calculation of $U^{B_T}$ does not rely on $U^{\neg B}$ ,

(29) $$\matrix{ {{U^{{B_T}}}} \hfill & {\!\!\!\! = {{\rm\gamma} _B}{P_{\pi, {B_T} \to {B_T}}}{U^{{B_T}}} + {{\rm\gamma} _B}{P_{\pi, {B_T} \to \neg {B_T}}}{U^{\neg {B_T}}} + {B_1}} \hfill \cr\cr {} \hfill & {\!\! \!\!= {{\rm\gamma} _B}P_\pi ^{{B_T}}{U^{{B_T}}} + {{\rm\gamma} _B}{P_{\pi, {B_T} \to \neg {B_T}}}{{(I - {P_{\pi, \neg {B_T} \to \neg {B_T}}})}^{ - 1}}} \hfill \cr\cr {} \hfill & {\left[ {\matrix{ {{P_{\pi, \neg {B_T} \to {B_A}}}} & {{P_{\pi, \neg {B_T} \to \neg {B_A}}}} \cr } } \right]{\mathbb I} + {B_1}.} \hfill \cr } $$

Here,

$$\matrix{ {P_\pi ^{{B_T}}} \hfill & {\!\!: = {P_{\pi, {B_T} \to {B_T}}}} \hfill \cr {} \hfill & {\!\! + {P_{\pi, {B_T} \to \neg {B_T}}}{{(I - {P_{\pi, \neg {B_T} \to \neg {B_T}}})}^{ - 1}}{P_{\pi, \neg {B_T} \to {B_T}}}} \hfill \cr } $$

where for any $i,j\in B_T$ , $(P_{\pi}^{B_T})_{ij}$ is the probability of visiting jth state in B without visiting any state in $B_T$ after leaving the ith state in $B_T$ . As $P_{\pi}^{B_T}$ consists of only the transition probabilities between the transient states, $\lim_{k\to\infty}{({P_{\pi}^{B_T}}^k)_{ij}}=0$ . Thus any eigenvalue $\lambda$ of ${P_{\pi}^{B_T}}$ satisfies $\vert \lambda\vert \lt 1$ . Since ${\rm\gamma}_B \lt 1$ , we are sure $(I-{\rm\gamma}_B P_{\pi}^{B_T})$ is invertible and $U^{B_T}$ has a unique solution as,

(30) $$\matrix{ {} \hfill & {{U^{{B_T}}} = {{(I - {{\rm\gamma} _B}P_\pi ^{{B_T}})}^{ - 1}}{{\rm\gamma} _B}{P_{\pi, {B_T} \to \neg {B_T}}}{{(I - {P_{\pi, \neg {B_T} \to \neg {B_T}}})}^{ - 1}}} \hfill \cr\cr {} \hfill & {\left[ {\matrix{ {{P_{\pi, \neg {B_T} \to {B_A}}}} & {{P_{\pi, \neg {B_T} \to \neg {B_A}}}} \cr } } \right]{\mathbb I} + {{(I - {{\rm\gamma} _B}P_\pi ^{{B_T}})}^{ - 1}}{B_1}.} \hfill \cr } $$

From (25), we have

(31) $$\matrix{ {{U^{\neg {B_T}}}} \hfill & {\!\! = {P_{\pi, \neg {B_T} \to {B_T}}}{U^{{B_T}}} + {P_{\pi, \neg {B_T} \to \neg {B_T}}}{U^{\neg {B_T}}} + {B_2}.} \hfill \cr } $$

Using the fact $I- P_{\pi,\neg B_T\rightarrow \neg B_T}$ is invertible, we show $U^{\neg B_T}$ is uniquely determined by $U^{B_T}$ as

(32) $${U^{\neg {B_T}}} = {(I - {P_{\pi, \neg {B_T} \to \neg {B_T}}})^{ - 1}}({P_{\pi, \neg {B_T} \to {B_T}}}{U^{{B_T}}} + {B_2}),$$

Thus, the equation (25) has a unique solution. $\square$

In Lemma 5, for the case ${\rm\gamma}=1$ , we have shown that the equation (25) with surrogate reward (3) has a unique solution. In order to complete the proof for theorem 1, what remains to be shown is the unique solution of the equation (25) is equal to the value function (5).

Proof. for Theorem 1. The solution to the equation (25) is unique. For all $s\in \neg B_R$ , the value function $V(s)=0$ , then the value function is the unique solution for equation (25). Under the condition that the solution for all rejecting BSCCs is zero, the Bellman equation (10) is equivalent to the equation (25). $\square$

Setup of the LTL planning problem

We select a suitable Bellman equation based on the following reasons. 1) The Bellman equation is encountered in a benchmark LTL planning problem, which was introduced in (Bozkurt et al., Reference Bozkurt, Wang, Zavlanos and Pajic2020). 2) The bellman equation contains all types of states $\{B_A, B_T, \neg B_A, \neg B_R, \neg B_T\}$ we have discussed previously.

The Bellman equation is encountered in an LTL planning problem. The planning problem considers finding the optimal policy on a 20-state MDP that maximizes the satisfaction probability of a complex LTL objective

$$\matrix{ {\unicode{x03C6}} \hfill & {\!\! = \square (\neg d \wedge (b \wedge \neg \ \bigcirc b) \to \ \bigcirc (\neg b \cup (a \vee c)) \wedge a} \hfill \cr {} \hfill & \!\! { \to \ \bigcirc (\neg a \cup b) \wedge (\neg b \wedge \ \bigcirc b \wedge \neg \ \bigcirc \ \bigcirc b) \to (\neg a \cup c) \wedge c} \hfill \cr {} \hfill & \!\! { \to (\neg a \cup b) \wedge (b \wedge \ \bigcirc b) \to \lozenge a).} \hfill \cr } $$

One can transform the planning problem into finding the optimal memoryless policy on a 1040 states product MDP, using the csrl package (Bozkurt et al., Reference Bozkurt, Wang, Zavlanos and Pajic2020). The product MDP is a product of the MDP and a limit-deterministic Büchi automaton representing the LTL objective. The limit-deterministic Büchi automaton with 52 statesFootnote ³ is constructed using Rabinizer 4 (Křetínský et al., Reference Kretínský, Meggendorfer, Sickert, Ziegler, Chockler and Weissenbacher2018). The MC is induced by a memoryless policy on the product MDP. We obtain the memoryless policy using Q-learning following the same setting in (Bozkurt et al., Reference Bozkurt, Wang, Zavlanos and Pajic2020) with state-dependent discounting ${\rm\gamma}_B = 0.99$ , ${\rm\gamma} = 0.99999$ , number of episodes $K=100000$ , length of each episode $T=1000$ , and a time-varying learning rate. Note that the values of ${\rm\gamma}_B$ and ${\rm\gamma}$ in this setting are used only in Q-learning. In this case study, we study the Bellman equation with different ${\rm\gamma}_B$ values while setting ${\rm\gamma}=1$ .

The Bellman equation describes the recursive formulation of the value function on the Markov chain. The Markov chain has a state space that covers the set of states $\{B_A, B_T, \neg B_A, \neg B_R, \neg B_T\}$ . The state space consists of 1040 states, where 53 are accepting and 987 are rejecting. The cardinalities for different sets of states are shown as $\vert B_A \vert = 13$ , $\vert B_T \vert = 40$ , $\vert \neg B_A\vert = 112$ , $\vert \neg B_R \vert = 4$ , $\vert \neg B_T\vert = 871$ . We identify all BSCCs using graph search (Tarjan algorithm) (Tarjan, Reference Tarjan1972) and classify a BSCC as accepting or rejecting based on whether it contains an accepting state or not. 3 BSCCs are found, 2 are accepting BSCCs consists of 125 states in total and 1 is rejecting consists of 4 states.

Validating the uniqueness condition for tabular value iteration

We study the Bellman equation through dynamic programming updates. Solving the Bellman equation with dynamic programming is fundamental because it forms the basis of RL. Many RL algorithms can be viewed as stochastic versions of dynamic programming. Thus, understanding how dynamic programming behaves reveals when RL algorithms can or cannot recover the correct value function. All computations were implemented in Python/NumPy and use double precision throughout (float64).

Here, we show that only by forcing the values of rejecting BSCCs to zero can we use dynamic programming to calculate the correct value function. Moreover, we use the satisfaction probability to verify if the numerical solution is the value function of the Bellman equation. We do not directly apply the matrix inverse to get the solution by equation (11) as $(I_{m+n}-\Gamma_BP_\pi)$ is no longer invertible when ${\rm\gamma} = 1$ .

Given an initialization of the approximate value function $U_{(0)}=U_0$ and the Bellman equation (7), we iteratively update the approximate value function $U_{(k)}$ as

$${U_{(k + 1)}} = (1 - {{\rm\gamma} _B})\left[ {\matrix{ {{{\mathbb I}_m}} \cr {{{\mathbb O}_n}} \cr } } \right] + \left[ {\matrix{ {{{\rm\gamma} _B}{I_{m \times m}}} & {} \cr {} & {{\rm\gamma} {I_{n \times n}}} \cr } } \right]{P_\pi }{U_{(k)}},$$

where $m= 53$ is the number of accepting states, $n= 987$ is the number of rejecting states, $P_\pi$ is the transition probability matrix of the MC. The approximate value function $U_{(k)}$ will converge to a solution to the Bellman equation during updates. We obtain different solutions to the different Bellman equations by changing ${\rm\gamma}_B$ and $U_0$ . We run sufficient iterations $K=6\times 10^6$ such that the approximate value function satisfies the Bellman equation with negligible numerical error less than $\varepsilon = 10^{-15}$ ,

$${\rm\delta}(k):=\left\Vert U_{(k+1)}-U_{(k)}\right\Vert_\infty \lt \varepsilon,\qquad \varepsilon=10^{-15},$$

We choose $\varepsilon$ small enough that floating-point errors do not affect our comparisons of the solutions. Figure 3(a) and Figure 2(b) show $U_{(k)}$ converges to a solution under all settings.

Figure 2.

Under different discount factors $\gamma_B = 1-10^{-3}, 1-10^{-4}, 1-10^{-5}$ , the change of $\delta(k) = \Vert U_{(k+1)} - U_{(k)}\Vert_\infty$ (Bellman error) and $e(k) = \Vert U_{(k)} - \bar{V} \Vert_\infty$ (Approximation error) during dynamic programming updates with $U_0 = \mathbb{O}$ . (a) The vanishing of $\delta(k)$ shows the approximate value function $U_{(k)}$ converges to a solution to the Bellman equation in all settings. (b) Different e(K) shows the accuracy of the approximation is determined by $1-\gamma_B$ . Final errors are $e(K)=0.4456,\ 0.0740,\ 0.0079$ for different $\gamma_B$ . When the discount factor $\gamma_B=0.999$ is less closer to 1, e(k) grows instead of decreasing. The reason is that $U_{(k)}$ is now converging to a value function that deviates from the satisfaction probability.

Figure 3.

Under different initial conditions $U_0 = \mathbb{O}$ and $U_0 \neq \mathbb{O}$ , the change of $\delta(k) = \Vert U_{(k+1)} - U_{(k)}\Vert_\infty$ (Bellman error) and $e(k) = \Vert U_{(k)} - \bar{V} \Vert_\infty$ (Approximation error) during dynamic programming updates with $\gamma_B = 0.99999$ (a) The vanishing of $\delta(k)$ confirms that the approximate value function $U_{(k)}$ converges to a solution to the Bellman equation in all settings. (b) The decrease of e(k) shows the numerical solution is close to the satisfaction probability only when the initial conditions $U_0 = \mathbb{O}$ . Even when $ U_{(k)}$ converges, $e(k) \gt 0$ remains because the true value function is an approximation to the satisfaction probability.

The satisfaction probability $P(s \models \square \lozenge B)$ on the MC is the ground truth we used to evaluate the numerical solution of the Bellman equation. The Bellman equation is constructed so that its solution approximates the satisfaction probability. In order to show the approximation, we consider the solutions to Bellman equations with different discount factors ${\rm\gamma}_B$ . Figure 2(b) shows closer the discount factor ${\rm\gamma}_B$ to 1, less the approximation error

$$e(k): = {\left\| {\bar V - {U_{(k)}}} \right\|_\infty },$$

where $\bar{V}(s)$ is the satisfaction probability $P(s \models \square \lozenge B)$ we computed. For ${\rm\gamma}_B\,\in\, \{1-10^{-5},\,1-10^{-4},\,1-10^{-3}\}$ , the errors are $0.0079$ , $0.0740$ , and $0.4456$ , respectively, as shown in Figure 2. This error is described in equation (5) as

$$e(s) = {{\mathbb E}_\pi }[{G_t}({\unicode{x03C3}} )\mid {\unicode{x03C3}} [t] = s,{\unicode{x03C3}} \not\models {\rm{\square}}\lozenge B] \cdot P(s \not\models {\rm{\square}}\lozenge B).$$

Taking ${\rm\gamma}_B\to 1$ makes the expectation of cumulative reward collected on the path not entering accepting BSCCs smaller. In Figure 2(b), we can see that when the discount factor ${\rm\gamma}_B=0.999$ is less close to 1, e(k) grows instead of decreasing. The reason is that $U_{(k)}$ is now converging to a value function that deviates from the satisfaction probability.

We compute the ground truth using model-checking techniques (Baier and Katoen, Reference Baier and Katoen2008). Specifically, we identify the set of states in accepting BSCCs $Acc:= B_A\cup\neg B_A$ using graph search. By the property of BSCCs in Definition 6, the reachability probability of each state $\bar{V}(s)$ to Acc is equivalent to the satisfaction probability $P(s \models \square \lozenge B)$ . Then, we compute the reachability probability $\bar{V}(s)$ , which is the least fixed point solution to the following equation (Baier and Katoen, Reference Baier and Katoen2008, Theorem 10.15.),

$$\matrix{ {\bar V(s)} \hfill & {\!\! = \left\{ {\matrix{ 1 \hfill & {s\,\in\, Acc} \hfill \cr {{\sum _{a\,\in\, S\backslash Acc}}P(s,a)\bar V(a) + \sum\limits_{b\,\in\, Acc} P (s,b)} \hfill & {s \,\notin\, Acc} \hfill \cr } } \right..} \hfill \cr } $$

The finding in Figure 3(b) shows that the numerical solution is the value function only when the condition $U^{\neg B_R} = \mathbb{O}$ is satisfied. We generate different solutions for the same Bellman equation ( ${\rm\gamma}_B = 1 - 10^{-5}$ ) with different $U_0$ . Numerical results show that only the solution satisfying $U_{(K)}^{\neg B_R} = \mathbb{O}$ is close to the satisfaction probability. Specifically, for $U_0 = \mathbb{O}$ , the solution $U_{(K)}$ satisfies the condition, leading to the smallest error of $\Vert U_{(K)} - \bar{V} \Vert_\infty = 0.0079$ . Such an error is inevitable since the value function is an approximation to the satisfaction probability. Meanwhile, a random initialization $U_0 \sim [0,1]^{m+n}$ or a nonzero constant $U_0 = c \mathbb{I}$ with $c\in (0,1)$ violates the condition, and $U_{(K)}$ converges to a solution not satisfying the condition. We obtain significantly larger errors $\Vert U_{(K)} - \bar{V} \Vert_\infty = 16.75$ and $3.238$ , respectively. In these cases, the solution does not coincide with the value function of the Bellman equation.

Figure 4 shows, the change of approximate value function $U_{(k)}$ for all four states $\neg B_R$ inside one rejecting BSCCs during updates, given $U_0 \sim [0,1]^{m+n}$ . For the case when $U_0=\mathbb{O}$ and $U_0=0.1\mathbb{I}$ , the approximate value function stays at 0 and 1, respectively. Since the Bellman update

$$U_{(k + 1)}^{\neg {B_R}} = {P_{\pi, \neg {B_R} \to \neg {B_R}}}U_{(k)}^{\neg {B_R}},$$

does not involve discounting within a rejecting BSCC. The approximate value function either converges to the same nonzero constant for all states inside one BSCC or remains unchanged. Given an ill-posed initial condition, this constant differs from the value function, demonstrating that the solution is incorrect when the condition in Theorem 1 does not hold.

Figure 4.

Given $U_0 \sim [0,1]^{m+n}$ , the change of approximate value function $U_{(k)}$ for all four states $s_1,s_2,s_3$ and $s_4$ inside one rejecting BSCCs during updates. Since the Bellman update does not involve discounting within a rejecting BSCC, the approximate value function converges to the same nonzero constant for all states inside the BSCC. This constant differs from the value function, demonstrating that the solution is incorrect when the condition in Theorem 1 does not hold.

Validating the uniqueness condition for neural network approximations

We extend the case study to value estimation with neural network function approximation, which is widely used in modern RL for temporal-logic planning. Our aim is to demonstrate that, when ${\rm\gamma}=1$ , the uniqueness condition in Theorem 1 is typically violated by neural network generalization. We therefore organize the experiments around three guiding questions: (i) what occurs under standard practice without special handling; (ii) If we mimic the tabular initialization that preserves zeros on rejecting BSCCs, does a neural network keep them at zero? and (iii) whether enforcing the uniqueness condition leads to a more accurate solution. These questions correspond to the following setups.

• Baseline: We train on all states with standard random initialization. This represents the most common practice in RL. It highlights that without any special treatment, the estimated values of rejecting BSCC states deviate from the true solution, yielding large errors.

• Init0: train on all states with the output bias initialized so that $V_\theta(s)= 0$ at the start. This shows that even if value function estimations for rejecting BSCCs start at zero, generalization during training makes their values drift away from zero.

• Subset: train only on states that are not in rejecting BSCCs. Rejecting BSCC states $\neg B_R$ are excluded from both the model’s input and its output. This represents the ideal case when our condition holds, even though current work using neural networks does not achieve this.

We approximate the value function with a neural network $V_\theta$ implemented in Python/PyTorch with double precision. After k optimization steps the parameters are $\theta_k$ and the output at state s is $V_{\theta_k}(s)$ . All losses and metrics are functions of $\theta_k$ . The neural network $V_\theta$ maps a discrete state index to a trainable 16-dimensional embedding, followed by two hidden layers of width 64 with LeakyReLU activations $\phi(x)=\max\{x,-0.01x\}$ and a linear output head. Training minimizes the mean squared Bellman error on the training set $S_{{\rm train}}$ ,

$${\cal L}({\theta _k}) = {1 \over {|{S_{{\rm{train}}}}|}}\sum\limits_{s\,\in\, {S_{{\rm{train}}}}} \Big( {V_{{\theta _k}}}(s) - R(s) - \Gamma (s)\sum\limits_{s'} P (s,s'){\mkern 1mu} {V_{{\theta _k}}}(s'){\Big)^2},$$

using AdamW with learning rate $10^{-4}$ . In Baseline and Init0 we use the full state space $S_{{\rm train}}=S$ (with Init0 initializing the output bias so that $V_{\theta_0}(s)\approx 0$ ). In Subset we train only on states outside rejecting BSCCs, $S_{{\rm train}}=S\setminus \neg B_R$ , and whenever a Bellman target requires a successor $s'\in\neg B_R$ we substitute 0 for its value.

Accuracy and the uniqueness condition are evaluated by mean squared errors (MSE) with respect to the previously computed true value function $\bar V$ , the solution of the Bellman equation with ${\rm\gamma}=1$ and ${\rm\gamma}_B=0.999$ . We report the mean squared error over the evaluation set

$${\rm{MS}}{{\rm{E}}_S}({\theta _k}) = {1 \over {|{S_{eval}}|}}\sum\limits_{s\,\in\, {S_{eval}}} ( {V_{{\theta _k}}}(s) - \bar V(s){)^2},$$

and the mean squared error on rejecting BSCC states

$${\rm{MS}}{{\rm{E}}_{\neg {B_R}}}({\theta _k}) = {1 \over {|\neg {B_R}|}}\sum\limits_{s\,\in\, \neg {B_R}} ( {V_{{\theta _k}}}(s) - \bar V(s){)^2},$$

where $S_{eval}=S$ for the setup Init0 and Baseline and $S_{eval}=S\backslash \neg B_R$ for the setup Subset. We use MSE because it measures errors across all states in S and $\neg B_R$ , whereas $L_\infty$ used in the tabular case reflects only the single worst state.

The experiments show that neural networks break the uniqueness condition. Training converges in all three setups, as shown in Figure 5. In Figure 5(a) the loss $\mathcal{L}(\theta_k)$ decreases in every setup and, after $k\geq 8\times10^{4}$ , remains below $10^{-5}$ . Figure 5(b) shows that ${\rm MSE}_{\neg B_R}(\theta_k)$ on the evaluation set changes only slightly once ${\mathcal{L}}(\theta_k)$ has stabilized. Throughout training, Subset yields a distinctly smaller ${\rm MSE}_{\neg B_R}(\theta_k)$ than Baseline and Init0 because, when computing Bellman targets, transitions into $\neg B_R$ use a fixed value 0. Therefore, errors on $\neg B_R$ do not propagate to the rest of the state space. In contrast, Figure 6 shows that the uniqueness condition is violated in both Baseline and Init0: ${\rm MSE}_{\neg B_R}(\theta_k)$ converges to a nonzero level, and although Init0 starts near zero by construction, its error quickly rises to a nonzero plateau due to training-induced generalization.

Figure 5.

Neural network value approximation under three setups: Subset (train only on $S\backslash \neg B_R$ , excluding $\neg B_R$ from inputs/outputs), Baseline (train on all states with standard initialization), and Init0 (train on all states with zero-bias initialization so $V_\theta(s)\approx 0$ at $k{=}0$ ). (a) Training loss $\mathcal{L}(\theta_k)$ , i.e., the mean squared Bellman residual on $S_{{\rm train}}$ . (b) Value error ${\rm MSE}_{\mathcal{S}}(\theta_k)$ on the evaluation set. Empirically, for all setups, the training loss drops and, after $k\geq 8\times 10^{4}$ , stabilizes below $10^{-5}$ , while the value error changes only marginally thereafter. The Subset setup yields substantially smaller value error than Baseline and Init0.

Figure 6.

${\rm MSE}_{\neg B_R}(\theta_k)$ on rejecting BSCCs. The error in both Baseline and Init0 converges to an incorrect, nonzero level, indicating the neural network outputs on $\neg B_R$ violate the uniqueness condition in Theorem 1. Although Init0 starts near zero by construction, its error quickly rises to a nonzero plateau due to generalization during training.