Qualitative Analysis of VASSInduced MDPs
 3 Citations
 439 Downloads
Abstract
We consider infinitestate Markov decision processes (MDPs) that are induced by extensions of vector addition systems with states (VASS). Verification conditions for these MDPs are described by reachability and Büchi objectives w.r.t. given sets of controlstates. We study the decidability of some qualitative versions of these objectives, i.e., the decidability of whether such objectives can be achieved surely, almostsurely, or limitsurely. While most such problems are undecidable in general, some are decidable for large subclasses in which either only the controller or only the random environment can change the counter values (while the other side can only change controlstates).
1 Introduction
Markov decision processes (MDPs) [14, 17] are a formal model for games on directed graphs, where certain decisions are taken by a strategic player (a.k.a. Player 1, or controller) while others are taken randomly (a.k.a. by nature, or the environment) according to predefined probability distributions. MDPs are thus a subclass of general 2player stochastic games, and they are equivalent to 1.5player games in the terminology of [10]. They are also called “games against nature”.
A run of the MDP consists of a sequence of visited states and transitions on the graph. Properties of the system are expressed via properties of the induced runs. The most basic objectives are reachability (is a certain (set of) controlstate(s) eventually visited?) and Büchi objectives (is a certain (set of) controlstate(s) visited infinitely often?).
Since a strategy of Player 1 induces a probability distribution of runs of the MDP, the objective of an MDP is defined in terms of this distribution, e.g., if the probability of satisfying a reachability/Büchi objective is at least a given constant. The special case where this constant is 1 is a key example of a qualitative objective. Here one asks whether Player 1 has a strategy that achieves an objective surely (all runs satisfy the property) or almostsurely (the probability of the runs satisfying the property is 1).
Most classical work on algorithms for MDPs and stochastic games has focused on finitestate systems (e.g., [11, 14, 19]), but more recently several classes of infinitestate systems have been considered as well. For instance, MDPs and stochastic games on infinitestate probabilistic recursive systems (i.e., probabilistic pushdown automata with unbounded stacks) [13] and on onecounter systems [6, 7] have been studied. Another infinitestate probabilistic model, which is incomparable to recursive systems, is a suitable probabilistic extension of Vector Addition Systems with States (VASS; a.k.a. Petri nets), which have a finite number of unbounded counters holding natural numbers.
Our Contribution. We study the decidability of probability1 qualitative reachability and Büchi objectives for infinitestate MDPs that are induced by suitable probabilistic extensions of VASS that we call VASSMDPs. (Most quantitative objectives in probabilistic VASS are either undecidable, or the solution is at least not effectively expressible in \((\mathbb {R},+,*,\le )\) [3]). It is easy to show that, for general VASSMDPs, even the simplest of these problems, (almost) sure reachability, is undecidable. Thus we consider two monotone subclasses: 1VASSMDPs and PVASSMDPs. In 1VASSMDPs, only Player 1 can modify counter values while the probabilistic player can only change controlstates, whereas for PVASSMDPs it is viceversa. Still these two models induce infinitestate MDPs. Unlike for finitestate MDPs, it is possible that the value of the MDP, in the game theoretic sense, is 1, even though there is no single strategy that achieves value 1. For example, there can exist a family of strategies \(\sigma _\epsilon \) for every \(\epsilon >0\), where playing \(\sigma _\epsilon \) ensures a probability \(\ge 1\epsilon \) of reaching a given target state, but no strategy ensures probability 1. In this case, one says that the reachability property holds limitsurely, but not almostsurely (i.e., unlike in finitestate MDPs, almostsurely and limitsurely do not coincide in infinitestate MDPs).
We show that even for PVASSMDPs, all sure/almostsure/limitsure reachability/Büchi problems are still undecidable. However, in the deadlockfree subclass of PVASSMDPs, the sure reachability/Büchi problems become decidable (while the other problems remain undecidable). In contrast, for 1VASSMDPs, the sure/almostsure/limitsure reachability problem and the sure/almostsure Büchi problem are decidable.
Our decidability results rely on two different techniques. For the sure and almost sure problems, we prove that we can reduce them to the modelchecking problem over VASS of a restricted fragment of the modal \(\mu \)calculus that has been proved to be decidable in [4]. For the limitsure reachability problem in 1VASSMDP, we use an algorithm which at each iteration reduces the dimension of the considered VASS while preserving the limitsure reachability properties.
Although we do not consider the class of qualitative objectives referring to the probability of (repeated) reachability being strictly greater than 0, we observe that reachability on VASSMDPs in such a setting is equivalent to reachability on standard VASS (though this correspondence does not hold for repeated reachability).
Outline. In Sect. 2 we define basic notations and how VASS induce MDPs. In Sects. 3 and 4 we consider verification problems for PVASSMDP and 1VASSMDP, respectively. In Sect. 5 we summarize the decidability results (Table 1) and outline future work. Omitted proofs can be found in [2].
2 Models and Verification Problems
Let \(\mathbb {N}\) (resp. \(\mathbb {Z}\)) denote the set of nonnegative integers (resp. integers). For two integers i, j such that \(i \le j\) we use \([i..j]\) to represent the set \(\{ k \in \mathbb {Z}\mid i \le k \le j \}\). Given a set X and \(n \in \mathbb {N}\setminus \{ 0 \}\), \(X^n\) is the set of ndimensional vectors with values in X. We use \(\mathbf{0 }\) to denote the vector such that \(\mathbf{0 }(i)=0\) for all \(i \in [1..n]\). The classical order on \(\mathbb {Z}^n\) is denoted \(\le \) and is defined by \(\mathbf{v } \le \mathbf{w }\) if and only if \(\mathbf{v }(i) \le \mathbf{w }(i)\) for all \(i \in [1..n]\). We also define the operation \(+\) over ndimensional vectors of integers in the classical way (i.e., for \(\mathbf{v }\), \(\mathbf{v }' \in \mathbb {Z}^n\), \(\mathbf{v } + \mathbf{v }'\) is defined by \((\mathbf{v }+\mathbf{v }')(i)=\mathbf{v }(i)+\mathbf{v }'(i)\) for all \(i \in [1..n]\)). Given a set S, we use \(S^*\) (respectively \(S^\omega \)) to denote the set of finite (respectively infinite) sequences of elements of S. We now recall the notion of wellquasiordering (which we abbreviate as wqo). A quasiorder \((A,\preceq )\) is a wqo if for every infinite sequence of elements \(a_1,a_2,\ldots \) in A, there exist two indices \(i<j\) such that \(a_i \preceq a_j\). For \(n>0\), \((\mathbb {N}^n,\le )\) is a wqo. Given a set A with an ordering \(\preceq \) and a subset \(B \subseteq A\), the set B is said to be upward closed in A if \(a_1 \in B\), \(a_2 \in A\) and \(a_1 \preceq a_2\) implies \(a_2 \in B\).
2.1 Markov Decision Processes
A probability distribution on a countable set X is a function \(f: X \mapsto [0,1]\) such that \(\sum _{x \in X}f(x)=1\). We use \(\mathcal{D}(X)\) to denote the set of all probability distributions on X. We first recall the definition of Markov decision processes.
Definition 1
(MDPs). A Markov decision process (MDP) M is a tuple \(\langle C,C_1, C_P,A,\rightarrow ,p \rangle \) where: C is a countable set of configurations partitioned into \(C_1\) and \(C_P\) (that is \(C=C_1 \cup C_P\) and \(C_1 \cap C_P=\emptyset \)); A is a set of actions; \(\rightarrow \subseteq C \times A \times C\) is a transition relation; \(p: C_P \mapsto \mathcal{D}(C)\) is a partial function which assigns to some configurations in \(C_P\) probability distributions on C such that \(p(c)(c')>0\) if and only if \(c \xrightarrow {a} c'\) for some \(a \in A\).
Note that our definition is equivalent as seeing MDPs as games played between a nondeterministic player (Player 1) and a probabilistic player (Player P). The set \(C_1\) contains the nondeterministic configurations (or configurations of Player 1) and the set \(C_P\) contains the probabilistic configurations (or configurations of Player P). Given two configurations \(c,c'\) in C, we write \(c\rightarrow c'\) whenever there exists \(a \in A\) such that \(c \xrightarrow {a} c'\). We will say that a configuration \(c \in C\) is a deadlock if there does not exist \(c' \in C\) such that \(c \rightarrow c'\). We use \(C^{df}_1\) (resp. \(C^{df}_P\)), to denote the configurations of Player 1 (resp. of Player P) which are not a deadlock (df stands here for deadlock free).
A play of the MDP \(M=\langle C,C_1,C_P,A,\rightarrow ,p \rangle \) is either an infinite sequence of the form \(c_0 \xrightarrow {a_0} c_1 \xrightarrow {a_1} c_2 \cdots \) or a finite sequence \(c_0 \xrightarrow {a_0} c_1 \xrightarrow {a_1} c_2 \cdots \xrightarrow {a_{k1}} c_k\). We call the first kind of play an infinite play, and the second one a finite play. A play is said to be maximal whenever it is infinite or it ends in a deadlock configuration. These latter plays are called deadlocked plays. We use \(\varOmega \) to denote the set of maximal plays. For a finite play \(\rho =c_0 \xrightarrow {a_0} c_1 \xrightarrow {a_1} c_2 \cdots \xrightarrow {a_{k1}} c_k\), let \(c_k= last (\rho )\). We use \(\varOmega ^{df}_1\) to denote the set of finite plays \(\rho \) such that \( last (\rho ) \in C^{df}_1\).
A strategy for Player 1 is a function \(\sigma : \varOmega ^{df}_1 \mapsto C\) such that, for all \(\rho \in \varOmega ^{df}_1\) and \(c \in C\), if \(\sigma (\rho )=c\) then \( last (\rho ) \rightarrow c\). Intuitively, given a finite play \(\rho \), which represents the history of the game so far, the strategy represents the choice of Player 1 among the different possible successor configurations from \( last (\rho )\). We use \(\varSigma \) to denote the set of all strategies for Player 1. Given a strategy \(\sigma \in \varSigma \), an infinite play \(c_0 \xrightarrow {a_0} c_1 \xrightarrow {a_1} c_2 \cdots \)respects\(\sigma \) if for every \(k \in \mathbb {N}\), we have that if \(c_k \in C_1\) then \(c_{k+1}=\sigma (c_0 \xrightarrow {a_0} c_1 \xrightarrow {a_1} c_2 \cdots c_{k})\) and if \(c_k \in C_P\) then \(p(c_k)(c_{k+1}) >0\). We define finite plays that respect \(\sigma \) similarly. Let \(\mathtt {Plays}(M,c,\sigma ) \subseteq \varOmega \) be the set of all maximal plays of M that start from c and that respect \(\sigma \).
Note that once a starting configuration \(c_0 \in C\) and a strategy \(\sigma \) have been chosen, the MDP is reduced to an ordinary stochastic process. We define an event \(\mathcal{A}\subseteq \varOmega \) as a measurable set of plays and we use \(\mathbb {P}(M,c,\sigma ,\mathcal{A})\) to denote the probability of event \(\mathcal{A}\) starting from \(c \in C\) under strategy \(\sigma \). The notation \(\mathbb {P}^+(M,c,\mathcal{A})\) will be used to represent the maximal probability of event \(\mathcal{A}\) starting from c which is defined as \(\mathbb {P}^+(M,c,\mathcal{A})=\text{ sup }_{\sigma \in \varSigma } \mathbb {P}(M,c,\sigma ,\mathcal{A})\).
2.2 VASSMDPs
Probabilistic Vector Addition Systems with States have been studied, e.g., in [3]. Here we extend this model with nondeterministic choices made by a controller. We call this new model VASSMDPs. We first recall the definition of Vector Addition Systems with States.
Definition 2
(VASS). For \(n>0\), an ndimensional Vector Addition System with States (VASS) is a tuple \(S=\langle Q,T \rangle \) where Q is a finite set of control states and \(T \subseteq Q \times \mathbb {Z}^n \times Q\) is the transition relation labelled with vectors of integers.
In the sequel, we will not always make precise the dimension of the considered VASS. Configurations of a VASS are pairs \(\langle q,\mathbf{v } \rangle \in Q \times \mathbb {N}^n\). Given a configuration \(\langle q,\mathbf{v } \rangle \) and a transition \(t=\langle q,\mathbf{z },q' \rangle \) in T, we will say that t is enabled at \(\langle q'',\mathbf{v } \rangle \), if \(q=q''\) and \(\mathbf{v } + \mathbf{z } \ge \mathbf{0 }\). Let then \(\mathtt {En}(q,\mathbf{v })\) be the set \(\{ t \in T \mid t \text{ is } \text{ enabled } \text{ at } \langle q,\mathbf{v }) \rangle \}\). In case the transition \(t=\langle q,\mathbf{z },q' \rangle \) is enabled at \(\langle q,\mathbf{v } \rangle \), we define \(t(q,\mathbf{v })=\langle q',\mathbf{v }' \rangle \) where \(\mathbf{v }'=\mathbf{v } + \mathbf{z }\). An ndimensional VASS S induces a labelled transition system \(\langle C,T,\rightarrow \rangle \) where \(C=Q \times \mathbb {N}^n\) is the set of configurations and the transition relation \(\rightarrow \subseteq C \times T \times C\) is defined as follows: \(\langle q,\mathbf{v } \rangle \xrightarrow {t} \langle q',\mathbf{v }' \rangle \text{ iff } \langle q',\mathbf{v }' \rangle =t(q,\mathbf{v })\). VASS are sometimes seen as programs manipulating integer variables, a.k.a. counters. When a transition of a VASS changes the ith value of a vector \(\mathbf{v }\), we will sometimes say that it modifies the value of the ith counter. We show now in which manner we add probability distributions to VASS.
Definition 3
(VASSMDP). A VASSMDP is a tuple \(S=\langle Q,Q_1,Q_P,T,\tau \rangle \) where \(\langle Q,T \rangle \) is a VASS for which the set of control states Q is partitioned into \(Q_1\) and \(Q_P\), and \(\tau : T \mapsto \mathbb {N}\setminus \{ 0 \}\) is a partial function assigning to each transition a weight which is a positive natural number.
Nondeterministic (resp. probabilistic) choices are made from control states in \(Q_1\) (resp. \(Q_P\)). The subset of transitions from control states of \(Q_1\) (resp. control states of \(Q_P\)) is denoted by \(T_1\) (resp. \(T_P\)). Hence \(T =T_1 \cup T_P\) with \(T_1 \subseteq Q_1 \times \mathbb {Z}^n \times Q\) and \(T_P \subseteq Q_P \times \mathbb {Z}^n \times Q\). A VASSMDP \(S=\langle Q,Q_1,Q_P,T,\tau \rangle \) induces an MDP \(M_S=\langle C,C_1,C_P,T,\rightarrow ,p \rangle \) where: \(\langle C,T,\rightarrow \rangle \) is the labelled transition system associated with the VASS \(\langle Q,T \rangle \); \(C_1=Q_1 \times \mathbb {N}^n\) and \(C_P=Q_P \times \mathbb {N}^n\); and for all \(c \in C^{df}_P\) and \(c' \in C\), if \(c \rightarrow c'\), the probability of going from c to \(c'\) is defined by \(p(c)(c')=(\sum _{\{ t \mid t(c)=c' \}} \tau (t)) / (\sum _{t \in \mathtt {En}(c)} \tau (t))\), whereas if \(c \not \rightarrow c'\), we have \(p(c)(c')=0\). Note that the MDP \(M_S\) is welldefined: when defining \(p(c)(c')\) in the case \(c \rightarrow c'\), there exists at least one transition in \(\mathtt {En}(c)\) and consequently the sum \(\sum _{t \in \mathtt {En}(c)} \tau (t)\) is never equal to 0. Also, we could have restricted the weights to be assigned only to transitions leaving from a control state in \(Q_P\) since we do not take into account the weights assigned to the other transitions. A VASSMDP is deadlock free if its underlying VASS is deadlock free.
Finally, as in [18] or [4], we will see that to gain decidability it is useful to restrict the power of the nondeterministic player or of the probabilistic player by restricting their ability to modify the counters’ values and hence letting them only choose a control location. This leads to the two following definitions: a PVASSMDP is a VASSMDP \(\langle Q,Q_1,Q_P,T,\tau \rangle \) such that for all \(\langle q,\mathbf{z },q' \rangle \in T_1\), we have \(\mathbf{z }=\mathbf{0 }\) and a 1VASSMDP is a VASSMDP \(\langle Q,Q_1,Q_P,T,\tau \rangle \) such that for all \(\langle q,\mathbf{z },q' \rangle \in T_P\), we have \(\mathbf{z }=\mathbf{0 }\). In other words, in a PVASSMDP, Player 1 cannot change the counter values when taking a transition and, in a 1VASSMDP, it is Player P which cannot perform such an action.
2.3 Verification Problems for VASSMDPs
We consider qualitative verification problems for VASSMDPs, taking as objectives controlstate reachability and repeated reachability. To simplify the presentation, we consider a single target controlstate \(q_F \in Q\). However, our positive decidability results easily carry over to sets of target controlstates (while the negative ones trivially do). Note however, that asking to reach a fixed target configuration like \(\langle q_F,\mathbf{0 } \rangle \) is a very different problem (cf. [3]).
 1.
The sure reachability problem: Does there exist a strategy \(\sigma \in \varSigma \) such that
\(\mathtt {Plays}(M_S,c_0,\sigma ) \subseteq \llbracket \diamondsuit q_F \rrbracket \)?
 2.
The almostsure reachability problem: Does there exist a strategy \(\sigma \in \varSigma \) such that \(\mathbb {P}(M_S,c_0,\sigma ,\llbracket \diamondsuit q_F \rrbracket )=1\)?
 3.
The limitsure reachability problem: Does \(\mathbb {P}^+(M_S,c_0,\llbracket \diamondsuit q_F \rrbracket )=1\)?
 4.
The sure repeated reachability problem: Does there exist a strategy \(\sigma \in \varSigma \) such that \(\mathtt {Plays}(M_S,c_0,\sigma ) \subseteq \llbracket \Box \diamondsuit q_F \rrbracket \)?
 5.
The almostsure repeated reachability problem: Does there exist a strategy \(\sigma \in \varSigma \) such that \(\mathbb {P}(M_S,c_0,\sigma ,\llbracket \Box \diamondsuit q_F \rrbracket )=1\)?
 6.
The limitsure repeated reachability problem: Does \(\mathbb {P}^+(M_S,c_0,\llbracket \Box \diamondsuit q_F \rrbracket )=1\)?
2.4 Undecidability in the General Case
It was shown in [1] that the sure reachability problem is undecidable for (2dimensional) two player VASS. From this we can deduce that the sure reachability problem is undecidable for VASSMDPs. We now present a similar proof to show the undecidability of the almostsure reachability problem for VASSMDPs.
We now show how to reduce the controlstate reachability problem to the almostsure and limitsure reachability problems in deadlockfree VASSMDPs. From a Minsky machine, we construct a deadlockfree 2dimensional VASSMDP for which the control states of Player 1 are exactly the control states of the Minsky machine. The encoding is presented in Fig. 2 where the circles (resp. squares) are the control states of Player 1 (resp. Player P), and for each edge the corresponding weight is 1. The state \(\bot \) is an absorbing state from which the unique outgoing transition is a self loop that does not affect the values of the counters. This encoding allows us to deduce our first result.
Theorem 1
The sure, almostsure and limitsure (repeated) reachability problems are undecidable problems for 2dimensional deadlockfree VASSMDPs.
In the special case of 1dimensional VASSMDPs, the sure and almostsure reachability problems are decidable [7].
2.5 ModelChecking \(\mu \)calculus on SingleSided VASS
It is wellknown that there is a strong connection between modelchecking branching time logics and games, and in our case we have in fact undecidability results for simple reachability games played on a VASS and for the modelchecking of VASS with expressive branchingtime logics [12]. However for this latter point, decidability can be regained by imposing some restrictions on the VASS structure [4] as we will now recall. We say that a VASS \(\langle Q,T \rangle \) is \((Q_1,Q_2)\)singlesided iff \(Q_1\) and \(Q_2\) represents a partition of the set of states Q such that for all transitions \(\langle q,\mathbf{z },q' \rangle \) in T with \(q \in Q_2\), we have \(\mathbf{z }=\mathbf{0 }\); in other words only the transitions leaving a state from \(Q_1\) are allowed to change the values of the counters. In [4], it has been shown that, thanks to a reduction to games played on a singlesided VASS with parity objectives, a large fragment of the \(\mu \)calculus called \(L^{\textit{sv}}_\mu \) has a decidable modelchecking problem over singlesided VASS. The idea of this fragment is that the “always" operator \(\Box \) is guarded with a predicate enforcing the current control states to belong to \(Q_2\). Formally, the syntax of \(L^{\textit{sv}}_\mu \) for \((Q_1,Q_2)\)singlesided VASS is given by the following grammar: \(\phi \,{:}{:}{=}\, q ~\mid ~ X ~\mid ~ \phi \wedge \phi ~\mid ~ \phi \vee \phi ~\mid ~ \diamondsuit \phi ~\mid ~ Q_2 \wedge \Box \phi ~\mid ~ \mu X.\phi ~\mid ~ \nu X.\phi \), where \(Q_2\) stands for the formula \(\bigvee _{q \in Q_2} q\) and X belongs to a set of variables \(\mathcal {X}\). The semantics of \(L^{\textit{sv}}_\mu \) is defined as usual: it associates to a formula \(\phi \) and to an environment \(\varepsilon : \mathcal {X}\rightarrow 2^C\) a subset of configurations \(\llbracket \phi \rrbracket _\varepsilon \). We use \(\varepsilon _0\) to denote the environment which assigns the empty set to any variable. Given an environment \(\varepsilon \), a variable \(X \in \mathcal {X}\) and a subset of configurations C, we use \(\varepsilon [X:=C]\) to represent the environment \(\varepsilon '\) which is equal to \(\varepsilon \) except on the variable X, where we have \(\varepsilon '(X)=C\). Finally the notation \(\llbracket \phi \rrbracket \) corresponds to the interpretation \(\llbracket \phi \rrbracket _{\varepsilon _0}\).
The problem of modelchecking singlesided VASS with \(L^{\textit{sv}}_\mu \) can then be defined as follows: given a singlesided VASS \(\langle Q,T \rangle \), an initial configuration \(c_0\) and a formula \(\phi \) of \(L^{\textit{sv}}_\mu \), do we have \(c_0 \in \llbracket \phi \rrbracket \)?
Theorem 2
[4]. Modelchecking singlesided VASS wrt. \(L^{\textit{sv}}_\mu \) is decidable.
3 Verification of PVASSMDPs
In [4] it is proved that parity games played on a singlesided deadlockfree VASS are decidable (this entails the decidability of model checking \(L^{\textit{sv}}_\mu \) over singlesided VASS). We will see here that in the case of PVASSMDPs, in which only the probabilistic player can modify the counters, the decidability status depends on the presence of deadlocks in the system.
3.1 Undecidability in Presence of Deadlocks
We point out that the reduction presented in Fig. 2 to prove Theorem 1 does not carry over to PVASSMDPs, because in that construction both players have the ability to change the counter values. However, it is possible to perform a similar reduction leading to the undecidability of verification problems for PVASSMDPs, the main difference being that we crucially exploit the fact that the PVASSMDP can contain deadlocks.
From this encoding we deduce the following result.
Theorem 3
The sure, almost sure and limit sure (repeated) reachability problems are undecidable for 2dimensional PVASSMDPs.
3.2 Sure (repeated) Reachability in DeadlockFree PVASSMDPs
Unlike in the case of general PVASSMDPs, we will see that the sure (repeated) reachability problem is decidable for deadlockfree PVASSMDPs. Let \(S=\langle Q,Q_1,Q_P,T,\tau \rangle \) be a deadlockfree PVASSMDP, \(M_S=(C,C_1,C_P,\rightarrow ,p)\) its associated MDP and \(q_F \in Q\) a control state. Note that because the PVASSMDP S is deadlock free, Player P cannot take the play to a deadlock to avoid the control state \(q_F\), but he has to deal only with infinite plays. Since S is a PVASSMDP, the VASS \(\langle Q,T \rangle \) is \((Q_P,Q_1)\)singlesided. In [1, 18], it has been shown that controlstate reachability games on deadlockfree singlesided VASS are decidable, and this result has been extended to parity games in [4]. This implies the decidability of sure (repeated) reachability in deadlockfree PVASSMDPs. However, to obtain a generic way of verifying these systems, we construct a formula of \(L^{\textit{sv}}_\mu \) that characterizes the sets of winning configurations and use then the result of Theorem 2. Let \(V^P_S\) be the set of configurations from which the answer to the sure reachability problem (with \(q_F\) as state to be reached) is negative, i.e., \(V^P_S=\{ c \in C \mid \not \exists \sigma \in \varSigma \text{ s.t. } \mathtt {Plays}(M_S,c,\sigma ) \subseteq \llbracket \diamondsuit q_F \rrbracket \}\) and similarly let \(W^P_S=\{ c \in C \mid \not \exists \sigma \in \varSigma \text{ s.t. } \mathtt {Plays}(M_S,c,\sigma ) \subseteq \llbracket \Box \diamondsuit q_F \rrbracket \}\). The next lemma relates these two sets with a formula of \(L^{\textit{sv}}_\mu \) (where \(Q_P\) corresponds to the formula \(\bigvee _{q \in Q_P}\) and \(Q_1\) corresponds to the formula \(\bigvee _{q \in Q_1} q\)).
Lemma 1

\(V^P_S=\llbracket \nu X. (\bigvee _{q \in Q \setminus \{ q_F \}} q) \wedge (Q_1 \vee \diamondsuit X) \wedge (Q_P \vee (Q_1 \wedge \Box X)) \rrbracket \).

\(W^P_S=\llbracket \mu Y. \nu X. \big ( (\bigvee _{q \in Q \setminus \{ q_F \}} q) \wedge (Q_1 \vee \diamondsuit X) \wedge (Q_P \vee (Q_1 \wedge \Box X)) \vee (q_F \wedge Q_P \wedge \diamondsuit Y) \vee (q_F \wedge Q_1 \wedge \Box Y) \big ) \rrbracket \)
We use \((Q_P \vee (Q_1 \wedge \Box X))\) instead of \((Q_P \vee \Box X)\) so that the formulae are in the guarded fragment of the \(\mu \)calculus. Since the two formulae belong to \(L^{\textit{sv}}_\mu \) for the \((Q_P,Q_1)\)singlesided VASS S, decidability follows directly from Theorem 2.
Theorem 4
The sure reachability and repeated reachability problem are decidable for deadlock free PVASSMDPs.
3.3 AlmostSure and LimitSure Reachability in DeadlockFree PVASSMDPs
This allows us to deduce the following result for deadlock free PVASSMDPs.
Theorem 5
The almostsure and limitsure (repeated) reachability problems are undecidable for 2dimensional deadlockfree PVASSMDPs.
4 Verification of 1VASSMDPs
In this section, we will provide decidability results for the subclass of 1VASSMDPs. As for deadlockfree PVASSMDPs, the proofs for sure and almostsure problems use the decidability of \(L^{\textit{sv}}_\mu \) over singlesided VASS, whereas the technique used to show decidability of limitsure reachability is different.
4.1 Sure Problems in 1VASSMDPs
First we show that, unlike for PVASSMDPs, deadlocks do not matter for 1VASSMDPs. The idea is that in this case, if the deadlock is in a probabilistic configuration, it means that there is no outgoing edge (because of the property of 1VASSMDPs), and hence one can add an edge to a new absorbing state, and the same can be done for the states of Player 1. Such a construction does not work for PVASSMDPs, because in that case deadlocks in probabilistic configurations may depend on the counter values, and not just on the current controlstate.
Lemma 2
The sure (resp. almost sure, resp. limit sure) (repeated) reachability problem for 1VASSMDPs reduces to the sure (resp. almost sure, resp. limitsure) (repeated) reachability problem for deadlockfree 1VASSMDPs.
Hence in the sequel we will consider only deadlockfree 1VASSMDPs. Let \(S=\langle Q,Q_1,Q_P,T,\tau \rangle \) be a deadlockfree 1VASSMDP. For what concerns the sure (repeated) reachability problems we can directly reuse the results from Lemma 1 and then show that the complement formulae of the ones expressed in this lemma belong to \(L^{\textit{sv}}_\mu \) for the \((Q_1,Q_P)\)singlesided VASS \(\langle Q,T \rangle \) (in fact the correctness of these two lemmas did not depend on the fact that we were considering PVASSMDPs). Theorem 2 allows us to retrieve the decidability results already expressed in [18] (for sure reachability) and [4] (for sure repeated reachability).
Theorem 6
The sure (repeated) reachability problem is decidable for 1VASSMDPs.
4.2 AlmostSure Problems in 1VASSMDPs
We now move to the case of almostsure problems in 1VASSMDPs. We consider a deadlock free 1VASSMDP \(S=\langle Q,Q_1,Q_P,T,\tau \rangle \) and its associated MDP \(M_S=\langle C,C_1,C_P,\rightarrow ,p \rangle \). We will see that, unlike for PVASSMDPs, it is here also possible to characterize by formulae of \(L^{\textit{sv}}_\mu \) the two following sets: \(V^1_{AS}=\{ c \in C \mid \exists \sigma \in \varSigma \text{ such } \text{ that } \mathbb {P}(M_S,c,\sigma ,\llbracket \diamondsuit q_F \rrbracket )=1 \}\) and \(W^1_{AS}=\{ c \in C \mid \exists \sigma \in \varSigma \text{ such } \text{ that } \mathbb {P}(M_S,c,\sigma ,\llbracket \Box \diamondsuit q_F \rrbracket )=1 \}\), i.e. the set of configurations from which Player 1 has a strategy to reach the control state \(q_F\), respectively to visit infinitely often \(q_F\), with probability 1.
We begin with introducing the following formula of \(L^{\textit{sv}}_\mu \) based on the variables X and Y: \(\mathtt {InvPre}(X,Y)= (Q_1 \wedge \diamondsuit (X \wedge Y)) \vee (\diamondsuit Y \wedge Q_P \wedge \Box X)\). Note that \(\mathtt {InvPre}(X,Y)\) is a formula of \(L^{\textit{sv}}_\mu \) for the \((Q_1,Q_P)\)singlesided VASS \(\langle Q,T \rangle \). Intuitively, this formula represents the set of configurations from which (i) Player 1 can make a transition to the set represented by the intersection of the sets characterized by the variables X and Y and (ii) Player P can make a transition to the set Y and cannot avoid making a transition to the set X.
Almost Sure Reachability. We will now prove that \(V^1_{AS}\) can be characterized by the following formula of \(L^{\textit{sv}}_\mu \): \(\nu X. \mu Y. ( q_F \vee \mathtt {InvPre}(X,Y))\). Note that a similar result exists for finitestate MDPs, see e.g. [9]; this result in general does not extend to infinitestate MDPs, but in the case of VASSMDPs it can be applied. Before proving this we need some intermediate results.
We denote by E the set \(\llbracket \nu X. \mu Y. \big ( q_F \vee \mathtt {InvPre}(X,Y)\big ) \rrbracket _{\varepsilon _0}\). Since \(\nu X. \mu Y. \big ( q_F \vee \mathtt {InvPre}(X,Y)\big )\) is a formula of \(L^{\textit{sv}}_\mu \) interpreted over the singlesided VASS \(\langle Q,T \rangle \), we can show that E is an upwardclosed set. We now need another lemma which states that there exists \(N \in \mathbb {N}\) and a strategy for Player 1 such that, from any configuration of E, Player 1 can reach the control state \(q_F\) in less than N steps and Player P cannot take the play outside of E. The fact that we can bound the number of steps is crucial to show that \(\llbracket \nu X. \mu Y. \big ( q_F \vee \mathtt {InvPre}(X,Y)\big ) \rrbracket _{\varepsilon _0}\) is equal to \(V^1_{AS}\). For infinitestate MDPs where this property does not hold, our techniques do not apply.
Lemma 3
There exists \(N \in \mathbb {N}\) and a strategy \(\sigma \) of Player 1 such that for all \(c \in E\), there exists a play \(c \cdot c_1 \cdot c_2 \cdot \ldots \) in \(\mathtt {Plays}(M_S,c,\sigma )\) satisfying the three following properties: (1) there exists \(0 \le i \le N\) such that \(c_i \in \llbracket q_F \rrbracket \); (2) for all \(0 \le j \le i\), \(c_j \in E\); (3) for all \(0 \le j \le i\), if \(c_j \in C_P\) then for all \(c'' \in C\) such that \(c_j \rightarrow c''\), we have \(c'' \in E\).
This previous lemma allows us to characterize \(V^1_{AS}\) with a formula of \(L^{\textit{sv}}_\mu \). The proof of the following result uses the fact that the number of steps is bounded, and also the fact that the sets described by closed \(L^{\textit{sv}}_\mu \) formulae are upwardclosed. This makes the fixpoint iteration terminate in a finite number of steps.
Lemma 4
\(V^1_{AS}=\llbracket \nu X. \mu Y. ( q_F \vee \mathtt {InvPre}(X,Y)) \rrbracket \).
Since \(\langle Q,T \rangle \) is \((Q_1,Q_P)\)singlesided and since the formula associated to \(V^1_{AS}\) belongs to \(L^{\textit{sv}}_\mu \), from Theorem 2 we deduce the following theorem.
Theorem 7
The almostsure reachability problem is decidable for 1VASSMDPs.
Almost Sure Repeated Reachability. For the case of almost sure repeated reachability we reuse the previously introduced formula \(\mathtt {InvPre}(X,Y)\). We can perform a reasoning similar to the previous ones and provide a characterization of the set \(W^1_{AS}\).
Lemma 5
\(W^1_{AS}=\llbracket \nu X. \mathtt {InvPre}(X,\mu Y.( q_F \vee \mathtt {InvPre}(X,Y))) \rrbracket \).
As previously, this allows us to deduce the decidability of the almost sure repeated reachability problem for 1VASSMDP.
Theorem 8
The almost sure repeated reachability problem is decidable for 1VASSMDPs.
4.3 LimitSure Reachability in 1VASSMDP
We consider a slightly more general version of the limitsure reachability problem with a set \(X \subseteq Q\) of target states instead of a single state \(q_F\), i.e., the standard case corresponds to \(X = \{q_F\}\).
We extend the set of natural numbers \(\mathbb {N}\) to \(\mathbb {N}_{*}= \mathbb {N} \bigcup \{*\}\) by adding an element \(*\notin \mathbb {N}\) with \(*+j=*j=*\) and \(j < *\) for all \(j \in \mathbb {N}\). We consider then the set of vectors \(\mathbb {N}_{*}^{d}\). The projection of a vector \(\mathbf{v }\) in \(\mathbb {N}^{d}\) by eliminating components that are indexed by a natural number k is defined by \(proj_k (\mathbf{v })(i) = \mathbf{v }(i) \) if \(i\ne k\) and \(proj_k (\mathbf{v })(i)=*\) otherwise
Let \(Q_c\) represent controlstates which are indexed by a color. The coloring functions \(col_i:Q \rightarrow Q_c\) create colored copies of controlstates by \(col_i(q)= q_i\).
We define the functions \(state: Q \times \mathbb {N}_{*}^{d} \rightarrow Q\) and \({ count}: Q \times \mathbb {N}_{*}^{d} \rightarrow \mathbb {N}_{*}^{d}\) s.t for a configuration \(c_i = \langle q, \mathbf{v } \rangle \), where \(q \in Q\) and \(\mathbf v \in \mathbb {N}_{*}^{d}\) we have that \(state(q, \mathbf v )=q\) and \(count(q, \mathbf v )= \mathbf v \). For any two configurations \(c_1\) and \(c_2\), we write \(c_1 \prec c_2\) to denote that \(state(c_1)=state(c_2)\), and there exists a nonempty set of indexes I where for every \(i \in I\) , \(count(c_1)(i) < count(c_2)(i)\), whereas for every index \(j \notin I\), \(0<j \le d\), \(count(c_1)(j) = count(c_2)(j)\).
By Dickson’s Lemma, the conditions on line 7 or line 19 of the algorithm must eventually hold on every branch of the explored computation tree. Thus, it will terminate.
Lemma 6
Algorithm 1 terminates.
The next lemma states the correctness of Algorithm 1. Let \(S=\langle Q,Q_1,Q_P,T,\tau \rangle \) be 1VASSMDP of dimension \(d > 0\) with initial configuration \(c_0 = \langle q_0,\mathbf{v } \rangle \) and \(X\subseteq Q\) a set of target states. Let \(S'=\langle Q',Q_1',Q_P',T',\tau ' \rangle \) with initial configuration \(c_0' = \langle q_0',\mathbf {0} \rangle \) and set of target states \(X' \subseteq Q'\) be the \((d1)\) dimensional 1VASSMDP produced by Algorithm 1. As described above we have the following relation between these two systems.
Lemma 7
\(\mathbb {P}^+(M_S, c_0, \llbracket \diamondsuit X \rrbracket ) = 1\) iff \(\mathbb {P}^+(M_{S'}, c_0', \llbracket \diamondsuit X' \rrbracket ) = 1\).
Decidability of verification problems for PVASSMDP, deadlockfree PVASSMDP and 1VASSMDP. A \(\checkmark \) stands for decidable and a \(\times \) for undecidable.
PVASSMDP  df PVASSMDP  1VASSMDP  

Sure reachability  \(\times \) (Theorem 3)  \(\checkmark \) (Theorem 4)  \(\checkmark \) (Theorem 6) 
Almostsure reachability  \(\times \) (Theorem 3)  \(\times \) (Theorem 5)  \(\checkmark \) (Theorem 7) 
Limitsure reachability  \(\times \) (Theorem 3)  \(\times \) (Theorem 5)  \(\checkmark \) (Theorem 9) 
Sure repeated reachability  \(\times \) (Theorem 3)  \(\checkmark \) (Theorem 4)  \(\checkmark \) (Theorem 6) 
Almostsure repeated reachability  \(\times \) (Theorem 3)  \(\times \) (Theorem 5)  \(\checkmark \) (Theorem 8) 
Limitsure repeated reachability  \(\times \) (Theorem 3)  \(\times \) (Theorem 5)  Open 
Theorem 9
The limitsure reachability problem for 1VASSMDP is decidable.
5 Conclusion and Future Work
Table 1 summarizes our results on the decidability of verification problems for subclasses of VASSMDP. The exact complexity of most problems is still open. Algorithm 1 relies on Dickson’s Lemma for termination, and the algorithm deciding the modelchecking problem of Theorem 2 additionally uses the ValkJantzen construction repeatedly. However, all these problems are at least as hard as controlstate reachability in VASS, and thus EXPSPACEhard [12].
The decidability of the limitsure repeated reachability problem for 1VASSMDP is open. A hint of its difficulty is given by the fact that there are instances where the property holds even though a small chance of reaching a deadlock cannot be avoided from any reachable configuration. In particular, a solution would require an analysis of the longrun behavior of multidimensional random walks induced by probabilistic VASS. However, these may exhibit strange nonregular behaviors for dimensions \(\ge 3\), as described in [8] (Sect. 5).
References
 1.Abdulla, P.A., Bouajjani, A., d’Orso, J.: Monotonic and downward closed games. J. Logic Comput. 18(1), 153–169 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
 2.Abdulla, P.A., Ciobanu, R., Mayr, R., Sangnier, A., Sproston, J.: Qualitative analysis of VASSinduced MDPs. CoRR (2015). abs/1512.08824Google Scholar
 3.Abdulla, P.A., Henda, N.B., Mayr, R.: Decisive Markov chains. Logical Meth. Comput. Sci. 3(4) (2007)Google Scholar
 4.Abdulla, P.A., Mayr, R., Sangnier, A., Sproston, J.: Solving parity games on integer vectors. In: D’Argenio, P.R., Melgratti, H. (eds.) CONCUR 2013 – Concurrency Theory. LNCS, vol. 8052, pp. 106–120. Springer, Heidelberg (2013)CrossRefGoogle Scholar
 5.Baier, C., Katoen, J.P.: Principles of Model Checking. MIT Press, Cambridge (2008)zbMATHGoogle Scholar
 6.Brázdil, T., Brozek, V., Etessami, K.: Onecounter stochastic games. In: FSTTCS 2010, LIPIcs, vol. 8, pp. 108–119. Schloss Dagstuhl  LeibnizZentrum für Informatik (2010)Google Scholar
 7.Brázdil, T., Brozek, V., Etessami, K., Kučera, A., Wojtczak, D.: Onecounter Markov decision processes. In: SODA 2010, pp. 863–874. SIAM (2010)Google Scholar
 8.Brázdil, T., Kiefer, S., Kučera, A., Novotný, P.: Longrun average behaviour of probabilistic vector addition systems. In: LICS 2015, pp. 44–55. IEEE (2015)Google Scholar
 9.Chatterjee, K., de Alfaro, L., Faella, M., Legay, A.: Qualitative logics and equivalences for probabilistic systems. Logical Meth. Comput. Sci. 5(2) (2009)Google Scholar
 10.Chatterjee, K., Jurdziński, M., Henzinger, T.A.: Simple stochastic parity games. In: Baaz, M., Makowsky, J.A. (eds.) CSL 2003. LNCS, vol. 2803, pp. 100–113. Springer, Heidelberg (2003)CrossRefGoogle Scholar
 11.Condon, A.: The complexity of stochastic games. Inf. Comput. 96(2), 203–224 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
 12.Esparza, J., Nielsen, M.: Decidability issues for Petri nets  a survey. Bull. EATCS 52, 244–262 (1994)zbMATHGoogle Scholar
 13.Etessami, K., Yannakakis, M.: Recursive Markov decision processes and recursive stochastic games. In: Caires, L., Italiano, G.F., Monteiro, L., Palamidessi, C., Yung, M. (eds.) ICALP 2005. LNCS, vol. 3580, pp. 891–903. Springer, Heidelberg (2005)CrossRefGoogle Scholar
 14.Filar, J., Vrieze, K.: Competitive Markov Decision Processes. Springer, New York (1997)zbMATHGoogle Scholar
 15.Karp, R., Miller, R.: Parallel program schemata. J. Comput. Syst. Sci. 3(2), 147–195 (1969)MathSciNetCrossRefzbMATHGoogle Scholar
 16.Minsky, M.L.: Computation: Finite and Infinite Machines. PrenticeHall, Upper Saddle River (1967)zbMATHGoogle Scholar
 17.Puterman, M.L.: Markov Decision Processes. Wiley, New York (1994)CrossRefzbMATHGoogle Scholar
 18.Raskin, J.F., Samuelides, M., Begin, L.V.: Games for counting abstractions. In: AVoCS 2004, Electronic Notes in Theoretical Computer Science, vol. 128(6), pp. 69–85 (2005)Google Scholar
 19.Shapley, L.S.: Stochastic games. Proc. Nat. Acad. Sci. 39(10), 1095–1100 (1953)MathSciNetCrossRefzbMATHGoogle Scholar