A Modest Approach to Dynamic Heuristic Search in Probabilistic Model Checking

Abstract

This paper presents Modysh, a probabilistic model checker which harvests and extends non-exhaustive exploration methods originally developed in the AI planning context. Its core functionality is based on enhancements of the heuristic search methods labeled real-time dynamic programming and find-revise-eliminate-traps and is capable of handling efficiently maximal and minimal reachability properties, expected reward properties as well as bounded properties on general MDPs. Modysh is integrated in the infrastructure of the Modest Toolset and extends the property types supported by it. We discuss the algorithmic particularities in detail and evaluate the competitiveness of Modysh in comparison to state-of-the-art model checkers in a large case study rooted in the well-established Quantitative Verification Benchmark Set. This study demonstrates that Modysh is especially attractive to use on very large benchmark instances which are not solvable by any other tool.

This work has received support by the ERC Advanced Investigators Grant 695614 POWVER, by the DFG Grant 389792660 as part of TRR 248 CPEC, and by the Key-Area Research and Development Grant 2018B010107004 of Guangdong Province.

Appendices

A Proof for $\textit{MinProb}$

As announced in Sect. 3.1, this appendix provides a proof that G$\text {LRTDP}$ solves $\textit{MinProb}$ properties on general MDP structures correctly by converging to the optimal fixpoint.

To show convergence to the optimal value function from below in case of an admissible initialization, we can argue along the invariant

$$\begin{aligned} \forall k, \sigma : V_k(s) \le P^{\sigma }_s(\diamond G), \text { where } \sigma \text { s.t. } P^{\sigma }_s(\diamond G) = V^*(s) \end{aligned}$$

stating that the value function in every iteration is always at most the value under the optimal policy. This means that an initially admissible value function always stays admissible. This is true for the admissible initialization when $k=0$, because then $V_0(s) = 1$ if $s \in G$ and 0 otherwise. For all other iterations it holds that for some action a and we can derive that

$$\begin{aligned}\begin{gathered} \sum _{s'} P(s, a, s') \cdot V_k(s') \le \sum _{s'} P(s, a, s') \cdot \min _{\sigma }P^{\sigma }_{s'}(\diamond G) \\ \le \sum _{s'} \min _{\sigma }(P(s, a, s') \cdot P^{\sigma }_{s'}(\diamond G)) \le \min _{\sigma } \sum _{s'} P(s, a, s') \cdot P^{\sigma }_{s'}(\diamond G)\text {.} \end{gathered}\end{aligned}$$

The second inequality holds because $\sigma _{opt}$ is memoryless and independent of $s'$.

Now assume $\sigma _{opt}$ is such that $P^{\sigma _{opt}}_s(\diamond G)$ is minimal for all s. Then for action $a = greedy (s, V_k)$ we have for any action b, and in particular for $b=\sigma _{opt}(s)$,

$$\begin{aligned} \sum _{s'}P(s, a, s') \cdot V_k(s') \le \sum _{s'}P(s, b, s')\cdot V_k(s'). \end{aligned}$$

Moreover $V_k(s') \le P^{\sigma _{opt}}_{s'}(\diamond G)$, which allows us to derive

$$\begin{aligned} \sum _{s'}P(s, a, s') \cdot V_k(s') \le \sum _{s'}P(s, \sigma _{opt}(s), s') \cdot P^{\sigma _{opt}}_{s'}(\diamond G) = P^{\sigma _{opt}}_{s}(\diamond G). \end{aligned}$$

Claim: $\text {If } V_k \text { is a fixpoint for } k \rightarrow \infty \text { then } P^\sigma (\diamond G) = V_\infty (s_0) \forall \sigma \text { greedy in } V.~(5)$

Since this means $V^*(s_0) \le V_\infty (s_0)$ and with the result from above ($\forall k: V_k \le V^*$) we can conclude $V^*(s_0) = V_\infty (s_0)$.

It remains to show that (5) holds: Let , i. e., a greedy policy with respect to the value function $V_k$ and $S_k = \{s | P^{\sigma _k}_{s_0}(\diamond s)>0\}$, i. e., all states reachable with this greedy policy, then $\max ( residual (S_k)) \le \delta _k$ and for $k \rightarrow \infty$ it holds that $\delta _k \rightarrow 0$.

To show that $\delta _k$ will approach 0 it is enough to argue about the states which will be updated an infinite number of times, i. e., in the end, about the states on optimal policies. These are the states in $S_\infty = \bigcap _{i \ge 0} \bigcup _{k \ge i} S_k$.

Let K be such that $\forall k \ge K: \bigcup _{i \ge k} S_i = S_\infty$, i. e. a step from which on we only consider states which will be infinitely often visited when running G$\text {LRTDP}$ infinitely long. Assume we are in a step $j+1 \ge K$. Let $s \in S_\infty$. We have to distinguish two cases:

• If s has not been updated then $V_{j+1}(s) = V_j(s)$.

• If s is the updated state then $V_{j+1}(s) = \min _{\alpha } \sum _{s'} P(s, \alpha , s') \cdot V_j(s')$

But this is the same as for simple synchronous value iteration, for which convergence against the optimal fixpoint is proven. For our asynchronous case in G$\text {LRTDP}$ we nevertheless have to guarantee fairness among the states in $S_\infty$, i. e., we have to make sure that they are updated infinitely often. This is the case because each possible trial of $S_\infty$ (there are finitely many trials) appears infinitely often, i. e. the states in this trial are updated infinitely often (by construction of G$\text {LRTDP}$ when choosing the next greedy action). All other states not in $S_\infty$ can be ignored because they will not influence the greedy policy and optimal values because they are already too large:

For any $s \in S\setminus S_{\infty }$ it holds that $V_{\infty }(s) = V_K(s) \le V^*(s)$ and for any $s \in S_{\infty }$ by definition of $S_{\infty }$ and K we know that an action leading again to a state in $S_{\infty }$ will be chosen, i. e., an $a \in \sigma _{\infty }$: $V_{\infty }(s) \le \sum _{s' \in S_{\infty }} P(s, \sigma _{\infty }, s') \cdot V_{\infty }(s')$ but for every action we choose the greedy one and for any $k \ge K$ it holds that $V_k(s) \le \sum _{s' \in S} P(s, a, s') \cdot V_k(s') \le V_{\infty }(s)$, i. e., the action in $\sigma _{\infty }$ must have been the greedy action not leading to $S\setminus S_{\infty }$. This means that $V_{\infty }$ defines an optimal strategy on $S_{\infty }$ for $s_0 \in S_{\infty }$ which is also an optimal strategy on S because no state $s' \in S \setminus S_{\infty }$ is visited even with $V_{\infty }(s')<V^*(s')$. In addition the initial state lies in $S_\infty$ by construction, i. e., $P_{\min }(\diamond G) = V^{\sigma _{opt}}(s_0)$ reaches the fixpoint and is updated infinitely often.

In summary, when running G$\text {LRTDP}$ in an infinite number of iterations, the value function for states in $S_\infty$ will approach the optimal values of the minimal probability to reach the goal from below, will never get larger than the optimal value and the difference between V and $V^*$ always becomes strictly smaller for these states. In addition, we can at some point stop updating the value function for parts of the state space because these values will not have an influence on the correct optimal result for the initial state. In our implementation G$\text {LRTDP}$ is designed in such a way that it stops when the values on the optimal policy only change by less than $\varepsilon$, which is the same convergence criterion as for simple value iteration.

B Proof for $\textit{MaxProb}$

Taking up our promise from Sect. 3.1, in the following we will first give an intuition about why the presented combination of G$\text {LRTDP}$ and $\text {FRET}$ solves $\textit{MaxProb}$ properties on general MDP structures correctly, not only on problems having at least one almost-sure policy as proven in [31], by converging to the optimal fixpoint. Afterwards we sketch a more formal proof.

All greedy policies inspected by G$\text {LRTDP}$ at some point end in a goal state or a dead-end state. This could be a real dead-end, i. e., a sink state with only a self-loop or a permanent trap which has been transformed to a dead-end by the cycle elimination of FRET. If it is a permanent trap identified by $\text {FRET}$, the values of all states in it are set to 0. Otherwise, when the sink state is discovered for the first time its value is also directly set to 0. This means we tag these states, do not explore them further and propagate their value back through the graph. Cycling forever is not possible because $\text {FRET}$ eventually eliminates all such cycles in greedy policies. With this, we can state that at some point no more states are left to explore in the current G$\text {LRTDP}$ trial because all relevant traps are eliminated or a goal or a sink has been found. Then G$\text {LRTDP}$ runs until the state values of the current greedy policies are converged up to $\varepsilon$. Even if the greedy policy is not the same in every iteration, at some point it will stay within a set of greedy states which are part of finitely many greedy policies. The values of these states will have converged close enough to the optimal ones such that the algorithm concentrates on these optimal policies. The value function used in G$\text {LRTDP}$ is initialized admissibly and therefore can only monotonically decrease and approach the optimal fixpoint from above. When this point is reached (up to $\varepsilon$), the entire procedure (G$\text {LRTDP}$ + $\text {FRET}$) terminates. This fixpoint must be the optimal one because the Bellman equation only admits a single fixpoint [7].

To show convergence to the optimal value function from above in case of an admissible initialization, we can argue along the invariant

$$\begin{aligned} \forall k, \sigma : V_k(s) \ge P^{\sigma }_s(\diamond G), \text { where } \sigma \text { s.t. } P^{\sigma }_s(\diamond G) = V^*(s) \end{aligned}$$

stating that the value function in every iteration is always greater or equal than the optimal value under the optimal policy. This means that an initially admissible value function always stays admissible. This is true for the admissible initialization when $k=0$, because then $V_0(s) = 0$ if $s \in \mathcal S_\bot$ and 1 otherwise. For all other iterations it holds that

for some action a and we can derive that

$$\begin{aligned}\begin{gathered} \sum _{s'} P(s, a, s') \cdot V_k(s') \ge \sum _{s'} P(s, a, s') \cdot \max _{\sigma }P^{\sigma }_{s'}(\diamond G) \\ \ge \sum _{s'} \max _{\sigma }(P(s, a, s') \cdot P^{\sigma }_{s'}(\diamond G)) \ge \max _{\sigma } \sum _{s'} P(s, a, s') \cdot P^{\sigma }_{s'}(\diamond G) \end{gathered}\end{aligned}$$

The second inequality holds because $\sigma _{opt}$ is memoryless and independent of $s'$.

Now assume $\sigma _{opt}$ is such that $P^{\sigma _{opt}}_s(\diamond G)$ is maximal for all s. Then for action $a = greedy (s, V_k)$ we have for any action b, and in particular for $b=\sigma _{opt}(s)$,

$$\begin{aligned} \sum _{s'}P(s, a, s') \cdot V_k(s') \ge \sum _{s'}P(s, b, s')\cdot V_k(s'). \end{aligned}$$

Moreover $V_k(s') \ge P^{\sigma _{opt}}_{s'}(\diamond G)$ and hence

$$\begin{aligned} \sum _{s'}P(s, a, s') \cdot V_k(s') \ge \sum _{s'}P(s, \sigma _{opt}(s), s') \cdot P^{\sigma _{opt}}_{s'}(\diamond G) = P^{\sigma _{opt}}_{s}(\diamond G). \end{aligned}$$

Claim: $\text {If } V_k \text { is a fixpoint for } k \rightarrow \infty \text { then } P^\sigma (\diamond G) = V_\infty (s_0)~ \forall \sigma \text { greedy in } V.~(6)$

Since this means $V^*(s_0) \ge V_\infty (s_0)$ and with the result from above ($\forall k: V_k \ge V^*$) we can conclude $V^*(s_0) = V_\infty (s_0)$.

It remains to show that (6) holds: Let , i. e., a greedy policy with respect to the value function $V_k$ and $S_k = \{s | P^{\sigma _k}_{s_0}(\diamond s)>0\}$, i. e., all states reachable with this greedy policy, then $\max ( residual (S_k)) \le \delta _k$ and for $k \rightarrow \infty$ it holds that $\delta _k \rightarrow 0$.

To show that $\delta _k$ will approach 0 it is enough to argue about the states which will be updated an infinite number of times, i. e., in the end, about the states on optimal policies. These are the states in $S_\infty = \bigcap _{i \ge 0} \bigcup _{k \ge i} S_k$.

Let K be such that $\forall k \ge K: \bigcup _{i \ge k} S_i = S_\infty$, i. e. a step from which on we only consider states which will be infinitely often visited when running $\text {FRET}$-$\text {LRTDP}$ infinitely long. Assume we are in a step $j+1 \ge K$. Let $s \in S_\infty$. We have to distinguish two cases:

• If s has not been updated then $V_{j+1}(s) = V_j(s)$.

• If s is the updated state then $V_{j+1}(s) = \max _{\alpha } \sum _{s'} P(s, \alpha , s') \cdot V_j(s')$

This is the same as for simple synchronous value iteration, for which convergence against the optimal fixpoint is proven. For our asynchronous case in G$\text {LRTDP}$ we are left with the duty to guarantee fairness among the states in $S_\infty$, i. e., we have to make sure that they are updated infinitely often. This is the case because each possible trial of $S_\infty$ (there are finitely many trials) appears infinitely often, i. e., the states in this trial are updated infinitely often (by construction of G$\text {LRTDP}$ when choosing the next greedy action). All other states not in $S_\infty$ can be ignored because they will not influence the greedy policy and optimal values because they are already too large:

For any $s \in S\setminus S_{\infty }$ it holds that $V_{\infty }(s) = V_K(s) \ge V^*(s)$ and for any $s \in S_{\infty }$ by definition of $S_{\infty }$ and K we know that an action leading again to a state in $S_{\infty }$ will be chosen, i. e., an $a \in \sigma _{\infty }$: $V_{\infty }(s) \ge \sum _{s' \in S_{\infty }} P(s, \sigma _{\infty }, s') \cdot V_{\infty }(s')$ but for every action we choose the greedy one and for any $k \ge K$ it holds that $V_k(s) \ge \sum _{s' \in S} P(s, a, s') \cdot V_k(s') \ge V_{\infty }(s)$, i. e., the action in $\sigma _{\infty }$ must have been the greedy action not leading to $S\setminus S_{\infty }$.

This means that $V_{\infty }$ defines an optimal strategy on $S_{\infty }$ for $s_0 \in S_{\infty }$ which is also an optimal strategy on S because no state $s' \in S \setminus S_{\infty }$ is visited even with $V_{\infty }(s')>V^*(s')$.

In addition the initial state lies in $S_\infty$ by construction, i. e., $P_{\max }(\diamond G) = V^{\sigma _{opt}}(s_0)$ reaches the fixpoint and is updated infinitely often.

Altogether, this shows that when running $\text {FRET}$-$\text {LRTDP}$ over an infinite number of iterations, the value function for states in $S_\infty$ will approach the optimal values of the maximal probability to reach the goal from above, will never get smaller than the optimal value and the difference between V and $V^*$ always becomes strictly smaller for these states. In addition, we can at some point stop updating the value function for parts of the state space because these values will not have an influence on the correct optimal result for the initial state. In our implementation $\text {FRET}$-$\text {LRTDP}$ is designed in such a way that it stops when the values on the optimal policy only change by less than $\varepsilon$, which is the same convergence criterion as for simple value iteration.