# Shepherding Hordes of Markov Chains

- 5 Citations
- 4.1k Downloads

## Abstract

This paper considers large families of Markov chains (MCs) that are defined over a set of parameters with finite discrete domains. Such families occur in software product lines, planning under partial observability, and sketching of probabilistic programs. Simple questions, like ‘does at least one family member satisfy a property?’, are NP-hard. We tackle two problems: distinguish family members that satisfy a given quantitative property from those that do not, and determine a family member that satisfies the property optimally, i.e., with the highest probability or reward. We show that combining two well-known techniques, MDP model checking and abstraction refinement, mitigates the computational complexity. Experiments on a broad set of benchmarks show that in many situations, our approach is able to handle families of millions of MCs, providing superior scalability compared to existing solutions.

## 1 Introduction

Randomisation is key to research fields such as dependability (uncertain system components), distributed computing (symmetry breaking), planning (unpredictable environments), and probabilistic programming. Families of alternative designs differing in the structure and system parameters are ubiquitous. Software dependability has to cope with configuration options, in distributed computing the available memory per process is highly relevant, in planning the observability of the environment is pivotal, and program synthesis is all about selecting correct program variants. The automated analysis of such families has to face a formidable challenge—in addition to the state-space explosion affecting each family member, the family size typically grows exponentially in the number of features, options, or observations. This affects the analysis of (quantitative) software product lines [18, 28, 43, 45, 46], strategy synthesis in planning under partial observability [12, 14, 29, 36, 41], and probabilistic program synthesis [9, 13, 27, 40].

This paper considers families of Markov chains (MCs) to describe configurable probabilistic systems. We consider finite MC families with finite-state family members. Family members may have different transition probabilities and distinct topologies—thus different reachable state spaces. The latter aspect goes beyond the class of parametric MCs as considered in parameter synthesis [10, 22, 24, 31] and model repair [6, 16, 42].

For an MC family \(\mathfrak {D}\) and quantitative specification \(\varphi \), with \(\varphi \) a reachability probability or expected reward objective, we consider the following synthesis problems: (a) does some member in \(\mathfrak {D}\) satisfy a threshold on \(\varphi \)? (aka: *feasibility* synthesis), (b) which members of \(\mathfrak {D}\) satisfy this threshold on \(\varphi \) and which ones do not? (aka: *threshold synthesis*), and (c) which family member(s) satisfy \(\varphi \) optimally, e.g., with highest probability? (aka: *optimal synthesis*).

The simplest synthesis problem, feasibility, is NP-complete and can naively be solved by analysing all individual family members—the so-called *one-by-one* approach. This approach has been used in [18] (and for qualitative systems in e.g. [19]), but is infeasible for large systems. An alternative is to model the family \(\mathfrak {D}\) by a single Markov decision process (MDP)—the so-called *all-in-one* MDP [18]. The initial MDP state non-deterministically chooses a family member of \(\mathfrak {D}\), and then evolves in the MC of that member. This approach has been implemented in tools such as ProFeat [18], and for purely qualitative systems in [20]. The MDP representation avoids the individual analysis of all family members, but its size is proportional to the family size. This approach therefore does not scale to large families. A symbolic BDD-based approach is only a partial solution as family members may induce different reachable state-sets.

This paper introduces an *abstraction-refinement* scheme over the MDP representation^{1}. The abstraction *forgets* in which family member the MDP operates. The resulting *quotient* MDP has a single representative for every reachable state in a family member. It typically provides a very compact representation of the family \(\mathfrak {D}\) and its analysis using off-the-shelf MDP model-checking algorithms yields a speed-up compared to the all-in-one approach. Verifying the quotient MDP yields under- and over-approximations of the \(\min \) and \(\max \) probability (or reward), respectively. These bounds are safe as all *consistent* schedulers, i.e., those that pick actions according to a single family member, are contained in all schedulers considered on the quotient MDP. (CEGAR-based MDP model checking for partial information schedulers, a slightly different notion than restricting schedulers to consistent ones, has been considered in [30]. In contrast to our setting, [30] considers history-dependent schedulers and in this general setting no guarantee can be given that bounds on suprema converge [29]).

Model-checking results of the quotient MDP do provide useful insights. This is evident if the resulting scheduler is consistent. If the verification reveals that the \(\min \) probability exceeds *r* for a specification \(\varphi \) with a \(\le r\) threshold, then—even for inconsistent schedulers—it holds that all family members violate \(\varphi \). If the model checking is inconclusive, i.e., the abstraction is too coarse, we iteratively refine the quotient MDP by splitting the family into sub-families. We do so in an efficient manner that avoids rebuilding the sub-families. Refinement employs a light-weight analysis of the model-checking results.

We implemented our abstraction-refinement approach using the Storm model checker [25]. Experiments with case studies from software product lines, planning, and distributed computing yield possible speed-ups of up to 3 orders of magnitude over the one-by-one and all-in-one approaches (both symbolic and explicit). Some benchmarks include families of millions of MCs where family members are thousands of states. The experiments reveal that—as opposed to parameter synthesis [10, 24, 31]—the threshold has a major influence on the synthesis times.

To summarise, this work presents: (a) MDP-based abstraction-refinement for various synthesis problems over large families of MCs, (b) a refinement strategy that mitigates the overhead of analysing sub-families, and (c) experiments showing substantial speed-ups for many benchmarks. Extra material can be found in [1, 11].

## 2 Preliminaries

We present the basic foundations for this paper, for details, we refer to [4, 5].

*Probabilistic models.* A *probability distribution* over a finite or countably infinite set \(X\) is a function \(\mu :X\rightarrow [0,1]\) with \(\sum _{x\in X}\mu (x)=\mu (X)=1\). The set of all distributions on \(X\) is denoted \( Distr (X)\). The support of a distribution \(\mu \) is \({{\,\mathrm{supp}\,}}(\mu ) = \{x\in X\,|\,\mu (x)>0\}\). A distribution is *Dirac* if \(|\!{{\,\mathrm{supp}\,}}(\mu )| = 1\).

### Definition 1

**(MC).** A *discrete-time Markov chain* (MC) *D* is a triple \((S,s_0, \mathbf {P})\), where *S* is a finite set of states, \(s_0\in S\) is an initial state, and \(\mathbf {P}:S \rightarrow Distr (S)\) is a transition probability matrix.

MCs have unique distributions over successor states at each state. Adding nondeterministic choices over distributions leads to Markov decision processes.

### Definition 2

**(MDP).** A *Markov decision process* (MDP) is a tuple \(M=(S,s_0, Act ,\mathcal {P})\) where \(S, s_0\) as in Definition 1, \( Act \) is a finite set of actions, and \(\mathcal {P}:S\times Act \nrightarrow Distr (S)\) is a partial transition probability function.

The *available actions* in \(s\in S\) are \( Act (s)=\{a\in Act \mid \mathcal {P}(s,a) \ne \bot \}\). An MDP with \(| Act (s)|=1\) for all \(s\in S\) is an MC. For MCs (and MDPs), a state-reward function is \(\textit{rew}:S\rightarrow \mathbb {R}_{\ge 0}\). The reward \(\textit{rew}(s)\) is earned upon leaving *s*.

A *path* of an MDP *M* is an (in)finite sequence \(\pi = s_0\xrightarrow {a_0}s_1\xrightarrow {a_1}\cdots \), where \(s_i\in S\), \(a_i\in Act (s_i)\), and \(\mathcal {P}(s_i,a_i)(s_{i+1})\ne 0\) for all \(i\in \mathbb {N}\). For finite \(\pi \), \(\mathrm {last}(\pi )\) denotes the last state of \(\pi \). The set of (in)finite paths of *M* is \(\mathsf {Paths}_{ fin }^{M}\) (\(\mathsf {Paths}^{M}\)). The notions of paths carry over to MCs (actions are omitted). Schedulers resolve all choices of actions in an MDP and yield MCs.

### Definition 3

**(Scheduler).** A *scheduler* for an MDP \(M=(S,s_0, Act ,\mathcal {P})\) is a function \(\sigma :\mathsf {Paths}_{ fin }^{M}\rightarrow Act \) such that \(\sigma (\pi )\in Act (\mathrm {last}(\pi ))\) for all \(\pi \in \mathsf {Paths}_{ fin }^{M}\). Scheduler \(\sigma \) is *memoryless* if \(\mathrm {last}(\pi )=\mathrm {last}(\pi ') \implies \sigma (\pi )=\sigma (\pi ')\) for all \(\pi ,\pi '\in \mathsf {Paths}_{ fin }^{M}\). The set of all schedulers of \(M\) is \(\varSigma ^{M}\).

### Definition 4

**(Induced Markov Chain).**The MC

*induced by MDP*\(M\)

*and*\(\sigma \in \varSigma ^M\) is given by \(M_\sigma = (\mathsf {Paths}_{ fin }^{M},s_0,\mathbf {P}^{\sigma })\) where:

*Specifications.* For a MC \(D\), we consider unbounded reachability specifications of the form \(\varphi =\mathbb {P}_{\sim \lambda }(\lozenge G)\) with \(G\subseteq S\) a set of goal states, \(\lambda \in [0,1]\subseteq {\mathbb {R}}\), and \({{}\sim {}} \in \{<,\le ,\ge ,>\}\). The probability to satisfy the path formula \(\phi =\lozenge G\) in \(D\) is denoted by \(\texttt {Prob}(D, \phi )\). If \(\varphi \) holds for \(D\), that is, \(\texttt {Prob}(D, \phi )\sim \lambda \), we write \( D\models \varphi \). Analogously, we define expected reward specifications of the form \(\varphi =\mathbb {E}_{\sim \kappa }(\lozenge G)\) with \(\kappa \in {\mathbb {R}_{\ge 0}}\). We refer to \(\lambda \)/\(\kappa \) as *thresholds*. While we only introduce reachability specifications, our approaches may be extended to richer logics like arbitrary PCTL [32], PCTL* [3], or \(\omega \)-regular properties.

For an MDP \(M\), a specification \(\varphi \) holds (\(M\models \varphi \)) if and only if it holds for the induced MCs of all schedulers. The maximum probability \(\texttt {Prob}^{\max }(M,\phi )\) to satisfy a path formula \(\phi \) for an MDP *M* is given by a maximising scheduler \(\sigma ^{\max }\in \varSigma ^M\), that is, there is no scheduler \(\sigma '\in \varSigma ^M\) such that \(\texttt {Prob}(M_{\sigma ^{\max }},\phi )<\texttt {Prob}(M_{\sigma '},\phi )\). Analogously, we define the minimising probability \(\texttt {Prob}^{\min }(M,\phi )\), and the maximising (minimising) expected reward \(\texttt {ExpRew}^{\max }(M, \phi )\) (\(\texttt {ExpRew}^{\min }(M, \phi )\)).

The probability (expected reward) to satisfy path formula \(\phi \) from state \(s\in S\) in MC \(D\) is \(\texttt {Prob}(D,\phi )(s)\) (\(\texttt {ExpRew}(D,\phi )(s)\)). The notation is analogous for maximising and minimising probability and expected reward measures in MDPs. Note that the expected reward \(\texttt {ExpRew}(D, \phi )\) to satisfy path formula \(\phi \) is only defined if \(\texttt {Prob}(D, \phi )=1\). Accordingly, the expected reward for MDP \(M\) under scheduler \(\sigma \in \varSigma ^M\) requires \(\texttt {Prob}(M_{\sigma }, \phi )=1\).

## 3 Families of MCs

We present our approaches on the basis of an explicit representation of a *family of MCs* using a parametric transition probability function. While arbitrary probabilistic programs allow for more modelling freedom and complex parameter structures, the explicit representation alleviates the presentation and allows to reason about practically interesting synthesis problems. In our implementation, we use a more flexible high-level modelling language, cf. Sect. 5.

### Definition 5

**(Family of MCs).** A *family of MCs* is defined as a tuple \(\mathfrak {D}=(S,s_0,K,\mathfrak {P})\) where *S* is a finite set of states, \(s_0\in S\) is an initial state, *K* is a finite set of discrete parameters such that the domain of each parameter \(k\in K\) is \(T_k\subseteq S\), and \(\mathfrak {P}:S \rightarrow Distr (K)\) is a family of transition probability matrices.

The transition probability function of MCs maps states to distributions over successor states. For families of MCs, this function maps states to distributions over parameters. Instantiating each of these parameters with a value from its domain yields a “concrete” MC, called a *realisation*.

### Definition 6

**(Realisation).** A *realisation* of a family \(\mathfrak {D}=(S,s_0,K,\mathfrak {P})\) is a function \(r:K \rightarrow S\) where \(\forall k\in K:r(k) \in T_k\). A realisation *r* yields a MC \(D_r = (S,s_0,\mathfrak {P}(r))\), where \(\mathfrak {P}(r)\) is the transition probability matrix in which each \(k\in K\) in \(\mathfrak {P}\) is replaced by *r*(*k*). Let \(\mathcal {R}^\mathfrak {D}\) denote the *set of all realisations* for \(\mathfrak {D}\).

As a family \(\mathfrak {D}\) of MCs is defined over finite parameter domains, the number of family members (i.e. realisations from \(\mathcal {R}^\mathfrak {D}\)) of \(\mathfrak {D}\) is finite, viz. Open image in new window , but exponential in |*K*|. Subsets of \(\mathcal {R}^\mathfrak {D}\) induce so-called *subfamilies* of \(\mathfrak {D}\). While all these MCs share the same state space, their *reachable* states may differ, as demonstrated by the following example.

### Example 1

*(Family of MCs).*Consider a family of MCs \(\mathfrak {D} = (S,s_0,K,\mathfrak {P})\) where \(S=\{0,1,2,3\}\), \(s_0= 0\), and \(K=\{k_0,k_1,k_2\}\) with domains \(T_{k_0}=\{0\}, T_{k_1}=\{0,1\}\), and \(T_{k_2}=\{2,3\}\). The parametric transition function \(\mathfrak {P}\) is defined by:

*threshold synthesis*and

*max/min synthesis*.

### Problem 1

**(Threshold synthesis).** Let \(\mathfrak {D}\) be a family of MCs and \(\varphi \) a probabilistic reachability or expected reward specification. The *threshold synthesis problem* is to partition \(\mathcal {R}^{\mathfrak {D}}\) into *T* and *F* such that \(\forall r \in T:D_r \vDash \varphi \) and \(\forall r \in F:D_r \nvDash \varphi \).

As a special case of the threshold synthesis problem, the *feasibility synthesis problem* is to find just one realisation \(r\in \mathcal {R}^{\mathfrak {D}}\) such that \(D_r \vDash \varphi \).

### Problem 2

**(Max synthesis).** Let \(\mathfrak {D}\) a family of MCs and \(\phi =\lozenge G\) for \(G\subseteq S\). The *max synthesis problem* is to find a realisation \(r^* \in \mathcal {R}^{\mathfrak {D}}\) such that \(\texttt {Prob }(D_{r^*}, \phi ) = \max _{r\in \mathcal {R}^{\mathfrak {D}}} \{\texttt {Prob }(D_{r}, \phi )\}\). The problem is defined analogously for an expected reward measure or minimising realisations.

### Example 2

*(Synthesis problems).* Recall the family of MCs \(\mathfrak {D}\) from Example 1. For the specification \(\varphi =\mathbb {P}_{\ge 0.1}(\lozenge \{1\})\), the solution to the threshold synthesis problem is \(T=\{r_2,r_3\}\) and \(F=\{r_1,r_4\}\), as the goal state 1 is not reachable for \(D_{r_1}\) and \(D_{r_4}\). For \(\phi =\lozenge \{1\}\), the solution to the max synthesis problem on \(\mathfrak {D}\) is \(r_2\) or \(r_3\), as \(D_{r_2}\) and \(D_{r_3}\) have probability one to reach state 1.

### Approach 1

**(One-by-one** [18]**).** A straightforward solution to both synthesis problems is to enumerate all realisations \(r\in \mathcal {R}^\mathfrak {D}\), model check the MCs \(D_r\), and either compare all results with the given threshold or determine the maximum.

We already saw that the number of realisations is exponential in |*K*|.

### Theorem 1

The feasibility synthesis problem is NP-complete.

The theorem even holds for almost-sure reachability properties. The proof is a straightforward adaption of results for augmented interval Markov chains [17, Theorem 3], partial information games [15], or partially observable MDPs [14].

## 4 Guided Abstraction-Refinement Scheme

In the previous section, we introduced the notion of a family of MCs, two synthesis problems and the one-by-one approach. Yet, for a sufficiently high number of realisations such a straightforward analysis is not feasible. We propose a novel approach allowing us to more efficiently analyse families of MCs.

### 4.1 All-in-one MDP

We first consider a single MDP that subsumes all individual MCs of a family \(\mathfrak {D}\), and is equipped with an appropriate action and state labelling to identify the underlying realisations from \(\mathcal {R}^{\mathfrak {D}}\).

### Definition 7

**(All-in-one MDP**[18, 28, 43]

**).**The

*all-in-one MDP*of a family \(\mathfrak {D} = (S,s_0,K,\mathfrak {P})\) of MCs is given as \(M^\mathfrak {D}=(S^\mathfrak {D},s_0^\mathfrak {D}, Act ^\mathfrak {D},\mathcal {P}^\mathfrak {D})\) where \(S^\mathfrak {D}=S \times \mathcal {R}^{\mathfrak {D}} \cup \{s_0^\mathfrak {D}\}\), \( Act ^\mathfrak {D}=\{a^r\mid r\in \mathcal {R}^{\mathfrak {D}}\}\), and \(\mathcal {P}^\mathfrak {D}\) is defined as follows:

### Example 3

*(All-in-one MDP).* Figure 2 shows the all-in-one MDP \(M^\mathfrak {D}\) for the family \(\mathfrak {D}\) of MCs from Example 1. Again, states that are not reachable from the initial state \(s_0^\mathfrak {D}\) are marked grey. For the sake of readability, we only include the transitions and states that correspond to realisations \(r_1\) and \(r_2\).

From the (fresh) initial state \(s_0^\mathfrak {D}\) of the MDP, the choice of an action \(a_r\) corresponds to choosing the realisation *r* and entering the concrete MC \(D_r\). This property of the all-in-one MDP is formalised as follows.

### Corollary 1

^{2}:

Consequently, the feasibility synthesis problem for \(\varphi \) has the solution \(r\in \mathcal {R}^\mathfrak {D}\) iff there exists a memoryless deterministic scheduler \(\sigma ^r\) such that \(M^\mathfrak {D}_{\sigma ^r} \vDash \varphi \).

### Approach 2

**(All-in-one** [18]**).** Model checking the all-in-one MDP determines max or min probability (or expected reward) for all states, and thereby for all realisations, and thus provides a solution to both synthesis problems.

As also the all-in-one MDP may be too large for realistic problems, we merely use it as formal starting point for our abstraction-refinement loop.

### 4.2 Abstraction

First, we define a predicate abstraction that at each state of the MDP *forgets* in which realisation we are, i.e., abstracts the second component of a state (*s*, *r*).

### Definition 8

**(Forgetting).**Let \(M^\mathfrak {D}=(S^\mathfrak {D},s_0^\mathfrak {D}, Act ^\mathfrak {D},\mathcal {P}^\mathfrak {D})\) be an all-in-one MDP.

*Forgetting*is an equivalence relation \(\sim _f\ \subseteq S^\mathfrak {D} \times S^\mathfrak {D}\) satisfying

Forgetting induces the *quotient MDP* \(M^\mathfrak {D}_\sim = ( S^\mathfrak {D}_\sim ,[s_0^\mathfrak {D}]_\sim , Act ^\mathfrak {D},\mathcal {P}^\mathfrak {D}_\sim )\), where \(\mathcal {P}^\mathfrak {D}_\sim ([s]_\sim ,a_r)([s']_\sim ) = \mathfrak {P}(r)(s)(s')\).

At each state of the quotient MDP, the actions correspond to any realisation. It includes states that are unreachable in every realisation.

### Remark 1

*(Action space).* According to Definition 8, for every state \([s]_\sim \) there are \(|\mathfrak {D}|\) actions. Many of these actions lead to the same distributions over successor states. In particular, two different realisations *r* and \(r'\) lead to the same distribution in *s* if \(r(k) = r'(k)\) for all \(k\in K\) where \(\mathfrak {P}(s)(k) \ne 0\). To avoid this spurious blow-up of actions, we *a-priori* merge all actions yielding the same distribution.

### Definition 9

**(Consistent realisation).**For a family \(\mathfrak {D}\) of MCs and \(k\in K\),

*k-realisation-consistency*is an equivalence relation \(\approx _k\ \subseteq \mathcal {R}^\mathfrak {D}{\times }\mathcal {R}^\mathfrak {D}\) satisfying:

### Definition 10

**(Consistent scheduler).**For quotient MDP \(M^\mathfrak {D}_\sim \) after forgetting and \(k\in K\), a scheduler \(\sigma \in \varSigma ^{M^\mathfrak {D}_\sim }\) is

*k-consistent*if for all \(\pi ,\pi '\in \mathsf {Paths}_{ fin }^{M^\mathfrak {D}_\sim }\):

*K-consistent*(short:

*consistent*) if it is

*k*-consistent for all \(k\in K\).

### Lemma 1

### Proof

*(Idea).* For \(\sigma ^r \in \varSigma ^{M^\mathfrak {D}}\), we construct \(\sigma ^{r^*} \in \varSigma ^{M^\mathfrak {D}_\sim }\) such that \(\sigma ^{r^*}([s]_\sim ) = a_r\) for all *s*. Clearly \(\sigma ^{r^*}\) is consistent and \(M^\mathfrak {D}_{\sigma ^r} = \left( M^\mathfrak {D}_{\sim }\right) _{\sigma ^{r^*}}\) is obtained via a map between (*s*, *r*) and \([s]_\sim \). For \(\sigma ^{r^*} \in \varSigma ^{M^\mathfrak {D}_\sim }\), we construct \(\sigma ^r \in \varSigma ^{M^\mathfrak {D}}\) such that if \(\sigma ^{r^*}([s]_\sim ) = a_r\) then \(\sigma ^{r}(s_0^{\mathfrak {D}}) = a_r\). For all other states, we define \(\sigma ^{r}((s,r')) = a^{r'}\) independently of \(\sigma ^{r^*}\). Then \(M^\mathfrak {D}_{\sigma ^r} = \left( M^\mathfrak {D}_{\sim }\right) _{\sigma ^{r^*}}\) is obtained as above.

The following theorem is a direct corollary: we need to consider exactly the consistent schedulers.

### Theorem 2

For all-in-one MDP \(M^\mathfrak {D}\) and specification \(\varphi \), there exists a memoryless deterministic scheduler \(\sigma ^r \in \varSigma ^{M^\mathfrak {D}}\) such that \(M^\mathfrak {D}_{\sigma ^r} \vDash \varphi \) iff there exists a consistent deterministic scheduler \(\sigma ^{r^*}\in \varSigma ^{M^\mathfrak {D}_\sim }\) such that \(\left( M^\mathfrak {D}_{\sim }\right) _{\sigma ^{r^*}} \vDash \varphi \).

### Example 4

Recall the all-in-one MDP \(M^\mathfrak {D}\) from Example 3. The quotient MDP \(M^\mathfrak {D}_\sim \) is depicted in Fig. 3. Only the transitions according to realisations \(r_1\) and \(r_2\) are included. Transitions from previously unreachable states, marked grey in Example 3, are now available due to the abstraction. The scheduler \(\sigma \in \varSigma ^{M^\mathfrak {D}_\sim }\) with \(\sigma ([s_0^\mathfrak {D}]_\sim )=a_{r_2}\) and \(\sigma ([1]_\sim )=a_{r_1}\) is *not *\(k_1\)*-consistent* as different values are chosen for \(k_1\) by \(r_1\) and \(r_2\). In the MC \(M^\mathfrak {D}_{\sim \sigma }\) induced by \(\sigma \) and \(M^\mathfrak {D}_\sim \), the probability to reach state \([2]_\sim \) is one, while under realisation \(r_1\), state 2 is not reachable.

### Approach 3

**(Scheduler iteration).** Enumerating all consistent schedulers for \(M^\mathfrak {D}_\sim \) and analysing the induced MC provides a solution to both synthesis problems.

However, optimising over exponentially many consistent schedulers solves the NP-complete feasibility synthesis problem, rendering such an iterative approach unlikely to be efficient. Another natural approach is to employ solving techniques for NP-complete problems, like satisfiability modulo linear real arithmetic.

### Approach 4

**(SMT).** A dedicated SMT-encoding (in [11]) of the induced MCs of consistent schedulers from \(M^\mathfrak {D}_\sim \) that solves the feasibility problem.

### 4.3 Refinement Loop

Although iterating over consistent schedulers (Approach 3) is not feasible, model checking of \(M^\mathfrak {D}_\sim \) still provides useful information for the analysis of the family \(\mathfrak {D}\). Recall the feasibility synthesis problem for \(\varphi =\mathbb {P}_{\le \lambda } (\phi )\). If \(\texttt {Prob}^{\max }(M^\mathfrak {D}_\sim ,\phi ) \le \lambda \), then all realisations of \(\mathfrak {D}\) satisfy \(\varphi \). On the other hand, \(\texttt {Prob}^{\min }(M^\mathfrak {D}_\sim ,\phi ) > \lambda \) implies that there is no realisation satisfying \(\varphi \). If \(\lambda \) lies between the \(\min \) and \(\max \) probability, and the scheduler inducing the \(\min \) probability is not consistent, we cannot conclude anything yet, i.e., the abstraction is too coarse. A natural countermeasure is to refine the abstraction represented by \(M^\mathfrak {D}_\sim \), in particular, split the set of realisations leading to two synthesis sub-problems.

### Definition 11

**(Splitting).**Let \(\mathfrak {D}\) be a family of MCs, and \(\mathcal {R} \subseteq \mathcal {R}^{\mathfrak {D}}\) a set of realisations. For \(k\in K\) and predicate \(A_k\) over

*S*,

*splitting*partitions \(\mathcal {R}\) into

Splitting the set of realisations, and considering the subfamilies separately, rather than splitting states in the quotient MDP, is crucial for the performance of the synthesis process as we avoid rebuilding the quotient MDP in each iteration. Instead, we only restrict the actions of the MDP to the particular subfamily.

### Definition 12

**(Restricting).** Let \(M^\mathfrak {D}_\sim = ( S^\mathfrak {D}_\sim ,[s_0^\mathfrak {D}]_\sim , Act ^\mathfrak {D},\mathcal {P}^\mathfrak {D}_\sim )\) be a quotient MDP and \(\mathcal {R} \subseteq \mathcal {R}^{\mathfrak {D}}\) a set of realisations. The *restriction* of \(M^\mathfrak {D}_\sim \) wrt. \(\mathcal {R}\) is the MDP \(M^\mathfrak {D}_\sim [\mathcal {R}] = ( S^\mathfrak {D}_\sim ,[s_0^\mathfrak {D}]_\sim , Act ^\mathfrak {D}[\mathcal {R}],\mathcal {P}^\mathfrak {D}_\sim )\) where \( Act ^\mathfrak {D}[\mathcal {R}] = \{a_r \mid r \in \mathcal {R}\}.\)^{3}

Algorithm 1 illustrates the *threshold synthesis* process. Recall that the goal is to decompose the set \(\mathcal {R}^{\mathfrak {D}}\) into realisations satisfying and violating a given specification, respectively. The algorithm uses a set *U* to store subfamilies of \(\mathcal {R}^{\mathfrak {D}}\) that have not been yet classified as satisfying or violating. It starts building the quotient MDP with merged actions. That is, we never construct the all-in-one MDP, and we merge actions as discussed in Remark 1. For every \(\mathcal {R} \in U\), the algorithm restricts the set of realisations to obtain the corresponding subfamily. For the restricted quotient MDP, the algorithm runs standard MDP model checking to compute the \(\max \) and \(\min \) probability and corresponding schedulers, respectively. Then, the algorithm either classifies \(\mathcal {R}\) as satisfying/violating, or splits it based on a suitable predicate, and updates *U* accordingly. We describe the splitting strategy in the next subsection. The algorithm terminates if *U* is empty, i.e., all subfamilies have been classified. As only a finite number of subfamilies of realisations has to be evaluated, termination is guaranteed.

### 4.4 Splitting Strategies

If verifying the quotient MDP \(M^\mathfrak {D}_\sim [\mathcal {R}]\) cannot classify the (sub-)realisation \(\mathcal {R}\) as satisfying or violating, we split \(\mathcal {R}\), while we guide the splitting strategy by using the obtained verification results. The splitting operation chooses a suitable parameter \(k \in K\) and predicate \(A_k\) that partition the realisations \(\mathcal {R}\) into \(\mathcal {R}_{\top }\) and \(\mathcal {R}_{\bot }\) (see Definition 11). A good splitting strategy globally reduces the number of model-checking calls required to classify all \(r\in \mathcal {R}\).

*k*are: (1) the

*variance*, that is, how the splitting may narrow the difference between \(\max = \texttt {Prob}^{\max }(M^\mathfrak {D}_\sim [\mathcal {X}],\phi )\) and \(\min = \texttt {Prob}^{\min }(M^\mathfrak {D}_\sim [\mathcal {X}],\phi )\) for both \(\mathcal {X} = \mathcal {R}_{\top }\) or \(\mathcal {X} = \mathcal {R}_{\bot }\), and (2) the

*consistency*, that is, how the splitting may reduce the inconsistency of the schedulers \(\sigma _{\max }\) and \(\sigma _{\min }\). These aspects cannot be evaluated precisely without applying all the split operations and solving the new MDPs \(M^\mathfrak {D}_\sim [\mathcal {R}_{\bot }]\) and \(M^\mathfrak {D}_\sim [\mathcal {R}_{\top }]\). Therefore, we propose an efficient strategy that selects

*k*and \(A_k\) based on a light-weighted analysis of the model-checking results for \(M^\mathfrak {D}_\sim [\mathcal {R}]\). The strategy applies two

*scores*\(\texttt {variance}(k)\) and \(\texttt {consistency}(k)\) that estimate the influence of

*k*on the two key aspects. For any

*k*, the scores are accumulated over all

*important states*

*s*(reachable via \(\sigma _{\max }\) or \(\sigma _{\min }\), respectively) where \(\mathfrak {P}(s)(k) \ne 0\). A state

*s*is important for \(\mathcal {R}\) and some \(\delta \in \mathbb {R}_{\ge 0}\) if

*s*. To reduce the overhead of computing the scores, we simplify the scheduler representation. In particular, for \(\sigma _{\max }\) and every \(k\in K\), we extract a map \(C_{\max }^k:T_k \rightarrow \mathbb {N}\), where \(C_{\max }^k(t)\) is the number of important states for which \(\sigma _{\max }(s) = a_r\) with \(r(k) = t\). The mapping \(C_{\min }^k\) represents \(\sigma _{\min }\).

We define \(\texttt {variance}(k) = \sum _{t\in T_k} |C_{\max }^k(t) - C_{\min }^k(t)|\), leading to high scores if the two schedulers vary a lot. Further, we define \( \texttt {consistency}(k) = \texttt {size}\left( C_{\max }^k\right) \cdot \texttt {max}\left( C_{\max }^k\right) + \texttt {size}\left( C_{\min }^k\right) \cdot \texttt {max}\left( C_{\min }^k\right) \), where \(\texttt {size}\left( C\right) = |\{t\in T_k \mid C(t) > 0\}|-1 \) and \(\texttt {max}\left( C\right) = \max _{t\in T_k}\{ C(t)\}\), leading to high scores if the parameter has clear favourites for \(\sigma _{\max }\) and \(\sigma _{\min }\), but values from its full range are chosen.

As indicated, we consider different strategies for the two synthesis problems. For threshold synthesis, we favour the impact on the variance as we principally do not need consistent schedulers. For the max synthesis, we favour the impact on the consistency, as we need a consistent scheduler inducing the \(\max \) probability.

Predicate \(A_k\) is based on reducing the variance: The strategy selects \(T' \subset T_k\) with \(|T'| = \frac{1}{2}\left\lceil {|T_k|}\right\rceil \), containing those *t* for which \(C_{\max }^k(t) - C_{\min }^k(t)\) is the largest. The goal is to get a set of realisations that induce a large probability (the ones including \(T'\) for parameter *k*) and the complement inducing a small probability.

### Approach 5

**(MDP-based abstraction refinement).** The methods underlying Algorithms 1 and 2, together with the splitting strategies, provide solutions to the synthesis problems and are referred to as *MDP abstraction* methods.

## 5 Experiments

We implemented the proposed synthesis methods as a Python prototype using Storm [25]. In particular, we use the Storm Python API for model-adaption, -building, and -checking as well as for scheduler extraction. For SMT solving, we use Z3 [39] via pySMT [26]. The tool-chain takes a PRISM [38] or JANI [8] model with open integer constants, together with a set of expressions with possible values for these constants. The model may include the parallel composition of several modules/automata. The open constants may occur in guards^{4}, probability definitions, and updates of the commands/edges. Via adequate annotations, we identify the parameter values that yield a particular action. The annotations are key to interpret the schedulers, and to restrict the quotient without rebuilding.

All experiments were executed on a Macbook MF839LL/A with 8 GB RAM memory limit and a 12 h time out. All algorithms can significantly benefit from coarse-grained parallelisation, which we therefore do not consider here.

### 5.1 Research Questions and Benchmarks

The goal of the experimental evaluation is to answer the research question: *How does the proposed MDP-based abstraction methods (Approaches* 3–5*) cope with the inherent complexity (i.e. the NP-hardness) of the synthesis problems (cf. Problems* 1 *and* 2*)?* To answer this question, we compare their performance with Approaches 1 and 2 [18], representing state-of-the-art solutions and the base-line algorithms. The experiments show that the performance of the MDP abstraction significantly varies for different case studies. Thus, we consider benchmarks from various application domains to *identify the key characteristics of the synthesis problems affecting the performance of our approach.*

*Benchmarks description.*We consider the following case studies:

*Maze*is a planning problem typically considered as POMDP, e.g. in [41]. The family describes all MCs induced by small-memory [14, 35] observation-based deterministic strategies (with a fixed upper bound on the memory). We are interested in the expected time to the goal. In [35], parameter synthesis was used to find randomised strategies, using [22].

*Pole*considers balancing a pole in a noisy and unknown environment (motivated by [2, 12]). At deploy time, the controller has a prior over a finite set of environment behaviours, and should optimise the expected behavior without depending on the actual (hidden) environment. The family describes schedulers that do not depend on the hidden information. We are interested in the expected time until failure.

*Herman*is an asynchronous encoding of the distributed Herman protocol for self-stabilising rings [33, 37]. The protocol is extended with a bit of memory for each station in the ring, and the choice to flip various unfair coins. Nodes in the ring are anonymous, they all behave equivalently (but may change their local memory based on local events). The family describes variations of memory-updates and coin-selection, but preserves anonymity. We are interested in the expected time until stabilisation.

*DPM*considers a partial information scheduler for a disk power manager motivated by [7, 27]. We are interested in the expected energy consumption.

*BSN*(Body sensor network, [43]) describes a network of connected sensors that identify health-critical situations. We are interested in the reliability. The family contains various configurations of the used sensors.

*BSN*is the largest software product line benchmark used in [18]. We drop some implications between features (parameters for us) as this is not yet supported by our modelling language. We thereby extended the family.

Bench. | Range | | | \(|\mathcal {D}|\) | Member size | Quotient size | Run time | |||||
---|---|---|---|---|---|---|---|---|---|---|---|

Avg. | | Avg. | | | | | | | | 1-by-1 | All-in-1 | Sched. Enum. | ||||

| [3.35, 3.82] | 17 | 1327104 | 5689 | 16896 | 6793 | 7897 | 22416 | 130k\(^{*}\) | MO | 26k |

| [9.8, 9800] | 20 | 1048576 | 134 | 211 | 203 | 277 | 409 | 28k\(^{*}\) | TO | 2.7k |

| [1.86, 2.44] | 9 | 576 | 5287 | 6948 | 21313 | 102657 | 184096 | 55\(^{*}\) | 72 | 246 |

| [68, 210] | 9 | 32768 | 5572 | 18147 | 35154 | 66096 | 160146 | 2.9k\(^{*}\) | MO | 7.2k |

| [0, 0.988] | 10 | 1024 | 116 | 196 | 382 | 457 | 762 | 31\(^{*}\) | 2 | 2 |

Table 1 shows the relevant statistics for each benchmark: the benchmark name, the (approximate) range of the \(\min \) and \(\max \) probability/reward for the given family, the number of non-singleton parameters |*K*|, and the number of family members \(|\mathfrak {D}|\). Then, for the family members the average number of states and transitions of the MCs, and the states, actions (\(= \sum _{s \in S} | Act (s)|\)), and transitions of the quotient MDP. Finally, it lists in seconds the run time of the base-line algorithms and the consistent scheduler enumeration^{5}. The base-line algorithms employ the one-by-one and the all-in-one technique, using either a BDD or a sparse matrix representation. We report the best results. MOs indicate breaking the memory limit. Only the all-in-one approach required significant memory. As expected, the SMT-based implementation provides an inferior performance and thus we do not report its results.

### 5.2 Results and Discussion

Results for threshold synthesis via abstraction-refinement

Inst | \(\lambda \) | # Below | # Subf below | # Above | # Subf above | Singles | # Iter | Time | Build | Check | Anal. | Speedup |
---|---|---|---|---|---|---|---|---|---|---|---|---|

| 3.37 | 697 | 176 | 1326407 | 2186 | 920 | 4723 | 308 | 117 | 60 | 118 | |

3.73 | 1307077 | 7854 | 20027 | 3279 | 1294 | 22265 | 1.7k | 576 | 317 | 396 | | |

3.76 | 1322181 | 3140 | 4923 | 1025 | 1022 | 8329 | 584 | 187 | 114 | 197 | | |

3.79 | 1326502 | 572 | 602 | 123 | 74 | 1389 | 58 | 23 | 10 | 23 | | |

| 10 | 4 | 3 | 1048572 | 92 | 4 | 189 | 5 | \({<}1\) | 3 | \({<}1\) | |

20 | 4247 | 2297 | 1044329 | 4637 | 3400 | 13867 | 114 | 21 | 43 | 29 | | |

30 | 18188 | 9934 | 1030388 | 18004 | 14010 | 55875 | 608 | 80 | 127 | 270 | | |

8000 | 1046285 | 846 | 2291 | 1125 | 969 | 3941 | 136 | 9 | 106 | 13 | | |

| 1.9 | 6 | 6 | 570 | 368 | 320 | 747 | 333 | 303 | 11 | 18 | |

1.71 | 0 | 0 | 576 | 258 | 184 | 515 | 232 | 206 | 8 | 17 | | |

| 80 | 160 | 141 | 32608 | 1292 | 356 | 2865 | 1.0k | 602 | 322 | 64 | |

70 | 6 | 6 | 32762 | 443 | 40 | 897 | 380 | 190 | 156 | 32 | | |

60 | 0 | 0 | 32768 | 104 | 6 | 207 | 99 | 42 | 48 | 8 | | |

| .965 | 544 | 81 | 480 | 81 | 25 | 321 | 2 | \({<}1\) | \({<}1\) | \({<}1\) | |

.985 | 994 | 41 | 30 | 8 | 5 | 97 | \({<}1\) | \({<}1\) | \({<}1\) | \({<}1\) | |

*Results.* Table 2 shows results for threshold synthesis. The first two columns indicate the benchmark and the various thresholds. For each threshold \(\lambda \), the table lists the number of family members below (above) \(\lambda \), each with the number of subfamilies that together contain these instances, and the number of singleton subfamilies that were considered. The last table part gives the number of iterations of the loop in Algorithm 1, and timing information (total, build/restrict times, model checking times, scheduler analysis times). The last column gives the speed-up over the best base-line (based on the estimates).

*Key observations.* The speed-ups drastically vary, which shows that the MDP abstraction often achieves a superior performance but may also lead to a performance degradation in some cases. We identify four key factors.

**Iterations.** As typical for CEGAR approaches, the key characteristic of the benchmark that affects the performance is the number *N* of iterations in the refinement loop. The abstract action introduces an overhead per iteration caused by performing two MDP verification calls and by the scheduler analysis. The run time for *BSN*, with a small \(|\mathfrak {D}|\) is actually significantly affected by the initialisation of various data structures; thus only a small speedup is achieved.

**Abstraction size.** The size of the quotient, compared to the average size of the family members, is relevant too. The quotient includes at least all reachable states of all family members, and may be significantly larger if an inconsistent scheduler reaches states which are unreachable under any consistent scheduler. The existence of such states is a common artefact from encoding families in high-level languages. Table 1, however, indicates that we obtain a very compact representation for *Maze* and *Pole*.

**Thresholds.** The most important aspect is the threshold \(\lambda \). If \(\lambda \) is closer to the optima, the abstraction requires a smaller number of iterations, which directly improves the performance. We emphasise that in various domains, thresholds that ask for close-to-optimal solutions are indeed of highest relevance as they typically represent the system designs developers are most interested in [44]. *Why do thresholds affect the number of iterations?* Consider a family with \(T_k = \{0,1\}\) for each *k*. Geometrically, the set \(\mathcal {R}^\mathfrak {D}\) can be visualised as |*K*|-dimensional cube. The cube-vertices reflect family members. Assume for simplicity that one of these vertices is optimal with respect to the specification. Especially in benchmarks where parameters are equally important, the induced probability of a vertex roughly corresponds to the Manhattan distance to the optimal vertex. Thus, vertices above the threshold induce a diagonal hyperplane, which our splitting method approximates with orthogonal splits. Splitting diagonally is not possible, as it would induce optimising over observation-based schedulers. Consequently, we need more and more splits the more the diagonal goes through the middle of the cube. *Even when splitting optimally, there is a combinatorial blow-up in the required splits when the threshold is further from the optimal values.* Another effect is that thresholds far from optima are more affected by the over-approximation of the MDP model-checking results and thus yield more inconclusive answers.

**Refinement strategy.** So far, we reasoned about optimal splits. Due to the computational overhead, our strategy cannot ensure optimal splits. Instead, the strategy depends mostly on information encoded in the computed MDP strategies. *In models where the optimal parameter value heavily depends on the state, the obtained schedulers are highly inconsistent and carry only limited information for splitting.* Consequently, in such benchmarks we split sub-optimally. The sub-optimality has a major impact on the performance for *Herman* as all obtained strategies are highly inconsistent – they take a different coin for each node, which is good to speed up the stabilisation of the ring.

*Summary.* MDP abstraction is not a silver bullet. It has a lot of potential in threshold synthesis when the threshold is close to the optima. Consequently, *feasibility synthesis with unsatisfiable specifications is handled perfectly well by MDP abstraction*, while this is the worst-case for enumeration-based approaches. Likewise, *max synthesis* can be understood as threshold synthesis with a shifting threshold \(\max ^{*}\): If the \(\max ^{*}\) is quickly set close to \(\max \), MDP abstraction yields superior performance. Roughly, we can quickly approximate \(\max ^{*}\) when some of the parameter values are clearly beneficial for the specification.

## 6 Conclusion and Future Work

We contributed to the efficient analysis of families of Markov chains. In particular, we discussed and implemented existing approaches to solve practically interesting synthesis problems, and devised a novel abstraction refinement scheme that mitigates the computational complexity of the synthesis problems, as shown by the empirical evaluation. In the future, we will include refinement strategies based on counterexamples as in [23, 34].

## Footnotes

- 1.
Classical CEGAR for model checking of software product lines has been proposed in [21]. This uses feature transition systems, is purely qualitative, and exploits existential state abstraction.

- 2.
The original initial state \(s_0\) of the family of MCs needs to be the initial state of \(M^\mathfrak {D}_{\sigma ^r}\).

- 3.
Naturally, \(\mathcal {P}^\mathfrak {D}_\sim \) in \(M^\mathfrak {D}_\sim [\mathcal {R}]\) is restricted to \( Act ^\mathfrak {D}[\mathcal {R}]\).

- 4.
Slight care by the user is necessary to avoid deadlocks.

- 5.
Values with a \(^{*}\) are estimated by sampling a large fraction of the family.

## References

- 1.Repository with benchmarks. https://github.com/moves-rwth/shepherd
- 2.Arming, S., Bartocci, E., Chatterjee, K., Katoen, J.-P., Sokolova, A.: Parameter-independent strategies for pMDPs via POMDPs. In: McIver, A., Horvath, A. (eds.) QEST 2018. LNCS, vol. 11024, pp. 53–70. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99154-2_4CrossRefGoogle Scholar
- 3.Aziz, A., Singhal, V., Balarin, F., Brayton, R.K., Sangiovanni-Vincentelli, A.L.: It usually works: the temporal logic of stochastic systems. In: Wolper, P. (ed.) CAV 1995. LNCS, vol. 939, pp. 155–165. Springer, Heidelberg (1995). https://doi.org/10.1007/3-540-60045-0_48CrossRefGoogle Scholar
- 4.Baier, C., de Alfaro, L., Forejt, V., Kwiatkowska, M.: Model checking probabilistic systems. In: Clarke, E., Henzinger, T., Veith, H., Bloem, R. (eds.) Handbook of Model Checking, pp. 963–999. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-10575-8_28CrossRefzbMATHGoogle Scholar
- 5.Baier, C., Katoen, J.: Principles of Model Checking. MIT Press, Cambridge (2008)zbMATHGoogle Scholar
- 6.Bartocci, E., Grosu, R., Katsaros, P., Ramakrishnan, C.R., Smolka, S.A.: Model repair for probabilistic systems. In: Abdulla, P.A., Leino, K.R.M. (eds.) TACAS 2011. LNCS, vol. 6605, pp. 326–340. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19835-9_30CrossRefzbMATHGoogle Scholar
- 7.Benini, L., Bogliolo, A., Paleologo, G., Micheli, G.D.: Policy optimization for dynamic power management. IEEE Trans. Comput.-Aided Des. Integr. Circ. Syst.
**8**(3), 299–316 (2000)Google Scholar - 8.Budde, C.E., Dehnert, C., Hahn, E.M., Hartmanns, A., Junges, S., Turrini, A.: JANI: quantitative model and tool interaction. In: Legay, A., Margaria, T. (eds.) TACAS 2017. LNCS, vol. 10206, pp. 151–168. Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-662-54580-5_9CrossRefGoogle Scholar
- 9.Calinescu, R., Češka, M., Gerasimou, S., Kwiatkowska, M., Paoletti, N.: Efficient synthesis of robust models for stochastic systems. J. Syst. Softw.
**143**, 140–158 (2018)CrossRefGoogle Scholar - 10.Češka, M., Dannenberg, F., Paoletti, N., Kwiatkowska, M., Brim, L.: Precise parameter synthesis for stochastic biochemical systems. Acta Informatica
**54**(6), 589–623 (2017)MathSciNetCrossRefGoogle Scholar - 11.Češka, M., Jansen, N., Junges, S., Katoen, J.P.: Shepherding hordes of Markov chains. CoRR abs/1902.xxxxx (2019)Google Scholar
- 12.Chades, I., Carwardine, J., Martin, T.G., Nicol, S., Sabbadin, R., Buffet, O.: MOMDPs: a solution for modelling adaptive management problems. In: AAAI. AAAI Press (2012)Google Scholar
- 13.Chasins, S., Phothilimthana, P.M.: Data-driven synthesis of full probabilistic programs. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 279–304. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_14CrossRefGoogle Scholar
- 14.Chatterjee, K., Chmelik, M., Davies, J.: A symbolic SAT-based algorithm for almost-sure reachability with small strategies in POMDPs. In: AAAI, pp. 3225–3232. AAAI Press (2016)Google Scholar
- 15.Chatterjee, K., Kößler, A., Schmid, U.: Automated analysis of real-time scheduling using graph games. In: HSCC, pp. 163–172. ACM (2013)Google Scholar
- 16.Chen, T., Hahn, E.M., Han, T., Kwiatkowska, M.Z., Qu, H., Zhang, L.: Model repair for Markov decision processes. In: TASE, pp. 85–92. IEEE (2013)Google Scholar
- 17.Chonev, V.: Reachability in augmented interval Markov chains. CoRR abs/1701.02996 (2017)Google Scholar
- 18.Chrszon, P., Dubslaff, C., Klüppelholz, S., Baier, C.: ProFeat: feature-oriented engineering for family-based probabilistic model checking. Formal Asp. Comput.
**30**(1), 45–75 (2018)MathSciNetCrossRefGoogle Scholar - 19.Classen, A., Cordy, M., Heymans, P., Legay, A., Schobbens, P.: Model checking software product lines with SNIP. STTT
**14**(5), 589–612 (2012)CrossRefGoogle Scholar - 20.Classen, A., Cordy, M., Heymans, P., Legay, A., Schobbens, P.: Formal semantics, modular specification, and symbolic verification of product-line behaviour. Sci. Comput. Program.
**80**, 416–439 (2014)CrossRefGoogle Scholar - 21.Cordy, M., Heymans, P., Legay, A., Schobbens, P.Y., Dawagne, B., Leucker, M.: Counterexample guided abstraction refinement of product-line behavioural models. In: SIGSOFT FSE, pp. 190–201. ACM (2014)Google Scholar
- 22.Cubuktepe, M., Jansen, N., Junges, S., Katoen, J.-P., Topcu, U.: Synthesis in pMDPs: a tale of 1001 parameters. In: Lahiri, S.K., Wang, C. (eds.) ATVA 2018. LNCS, vol. 11138, pp. 160–176. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01090-4_10CrossRefGoogle Scholar
- 23.Dehnert, C., Jansen, N., Wimmer, R., Ábrahám, E., Katoen, J.-P.: Fast debugging of PRISM models. In: Cassez, F., Raskin, J.-F. (eds.) ATVA 2014. LNCS, vol. 8837, pp. 146–162. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11936-6_11CrossRefGoogle Scholar
- 24.Dehnert, C., et al.: PROPhESY: a PRObabilistic ParamEter SYnthesis tool. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 214–231. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21690-4_13CrossRefGoogle Scholar
- 25.Dehnert, C., Junges, S., Katoen, J.-P., Volk, M.: A storm is coming: a modern probabilistic model checker. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10427, pp. 592–600. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63390-9_31CrossRefGoogle Scholar
- 26.Gario, M., Micheli, A.: PySMT: a solver-agnostic library for fast prototyping of SMT-based algorithms. In: SMT Workshop 2015 (2015)Google Scholar
- 27.Gerasimou, S., Calinescu, R., Tamburrelli, G.: Synthesis of probabilistic models for quality-of-service software engineering. Autom. Softw. Eng.
**25**(4), 785–831 (2018)CrossRefGoogle Scholar - 28.Ghezzi, C., Sharifloo, A.M.: Model-based verification of quantitative non-functional properties for software product lines. Inf. Softw. Technol.
**55**(3), 508–524 (2013)CrossRefGoogle Scholar - 29.Giro, S., D’Argenio, P.R., Fioriti, L.M.F.: Distributed probabilistic input/output automata: expressiveness, (un)decidability and algorithms. Theor. Comput. Sci.
**538**, 84–102 (2014)MathSciNetCrossRefGoogle Scholar - 30.Giro, S., Rabe, M.N.: Verification of partial-information probabilistic systems using counterexample-guided refinements. In: Chakraborty, S., Mukund, M. (eds.) ATVA 2012. LNCS, vol. 7561, pp. 333–348. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33386-6_26CrossRefzbMATHGoogle Scholar
- 31.Hahn, E.M., Hermanns, H., Zhang, L.: Probabilistic reachability for parametric Markov models. Softw. Tools Technol. Transfer
**13**(1), 3–19 (2011)CrossRefGoogle Scholar - 32.Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Formal Aspects Comput.
**6**(5), 512–535 (1994)CrossRefGoogle Scholar - 33.Herman, T.: Probabilistic self-stabilization. Inf. Process. Lett.
**35**(2), 63–67 (1990)MathSciNetCrossRefGoogle Scholar - 34.Jansen, N., et al.: Symbolic counterexample generation for large discrete-time Markov chains. Sci. Comput. Program.
**91**, 90–114 (2014)CrossRefGoogle Scholar - 35.Junges, S., et al.: Finite-state controllers of POMDPs using parameter synthesis. In: UAI, pp. 519–529. AUAI Press (2018)Google Scholar
- 36.Kochenderfer, M.J.: Decision Making Under Uncertainty: Theory and Application, 1st edn. The MIT Press, Cambridge (2015)CrossRefGoogle Scholar
- 37.Kwiatkowska, M., Norman, G., Parker, D.: Probabilistic verification of Herman’s self-stabilisation algorithm. Formal Aspects Comput.
**24**(4), 661–670 (2012)MathSciNetCrossRefGoogle Scholar - 38.Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22110-1_47CrossRefGoogle Scholar
- 39.de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78800-3_24CrossRefGoogle Scholar
- 40.Nori, A.V., Ozair, S., Rajamani, S.K., Vijaykeerthy, D.: Efficient synthesis of probabilistic programs. In: PLDI, pp. 208–217. ACM (2015)Google Scholar
- 41.Norman, G., Parker, D., Zou, X.: Verification and control of partially observable probabilistic systems. Real-Time Syst.
**53**(3), 354–402 (2017)CrossRefGoogle Scholar - 42.Pathak, S., Ábrahám, E., Jansen, N., Tacchella, A., Katoen, J.-P.: A greedy approach for the efficient repair of stochastic models. In: Havelund, K., Holzmann, G., Joshi, R. (eds.) NFM 2015. LNCS, vol. 9058, pp. 295–309. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-17524-9_21CrossRefGoogle Scholar
- 43.Rodrigues, G.N., et al.: Modeling and verification for probabilistic properties in software product lines. In: HASE, pp. 173–180. IEEE (2015)Google Scholar
- 44.Skaf, J., Boyd, S.: Techniques for exploring the suboptimal set. Optim. Eng.
**11**(2), 319–337 (2010)MathSciNetCrossRefGoogle Scholar - 45.Vandin, A., ter Beek, M.H., Legay, A., Lluch-Lafuente, A.: QFLan: a tool for the quantitative analysis of highly reconfigurable systems. In: Havelund, K., Peleska, J., Roscoe, B., de Vink, E. (eds.) FM 2018. LNCS, vol. 10951, pp. 329–337. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-95582-7_19CrossRefGoogle Scholar
- 46.Varshosaz, M., Khosravi, R.: Discrete time Markov chain families: modeling and verification of probabilistic software product lines. In: SPLC Workshops, pp. 34–41. ACM (2013)Google Scholar

## Copyright information

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.