Parameter synthesis for Markov models: covering the parameter space

Abstract

Markov chain analysis is a key technique in formal verification. A practical obstacle is that all probabilities in Markov models need to be known. However, system quantities such as failure rates or packet loss ratios, etc. are often not—or only partially—known. This motivates considering parametric models with transitions labeled with functions over parameters. Whereas traditional Markov chain analysis relies on a single, fixed set of probabilities, analysing parametric Markov models focuses on synthesising parameter values that establish a given safety or performance specification $\varphi$. Examples are: what component failure rates ensure the probability of a system breakdown to be below 0.00000001?, or which failure rates maximise the performance, for instance the throughput, of the system? This paper presents various analysis algorithms for parametric discrete-time Markov chains and Markov decision processes. We focus on three problems: (a) do all parameter values within a given region satisfy $\varphi$?, (b) which regions satisfy $\varphi$ and which ones do not?, and (c) an approximate version of (b) focusing on covering a large fraction of all possible parameter values. We give a detailed account of the various algorithms, present a software tool realising these techniques, and report on an extensive experimental evaluation on benchmarks that span a wide range of applications.

1 Introduction

Uncertainty. Probabilistic model checking subsumes a multitude of formal verification techniques for systems that exhibit uncertainties [16, 54, 98]. Such systems are typically modeled by Markov chains or Markov decision processes [121]. Applications range from reliability, dependability and performance analysis to systems biology, take for instance reliability measures such as the mean time between failures in fault trees [28, 123] and the probability of a system breakdown within a time limit.

The results of probabilistic model checking algorithms are rigorous, their quality depends solely on the system models. Yet, there is one major practical obstacle: All probabilities (or rates) in the Markov model are precisely known a priori. In many cases, this assumption is too severe. System quantities such as component fault rates, molecule reaction rates, packet loss ratios, etc. are often not, or at best partially, known. Let us give a few examples. The quality of service of a (wireless) communication channel may be modelled by e.g., the popular Gilbert-Elliott model, a two-state Markov chain in which packet loss has an unknown probability depending on the channel’s state [112]. Other examples include the back-off probability in CSMA/CA protocols determining a node’s delay before attempting a transmission [1], the bias of used coins in self-stabilising protocols [88, 105], and the randomised choice of selecting the type of time-slots (sleeping, transmit, or idle) in the birthday protocol, a key mechanism used for neighbour discovery in wireless sensor networks [110] to lower power consumption. In particular, in early stages of reliable system design, the concrete failure rate of components [55] is left unspecified. Optimally, analyses in this stage may even guide the choice of a concrete component from a particular manufacturer.

The probabilities in all these systems are deliberately left unspecified. They can later be determined in order to optimise some performance or dependability measure. Dually, some systems should be robust for all (reasonable) failure rates. For example, a network protocol should ensure a reasonable quality of service for each reasonable channel quality.

Parametric probabilistic models. What do these examples have in common? The random variables for packet loss, failure rate etc. are not fully defined, but are parametric. Whether a parametric system satisfies a given property or not—“is the probability that the system goes down within k steps below $10^{{-}8}$”—depends on these parameters. Relevant questions are then: for which concrete parameter values is such a property satisfied—the (parameter) synthesis problem—and, in case of decision-making models, which parameter values yield optimal designs? That is, for which fixed probabilities do such protocols work in an optimal way, i.e., lead to maximal reliability, maximise the probability for nodes to be discovered, or minimise the time until stabilisation, and so on. These questions are intrinsically hard as parameters can take infinitely many different values that, in addition, can depend on each other.

This paper faces these challenges and presents various algorithmic techniques to treat different variations of the (optimal) parameter synthesis problem. To deal with uncertainties in randomness, parametric probabilistic models are adequate. These models are just like Markov models except that the transition probabilities are specified by arithmetic expressions over real-valued parameters. Transition probabilities are thus functions over a set of parameters. A simple instance is to use intervals over system parameters imposing constant lower and upper bounds on every parameter [74, 100]. The general setting as considered here is more liberal as it e.g., includes the possibility to express complex parameter dependencies. We address the analysis of parametric Markov models where probability distributions are functions over system parameters, specifically, parametric discrete-time Markov chains (pMCs) and parametric discrete-time Markov decision processes (pMDPs).

Fig. 1

A a biased and b parametric variant of Knuth-Yao’s algorithm. In gray states an unfair coin is flipped with probability $\nicefrac {2}{5}$ for ‘heads’; for the unfair coin in the white states this probability equals $\nicefrac {7}{10}$. On the right, the two biased coins have parametric probabilities

Example 1

The Knuth-Yao randomised algorithm [99] uses repeated coin flips to model a six-sided die. It uses a fair coin to obtain each possible outcome (‘one’, ‘two’, ..., ‘six’) with probability $\nicefrac {1}{6}$. Figure 1a depicts a Markov chain (MC) of a variant in which two unfair coins are flipped in an alternating fashion. Flipping the unfair coins yields heads with probability $\nicefrac {2}{5}$ (gray states) or $\nicefrac {7}{10}$ (white states), respectively. Accordingly, the probability of tails is $\nicefrac {3}{5}$ and $\nicefrac {3}{10}$, respectively. The event of throwing a ‘two’ corresponds to reaching the state  in the MC. Assume now a specification that requires the probability to obtain ‘two’ to be larger than $\nicefrac {3}{20}$. Knuth-Yao ’s original algorithm accepts this specification as using a fair coin results in $\nicefrac {1}{6}$ as probability to end up in . The biased model, however, does not satisfy the specification; in fact, a ‘two’ is reached with probability $\nicefrac {1}{10}$.

Probabilistic model checking. The analysis algorithms presented in this paper are strongly related to (and presented as) techniques from probabilistic model checking. Model checking [13, 46] is a popular approach to verify the correctness of a system by systematically evaluating all possible system runs. It either certifies the absence of undesirable (dangerous) behaviour or delivers a system run witnessing a violating system behaviour. Traditional model checking typically takes two inputs: a finite transition system modelling the system at hand and a temporal logic formula specifying a system requirement. Model checking then amounts to checking whether the transition system satisfies the logical specification, which in its simplest form describes that a particular state can (not) be reached. Model checking is nowadays a successful analysis technique adopted by mainstream hardware and software industry [49, 101].

To cope with real-world systems exhibiting random behaviour, model checking has been extended to deal with probabilistic, typically Markov, models. Probabilistic model checking [13, 16, 98] takes as input a Markov model of the system at hand together with a quantitative specification specified in some probabilistic extension of LTL or CTL. Example specifications are e.g., “is the probability to reach some bad (or degraded) state below a safety threshold $\lambda$?” or “is the expected time until the system recovers from a fault bounded by some threshold $\kappa$”. Efficient probabilistic model-checking techniques do exist for models such as discrete-time Markov chains (MCs), Markov decision processes (MDPs), and their continuous-time counterparts [98]. Probabilistic model checking extends and complements long-standing analysis techniques for Markov models.

It has been adopted in the field of performance analysis to analyse stochastic Petri nets [4, 38], in dependability analysis for analysing architectural system descriptions [29], in reliability engineering for fault tree analysis [27, 139], as well as in security [114], distributed computing [105], and systems biology [104]. Unremitting algorithmic improvements employing the use of symbolic techniques to deal with large state spaces have led to powerful and popular software tools realising probabilistic model checking techniques such as PRISM [102] and Storm [66].

1.1 Problem statements

We now give a more detailed description of the parameter synthesis problems considered in this paper. We start off by establishing the connection between parametric Markov models and concrete ones, i.e., ones in which the probabilities are fixed such as MCs and MDPs. Each parameter in a pMC or pMDP (where p stands for parametric) has a given parameter range. The parameter space of the parametric model is the Cartesian product of these parameter ranges. Instantiating the parameters with a concrete value in the parameter space to the parametric model results in an instantiated model. The parameter space defines all possible parameter instantiations, or equivalently, the instantiated models. A parameter instantiation that yields a Markov model, e.g., results in probability distributions, is called well-defined. In general, a parametric Markov model defines an uncountably infinite family of Markov models, where each family member is obtained by a well-defined instantiation. A region R is a fragment of the parameter space; it is well-defined if all instantiations in R are well-defined.

Example 2

(pMC) Figure 1b depicts a parametric version of the biased Knuth-Yao die from Example 1. It has parameters $V= \{p,q\}$, where p is the probability of outcome heads in gray states and q the same for white states. The parameter space is $\{ (p,q) \mid 0< p,q < 1 \}$. The probability for tails is $1{-}p$ and $1{-}q$, respectively. The sample instantiation u with $u(p) = \nicefrac {2}{5}$ and $u(q) = \nicefrac {7}{10}$ is well-defined and results in the MC in Fig. 1a. The region

$$\begin{aligned} R= \{ u :V\rightarrow \mathbb {R}\mid \nicefrac {1}{10} \le u(p) \le \nicefrac {9}{10} \text { and } \nicefrac {3}{4} \le u(q)\le \nicefrac {5}{6} \} \end{aligned}$$

is well-defined. Contrarily, region

$$\begin{aligned} R' = \{ u \mid \nicefrac {1}{5} \le u(p) \le \nicefrac {6}{5} \text{ and } \nicefrac {2}{5} \le u(q) \le \nicefrac {7}{10} \} \end{aligned}$$

is not well-defined, as it contains the instantiation $u'$ with $u'(p) = \nicefrac {6}{5}$ which does not yield an MC. For pMCs whose transition probabilities are high-degree polynomials, it is not always obvious whether a region is well-defined.

We are now in a position to describe the three problems considered in this paper.

The verification problem is defined as follows:

Consider the following possible outcomes:

• If R only contains instantiations of $\mathcal {D}$ satisfying $\varphi$, then the verification problem evaluates to true and the Markov model $\mathcal {D}$ on region R accepts specification $\varphi$. Whenever $\mathcal {D}$ and $\varphi$ are clear from the context, we call R accepting.

• If R contains an instantiation of $\mathcal {D}$ refuting $\varphi$, then the problem evaluates to false. If R contains only instantiations of $\mathcal {D}$ refuting $\varphi$, then $\mathcal {D}$ on R rejects $\varphi$. Whenever $\mathcal {D}$ and $\varphi$ are clear from the context, we call R rejecting.

• If R contains instantiations satisfying $\varphi$ as well as instantiations satisfying $\lnot \varphi$, then $\mathcal {D}$ on R is inconclusive w. r. t. $\varphi$. In this case, we call R inconsistent.

In case the verification problem yields $\texttt {false}$ for $\varphi$, one can only infer that the region R is not accepting, but not conclude whether R is inconsistent or rejecting. To determine whether R is rejecting, we need to consider the verification problem for the negated specification $\lnot \varphi$. Inconsistent regions for $\varphi$ are also inconsistent for $\lnot \varphi$.

Example 3

(Verification problem) Consider the pMC $\mathcal {D}$, the well-defined region R from Example 2, and the specification $\varphi ' := \lnot \varphi$ that constrains the probability to reach to be at most $\nicefrac {3}{20}$. The verification problem is to determine whether all instantiations of $\mathcal {D}$ in R satisfy $\varphi '$. As there is no instantiation within R for which the probability to reach is above $\nicefrac {3}{20}$, the verification problem evaluates to true. Thus, R accepts $\varphi '$.

Typical structurally simple regions are described by hyperrectangles or given by linear constraints, rather than non-linear constraints; we refer to such regions as simple. A simple region comprising a large range of parameter values may likely be inconsistent, as it contains both instantiations satisfying $\varphi$, and some satisfying $\lnot \varphi$. Thus, we generalise the problem to synthesise a partition of the parameter space.

The exact synthesis problem is described as follows:

The aim is to obtain such a partition in an automated manner. A complete sub-division of the parameter space into accepting and rejecting regions provides deep insight into the effect of parameter values on the system’s behaviour. The exact division typically is described by non-linear functions over the parameters, referred to as solution functions.

Example 4

Consider the pMC $\mathcal {D}$, the region R, and the specification $\varphi$ as in Example 3. The solution function:

$$\begin{aligned} f_{\varphi }(p, q) = \frac{p \cdot (1-q) \cdot (1-p)}{1-p\cdot q} \end{aligned}$$

describes the probability to eventually reach . Given that $\varphi$ imposes a lower bound of $\nicefrac {3}{20}$, we obtain

$$\begin{aligned} R_a = \{ u \mid f(u(p), u(q)) \ge \nicefrac {3}{20} \}\text { and }R_r = R \setminus R_a. \end{aligned}$$

The example illustrates that exact symbolic representations of the accepting and rejecting regions may be complex and hard to compute algorithmically. The primary reason is that the boundaries are described by non-linear functions. A viable alternative therefore is to consider an approximative version of the synthesis problem.

The approximate synthesis problem: As argued before, the regions obtained via exact synthesis are typically not simple. The aim of the approximate synthesis problem is to use simpler and more tractable representations of regions. As such shapes ultimately approximate the exact solution function, simple regions become infinitesimally small when getting close to the border between accepting and rejecting areas. For computational tractability, we are thus interested in approximating a partition of the parameter space in accepting and rejecting regions, where we allow also for a (typically small) part to be covered by possibly inconsistent regions. Practically this means that $c\,\%$ of the entire parameter space is covered by simple regions that are either accepting or rejecting, for some adequate value of c. Altogether this results in the following problem description:

Example 5

Consider the pMC $\mathcal {D}$, the region R, and the specification $\varphi$ as in Example 3. The parameter space in Fig. 2 is partitioned into simple regions (rectangles). The green (dotted) area—the union of a number of smaller rectangular accepting regions—indicates the parameter values for which $\varphi$ is satisfied, whereas the red (hatched) area indicates the set of rejecting regions for $\varphi$. The white area indicates the unknown regions. The indicated partition covers 95% of the parameter space. The sub-division into accepting and rejecting (simple) regions approximates the solution function $f_{\varphi }(p,q)$ given before.

Fig. 2

Parameter space partitioning into accepting (green), rejecting (red), and unknown (white) regions

1.2 Solution approaches

We now outline our approaches to solve the verification problem and the two synthesis problems. For the sake of convenience, we start with the synthesis problem.

Synthesis. The most straightforward description of the sets $R_a$ and $R_r$ is of the form:

$$\begin{aligned} R_a&= \{ u \mid \mathcal {D}[u] \text { satisfies } \varphi \} \quad \text{ and } \\ R_r&= \{ u \mid \mathcal {D}[u] \text { satisfies } \lnot \varphi \}. \end{aligned}$$

The satisfaction relation (denoted $\models$) can be concisely described by a set of linear equations over the transition probabilities [13]. As in the parametric setting the transition probabilities are no longer fixed, but rather defined over a set of parameters, the equations become non-linear.

Example 6

(Non-linear equations for reachability) Take the MC from Fig. 1a. To compute the probability of eventually reaching, e.g., state , one introduces a variable $p_s$ for each transient state s encoding that probability for s. For state $s_0$ and variable $p_{s_0}$, the corresponding linear equation reads:

$$\begin{aligned} p_{s_0} = \nicefrac {2}{5}\cdot p_{s_1} + \nicefrac {3}{5}\cdot p_{s_2}, \end{aligned}$$

where $p_{s_1}$ and $p_{s_2}$ are the variables for $s_1$ and $s_2$, respectively.

The corresponding equation for the pMC from Fig. 1b reads:

$$\begin{aligned} p_{s_0} = p\cdot p_{s_1} + (1-p)\cdot p_{s_2}. \end{aligned}$$

The multiplication of parameters in the model and equation variables leads to a non-linear equation system.

Thus, we can describe the sets $R_a$ and $R_r$ colloquially as:

$$\begin{aligned} R_a, R_r&= \{ u \mid u \text { satisfies a set of non-linear constraints} \}. \end{aligned}$$

We provide further details on these constraint systems in Sect. 6.

A practical drawback of the resulting equation system is the substantial number of auxiliary variables $p_s$, one for each state in the pMC. A viable possibility for pMCs is to simplify the equations by (variants of) state elimination [64]. This procedure successively removes states from the pMC until only a start and final state (representing the reachability objective) remain that are connected by a transition whose label is (a mild variant of) the solution function $f_\varphi$ that exactly describes the probability to reach a target state:

$$\begin{aligned} R_a = \{ u \mid f_\varphi (u)> 0 \} \quad \text{ and }\quad R_r = \{ u \mid f_{\lnot \varphi }(u) > 0 \}. \end{aligned}$$

We recapitulate state elimination and present several alternatives in Sect. 5.

Fig. 3

Verification via exact synthesis

Verification. The basic approach to the verification problem is depicted in Fig. 3. We use a description of the accepting region as computed via the synthesis procedure above. Then, we combine the description of the accepting region with the region R to be verified, as follows: A region R accepts a specification, if $R \cap R_a = R$, or equivalently, if $R \cap R_r = \emptyset$. The existence of a rejecting instance in R is thus of relevance; if such a point does not exist, the region is accepting. Using $R_a$ and $R_r$ as obtained above, the query “is $R \cap R_r = \emptyset$?” can be solved via satisfiability modulo theories (SMT) over non-linear arithmetic, checking the conjunction over the corresponding constraints for unsatisfiability. With the help of SMT solvers over this theory like Z3 [93], MathSAT [31], or SMT-RAT [52], this can be solved in a fully automated manner. This procedure is complete, and is computationally involved. Details of the procedure are discussed in Sect. 6.

Parameter lifting [122] is an alternative, approximative solution to the verification problem. Intuitively, this approach over-approximates $R_r$ for a given R, by ignoring parameter dependencies. Region R is accepted if the intersection with the over-approximation of $R_r$ is empty. This procedure is sound but may yield false negatives as a rejecting point may lie in the over-approximation but not in $R_r$. Tightening the over-approximation makes the approach complete. A major benefit of parameter lifting (details in Sect. 7 and Sect. 8) is that the intersection with the over-approximation of $R_r$ can be investigated by standard probabilistic model-checking procedures. This applicability of mature tools results—as will be shown in Sect. 11—in a practically efficient procedure.

Approximate synthesis. We solve the approximate synthesis problem with an iterative synthesis loop. Here, the central issue is to obtain representations of $R_a$ and $R_r$ by simple regions. Our approach for this parameter space partitioning therefore iteratively obtains partial partitions of the parameter space. The main idea is to compute a sequence $\left( R^i_a \right) _i$ of simple accepting regions that successively extend each other. Similarly, an increasing sequence $\left( R^i_r \right) _i$ of simple rejecting regions is computed. The typical approach is to let $R^{i+1}_a$ be the union of $R^i_a$, the approximations in the previous iteration, together with some accepting region with a simple representation. Rejecting regions are handled analogously. At the i-th iteration, $R^i_a \cup R^i_r$ is the covered fragment of the parameter space. The iterative approach halts when this fragment forms at least $c\,\%$ of the entire parameter space. Termination is guaranteed. In the limit, the accepting and rejecting regions converge to the exact solution, $\lim _{i \rightarrow \infty } R_a^i = R_a$ and $\lim _{i \rightarrow \infty } R_r^i = R_r$, under some mild constraints on the ordering of the regions $R^i$.

Figure 4 outlines a procedure to address the approximate synthesis problem. As part of our synthesis method, we algorithmically guess a (candidate) region R and guess whether it is accepting or rejecting. We then exploit one of our verification methods to verify whether R is indeed accepting (or rejecting). If it is not accepting (rejecting), we exploit this information together with any additional information obtained during verification to refine the candidate region. This process is repeated until an accepting or rejecting region results. We discuss the method and essential improvements in Sect. 9.

Example 7

Consider the pMC $\mathcal {D}$ and the specification $\varphi$ as in Example 2. The parameter space in Figure 2 is partitioned into regions. The green (dotted) area—the union of a number of smaller rectangular accepting regions—indicates the parameter values for which $\varphi$ is satisfied, whereas the red (hatched) area indicates the set of rejecting regions for $\varphi$. Checking whether a region is accepting, rejecting, or inconsistent is done by verification. The small white area consists of regions that are unknown (i.e., not yet considered) or inconsistent.

Fig. 4

Approximate synthesis process using verification as black box

1.3 Overview of the paper

Section 2 introduces the required formalisms and concepts. Section 3 defines the notion of a region and formalises the three problems: the verification problem and the two synthesis problems. It ends with a bird’s eye view of the verification approaches that are later discussed in detail. Section 4 details specific region structures and procedures to check elementary region properties such as well-definedness and graph-preservedness, two prerequisites for the verification procedures. Section 5 shows how to do exact synthesis by computing the solution function. Sections 6–8 present algorithms for the verification problem. Section 9 details the approach to reduce the synthesis problem to a series of verification problems. Sections 10 and 11 contain information about the implementation of the approaches, as well as an extensive experimental evaluation. Section 12 contains a discussion of the approaches and related work. Section 13 concludes with an outlook.

1.4 Contributions of this paper

The paper is loosely based on the conference papers [65] and [122] and extends these works in the following ways. It gives a uniform treatment of the solution techniques to the synthesis problem, and treats all techniques uniformly for all different objectives—bounded and unbounded reachability as well as expected reward specifications. The material on SMT-based region verification has been extended in the following way: The paper gives the complete characterisations of the SMT encoding with or without solution function. Furthermore, it is the first to extend this encoding to MDPs under angelic and demonic non-determinism and includes an explicit and in-depth discussion on exact region checking via SMT checkers. It presents a uniform treatment of the linear equation system for Markov chains and its relation to state elimination and Gaussian elimination. It presents a novel and simplified description of state elimination for expected rewards, and a version of state elimination that is targeted towards MTBDDs. The paper contains a correctness proof of approximate verification for a wider range of pMDPs and contains proofs for expected rewards. It also supports expected-time properties for parametric continuous-time MDPs (via the embedded pMDP). Novel heuristics have been developed to improve the iterative synthesis loop. All presented techniques, models, and specifications are realised in the state-of-the-art tool PROPhESY¹.

2 Preliminaries

2.1 Basic notations

We denote the set of real numbers by $\mathbb {R}$, the rational numbers by $\mathbb {Q}$, and the natural numbers including 0 by $\mathbb {N}$. Let $[0,1]\subseteq \mathbb {R}$ denote the closed interval of all real numbers between 0 and 1, including the bounds; $(0,1)\subseteq \mathbb {R}$ denotes the open interval of all real numbers between 0 and 1 excluding 0 and 1.

Let X, Y denote arbitrary sets. If $X\cap Y=\emptyset$, we write $X\uplus Y$ for the disjoint union of the sets X and Y. We denote the power set of X by $2^X=\{X'\mid X'\subseteq X\}$. Let X be a finite or countably infinite set. A probability distribution over $X$ is a function $\mu :X\rightarrow [0,1]\subseteq \mathbb {R}$ with $\sum _{x\in X}\mu (x)=\mu (X)=1$.

2.2 Polynomials, rational functions

Let V denote a finite set of parameters over $\mathbb {R}$ and ${{\,\textrm{dom}\,}}(p)\subseteq \mathbb {R}$ denote the domain of parameter $p\in V$.

Definition 1

(Polynomial, rational function) For a finite set $V=\{p_1,\ldots , p_{n}\}$ of n parameters, a monomial m is

$$\begin{aligned} m = p_{1}^{e_{1}}\cdot \ldots \cdot p_{n}^{e_{n}}\text { with } e_i\in \mathbb {N}. \end{aligned}$$

Let $Mon [V]$ denote the set of monomials over V. A polynomial $g$ (over V) with t terms is a weighted sum of monomials:

$$\begin{aligned} g= \sum _{j=1}^{t}a_j\cdot m_j \text { with } a_j\in \mathbb {Q}\setminus \{ 0 \},\; m_j \in Mon [V]. \end{aligned}$$

Let $\mathbb {Q}[V]$ be the set of polynomials over V. A rational function $f=\frac{g_1}{g_2}$ over V is a fraction of polynomials $g_1, g_2\in \mathbb {Q}[V]$ with $g_2 \not \equiv 0$ (where $\equiv$ states equivalence). Let $\mathbb {Q}(V)$ be the set of rational functions over V.

A monomial is linear, if $\sum _{i=1}^{|V|} e_{i} \le 1$, and multi-linear, if $e_{i} \le 1$ for all $1 \le i \le |V|$. A polynomial $g$ is (multi-)linear, if all monomials occurring in $g$ are (multi-)linear.

Instantiations replace parameters by constant values in polynomials or rational functions.

Definition 2

(Parameter instantiations) A (parameter) instantiation u of parameters V is a function $u:V \rightarrow \mathbb {R}$.

We abbreviate the parameter instantiation u with $u(p_i) = a_i \in \mathbb {R}$ by the n-dimensional vector $(a_1,\ldots ,a_n) \in \mathbb {R}^n$ for ordered parameters $p_1,\ldots ,p_n$. Applying the instantiation u on V to polynomial $g \in \mathbb {Q}[V]$ yields $g[u]$ which is obtained by replacing each $p \in {V}$ in g by u(p), with subsequent application of $+$ and $\cdot$. For rational function $f=\frac{g_1}{g_2}$, let $f[u]=\frac{g_1[u]}{g_2[u]}\in {\mathbb {R}}$ if $g_2[u]\not \equiv 0$, and otherwise $f[u] = \bot$.

2.3 Probabilistic models

Let us now introduce the probabilistic models used in this paper. We first define parametric Markov models and present conditions such that their instantiations result in Markov models with constant probabilities. Then, we discuss how to resolve non-determinism in decision processes.

2.3.1 Parametric Markov models

The transitions in parametric Markov models are equipped with rational functions over the set of parameters. Although this is the general setting, for some of our algorithmic techniques we will restrict ourselves to linear polynomials². We consider parametric MCs and MDPs as sub-classes of a parametric version of classical two-player stochastic games [128]. The state space of such games is partitioned into two parts, $S_{\bigcirc }$ and $S_{\Box }$. At each state, a player chooses an action upon which the successor state is determined according to the (parametric) probabilities. Choices in $S_{\bigcirc }$ and $S_{\Box }$ are made by player $\bigcirc$ and ${\Box }$, respectively. pMDPs and pMCs are parametric stochastic one- and zero-player games respectively.

Definition 3

(Parametric models) A parametric stochastic game (pSG) is a tuple $\mathcal {G}{}=(S{}, V{}, s_{ I }{}, Act {},\mathcal {P}{})$ with a finite set S of states with $S = S_{\bigcirc }\uplus S_{\Box }$, a finite set $V$ of parameters over $\mathbb {R}$, an initial state $s_{ I }\in S$, a finite set $Act$ of actions, and a transition function $\mathcal {P}:S \times Act \times S \rightarrow \mathbb {Q}(V) \cup \mathbb {R}\cup \{ \bot \}$ with $| Act (s)|\ge 1$ for all $s \in S$, where $Act (s) = \{\alpha \in Act \mid \exists s'\in S.\,\mathcal {P}(s,\alpha ,s') \not \equiv 0\}$ is the set of enabled actions at state s.

• A pSG is a parametric Markov decision process (pMDP) if $S_{\bigcirc }=\emptyset$ or $S_{\Box }=\emptyset$.

• A pMDP is a parametric Markov chain (pMC) if $| Act (s)|=1$ for all $s \in S$.

A parametric state-action reward function $\text {rew}:S \times Act \rightarrow \mathbb {Q}(V) \cup \mathbb {R}\cup \{\bot \}$ associates rewards with state-action pairs³. It is assumed that deadlock states are absent, i.e., $Act (s)\ne \emptyset$ for all $s \in S$. Entries in $\mathbb {R}\cup \{\bot \}$ in the co-domains of the functions $\mathcal {P}$ and $\text {rew}$ ensure that the model is closed under instantiations, see Definition 5 below. Throughout the rest of this paper, we silently assume that any given pSGs only uses constants from $\mathbb {Q}$ and rational functions $\mathbb {Q}(V)$, but no elements from $\mathbb {R}\setminus \mathbb {Q}$ or $\bot$. A model is called parameter-free if all its transition probabilities are constant.

A pSG intuitively works as follows. In state $s \in S_{\bigcirc }$, player $\bigcirc$ non-deterministically selects an action $\alpha \in Act (s)$. With (parametric) probability $\mathcal {P}(s,\alpha ,s')$ the play then evolves to state $s'$. On leaving state s via action $\alpha$, the reward $\text {rew}(s, \alpha )$ is earned. If $s \in S_{\Box }$, the choice is made by player ${\Box }$, and as for player $\bigcirc$, the next state is determined in a probabilistic way. As by assumption no deadlock states occur, this game goes on forever. A pMDP is a game with one player, whereas a pMC has no players; a pMC thus evolves in a fully probabilistic way. Let $\mathcal {D}$ denote a pMC, $\mathcal {M}$ a pMDP, and $\mathcal {G}$ a pSG.

Fig. 5

The considered types of parametric probabilistic models (a)–(c) and an instantiated model (d)

Example 8

Figure 5a–c depict a pSG, a pMDP, and a pMC respectively over parameters $V= \{p,q\}$. The states of the players $\bigcirc$ and ${\Box }$ are drawn as circles and rectangles, respectively. The initial state is indicated by an incoming arrow without source. We omit actions in state s if $| Act (s)| = 1$. In state $s_0$ of Fig. 5a, player $\bigcirc$ can select either action $\alpha$ or $\beta$. On selecting $\alpha$, the game moves to state $s_1$ with probability p, and to $s_2$ with probability $1{-}p$. In state $s_2$, player ${\Box }$ can select $\alpha$ or $\beta$; in $s_1$ there is a single choice only.

A transition $(s,\alpha ,s')$ exists if $\mathcal {P}(s,\alpha ,s') \not \equiv 0$. As pMCs have a single enabled action at each state, we omit this action and just write $\mathcal {P}(s,s')$ for $\mathcal {P}(s,\alpha ,s')$ if $Act (s) = \{\alpha \}$. A state $s'$ is a successor of s, denoted $s' \in \text {succ}(s)$, if $\mathcal {P}(s, \alpha , s') \not \equiv 0$ for some $\alpha$; in this case, $s \in \text {pred}(s')$ is a predecessor of $s'$.

Remark 1

Parametric stochastic games are the most general model used in this paper. They subsume pMDPs and pMCs and parameter-free SGs, which are used throughout this paper. We concisely introduce the formal foundations on this general class and indicate how these apply to subclasses. Most algorithmic approaches in this paper are not directly applicable to pSGs, but tailored to either pMDPs or pMCs. This is indicated when introducing these techniques.

Definition 4

(Stochastic game) A pSG $\mathcal {G}{}=(S{}, V{}, s_{ I }{}, Act {},\mathcal {P}{})$ is a stochastic game (SG) if $\mathcal {P}:S \times Act \times S \rightarrow [0,1]$ and $\sum _{s'\in S}\mathcal {P}(s,\alpha ,s') = 1$ for all $s \in S$ and $\alpha \in Act (s)$.

A state-action reward function $\text {rew}:S \times Act \rightarrow \mathbb {R}_{\ge 0}$ associates (non-negative, finite) rewards to outgoing actions. Analogously, Markov chains (MCs) and Markov decision processes (MDPs) are defined as special cases of pMCs and pMDPs, respectively. We use $D$ to denote a MC, $M$ for an MDP and $G$ for an SG.

2.3.2 Paths and reachability

An infinite path of a pSG $\mathcal {G}$ is an infinite sequence $\pi = s_0 \alpha _0 s_1 \alpha _1 \ldots$ of states $s_i\in S$ and actions $\alpha _i\in Act (s_i)$ with $\mathcal {P}(s_i,\alpha _i,s_{i+1})\not \equiv 0$ for $i\ge 0$. A finite path of a pSG $\mathcal {G}$ is a non-empty finite prefix $s_0 \alpha _0 \ldots s_n$ of an infinite path $s_0 \alpha _0 \ldots s_n \alpha _n\ldots$ of $\mathcal {G}$ for some $n\in \mathbb {N}$. Let $Paths ^{\mathcal {G}}$ denote the set of all finite or infinite paths of $\mathcal {G}$ while $Paths _{ fin }^{\mathcal {G}}\subseteq Paths ^{\mathcal {G}}$ denotes the set of all finite paths. For paths in (p)MCs, we omit the actions. The set $Paths ^{\mathcal {G}}(s)$ contains all paths that start in state $s\in S$. For a finite path $\pi \in Paths _{ fin }^{\mathcal {G}}$, $last (\pi )=s_n$ denotes the last state of $\pi$. The length $|\pi |$ of a path $\pi$ is $|\pi |=n$ for $\pi \in Paths _{ fin }^{\mathcal {G}{}}$ and $|\pi |=\infty$ for infinite paths. The accumulated reward along the finite path $s_0 \alpha _0 \ldots \alpha _{n-1} s_n$ is given by the sum of the rewards $\text {rew}(s_i,\alpha _i)$ for $0 \le i < n$.

We denote the set of states that can reach a set of states T as follows: $\lozenge T = \{s\in S\mid \exists \pi \in Paths _{ fin }^{\mathcal {G}}(s).\ last (\pi )\in T\}$. A set of states $T \subseteq S$ is reachable from $s\in S$, written $s\in \lozenge T$, iff there is a path from s to some $s'\in T$. A state s is absorbing iff $\mathcal {P}(s,\alpha ,s)=1$ for all $\alpha \in Act (s)$.

Example 9

The pMC in Fig. 5c has a path $\pi = s_0s_1s_3s_3$ with $|\pi | = 3$. Thus $s_0 \in \lozenge \{ s_3 \}$. There is no path from $s_4$ to $s_3$, so $s_4 \not \in \lozenge \{ s_3 \}$. States $s_3$ and $s_4$ are the only absorbing states.

2.3.3 Model instantiation

Instantiated parametric models are obtained by instantiating the rational functions in all transitions as in Definition 2.

Definition 5

(Instantiated pSG) For a pSG $\mathcal {G}{}=(S{}, V{}, s_{ I }{}, Act {},\mathcal {P}{}){}$ and instantiation u of V, the instantiated pSG at u is given by $\mathcal {G}[u]=(S,s_{ I }, Act ,\mathcal {P}[u])$ with $\mathcal {P}[u](s,\alpha ,s')=\mathcal {P}(s,\alpha ,s')[u]$ for all $s, s' \in S$ and $\alpha \in Act$.

The instantiation of the parametric reward function $\text {rew}$ at u is $\text {rew}[u]$ with $\text {rew}[u](s,\alpha ) = \text {rew}(s,\alpha )[u]$ for all $s\in S, \alpha \in Act$. Instantiating pMDP $\mathcal {M}$ and pMC $\mathcal {D}$ at u is denoted by $\mathcal {M}[u]$ and $\mathcal {D}[u]$, respectively.

Remark 2

The instantiation of a pSG at u is a pSG, but not necessarily an SG. This is due to the fact that an instantiation does not ensure that $\mathcal {P}(s, \alpha , \cdot )$ is a probability distribution. In fact, instantiation yields a transition function of the form $\mathcal {P}:S\times Act \times S\rightarrow \mathbb {R}\cup \{ \bot \}$. Similarly, there is no guarantee that the rewards $\text {rew}[u]$ are non-negative. Therefore, we impose restrictions on the parameter instantiations.

Definition 6

(Well-defined instantiation) An instantiation u is well-defined for a pSG $\mathcal {G}$ if the pSG $\mathcal {G}[u]$ is an SG.

The reward function $\text {rew}$ is well-defined on u if it does only associate non-negative reals to state-action pairs.

Example 10

Consider again the pMC in Fig. 5c. The instantiation u with $u(p)=\nicefrac 4 5$ and $u(q)=\nicefrac 3 5$ is well-defined and induces the MC $\mathcal {D}[u]$ depicted in Fig. 5d.

From now on, we silently assume that every pSG we consider has at least one well-defined instantiation. This condition can be assured through checking the satisfiability of the conditions in Def. 4, which we discuss in Sect. 4.2.

Our methods necessitate instantiations that are not only well-defined, but also preserve the topology of the pSG. In particular, we are interested in the setting where reachability between two states coincides for the pSG and the set of instantiations u we consider. We detail this discussion in Sect. 4.2.

Definition 7

(Graph preserving) A well-defined instantiation u for pSG $\mathcal {G}{}=(S{}, V{}, s_{ I }{}, Act {},\mathcal {P}{})$ is graph preserving if for all $s,s' \in S$ and $\alpha \in Act$,

$$\begin{aligned} \mathcal {P}(s,\alpha ,s') \not \equiv 0 \implies \mathcal {P}(s,\alpha ,s')[u] \in \mathbb {R}\setminus \{0\}. \end{aligned}$$

Example 11

The well-defined instantiation u with $u(p)=1$ and $u(q)=\nicefrac 3 5$ for the pMC in Fig. 5c is not graph preserving.

2.3.4 Resolving non-determinism

Strategies⁴ resolve the non-deterministic choices in stochastic games with at least one player. For the objectives considered here, it suffices to consider so-called deterministic strategies [136]; more general strategies can be found in [13, Ch. 10]. We define strategies for pSGs and assume well-defined instantiations as in Definition 6.

Definition 8

(Strategy) A (deterministic) strategy $\sigma _i$ for player $i\in \{\bigcirc ,{\Box }\}$ in a pSG $\mathcal {G}$ with state space $S = S_{\bigcirc }\uplus S_{\Box }$ is a function

$$\begin{aligned} \sigma _i:\{\pi \in Paths _{ fin }^{\mathcal {G}} \mid last (\pi )\in S_i\}\rightarrow Act \end{aligned}$$

such that $\sigma _i(\pi )\in Act ( last (\pi ))$. Let $Str ^\mathcal {G}$ denote the set of strategies $\sigma = (\sigma _{\bigcirc },\sigma _{\Box })$ for pSG $\mathcal {G}$ and $Str ^{\mathcal {G}}_i$ the set of strategies of player i.

A pMDP has only a player-i strategy for the player with $S_i \ne \emptyset$; in this case the index i is omitted. A player-i strategy $\sigma _i$ is memoryless if $last (\pi )= last (\pi ')$ implies $\sigma _i(\pi )=\sigma _i(\pi ')$ for all finite paths $\pi ,\pi '$. A memoryless strategy can thus be written in the form $\sigma _i:S_i\rightarrow Act$. A pSG-strategy $\sigma =(\sigma _{\bigcirc },\sigma _{\Box })$ is memoryless if both $\sigma _{\bigcirc }$ and $\sigma _{\Box }$ are memoryless.

Remark 3

From now on, we only consider memoryless strategies and refer to them as strategies.

A strategy $\sigma$ for a pSG resolves all non-determinism and results in an induced pMC.

Definition 9

(Induced pMC) The pMC $\mathcal {G}^\sigma$ induced by strategy $\sigma = (\sigma _{\bigcirc },\sigma _{\Box })$ on pSG $\mathcal {G}{}=(S{}, V{}, s_{ I }{}, Act {},\mathcal {P}{})$ equals $(S, V, s_{ I }, P^\sigma )$ with:

$$\begin{aligned} P^\sigma (s,s')= {\left\{ \begin{array}{ll} \mathcal {P}(s,\sigma _{\bigcirc }(s),s') \quad \text{ if } s \in S_{\bigcirc }\\ \mathcal {P}(s,\sigma _{\Box }(s),s') \quad \text{ if } s\in S_{\Box }. \end{array}\right. } \end{aligned}$$

Example 12

Let $\sigma$ be a strategy for the pSG $\mathcal {G}$ in Fig. 5a with $\sigma _{\bigcirc }(s_0) = \alpha$ and $\sigma _{\Box }(s_2)=\beta$. The induced pMC $\mathcal {G}^{\sigma }$ equals pMC $\mathcal {D}$ in Fig. 5c. Analogously, imposing strategy $\sigma '$ with $\sigma '(s_0) = \alpha$ on the pMDP in Fig. 5b yields $\mathcal {M}^{\sigma '} = \mathcal {D}$.

The notions of strategies for pSGs and pMDPs and of induced pMCs naturally carry over to non-parametric models; e.g., the MC $G^\sigma$ is induced by strategy $\sigma \in Str ^G$ on SG $G$.

2.4 Specifications and solution functions

2.4.1 Specifications

Specifications constrain the measures of interest for (parametric) probabilistic models. Before considering parameters, let us first consider MCs. Let $D{}=(S{},s_{ I }{},\mathcal {P}{})$ be an MC and $T\subseteq S$ a set of target states that (without loss of generality) are assumed to be absorbing. Let $\lozenge T$ denote the path property to reach T⁵. Furthermore, the probability measure $\text {Pr}_s$ over sets of paths can be defined using a cylinder construction with $\text {Pr}_s(s_0\alpha _0\ldots s_n)=\Pi _{i=0}^{n-1}\mathcal {P}(s_i,\alpha _i,s_{i+1})$, see [13, Ch. 10].

We consider three kinds of specifications:

• 1. Unbounded probabilistic reachability A specification $\mathbb {P}_{\le \lambda }(\lozenge \, T)$ asserts that the probability to reach T from the initial state $s_{ I }$ shall be at most $\lambda$, where $\lambda \in {\mathbb {Q}} \cap [0,1]$. More generally, specification $\varphi ^r$ is satisfied by MC $D$, written:

  $$\begin{aligned} D\models \mathbb {P}_{\sim \lambda }(\lozenge \, T) \quad \text{ iff } \quad \text {Pr}^{D}_{s_{ I }}(\lozenge \, T) \sim \lambda , \end{aligned}$$

  where $\text {Pr}_{s_{ I }}^D(\lozenge \, T)$ is the probability mass of all infinite paths that start in $s_{ I }$ and visit any state from T.

• 2. Bounded probabilistic reachability In addition to reachability, these specifications impose a bound on the maximal number of steps until reaching a target state. Specification $\varphi ^b = \mathbb {P}_{\sim \lambda }(\lozenge ^{\le n}\, T)$ asserts that in addition to $\mathbb {P}_{\sim \lambda }(\lozenge \, T)$, states in T should be reached within $n \in {\mathbb {N}}$ steps. The satisfaction of $\mathbb {P}_{\sim \lambda }(\lozenge ^{\le n}\, T)$ is defined similar as above.

• 3. Expected reward until a target The specification $\mathbb {E}_{\le \kappa }(\lozenge \, T)$ asserts that the expected reward until reaching a state in T shall be at most $\kappa \in {\mathbb {R}}$. Let $\text {ER}_{s_{ I }}^{D}(\lozenge \, T)$ denote the expected accumulated reward until reaching a state in $T \subseteq S$ from state $s_{ I }$. We obtain this reward by multiplying the probability of every path reaching T with the accumulated reward of that path, up until reaching T. Details are given in [13, Chapter 10].⁶. Then we define

  $$\begin{aligned} D\models \mathbb {E}_{\sim \kappa }(\lozenge \, T) \quad \text{ iff } \quad \text {ER}_{s_{ I }}^{D}(\lozenge \, T)\sim \kappa , \end{aligned}$$

  We do not treat the accumulated reward to reach a target within n steps, as this is not a very useful measure. In case there is a possibility to not reach the target within n steps, this yields $\infty$.

We omit the superscript $D$ if it is clear from the context. We write $\lnot \varphi$ to invert the relation: $D\models \lnot \mathbb {P}_{\le \lambda }(\lozenge \, T)$ is thus equivalent to $D\models \mathbb {P}_{>\lambda }(\lozenge \, T)$. An SG $G$ satisfies specification $\varphi$ under strategy $\sigma$ if the induced MC $G^{\sigma } \models \varphi$. Unbounded reachability and expected rewards are prominent examples of indefinite-horizon properties – they measure behaviour up-to some specified event (the horizon) which may be reached after arbitrarily many steps.

Remark 4

Bounded reachability in MDPs can be reduced to unbounded reachability by a technique commonly referred to as unrolling [5]. For performance reasons, it is sometimes better to avoid this unrolling, and present dedicated approaches.

2.4.2 Solution functions

Computing (unbounded) reachability probabilities and expected rewards for MCs reduces to solving linear equation systems [13] over the field of reals (or rationals). For parametric MCs, we obtain a linear equation system over the field of the rational functions over V instead. The solution to this equation system is a rational function. (See Examples 4 and 6 on pages 6 and 7). More details on the the solution function and the equation system follow in Sects. 5 and 6, respectively.

Definition 10

(Solution functions) For a pMC $\mathcal {D}{}=(S{}, V{}, s_{ I }{}, \mathcal {P}{})$, $T \subseteq S$ and $n\in \mathbb {N}$, a solution function for a specification $\varphi$ is a rational function

$$\begin{aligned} \begin{array}{rcl} f_{\mathcal {D},T}^r \in \mathbb {Q}(V) &{} \text{ for } &{} \varphi =\mathbb {P}_{\sim \lambda }(\lozenge \, T)\\ f_{\mathcal {D},T,n}^b \in \mathbb {Q}(V) &{} \text{ for } &{} \varphi =\mathbb {P}_{\sim \lambda }(\lozenge ^{\le n}\, T) \text{, } \text{ and }\\ f_{\mathcal {D},T}^e\in \mathbb {Q}(V) &{} \text{ for } &{} \varphi =\mathbb {E}_{\sim \kappa }(\lozenge \, T), \end{array} \end{aligned}$$

such that for every well-defined graph-preserving instantiation u:

$$\begin{aligned} f_{\mathcal {D},T}^r[u]&\ = \ \text {Pr}^{\mathcal {D}[u]}_{s_{ I }}(\lozenge \, T),\\ f_{\mathcal {D},T,n}^b[u]&\ = \ \text {Pr}^{\mathcal {D}[u]}_{s_{ I }}(\lozenge ^{\le n}\, T) \text{, } \text{ and } \\ f_{\mathcal {D},T}^e[u]&\ = \ \text {ER}_{s_{ I }}^{\mathcal {D}[u]}(\lozenge \, T). \end{aligned}$$

Fig. 6

Two sample parametric models

Example 13

Consider the reachability probability to reach $s_2$ for the pMC in Fig. 6a. Any instantiation u with $u(p),u(q)\in (0,1)$ is well-defined and graph-preserving. As the only two finite paths to reach $s_2$ are $s_0 s_2$ and $s_0 s_1 s_2$, we have $f_{\mathcal {D},\{s_2\}}^r = 1-p + p \cdot q$.

For pSGs (and pMDPs), the solution function depends on the resolution of non-determinism by strategies, i. e., they are defined on the induced pMCs. Formally, a solution function for a pSG $\mathcal {G}$, a reachability specification $\varphi ^r=\mathbb {P}_{\le \lambda }(\lozenge \, T)$, and a strategy $\sigma \in Str ^\mathcal {G}$ is a function $f^r_{\mathcal {G},\sigma ,T} \in \mathbb {Q}(V)$ such that for each well-defined graph-preserving instantiations u it holds:

$$\begin{aligned} f^r_{\mathcal {G},\sigma ,T}[u] = \text {Pr}^{\mathcal {G}^\sigma [u]}_{s_{ I }}(\lozenge \, T). \end{aligned}$$

These notions are defined analogously for bounded reachability (denoted $f^b_{\mathcal {G},\sigma ,T,n}$) and expected reward (denoted $f^e_{\mathcal {G},\sigma ,T}$) specifications.

Example 14

For the pMDP in Fig. 6b, the solution functions for reaching $s_2$ are $1{-}p + p \cdot q$, for the strategy $\sigma _\alpha =\{ s_0 \mapsto \alpha \}$, and 1 for the strategy $\sigma _\beta = \{ s_0 \mapsto \beta \}$.

Remark 5

We define solution functions only for graph-preserving valuations. For the more general well-defined solutions, a similar definition can be given [94] where (solution) functions are no longer rational functions but instead a collection of solution functions obtained on the graph-preserving subsets. In particular, unless a pMC is acyclic, such a function is only semi-continuous [97]. A key reason for the discontinuity is the change of states that are in $\lozenge T$, e.g., consider instantiations with $q=1$ in Fig. 5c. We provide the decomposition into graph-preserving subsets in Sect. 4.3.

2.5 Constraints and formulas

We consider (polynomial) constraints of the form $g \sim g'$ with $g,g' \in \mathbb {Q}[V]$ and $\sim \in \, \{<,\le ,=,\ge ,>\}$. We denote the set of all constraints over V with ${\mathcal {C}}[V]$. A constraint $g \sim g'$ can be equivalently formulated as $g - g' \sim 0$. A formula $\psi$ over a set of polynomial constraints is recursively defined: Each polynomial constraint is a formula, and the Boolean combination of formulae is also a formula.

Example 15

Let p, q be variables. $1-p\cdot q > 0$ and $p^2 < 0$ are constraints, $\lnot \left( p^2 < 0\right)$ and $\left( 1-p\cdot q > 0\right) \vee \left( p^2 < 0\right)$ are formulae.

The semantics of constraints are standard: i.e., an instantiation u satisfies $g \sim g'$ if $g[u] \sim g'[u]$. An instantiation satisfies $\psi \wedge \psi '$ if u satisfies both $\psi$ and $\psi '$. The semantics for other Boolean connectives are defined analogously. Moreover, we will write $g \ne g'$ to denote the formula $g < g' \vee g > g'$.

Checking whether there exists an instantiation that satisfies a formula is equivalent to checking membership of the existential theory of the reals [21]. Such a check can be automated using SMT-solvers capable of handling quantifier-free non-linear arithmetic over the reals [93], such as [52, 63].

Statements of the form $f \sim f'$ with $f,f' \in \mathbb {Q}(V)$ are not necessarily polynomial constraints: however, we are not interested in instantiations u with $f[u] = \bot$, and thus later (in Sect. 4.2.2) we can transform such constraints into formulae over polynomial constraints.

3 Formal problem statements

This section formalises the three problem statements mentioned in the introduction: the verification problem and two synthesis problems. We start off by making precise what regions are and how to represent them. We then define what it means for a region to satisfy a given specification. This puts all in place to making the three problem statements precise. Finally, it surveys the verification approaches that are detailed later in the paper.

3.1 Regions

Instantiated parametric models are amenable to standard probabilistic model checking. However, sampling an instantiation is very restrictive—verifying an instantiated model gives results for a single point in the (uncountably large) parameter space. A more interesting problem is to determine which parts of the parameter space give rise to a model that complies with the specification. Such sets of parameter values are, inspired by their geometric interpretation, called regions. Regions are solution sets of conjunctions of constraints over the set V of parameters.

Definition 11

(Region) A region R over V is a set of instantiations of V (or dually a subset of $\mathbb {R}^{|V|}$) for which there exists a set $C(R) \subseteq {\mathcal {C}}[V]$ of polynomial constraints such that for their conjunction $\Upphi (R)=\bigwedge _{c \in C(R)} c$ we have

$$\begin{aligned} R \ = \ \{ u \mid \Upphi (R)[u]\}. \end{aligned}$$

We call C(R) the representation of R.

Any region which is a subset of a region R is called a subregion of R.

Example 16

Let the region R over $V=\{ p, q\}$ be described by

$$\begin{aligned} C(R)=\{ p^2 + q^2 - 1 \le 0 ,\ p+q-1 \le 0\}. \end{aligned}$$

Thus, $R = \{\, u \mid (p^2{+}q^2{-}1)[u] \le 0 \wedge (p{+}q{-}1)[u] \le 0 \, \}$. The region R contains the instantiation $u = (\nicefrac 2 5, \nicefrac 3 5)$ as $(\nicefrac 2 5)^2 + (\nicefrac 3 5)^2 - 1 \le 0$ and $\nicefrac 2 5+\nicefrac 3 5 - 1 \le 0$. The instantiation $u' = (\nicefrac 1 2, \nicefrac 3 5) \not \in R$ as $\nicefrac 1 2 + \nicefrac 3 5 - 1 > 0$. Regions do not have to describe a contiguous area of the parameter space; e.g., consider the region $R'$ described by $\{{-}p^2 + 1 < 0\}$ is $R'=(-\infty ,-1]\cup [1,+\infty )$.

Regions are semi-algebraic sets [21] which yield the theoretical formalisation of notions such as distance, convexity, etc. It also ensures that regions are well-behaved: Informally, a region in the space $\mathbb {R}^n$ is given by a finite number of connected semialgebraic sets (cells⁷), and (the boundaries of) each cell can be described by a finite set of polynomials. The size $\Vert R \Vert$ of a region R is given by the Lebesgue measure. All regions are Lebesgue measurable.

A region is called well-defined if all its instantiations are well defined.

Definition 12

(Well-defined region) Region R is well defined for pSG $\mathcal {G}$ if for all $u\in R$, u is a well-defined valuation for $\mathcal {G}$.

3.2 Angelic and demonic satisfaction relations

As a next step towards our formal problem statements, we have to define what it means for a region to satisfy a specification. We first introduce two satisfaction relations—angelic and demonic—for parametric Markov models for a single instantiation. We then lift these two notions to regions.

Definition 13

(Angelic and demonic satisfaction relations) For pSG $\mathcal {G}$, well-defined instantiation u, and specification $\varphi$, the satisfaction relations $\models _a$ and $\models _d$ are defined by:

$$\begin{aligned}&\mathcal {G}, u \models _a \varphi \quad \text{ iff } \quad \exists \sigma \in Str ^{\mathcal {G}}.\ \mathcal {G}[u]^\sigma \models \varphi \quad (\text {angelic})\\&\mathcal {G}, u \models _d \varphi \quad \text{ iff } \quad \forall \sigma \in Str ^{\mathcal {G}}.\ \mathcal {G}[u]^\sigma \models \varphi \quad (\text {demonic}). \end{aligned}$$

The angelic relation $\models _a$ refers to the existence of a strategy to fulfil the specification $\varphi$, whereas the demonic counterpart $\models _d$ requires all strategies to fulfil $\varphi$. Observe that $\mathcal {G}, u \not \models _a \varphi$ if and only if $\mathcal {G}, u \models _d \lnot \varphi$. Thus, demonic and angelic can be considered to be dual. By $\models _\heartsuit$ we denote the dual of $\models _\clubsuit$, that is, if $\clubsuit =a$ then $\heartsuit =d$ and vice versa. For pMCs, the relations $\models _a$ and $\models _d$ coincide and the subscripts a and d are omitted.

Example 17

Consider the pMDP $\mathcal {M}$ in Fig. 6b, instantiation $u = (\nicefrac 1 2, \nicefrac 1 2)$ and $\varphi = \mathbb {P}_{> \nicefrac 4 5}(\lozenge \{s_2\})$. We have $\mathcal {M}, u \models _a \varphi$, as for strategy $\sigma _\beta = \{ s_0 \mapsto \beta \}$ the state $s_2$ is reached with probability one; thus, $\mathcal {M}[u]^{\sigma _\beta } \models \varphi$. However, $\mathcal {M}, u \not \models _d \varphi$, as for strategy $\sigma _\alpha = \{ s_0 \mapsto \alpha \}$, we have $(1{-}p+p \cdot q)[u] = \nicefrac 3 4 \not > \nicefrac 4 5$; thus, $\mathcal {M}[u]^{\sigma _\alpha } \not \models \varphi$. By duality, $\mathcal {M}, u \models _a \lnot \varphi$.

We now lift these two satisfaction relations to regions. The aim is to consider specifications $\varphi$ that hold for all instantiations represented by a region R of a parametric model $\mathcal {G}$. This is captured by the following satisfaction relation.

Definition 14

(Satisfaction relation for regions) For pSG $\mathcal {G}$, well-defined region R, and specification $\varphi$, the relation $\models _\clubsuit$, $\clubsuit \in \{a, d\}$, is defined as:

$$\begin{aligned} \mathcal {G}, R \models _\clubsuit \varphi \quad \text{ iff } \quad \mathcal {G}, u \models _\clubsuit \varphi \text { for all } u\in R. \end{aligned}$$

Before we continue, we note the difference between $\mathcal {G}, R \not \models _\clubsuit \varphi$ and $\mathcal {G}, R \models _\clubsuit \lnot \varphi$:

$$\begin{aligned} \mathcal {G}, R \models _\clubsuit \lnot \varphi \text { implies }\mathcal {G}, u \models _\clubsuit \lnot \varphi \text { for} ~all ~ u\in R, \end{aligned}$$

whereas in constrast,

$$\begin{aligned} \mathcal {G}, R \not \models _\clubsuit \varphi \text { implies }\mathcal {G}, u \not \models _\clubsuit \varphi \text { for}~ some ~ u \in R. \end{aligned}$$

Definition 15

(Accepting/rejecting/inconsistent region) A well-defined region R is accepting (for $\mathcal {G}$, $\varphi$, $\clubsuit$) if $\mathcal {G}, R \models _\clubsuit \varphi$. Region R is rejecting (for $\mathcal {G}$, $\varphi$, $\clubsuit$) if $\mathcal {G}, R \models _{\heartsuit } \lnot \varphi$. Region R is inconsistent if it is neither accepting nor rejecting.

By the duality of $\models _a$ and $\models _d$, a region is thus rejecting iff $\forall u\in R.~\mathcal {G}, u \not \models _{\clubsuit } \varphi$. Note that this differs from $\mathcal {G}, R \not \models _\clubsuit \varphi$.

Example 18

Reconsider the pMDP in Fig. 6b, with $R = [\nicefrac 2 5, \nicefrac 1 2] \times [\nicefrac 2 5, \nicefrac 1 2]$ and $\varphi = \mathbb {P}_{>\nicefrac 4 5}(\lozenge \{s_2\})$. The corresponding solution functions are given in Example 14. It follows that:

• $\mathcal {M}, R \models _a \varphi$, as for strategy $\sigma _\beta = \{ s_0 \mapsto \beta \}$, we have $\mathcal {M}^{\sigma _\beta }, u \models \varphi$ for all $u\in R$.

• $\mathcal {M}, R \not \models _d \varphi$, as for strategy $\sigma _\alpha = \{ s_0 \mapsto \alpha \}$, $\mathcal {M}^{\sigma _\alpha }, u \not \models \varphi$ for $u = (\nicefrac 1 2, \nicefrac 1 2)$.

• $\mathcal {M}, R \models _a \lnot \varphi$ using strategy $\sigma _\alpha$.

Regions can be inconsistent w. r. t. a relation, and consistent w. r. t. its dual relation. The region $(0,1) \times (0,1)$ is inconsistent for $\mathcal {M}$ and $\models _d$, as for both $\varphi$ and $\lnot \varphi$, there is a strategy that is not accepting. For $\models _a$, there is a single strategy which accepts $\varphi$; other strategies do not affect the relation.

As an example of an accepting region under the demonic relation, consider $R' = [\nicefrac 4 5, \nicefrac 9 {10}] \times [\nicefrac 2 5, \nicefrac 9 {10}]$. We have $\mathcal {M}, R' \models _d \varphi$, as for both strategies, the induced probability is always exceeding $\nicefrac 4 5$.

3.3 Formal problem statements

We are now in a position to formalise the two synthesis problems and the verification problem from the introduction, page 5. We present the formal problem statements in the order of treatment in the rest of the paper.

Remark 6

The solution function for pMCs precisely describes how (graph-preserving) instantiations map to the relevant measure. Therefore, comparing the solution function with the threshold divides the parameter space into an accepting region $R_a$ and a rejecting region $R_r$ and defines the exact result for the formal synthesis problem. Recall also Example 4.

The verification procedure allows us to utilise an approximate synthesis problem in which verification procedures are used as a backend.

Note that no requirements are imposed on the (unknown, open) region $R_o$.

Remark 7

By definition, the angelic satisfaction relation for region R and pSG $\mathcal {G}$ is equivalent to:

$$\begin{aligned} \mathcal {G}, R \models _a \varphi \quad \text {if and only if} \quad \forall u \in R.~\exists \sigma \in Str ^\mathcal {G}.~\mathcal {G}^\sigma , u \models \varphi . \end{aligned}$$

An alternative notion in parameter synthesis is the existence of a robust strategy:

$$\begin{aligned} \exists \sigma \in Str ^\mathcal {G}.~\forall u \in R.~ \mathcal {G}^\sigma , u \models \varphi . \end{aligned}$$

Note the swapping of quantifiers compared to $\models _a$. That is, $\mathcal {G}, R \models _a \varphi$ considers potentially different strategies for different parameter instantiations $u \in R$. The notion of robust strategies leads to a series of quite orthogonal challenges. For instance, the notion is not compositional, i.e., if robust strategies exist in $R_1$ and $R_2$, then we cannot conclude the existence of a robust strategy in $R_1 \cup R_2$. Moreover, memoryless strategies are not sufficient, see [9]. Robust strategies are outside the scope of this paper and are only shortly mentioned in Sect. 8.

3.4 A bird’s eye view on the verification procedures

In the later sections, we will present several techniques that decide the verification problem for pMCs and pMDPs. (Recall that stochastic games were only used to define the general setting.)

The verification problem is used to analyse the regions of interest. The assumption that this region contains only well-defined instantiations is therefore natural. It can be checked algorithmically as described in Sect. 4.2 below. Many verification procedures require that the region is graph preserving. A decomposition result of well-defined into graph-preserving regions is given in Sect. 4.3.

Section 6 presents two verification procedures. The first one directly solves the non-linear equation system, see Example 6, as an SMT query. The second procedure reformulates the SMT query using the solution function. While this reformulation drastically reduces the number of variables in the query, it requires an efficient computation of the solution function, as described in Sect. 5.

Section 7 covers an approximate and more efficient verification procedure, called parameter lifting, which is tailored to multi-linear functions and closed rectangular regions. Under these mild restrictions, the verification problem for pMCs (pMDPs) can be approximated using a sequence of standard verification analyses on non-parametric MDPs (SGs) of similar size, respectively. The key steps here are to relax the parameter dependencies, and consider lower- and upper-bounds of parameters as worst and best cases.

4 Regions

Section 3.1 already introduced regions. This section details specific region structures such as linear, rectangular and graph-preserving regions. It then presents procedures to check whether a region is graph preserving. Finally, we describe how well-defined but not graph-preserving regions can be turned into several regions that are graph preserving.

4.1 Regions with specific structure

As defined before, a region R is a (typically uncountably infinite) set of parameter valuations described by a set C(R) of polynomial constraints. Two classes of regions are particularly relevant: linear and rectangular regions.

Definition 16

(Linear region) A region with representation C(R) is linear if for all $g\sim 0 \in C(R)$, the polynomial $g$ is linear.

Linear regions describe convex polytopes. We refer to the vertices (or angular points) of the polytope as the region vertices.

Definition 17

(Rectangular region) A region R with representation

$$\begin{aligned} C(R) \ = \ \bigcup _{i=1}^{|V|} \{ \, {-}p_i + a_i \unlhd _i^1 0, p_i + b_i \unlhd _i^2 0 \, \} \end{aligned}$$

with $a_i \le b_i \in \mathbb {Q}$ and $\unlhd ^j_i \in \{ <, \le \}$ for $0 < i \le |V|$ and $j \in \{ \, 1,2 \, \}$ is called rectangular. A rectangular region is closed if all inequalities $\unlhd _i^j$ in the constraints in C(R) are non-strict.

Rectangular regions are hyper-rectangles and a subclass of linear regions. A closed rectangular region R can be represented as with parameter intervals $[a_p, b_p]$ described by the bounds $a_p$ and $b_p$ for all $p \in V$. For a region R, we refer to the bounds of parameter p by $B_R(p) = \{ a_p, b_p \}$ and to the interval of parameter p by $I_R(p) = [a_p,b_p]$. We may omit the subscript R, if it is clear from the context. For a rectangular region R, the size $\Vert R\Vert$ equals $\prod _{p \in V} (b_p - a_p)$.

Regions represent sets of instantiations $\mathcal {G}[u]$ of a pSG $\mathcal {G}$. The notion of graph-preservation from Definition 7 lifts to regions in a straightforward manner:

Definition 18

(Graph-preserving region) Region R is graph preserving for pSG $\mathcal {G}$ if for all $u\in R$, u is a graph-preserving valuation for $\mathcal {G}$.

By this definition, all instantiations from graph-preserving regions have the same topology as the parametric model, cf. Remark 8 below. In addition, all such instantiations are well-defined.

Example 19

Let $\mathcal {D}$ be the pMC in Fig. 5c, $R = [\nicefrac 1 {10}, \nicefrac 4 5] \times [\nicefrac 2 5, \nicefrac 7 {10}]$ be a (closed rectangular) region, and instantiation $u = (\nicefrac 4 5, \nicefrac 3 5) \in R$. Figure 5d depicts the instantiation $\mathcal {D}[u]$, an MC with the same topology as $\mathcal {D}$. As the topology is preserved for all possible instantiations $\mathcal {D}[u']$ with $u' \in R$, the region R is graph preserving. The region $R'=[0,1] \times [0,1]$ is not graph preserving as, e.g., the instantiation $(0,0) \in R'$ results in an MC that has no transition from state $s_1$ to $s_2$.

Remark 8

Graph-preserving regions have the nice property that if

$$\begin{aligned} \exists u \in R, \mathcal {G},u \models _\clubsuit \mathbb {P}_{=1}(\lozenge \, T)\text { implies }\mathcal {G}, R \models _\clubsuit \mathbb {P}_{=1}(\lozenge \, T). \end{aligned}$$

This property can be checked by standard graph analysis [13, Ch. 10]. It is thus straightforward to check $\mathcal {G}, R \models _\clubsuit \mathbb {P}_{=1}(\lozenge T)$, an important precondition for computing expected rewards. In the rest of this paper when considering expected rewards, it is assumed that within a region the probability to reach a target is one.

The following two properties of regions are frequently (and often implicitly) used in this paper.

Lemma 1

(Characterisation for inconsistent regions) For any inconsistent region R it holds that $R = R_a \cup R_r$ for some accepting $R_a \ne \emptyset$ and rejecting $R_r \ne \emptyset$.

Lemma 2

(Compositionality) Region $R = R_1 \cup R_2$ is accepting (rejecting) if and only if both $R_1$ and $R_2$ are accepting (rejecting).

The statements follow from the universal quantification over all instantiations in the definition of $\models _\clubsuit$.

4.2 Checking whether a region is graph preserving

The verification problem for region R requires R to be well-defined. We first address the problem on how to check this condition. In fact, we present a procedure to check graph preservation which is slightly more general and useful later, see also Remark 8. To show that region R is not graph preserving, a point in R suffices that violates the conditions in Definition 7. Using the representation of region R, the implication

$$\begin{aligned} \Upphi (R) \implies R \text { graph preserving} \end{aligned}$$

needs to be valid since any violating assignment corresponds to a non-graph-preserving instantiation inside R. Technically, we consider satisfiability of the conjunction of:

• the inequalities C(R) representing the candidate region, and

• a disjunction of (in)equalities describing violating graph-preserving.

This conjunction is satisfiable if and only if the region is not graph preserving.

4.2.1 Graph preservation for polynomial transition functions

Let us consider the above for pSGs with polynomial transition functions. The setting for pSGs with rational functions is discussed at the end of this section. The following constraints (1)–(4), which we denote $\textit{GP}$, capture the notion of graph preservation:

$$\begin{aligned}&\bigwedge _{\begin{array}{c} s,s'\in S,\alpha \in Act (s)\\ \mathcal {P}(s,\alpha ,s') \not \equiv 0 \end{array}} 0 \le \mathcal {P}(s,\alpha ,s') \le 1 \, \end{aligned}$$

(1)

$$\begin{aligned}&\quad \wedge \bigwedge _{s\in S,\alpha \in Act (s)} \sum _{s'\in S} \mathcal {P}(s,\alpha ,s') = 1 \end{aligned}$$

(2)

$$\begin{aligned}&\quad \wedge \bigwedge _{s\in S,\alpha \in Act (s)} \text {rew}(s,\alpha ) \ge 0 \end{aligned}$$

(3)

$$\begin{aligned}&\quad \wedge \bigwedge _{\begin{array}{c} s,s'\in S,\alpha \in Act (s)\\ \mathcal {P}(s,\alpha ,s') \not \equiv 0 \end{array}} 0 < \mathcal {P}(s,\alpha ,s') . \end{aligned}$$

(4)

The constraints ensure that (1) all non-zero entries are evaluated to a probability, (2) transition probabilities are probability distributions, (3) rewards are non-negative, and (4) non-zero entries remain non-zero. The constraints (1)–(3) suffice to ensure well-definedness. The constrains (1)–(4) can be simplified to:

$$\begin{aligned}&\bigwedge _{\begin{array}{c} s,s'\in S,\alpha \in Act (s)\\ \mathcal {P}(s,\alpha ,s') \not \equiv 0 \end{array}}{} & {} \mathcal {P}(s,\alpha ,s') > 0 \\ \wedge&\bigwedge _{s\in S,\alpha \in Act (s)}{} & {} \sum _{s'\in S} \mathcal {P}(s,\alpha ,s') = 1 \\ \wedge&\bigwedge _{s\in S,\alpha \in Act (s)}{} & {} \text {rew}(s,\alpha ) \ge 0. \end{aligned}$$

Example 20

Recall the pMC from Fig. 5c.

$$\begin{aligned} \textit{GP} = \quad&p> 0 \,\wedge \, 1{-}p> 0 \, \wedge \, p{+}1{-}p = 1 \ \wedge \ q> 0 \, \wedge \, 1{-}q > 0 \, \wedge \, q{+}1{-}q = 1. \end{aligned}$$

This equation simplifies to $0< p< 1 \wedge 0< q < 1$. To check whether the region R described by $\Upphi (R) = \nicefrac 1 {{10}} \le p \le \nicefrac 4 5 \wedge \nicefrac 2 5 \le q \le \nicefrac 7 {10}$ is graph preserving, we check whether the conjunction $\Upphi (R) \wedge \lnot \textit{GP}$ is satisfiable, with

$$\begin{aligned} \lnot \textit{GP} \ = \ p \le 0 \vee p \ge 1 \vee q \le 0 \vee q \ge 1. \end{aligned}$$

As the conjunction is not satisfiable, the region R is graph preserving. Contrary, $R' = [0,1] \times [0,1]$ is not graph preserving as $u = (0,0)$ satisfies the conjunction $\Upphi (R') \wedge \lnot \textit{GP}$.

Satisfiability of $\textit{GP}$, or equivalently, deciding whether a region is graph preserving, is as hard as the existential theory of the reals [21], if no assumptions are made about the transition probability and reward functions. This checking can be automated using SMT-solvers capable of handling quantifier-free non-linear arithmetic over the reals [93]. The complexity drops to polynomial time once both the region R and all transition probability (and reward) functions are linear as linear programming has a polynomial complexity and the formula is then a disjunction over linear programs (with trivial optimisation functions).

4.2.2 Graph preservation for rational transition functions

In case the transition probability and reward function of a pSG are not polynomials, the left-hand side of the statements in (1)–(4) are not polynomials, and the statements would not be constraints. We therefore perform the following transformations on (1)–(4):

• Transforming equalities:

  

• Transforming inequalities $\unrhd \, \in \{ >, \ge \}$:

  

  with $c \in \mathbb {Q}$, and equals < for $\not>$ and $\le$ for $\not \ge$.

• Transforming $<, \le$ is analogous.

• Transforming $g \ne g'$ (i.e., $g < g' \vee g > g'$) involves transforming both disjuncts.

The result is a formula with polynomial constraints that correctly describes graph preservation (or well-definedness).

Example 21

Consider a state with outgoing transition probabilities q and $\frac{p}{1+p}$. The graph preservation statements are (after some simplification):

$$\begin{aligned} q> 0 \text { and } \frac{p}{1+p} > 0 \text { and } q + \frac{p}{1+p} = 1. \end{aligned}$$

Transforming the second item as explained above yields:

$$\begin{aligned} 1+p \ne 0 \wedge \Big ((1+p> 0 \wedge p > 0) \vee (1+p< 0 \wedge p < 0) \Big ) \end{aligned}$$

while transforming the third item yields:

$$\begin{aligned} (1+p \ne 0) \wedge q \cdot (1{+}p) -1 = 0. \end{aligned}$$

Finally, we obtain the following formula (after some further simplifications):

$$\begin{aligned} q>0 \;\wedge \; \left( p>0 \vee p<-1 \right) \;\wedge \; q \cdot (1+p) - 1 = 0. \end{aligned}$$

4.3 Reduction to graph-preserving regions

Fig. 7

Ensuring graph-preservation on subregions

In this section, we show how we can partition a well-defined region into a set of graph-preserving regions. This is useful, e.g., as we only define solution functions for graph-preserving regions. The decomposition in this section allows to define solution functions on each of these partitions, see also Remark 5. Before we illustrate the decomposition, we define sub-pSGs: Given two pSGs $\mathcal {G}{}=(S{}, V{}, s_{ I }{}, Act {},\mathcal {P}{})$ and $\mathcal {G}'=(S', V', s_{ I }', Act ',\mathcal {P}')$, $\mathcal {G}'$ is a sub-pSG of $\mathcal {G}$ if $S' \subseteq S$, $V'\subseteq V$, $s_{ I }'=s_{ I }\in S'$, $Act ' \subseteq Act$, and $\mathcal {P}'(s,\alpha ,s')\in \{ \mathcal {P}(s,\alpha ,s'), 0 \}$ for all $s,s'\in S'$ and $\alpha \in Act '$. Note that for a given state $s\in S$ and action $\alpha \in Act (s)$, the sub-pSG might not contain s or $\alpha$ might not be enabled in s, but it is also possible that the sub-pSG omits some but not all successors of $\alpha$ in s.

Example 22

Reconsider the pMC $\mathcal {D}$ from Fig. 5c, and let $R=[0,1] \times [0,1]$, which is well-defined but not graph preserving. Region R can be partitioned into 9 regions, see Fig. 7a where each dot, line segment, and the inner region are subregions of R. All subregions are graph preserving on some sub-pMC of $\mathcal {D}$. Consider, e.g., the line-region $R' = \{ u \in R \mid p[u] = 0 \}$. The subregion $R'$ is not graph preserving on pMC $\mathcal {D}$, as the transition $s_0 \xrightarrow {p} s_1$ vanishes when $p=0$. However, $R'$ is graph preserving on the sub-pMC $\mathcal {D}'$ in Fig. 7b, which is obtained from $\mathcal {D}$ by removing the transitions on the line-region $p{=}0$.

Let us formalise the construction from this example. For a given well-defined region R, and pSG $\mathcal {G}$, let ${\mathcal {Z}}_R$ describe the set of constraints:

$$\begin{aligned} \begin{array}{l@{}l} \{ \mathcal {P}(s,\alpha ,s') {=} 0\ |\&s, s' \in S\wedge \alpha \in Act (s)\wedge \mathcal {P}(s,\alpha ,s')\not \equiv 0\wedge \exists u \in R.\,\mathcal {P}(s,\alpha ,s')[u] = 0 \}. \end{array} \end{aligned}$$

For $X \subseteq {\mathcal {Z}}_R$, the subregion $R_X \subseteq R$ is defined as:

$$\begin{aligned} \Upphi (R_X) \ = \Upphi (R) \wedge \bigwedge _{c \in X} c \wedge \bigwedge _{c \in {\mathcal {Z}}_R \setminus X} \lnot c. \end{aligned}$$

It follows that X uniquely characterises which transition probabilities in $\mathcal {G}$ are set to zero. In fact, each instance in $R_X$ is graph preserving for the unique sub-pSG $\mathcal {G}'$ of $\mathcal {G}$ obtained from $\mathcal {G}$ by removing all zero-transitions in $R_X$. The pSG $\mathcal {G}'$ is well-defined as R on $\mathcal {G}$ is well-defined. By construction, it holds that $\mathcal {G}[u] = \mathcal {G}'[u]$ for all instantiations $u \in R'$.

5 Exact synthesis by computation of the solution function

This section discusses how to compute the solution function. The solution function for pMCs describes the exact accepting and rejecting regions, as discussed in Sect. 3.3⁸. This section thus provides an algorithmic approach to the exact synthesis problem. In Sect. 6, we will also see that the solution function may be beneficial for the performance of SMT-based (region) verification.

The original approach to compute the solution function of pMCs is via state elimination [64, 78], and is analogous to the computation of regular expressions from nondeterministic finite automata (NFAs) [90]. It is suitable for a range of indefinite-horizon properties. The core idea behind state elimination and the related approaches presented here is based on two operations:

Fig. 8

Essential ideas for state elimination

• Adding short-cuts: Consider the pMC-fragment in Fig. 8a. The reachability probabilities from any state to t are as in Fig. 8b, where we replaced the transition from s to $s'$ by shortcuts from s to t and all other successors of $s'$, bypassing $s'$. By successive application of shortcuts, any path from the initial state to the target state eventually has length 1.

• Elimination of self-loops: A prerequisite for introducing a short-cut is that the bypassed state is loop-free. Recall that the probability of staying forever in a non-absorbing state is zero, and justifies elimination of self-loops by rescaling all other outgoing transitions, as depicted in the transition from Fig. 8c, d.

The remainder of this section is organised as follows: Sect. ?? recaps the original state elimination approach in Sect. 5.1, albeit slightly rephrased. The algorithm is given for (indefinite) reachability probabilities, expected rewards, and bounded reachability probabilities. In the last part, we present alternative, equivalent formulations which sometimes allow for superior performance. In particular, Sect. 5.2 clarifies the relation to solving a linear equation system over a field of rational functions, and Sect. 5.3 discusses a variation of state elimination applicable to pMCs described by multi-terminal binary decision diagrams.

5.1 Algorithm based on state elimination

Let $T \subseteq S$ be a set of target states and assume w. l. o. g.  that all states in T are absorbing and that $s_{ I }\not \in T$.

5.1.1 Reachability probabilities

We describe the algorithm to compute reachability probabilities based on state elimination in Algorithm 1. In the following, $\mathcal {P}$ is the transition matrix. The function eliminate_selfloop$(\mathcal {P}, s)$ rescales all outgoing probabilities of a non-absorbing state s by eliminating its self-loop. The function eliminate_transition($\mathcal {P}, s_{1}, s_{2}$) adds a shortcut from $s_1$ to the successors of $s_2$. Both operations preserve reachability to T. The function eliminate_state$(\mathcal {P}, s)$ “bypasses” a state s by adding shortcuts from all its predecessors. More precisely, we eliminate the incoming transitions of s, and after all incoming transitions are removed, the state s is unreachable. It is thereby effectively removed from the model.

After removing all non-absorbing, non-initial states $S^?$, the remaining model contains only self-loops at the absorbing states and transitions emerging from the initial state. Eliminating the self-loop on the initial state (by rescaling) yields a pMC. In this pMC, after a single step, an absorbing state is reached. These absorbing states are either a target or a sink. The solution function is then the sum over all (one-step) transition probabilities to target states.

Algorithm 1

State elimination for pMCs

Fig. 9

State elimination exemplified

Example 23

Consider again the pMC from Example 8, also depicted in Fig. 9a. Assume state $s_2$ is to be eliminated. Applying the function eliminate_state($\mathcal {P}, s_2$), we first eliminate the transition $s_1 \rightarrow s_2$, which yields Fig. 9b, and subsequently eliminate the transition $s_0 \rightarrow s_2$ (Fig. 9c). State $s_2$ is now unreachable, so we can eliminate $s_2$, reducing computational effort when eliminating state $s_1$. For state $s_1$, we first eliminate the self-loop (Fig. 9e) and then eliminate the transition $s_0 \rightarrow s_1$. The final result, after additionally removing the now unreachable $s_1$, is depicted in Fig. 9f. The result, i.e., the probability to eventually reach $s_3$ from $s_0$ in the original model, can now be read from the single transition between these two states.

As for computing of regular expressions from NFAs, the order in which the states are eliminated is essential. Computing an optimal order with respect to minimality of the result, however, is already NP-hard for acyclic NFAs, see [84]. For state elimination on pMCs, the analysis is more intricate, as the cost of every operation crucially depends on the size and the structure of the rational functions. We briefly discuss the implemented heuristics in Sect. 10.2.1.

Remark 9

The elimination of self-loops yields a rational function. In order to keep these functions as small as possible, it is natural to eliminate common factors of the numerator and the denominator. Such a reduction, however, involves the computation of greatest common divisors (gcds). This operation is expensive for multivariate polynomials. In [91], data structures to avoid their computation are introduced, in [17] a method is presented that mostly avoids introducing common factors.

5.1.2 Expected rewards

The state elimination approach can also be adapted to compute expected rewards [78]. When eliminating a state s, in addition to adjusting the probabilities of the transitions from all predecessors $s_1$ of s to all successors $s_2$ of s, it is also necessary to “summarise” the reward that would have been gained from $s_1$ to $s_2$ via s. The presentation in [78] describes these operations on so-called transition rewards. Observe that for the analysis of expected rewards in MCs, we can always reformulate transition rewards in terms of state rewards. We preprocess pMCs to only have rewards at the states: this adjustment simplifies the necessary operations considerably.

The treatment of the expected reward computation is easiest from an adapted (and more performant) implementation of state elimination, as outlined in Algorithm 2. Here, we eliminate the probabilities to reach a target state in exactly one step, and collect these probabilities in a vector x which we refer to as one-step-probabilities. Then, we proceed similar as before. However, the elimination of a transition from $s_1$ to s now has two effects: it updates the probabilities within the non-target states as before, and (potentially) updates the probability $x(s_1)$ to reach the target within one step from $s_1$ (with the probability that the target was reached via s in two steps). Upon termination of the outer loop, the vector x contains the probabilities from all states to reach the target, that is, $x(s_i) = x_{s_i}$.

Finally, when considering rewards, the one-step-probabilities contain initially the rewards for the states. Eliminating a transition then moves the (expected) reward to the predecessors by the same sequence of arithmetic operations.

Algorithm 2

State elimination with one-step probabilities

5.1.3 Bounded reachability

As discussed in Remark 4, bounded reachability can typically be considered by an unfolding of the Markov model and considering an unbounded reachability property on that (acyclic) unfolding. In combination with state elimination, that yields the creation of many states that are eliminated afterwards, and does not take into account any problem-specific properties. Rather, and analogous to the parameter-free case [13], it is better to do the adequate matrix-vector multiplication (# number of steps often). The matrix originates from the transition matrix, the vector (after i multiplications) encodes the probability to reach a state within i steps.

5.2 Algorithm based on solving the linear equation system

The following set of equations is a straightforward adaption of the Bellman linear equation system for MCs found in, e.g., [13, 121] to pMCs. For each state s, a variable $x_s$ is used to express the probability $\text {Pr}_s(\lozenge T)$ to reach a state in T from the state s. Recall that we overloaded $\lozenge T$ to also denote the set of states from which T is reachable (with positive probability). Analogously, we use $\lnot \lozenge T$ to denote the set of states from which T is not reachable, i. e., $\lnot \lozenge T=S\setminus \lozenge T$. We have:

$$\begin{aligned} x_s&= 0&\quad&\forall s \in \lnot \lozenge T \end{aligned}$$

(5)

$$\begin{aligned} x_s&= 1&\quad&\forall s \in T \end{aligned}$$

(6)

$$\begin{aligned} x_s&= \sum _{s' \in S} \mathcal {P}(s,s') \cdot x_{s'}&\quad&\forall s \in \lozenge T \setminus T. \end{aligned}$$

(7)

This system of equations has a unique solution for every well-defined parameter instantiation. In particular, the set of states satisfying $\lnot \lozenge T$ is the same for all well-defined graph-preserving parameter instantiations, as instantiations that maintain the graph of the pMC do not affect the reachability of states in T.

For pMCs, the coefficients are no longer from the field of the real numbers, but rather from the field of rational functions.

Example 24

Consider the equations for the pMC from Fig. 9a.

$$\begin{aligned} x_0 =&~ p \cdot x_1 + (1-p) \cdot x_2 \\ x_1 =&~ q \cdot x_2 + (1-q) \cdot x_3 \\ x_2 =&~ q \cdot x_1 + (1-q) \cdot x_4 \\ x_3 =&~ 1 \\ x_4 =&~ 0. \end{aligned}$$

Bringing the system in normal form yields:

$$\begin{aligned} x_0 - p \cdot x_1 - (1-p) \cdot x_2 =&~0 \\ x_1 - q \cdot x_2 - (1-q) \cdot x_3=&~0 \\ -q \cdot x_1 + x_2 - (1-q) \cdot x_4=&~0 \\ x_3 =&~1 \\ x_4 =&~0. \end{aligned}$$

Adding q times the second equation to the third equation (concerning state $s_2$) brings the left-hand side matrix in upper triangular form:

$$\begin{aligned} x_0 - p \cdot x_1 - (1-p) \cdot x_2 =&~0 \\ x_1 - q \cdot x_2 - (1-q) \cdot x_3=&~0 \\ (1 - q^2) \cdot x_2 - q(1-q) \cdot x_3 - (1-q) \cdot x_4=&~0 \\ x_3 =&~1\\ x_4 =&~0 . \end{aligned}$$

The equation system yields the same result as the elimination of the transition from $s_2$ to $s_1$ (notice the symmetry between $s_1$ and $s_2$).

The example illustrates that there is no elementary advantage in doing state elimination over resorting to solving the linear equation sytem by (some variant of) Gaussian elimination. If we are only interested in the probability from the initial state, we do not need to solve the full equation system. The state-elimination algorithm, in which we can remove unreachable states, optimises for this observation, in contrast to (standard) linear equation solving. As in state elimination, the elimination order of the rows has a significant influence.

5.3 Algorithm based on set-based transition elimination

To succinctly represent large state spaces, Markov chains are often represented by multi-terminal binary decision diagrams (or variants thereof) [14]. Such a symbolic representation handles sets of states instead of single states (and thus also sets of transitions), and thereby exploits symmetries and similarities in the underlying graph of a model. To support efficient elimination, we describe how to eliminate sets of transitions at once. The method is similar to the Floyd-Warshall algorithm for all-pair shortest paths [51]. The transition matrix contains one-step probabilities for every pair of source and target states. Starting with a self-loop-free pMC (obtained by eliminating all self-loops from the original pMC), we iterate two operations until convergence. By doing a matrix-matrix multiplication, we effectively eliminate all transitions emanating from all non-absorbing states simultaneously. As this step may reintroduce self-loops, we eliminate them in a second step. As before, eventually only direct transitions to absorbing states remain, which effectively yield the unbounded reachability probabilities. The corresponding pseudo-code is given in Algorithm 3.

The approach of this algorithm can conveniently be explained in the equation system representation. Let us therefore conduct one step of the algorithm as an example, where we use the observation that the matrix-matrix multiplication corresponds to replacing the variables $x_s$ by their defining equations in all other equations.

Fig. 10

One step of set-based transition elimination exemplified

Algorithm 3

Set-based transition elimination for pMCs

Example 25

Reconsider the equations from Example 24:

$$\begin{aligned} x_0 =&~ p \cdot x_1 + (1-p) \cdot x_2 \\ x_1 =&~ q \cdot x_2 + (1-q) \cdot x_3 \\ x_2 =&~ q \cdot x_1 + (1-q) \cdot x_4 \\ x_3 =&~ 1 \\ x_4 =&~ 0. \end{aligned}$$

Using the equations for $x_0, x_1, x_2$ to replace their occurrences in all other equations yields:

$$\begin{aligned} x_0 =&~ p \cdot (q \cdot x_2 + (1-q) \cdot x_3) + (1-p)(q \cdot x_1 + (1-q) \cdot x_4) \\ x_1 =&~ q \cdot (q \cdot x_1 + (1-q) \cdot x_4) + (1-q) \cdot x_3 \\ x_2 =&~ q \cdot (q \cdot x_2 + (1-q) \cdot x_3) + (1-q) \cdot x_4 \\ x_3 =&~ 1 \\ x_4 =&~ 0 \end{aligned}$$

which simplifies to

$$\begin{aligned} x_0 =&~ (1-p) \cdot q \cdot x_1 + p \cdot q \cdot x_2 + p \cdot (1-q) \cdot x_3 \\ {}&\quad + (1-p)(1-q)\cdot x_4 \\ x_1 =&~ \frac{1}{1+q} \cdot x_3 + \frac{q}{1+q} \cdot x_4 \\ x_2 =&~ \frac{q}{1+q} \cdot x_3 + \frac{1}{1+q} \cdot x_4 \\ x_3 =&~ 1 \\ x_4 =&~ 0. \end{aligned}$$

We depict the pMC which corresponds to this equation system in Fig. 10a. Again, notice the similarity to state elimination. For completeness, the result after another iteration is given in Fig. 10b.

The correctness follows from the following argument: After every iteration, the equations describe a pMC over the same state space as before. As all absorbing states have defining equations $x_i \in \{0,1\}$, the equation system is known to have a unique solution [13]. Moreover, as the equation system in iteration i implies the equation system in iteration $i+1$, they preserve the same (unique) solution.

6 SMT-based region verification

In this section, we discuss a complete procedure to verify regions by encoding them as queries for an SMT solver, or more precisely, in the existential theory of the reals (the QF_NRA theory in the SMT literature). We first introduce the constraints for verifying regions on pMCs in Sect. 6.1. The constraints are either based on the equation system encoding from Sect. 5.2 or use the solution function, which yields an equation system with less variables at the cost of precomputing the solution function. In Sect. 6.2, we then introduce the encodings for region verification on pMDPs under angelic and demonic strategies.

Throughout the section, we focus on unbounded reachability, that is, we assume $\varphi =\mathbb {P}_{\le \lambda }(\lozenge T)$. As expected rewards can be described by a similar equation system, lifting the concepts is straightforward. We assume a graph-preserving region R: Assuming that R is graph preserving eases the encodings significantly, but is not strictly necessary: In [94, Ch. 4], we provide encodings for well-defined regions R.

Fig. 11

Toy-examples (repeated from Fig. 6)

6.1 Satisfiability checking for pMC region checking

Recall from Sect. 5.2 the equation system for pMCs, exemplified by the following running example.

Example 26

Reconsider the pMC $\mathcal {D}$ from Fig. 6a, repeated in Fig. 11a for convenience. The concrete equation system of (5)–(7) on page 32 for reaching $T=\{s_2\}$, using $x_i$ to denote $x_{s_i}$, is given by:

$$\begin{aligned} x_0&= p \cdot x_1 + (1{-}p) \cdot x_2 \\ x_1&= q \cdot x_2 + (1{-}q) \cdot x_3 \\ x_2&= 1 \\ x_3&= 0. \end{aligned}$$

The conjunction of the equation system for the pMC, (5)–(7) on page 32, is an implicitly existential quantified formula to which we refer by $\Upphi (\mathcal {D})$—consider the remark below. By construction, this formula is satisfiable.

Remark 10

If transitions in the pMC are not polynomial but rational functions, the equations are not polynomial constraints, hence their conjunction is not a formula (Sect. 2.5). Instead, each $x = \sum \mathcal {P}(s,s')$ has to be transformed by the rules in Sect. 4.2.2: then, their conjunction is a formula. This transformation can always be applied, in particular, in the equalities we are never interested in the evaluation of instantiations $u \in R$ with $\mathcal {P}(s,s')[u] = \bot$: Recall that we are interested in analysing this equation system on a well-defined parameter region R: Therefore, for any $u \in R$, $\mathcal {P}(s,s')[u] \ne \bot$ for each $s, s'\in S$. Thus, when $\Upphi (\mathcal {D})$ is used in conjunction with $\Upphi (R)$, we do not need to consider this special case.

We consider the conjunction of the equation system, a property and a region. Concretely, let us first consider the conjunction of:

• the equation system $\Upphi (\mathcal {D})$,

• a comparison of the initial state $s_{ I }$ with the threshold $\lambda$, and

• a formula $\Upphi (R)$ describing the parameter region R.

Satisfiability of this conjunction means that—for some parameter instantiation within the region R—the reachability probability from the initial state $s_{ I }$ satisfies the bound. Unlike $\Upphi (\mathcal {D})$, this conjunction may be unsatisfiable.

Example 27

We continue with Example 26. Let $\varphi =\mathbb {P}_{\le 0.4}(\lozenge \{ s_2 \})$ and $R = \{ (p,q) \in [0.4, 0.6] \times [0.2, 0.5]\}$. We have $\Upphi (R)=0.4 \le p \wedge p \le 0.6 \wedge 0.2 \le q \wedge q \le 0.5$. We obtain the following conjunction:

$$\begin{aligned} \Upphi (\mathcal {D}) \wedge x_0&\le 0.4 \wedge \Upphi (R) \end{aligned}$$

(8)

where $\Upphi (\mathcal {D})$ is the conjunction of the equation system, i.e.:

$$\begin{aligned} \Upphi (\mathcal {D})= & {} \Big ( x_0&= p \cdot x_1 + (1{-}p) \cdot x_2&\wedge \\{} & {} x_1&= q \cdot x_2 + (1{-}q) \cdot x_3&\wedge \\{} & {} x_2&= 1~\wedge ~x_3 = 0&\Big ). \end{aligned}$$

Formula (8) is unsatisfiable, thus, no instance of p and q within the region R induces a reachability probability of at most $\nicefrac {2}{5}$.

Towards region verification, consider that the satisfaction relations $\models _a$⁹ as defined in Definition 13, we have to certify that all parameter values within a region yield a reachability probability that satisfies the threshold. Thus, we have to quantify over all instantiations u, (roughly) leading to a formula of the form $\forall u \ldots \models \varphi$. By negating this statement, we obtain the proof obligation $\lnot \exists u \ldots \models \lnot \varphi$: no parameter value within the region R satisfies the negated comparison with the initial state. This intuition leads to the following conjunction of:

• the equation system $\Upphi (\mathcal {D})$,

• a comparison of the initial state with the threshold, by inverting the given threshold-relation, and

• a formula $\Upphi (R)$ describing the parameter region.

This conjunction is formalised in the following definition.

Definition 19

(Equation system formula) Let $\mathcal {D}$ be a pMC, $\varphi =\mathbb {P}_{\sim \lambda }(\lozenge T)$, and R a region. The equation system formula is given by:

$$\begin{aligned} \Upphi (\mathcal {D}) \wedge x_{s_{ I }} \not \sim \lambda \wedge \Upphi (R). \end{aligned}$$

Theorem 1

The equation system formula is unsatisfiable iff $\mathcal {D},R\models \varphi$.

Otherwise, a satisfying solution is a counterexample.

Example 28

We continue Example 27. We invert the relation $x_0 \le 0.4$ and obtain:

$$\begin{aligned} \Upphi (\mathcal {D}) \wedge x_0&> 0.4 \wedge \Upphi (R). \end{aligned}$$

By SMT-checking, we determine that the formula is satisfiable, e.g., with $p = 0.5$ and $q = 0.3$. Thus, $\mathcal {D},R\not \models \varphi$. If we consider instead the region $R' = \{ (p,q) \in [0.8, 0.9] \times [0.1, 0.2]\}$ with $\Upphi (R')=0.8 \le p \wedge p \le 0.9 \wedge 0.1 \le q \wedge q \le 0.2$, we obtain:

$$\begin{aligned} \Upphi (\mathcal {D}) \wedge x_0&> 0.4 \wedge \Upphi (R') \end{aligned}$$

which is unsatisfiable. Hence, no point in $R'$ induces a probability larger than $\nicefrac {2}{5}$ and, equivalently, all points in $R'$ induce a probability of at most $\nicefrac {2}{5}$. Thus, $\mathcal {D},R'\models \varphi$.

We observe that the number of variables in this encoding is $|S_{}| + |V|$. In particular, we are often interested in systems with at least thousands of states. The number of variables is therefore often too large for SMT-solvers dealing with non-linear real arithmetic. However, many of the variables are auxiliary variables that encode the probability to reach target states from each individual state. We can get rid of these variables by replacing the full equation system by the solution function (Definition 10).

Definition 20

(Solution function formula) Let $\mathcal {D}$ be a pMC, $\varphi =\mathbb {P}_{\sim \lambda }(\lozenge T)$, and R a region. The solution function formula¹⁰ is given by:

$$\begin{aligned} f^r_{\mathcal {D},T} \not \sim \lambda \wedge \Upphi (R). \end{aligned}$$

Corollary 1

The solution function formula is unsatisfiable iff $\mathcal {D},R\models \varphi$.

Example 29

We consider the same scenario as in Example 27. The solution function is given in Example 13. The solution function formula is:

$$\begin{aligned} 1 - p + p\cdot q > 0.4 \wedge \Upphi (R). \end{aligned}$$

By construction, the equation system formula and the solution function formula for pMC $\mathcal {D}$ and reachability property $\varphi$ are equisatisfiable.

6.2 Existentially quantified formula for parametric MDPs

We can also utilise an SMT solver to tackle the verification problem on pMDPs. For parametric MDPs, we distinguish between the angelic and the demonic case, cf. Definition 14. We use the fact that optimal strategies for unbounded reachability objectives are memoryless and deterministic [121].

6.2.1 Demonic strategies

The satisfaction relation $\models _d$ is defined by two universal quantifiers, $\forall u \forall \sigma \ldots \models \varphi$. We therefore try to refute satisfiability of $\exists u \exists \sigma \ldots \models \lnot \varphi$. Put in a game-theoretical sense, the same player can choose both the parameter instantiation u and the strategy $\sigma$ to resolve the non-determinism. We generalise the set of linear equations from the pMC to an encoding for pMDPs, where we define a disjunction over all possible nondeterministic choices:

$$\begin{aligned}&x_s = 0&\quad&\forall s \in \lnot \lozenge T \end{aligned}$$

(9)

$$\begin{aligned}&x_s = 1&\quad&\forall s \in T \end{aligned}$$

(10)

$$\begin{aligned}&\bigvee _{\alpha \in Act (s)} \Big ( x_s = \sum _{s' \in S} \mathcal {P}(s,\alpha ,s') \cdot x_{s'}\Big )&\quad&\forall s \in \lozenge T \setminus T. \end{aligned}$$

(11)

We denote the conjunction of (9)–(11) as $\Upphi _d(\mathcal {M})$ for pMDP $\mathcal {M}$¹¹. Instead of a single equation for the probability to reach the target from state s, we get one equation for each action. The solver can now freely choose which (memoryless deterministic) strategy it uses to refute the property.

Definition 21

(Demonic equation system formula) Let $\mathcal {M}$ be a pMDP, $\varphi =\mathbb {P}_{\le \lambda }(\lozenge T)$, and R a region. The demonic equation system formula is given by:

$$\begin{aligned} \Upphi _d(\mathcal {M}) \wedge x_{s_{ I }} > \lambda \wedge \Upphi (R). \end{aligned}$$

Theorem 2

The demonic equation system formula is unsatisfiable iff $\mathcal {M},R\models _d \varphi$.

Example 30

Let $\mathcal {M}$ be the pMDP from Fig. 11b. Let $R, \varphi$ be as in Example 27. The demonic equation system formula is

$$\begin{aligned} \Upphi _d(\mathcal {M}) \wedge x_0 > 0.4 \wedge \Upphi (R) \end{aligned}$$

with $\Upphi (R)$ as before, and

$$\begin{aligned} \Upphi _d(\mathcal {M})= & {} \Big ( \big ( x_0&= p \cdot x_1 + (1{-}p) \cdot x_2 \quad \vee \quad x_0 = x_2\big )&\wedge \\{} & {} x_1&= q \cdot x_2 + (1{-}q) \cdot x_3&\wedge \\{} & {} x_2&= 1~\wedge ~x_3 = 0&\Big ). \end{aligned}$$

Similarly, when using the (potentially exponential) set of solution functions, we let the solver choose:

Definition 22

(Demonic solution function formula) Let $\mathcal {M}$ be a pMDP, $\varphi =\mathbb {P}_{\sim \lambda }(\lozenge T)$, and R a region. The demonic solution function formula is given by:

$$\begin{aligned} \bigvee _{\sigma \in Str ^\mathcal {M}} f^r_{\mathcal {M}^\sigma ,T} \not \sim \lambda \wedge \Upphi (R). \end{aligned}$$

Corollary 2

The demonic solution function formula is unsatisfiable iff $\mathcal {M},R\models _d \varphi$.

As the set of solution functions can be exponential, the demonic solution function formula can grow exponentially.

Example 31

The demonic solution function formula for $\mathcal {M}, \varphi , R$ as in Example 30, is given by:

$$\begin{aligned}&\Big (1> 0.4 \vee 1 - p + p\cdot q > 0.4 \Big ) \wedge \Upphi (R). \end{aligned}$$

6.2.2 Angelic strategies

The satisfaction relation $\models _a$ has two different quantifiers, $\forall u \exists \sigma \ldots \models \varphi$. Again, we equivalently try to refute the satisfiability of $\exists u \forall \sigma \ldots \models \lnot \varphi$. The quantifier alternation can be circumvented by lifting the linear programming (LP) formulation for MDPs [121], where for each nondeterministic choice an upper bound on the probability variables $x_s$ is obtained:

$$\begin{aligned}&x_s = 0&\quad&\forall s \in \lnot \lozenge T \end{aligned}$$

(12)

$$\begin{aligned}&x_s = 1&\quad&\forall s \in T \end{aligned}$$

(13)

$$\begin{aligned}&\bigwedge _{\alpha \in Act (s)} \Big ( x_s \le \sum _{s' \in S} \mathcal {P}(s,\alpha ,s') \cdot x_{s'}\Big )&\quad&\forall s \in \lozenge T \setminus T . \end{aligned}$$

(14)

Intuitively, the conjunction in constraint (14) eliminates the freedom of choosing any strategy from the solver and forces it to use the strategy that minimises the reachability probability. This means that the constraint system is only satisfiable if all strategies violate the probability bound. We denote the conjunction of (12)–(14) as $\Upphi _a(\mathcal {M})$. Notice that, as for parameter-free MDPs, the optimisation objective of the LP formulation can be substituted by a constraint on probability in the initial state.

Definition 23

(Angelic equation system formula) Let $\mathcal {M}$ be a pMDP, $\varphi =\mathbb {P}_{\le \lambda }(\lozenge T)$, and R a region. The angelic equation system formula is given by:

$$\begin{aligned} \Upphi _a(\mathcal {M}) \wedge x_{s_{ I }} > \lambda \wedge \Upphi (R). \end{aligned}$$

Theorem 3

The angelic equation system formula is unsatisfiable iff $\mathcal {M},R\models _a \varphi$.

Example 32

Let $\mathcal {M}, \varphi , R$ as in Example 30. The angelic equation system formula is given by

$$\begin{aligned} \Upphi _a(\mathcal {M}) \wedge x_0 > 0.4 \wedge \Upphi (R) \end{aligned}$$

with

$$\begin{aligned} \Upphi _a(\mathcal {M})= & {} \Big ( \big ( x_0&\le p \cdot x_1 + (1{-}p) \cdot x_2 \wedge x_0 \le x_2\big )&\wedge \\{} & {} x_1&\le q \cdot x_2 + (1{-}q) \cdot x_3&\wedge \\{} & {} x_2&= 1~\wedge ~x_3 = 0&\Big ). \end{aligned}$$

When using the set of solution functions, all strategies have to be considered. Again, for most pMDPs, this set is prohibitively large.

Definition 24

(Angelic solution function formula) Let $\mathcal {M}$ be a pMDP, $\varphi =\mathbb {P}_{\le \lambda }(\lozenge T)$, and R a region. The angelic solution function formula is given by:

$$\begin{aligned} \bigwedge _{\sigma \in Str ^\mathcal {M}} f^r_{\mathcal {M}^\sigma ,T} > \lambda \wedge \Upphi (R). \end{aligned}$$

Corollary 3

The angelic solution function formula is unsatisfiable iff $\mathcal {M},R\models _a \varphi$.

Example 33

The angelic solution function formula for $\mathcal {M}, \varphi , R$ as in Example 30 is given by:

$$\begin{aligned}&\Big (1> 0.4 \wedge 1 - p + p\cdot q > 0.4\Big ) \wedge \Upphi (R). \end{aligned}$$

7 Model-checking-based region verification of parametric MCs

This section discusses an abstraction (and refinement) procedure for region verification of pMCs. Intuitively, in order to bound the probability in a region from above, we bound the value induced by any instantation from above. We aim to do this by finding an instantiation that maximises the reachability probability in the region. This problem is particularly hard, as there are dependencies between the different parameters:

Fig. 12

A pMC $\mathcal {D}$ and its substitution $\textsf {sub}_{R}(\mathcal {D})$ and its relaxation $\textsf {rel}_{}(\mathcal {D})$

Example 34

Consider the pMC $\mathcal {D}$ in Fig. 12a—repeating Fig. 5c— and region $R = [\nicefrac {1}{10}, \nicefrac {4}{5}] \times [\nicefrac {2}{5}, \nicefrac {7}{10}]$. We again aim to reach $s_3$. We make two observations: $s_4$ is the only state from which we cannot reach $s_3$, furthermore, $s_4$ is only reachable via $s_2$. Hence, it is best to avoid $s_2$. From state $s_0$, it is thus beneficial if the transition probability to $s_2$ is as small as possible. Equivalently, it is beneficial if p is as large as possible, as this minimises the probability of reaching $s_2$ and as p does not occur elsewhere. Now we consider state $s_1$: As we want to reach $s_3$, the value of q should be preferably low. However, q occurs also at transitions leaving $s_2$. From $s_2$, q should be assigned a high value as we want to avoid $s_4$. In particular, the optimal value for q depends on the probability that we ever visit $s_2$, which is directly influenced by the value of p.

In a nutshell, the abstraction we propose in this section ignores the dependencies between the same occurence of a parameter. Conveniently, the abstraction transforms a pMC into an (parameter-free!) MDP whose minimal (maximal) reachability probability under-approximates (over-approximates) the reachability probability of the pMC. This result is formalised in Theorem 5, below.

Example 35

Consider the pMC in Fig. 12a and a region $R = [\nicefrac {1}{10}, \nicefrac {4}{5}] \times [\nicefrac {2}{5}, \nicefrac {7}{10}]$. The method creates the MDP in Fig. 12b, where different types of arrows reflect different actions. The MDP is created by adding in each state two actions: One reflecting the lower bound of the parameter range, one reflecting the upper bound. Model checking on this MDP yields a maximal probability of $\nicefrac {47}{60}$. From this result, we infer that $\max _{u \in R} \text {Pr}^{\mathcal {D}[u]}(\lozenge T) \le \nicefrac {47}{60}$.

The essence of this construction is to consider parameter values as a local, discrete choice that we can capture with nondeterminism. To support the discretisation, we must ensure that the optimal values are taken at the bounds of the region. While this is not true in general due to the nonlinearity of the solution function, creating a suitable over-approximation, called the relaxation, enforces this property, as we show in Theorem 4, also below.

In the remainder of this section, we first clarify helpful assumptions on the type of pMCs we support in Sect. 7.1. We then construct so-called relaxed pMCs in Sect. 7.2. In Sect. 7.3, we translate relaxed pMCs to parameter-free MDPs to allow off-the-shelf MDP analysis for region verification of pMCs.

7.1 Preliminaries

We formalise the perspective that underpins our approach to region verification and introduce some assumptions.

7.1.1 A perspective for region verification

The probability $\text {Pr}^{\mathcal {D}}(\lozenge T)$ can be expressed as a rational function with polynomials $g_1, g_2$ due to Definition 10. Recall that we assume region R to be graph preserving. Therefore, $g_2[u] \ne 0$ for all $u \in R$ and f is continuous on any closed region R. Hence, there is an instantiation $u \in R$ that induces the maximal (or minimal) reachability probability:

$$\begin{aligned}&\sup _{u\in R}\text {Pr}^{\mathcal {D}[u]}(\lozenge T) = \max _{u\in R} \text {Pr}^{\mathcal {D}[u]}(\lozenge T) \quad \text {and}\quad \inf _{u\in R} \text {Pr}^{\mathcal {D}[u]}(\lozenge T) = \min _{u\in R}\text {Pr}^{\mathcal {D}[u]}(\lozenge T). \end{aligned}$$

To infer that R is accepting (i.e. all instantiations $u \in R$ induce probabilities at most $\lambda$), it suffices to show that the maximal reachability probability over all instantiations is at most $\lambda$:

$$\begin{aligned} \mathcal {D}, R \models \mathbb {P}_{\le \lambda }(\lozenge T)&\iff \big (\max _{u\in R} \text {Pr}^{\mathcal {D}[u]}(\lozenge T) \big ) \le \lambda , \text { and } \\ \mathcal {D}, R \models \lnot \mathbb {P}_{\le \lambda }(\lozenge T)&\iff \big (\min _{u\in R} \text {Pr}^{\mathcal {D}[u]}(\lozenge T) \big ) > \lambda . \end{aligned}$$

One way to determine the maximum reachability probability is to first determine which $u \in R$ induces the maximum, and then compute the probability on the instantiated model $\mathcal {D}[u]$. While we only discuss upper-bounded specifications here, the results can be analogously described for lower-bounded specifications.

Example 36

Consider $\mathcal {D}$ depicted in Fig. 11a, $\varphi =\mathbb {P}_{\le \nicefrac {9}{10}}(\lozenge \{ s_2 \})$, and $R' = \{ (p,q) \in [\nicefrac {2}{5}, \nicefrac {3}{5}] \times [\nicefrac {1}{5}, \nicefrac {1}{2}]\}$ as in Example 27. The maximum is obtained at $u = (\nicefrac {2}{5},\nicefrac {1}{2})$ (via some oracle). We have $\mathcal {D}[u] \models \mathbb {P}_{\le \nicefrac {9}{10}}(\lozenge \{ s_2 \})$, and thus, $\mathcal {D}, R' \models \mathbb {P}_{\le \nicefrac {9}{10}}(\lozenge \{ s_2 \})$.

However, constructing an oracle that determines the u that induces the maximum is difficult in general. We focus on the essential idea an therefore make the following assumptions throughout the rest of this section:

Assumption 1

• We restrict the (graph-preserving) region R to be (i) rectangular, and (ii) closed. This restriction makes the bounds of the parameters independent of other parameter instantiations, and ensures that the maximum over the region exists.

• We restrict the pMC $\mathcal {D}$ to be locally monotone—explained in Sect. 7.1.2—to exclude difficulties from analysing single transitions.

The first assumption can be a nuisance. In particular, it is not always clear how to create an adequate closed region from an open region. The second assumption is very mild and can be accomodated for using adequate preprocessing [94, Section 5.1] that introduced additional states.

7.1.2 Locally monotone pMCs

Recall that the solution function is nonlinear. We aim to approximate this u and therefore want to exploit the structure of the pMC. Therefore, we want to make an assumption on the transition relation.

Example 37

Consider a three-state pMC where the probability from initial state $s_{ I }$ to target state t is a non-linear, non-monotone transition function, as, e.g., the transition probability from $s_0$ to $s_3$ of the pMC in Fig. 9f. Finding the maximum requires an analysis of the derivative of the solution function, and is (approximately) as hard as the exact verification problem.

Instead, we assume monotonic transition probabilities, and consider a slightly restricted class of pMCs.

Definition 25

(Locally monotone pMCs) A pMC $\mathcal {D}{}=(S{}, V{}, s_{ I }{}, \mathcal {P}{})$ is locally monotone iff for all $s \in S$ there is a multilinear polynomial $g_s \in \mathbb {Q}[V]$ satisfying

$$\begin{aligned} \mathcal {P}(s, s') \in \left\{ \nicefrac {f}{g_s} \mid f\in \mathbb {Q}[V] \text { is multilinear} \right\} \end{aligned}$$

for all $s' \in S$.

Locally monotone pMCs include most pMCs from the literature [122]¹². Examples of the egligible transition probabilities are $p,pq,\nicefrac {1}{p}$ and their complements formed by $1-p$ etc.

Thanks to monotonicity, for a locally monotone pMC $\mathcal {D}{}=(S{}, V{}, s_{ I }{}, \mathcal {P}{})$, and a closed rectangular region R we have that for all $s,s' \in S:$

$$\begin{aligned} \max _{u \in R} \mathcal {P}(s,s') = \max _{u \in B(V)} \mathcal {P}(s,s') \end{aligned}$$

where $B(V) = \{ u \mid \forall p \in V. u(p) \in B_R(p) \}$, i.e., all maxima of the individual transition probabilities are attained at the bounds of the region. However, the restriction to local monotonicity does not immediately overcome the challenge of constructing an oracle. The resulting solution function may still be highly nonlinear. In particular, Example 34 uses a locally monotone pMC and a closed rectangular region. However, as the example indicates, trade-offs in locally monotone pMCs occur due to dependencies where parameters occur at multiple states.

7.2 Relaxation

The idea of our approach, inspired by [30], is to drop the aforementioned dependencies between parameters by means of a relaxation of the pMC. We want to highlight that this relaxed pMC is very similar to so-called interval MCs, a detailed discussion is given in [94, Section 5.1.1.3]. Intuitively, the relaxation $\textsf {rel}_{}(\mathcal {D})$ is a pMC that arises from $\mathcal {D}$ to a pMC with the same state space but an updated transition relation. In particular, it introduces a fresh copy of every parameter in every state, thereby eliminating parameter dependencies between different states (if any). This step simplifies finding an optimal instantiation (in the relaxation). However, the set of instantiated pMCs grows: some of the instantiations cannot be obtained from the original pMC. In this subsection, we first formalize the relaxation, then clarify the relation between properties being satisfied on the pMC and properties satisfied on the relaxation. We finish the subsection by discussing how to efficiently analyze a relaxed pMC.

Definition 26

(Relaxation) The relaxation of pMC $\mathcal {D}{}=(S{}, V{}, s_{ I }{}, \mathcal {P}{})$ is the pMC $\textsf {rel}_{}(\mathcal {D}) = (S, \textsf {rel}_{\mathcal {D}}(V), s_{ I }, \mathcal {P}')$ with $\textsf {rel}_{\mathcal {D}}(V)=\{p_i^s \mid p_i \in V, s\in S\}$ and $\mathcal {P}'(s,s')=\mathcal {P}(s,s')[p_1, \dots , p_n / p_1^s, \dots , p_n^s]$.

We extend an instantiation u for $\mathcal {D}$ to the relaxed instantiation $\textsf {rel}_{\mathcal {D}}(u)$ for $\textsf {rel}_{}(\mathcal {D})$ by $\textsf {rel}_{\mathcal {D}}(u)(p_i^s) = u(p_i)$ for every s. We have that for all u, $\mathcal {D}[u] = \textsf {rel}_{}(\mathcal {D})[\textsf {rel}_{\mathcal {D}}(u)]$. We lift the relaxation to regions such that $B(p_i^s) = B(p_i)$ for all s, i. e., . We drop the subscript $\mathcal {D}$, whenever it is clear from the context.

Example 38

Figure 12c depicts the relaxation $\textsf {rel}_{}(\mathcal {D})$ of the pMC $\mathcal {D}$ from Fig. 12a. For $R=[\nicefrac {1}{10}, \nicefrac {4}{5}] \times [\nicefrac {2}{5}, \nicefrac {7}{10}]$ and $u=(\nicefrac {4}{5}, \nicefrac {3}{5}) \in R$ from Example 19, we obtain $\textsf {rel}_{}(R)=[\nicefrac {1}{10}, \nicefrac {4}{5}] \times [\nicefrac {2}{5}, \nicefrac {7}{10}] \times [\nicefrac {2}{5}, \nicefrac {7}{10}]$ and $\textsf {rel}_{}(u)=(\nicefrac {4}{5}, \nicefrac {3}{5}, \nicefrac {3}{5})$. An instantiation $\textsf {rel}_{}(\mathcal {D})[\textsf {rel}_{}(u)]$ corresponds to $\mathcal {D}[u]$ as depicted in Fig. 5d. The relaxed region $\textsf {rel}_{}(R)$ contains also instantiations, e.g., $(\nicefrac {4}{5}, \nicefrac {1}{2}, \nicefrac {3}{5})$ which are not realisable in R.

For a pMC $\mathcal {D}$ and a graph-preserving region R, relaxation increases the set of possible instantiations: $\{\mathcal {D}[u] \mid u \in R\} \subseteq \{\textsf {rel}_{}(\mathcal {D})[u] \mid u \in \textsf {rel}_{}(R)\}$. Thus, the maximal reachability probability over all instantiations of $\mathcal {D}$ within R is bounded by the maximum over the instantiations of $\textsf {rel}_{}(\mathcal {D})$ within $\textsf {rel}_{}(R)$.

Lemma 3

For pMC $\mathcal {D}$ and region R:

$$\begin{aligned} \max _{u\in R}\big ( \text {Pr}^{\mathcal {D}[u]}(\lozenge T) \big ) \ = \ \max _{u\in R}\big ( \text {Pr}^{\textsf {rel}_{}(\mathcal {D})[\textsf {rel}_{}(u)]}(\lozenge T) \big ) \le \ \max _{u\in \textsf {rel}_{}(R)} \big ( \text {Pr}^{\textsf {rel}_{}(\mathcal {D})[u]}(\lozenge T) \big ). \end{aligned}$$

Consequently, if $\textsf {rel}_{}(\mathcal {D})$ satisfies a reachability property, so does $\mathcal {D}$.

Corollary 4

For pMC $\mathcal {D}$ and region R:

$$\begin{aligned} \max _{u\in \textsf {rel}_{}(R)}\big ( \text {Pr}^{\textsf {rel}_{}(\mathcal {D})[u]}(\lozenge T)\big ) \le \lambda \text { implies }\mathcal {D}, R \models \mathbb {P}_{\le \lambda }(\lozenge T). \end{aligned}$$

We now formalise the earlier observation: Without parameter dependencies, finding optimal instantiations in a pMC is simpler. Although $\textsf {rel}_{}(\mathcal {D})$ has (usually) more parameters than $\mathcal {D}$, finding an instantiation $u \in \textsf {rel}_{}(R)$ that maximises the reachability probability is simpler than in $u \in R$: For any $p_i^s \in \textsf {rel}_{}(V)$, we can in state s pick a value in $I(p^s_i)$ that maximises the probability to reach T from state s. There is no (negative) effect for the reachability probability at the other states as $p_i^s$ only occurs at s. Optimal instantiations can thus be determined locally (at the states).

Furthermore, as both $\mathcal {D}$ is locally monotone, and there are no parameter dependencies, the maximum reachability probability is relatively easy to find: We only need to consider instantiations u that set the value of each parameter to either the lowest or highest possible value, i. e., $u(p_i^s) \in B(p_i^s)$ for all $p_i^s \in \textsf {rel}_{}(V)$:

Theorem 4

Let $\mathcal {D}$ be a pMC with states S and $T\subseteq S$ and R a region subject subject to Assumption 1. There exists an instantiation $u \in \textsf {rel}_{}(R)$ satisfying $u(p_i^s) \in B(p_i^s)$ for all $p_i^s\in \textsf {rel}_{}(V)$ such that:

$$\begin{aligned} \text {Pr}^{\textsf {rel}_{}(\mathcal {D})[u]}(\lozenge T) = \max _{v\in \textsf {rel}_{}(R)}\text {Pr}^{\textsf {rel}_{}(\mathcal {D})[v]}(\lozenge T). \end{aligned}$$

To prove this statement, we consider an instantiation which assigns a value to a parameter strictly between its bounds. Any such instantiation can be modified such that all parameters are assigned to its bound, without decreasing the induced reachability probability. The essential statement is the monotonicity of a parameter without any further dependencies. The number of instantiations that must be analysed is therefore finite, compared for infinitely many candidates for non-relaxed pMCs.

Lemma 4

Let $\mathcal {D}$ be a locally monotone pMC with a single parameter p that only occurs at one state $s \in S$, i.e. $\mathcal {P}(\hat{s},s') \in [0,1]$ for all $\hat{s}, s' \in S$ with $\hat{s} \ne s$. For region R and $T\subseteq S$, the probability $\text {Pr}^{\mathcal {D}}(\lozenge T)$ is monotonic on R.

Proof

W. l. o. g. let $s\notin T$ be the initial state of $\mathcal {D}$ and let T be reachable from s. Furthermore, let $\mathcal {U}$ denote the standard until-modality and $\lnot T$ denote $S\setminus T$. Using the characterisation of reachability probabilities as linear equation system (cf. [13]), the reachability probability w. r. t. T (from the initial state) in $\mathcal {D}$ is given by:

$$\begin{aligned}&\text {Pr}^{\mathcal {D}}(\lozenge T) \\&\quad = \sum _{s'\in S} \mathcal {P}(s,s') \cdot \text {Pr}^{\mathcal {D}}_{s'}(\lozenge T) \\&\quad = \sum _{s'\in S} \mathcal {P}(s,s') \cdot \Big (\text {Pr}_{s'}^{\mathcal {D}}(\lnot s \, \mathcal {U}\, T) + \text {Pr}_{s'}^{\mathcal {D}}(\lnot T \, \mathcal {U}\, s) \cdot \text {Pr}^{\mathcal {D}}(\lozenge T) \Big )\\&\quad = \sum _{s'\in S} \mathcal {P}(s,s') \cdot \text {Pr}_{s'}^{\mathcal {D}}(\lnot s \, \mathcal {U}\, T) + \sum _{s'\in S} \mathcal {P}(s,s') \cdot \text {Pr}_{s'}^{\mathcal {D}}(\lnot T \, \mathcal {U}\, s) \cdot \text {Pr}^{\mathcal {D}}(\lozenge T). \end{aligned}$$

Transposing the equation yields

$$\begin{aligned} \text {Pr}^{\mathcal {D}}(\lozenge T) = \frac{\sum _{s'\in S} \mathcal {P}(s,s') \cdot \text {Pr}_{s'}^{\mathcal {D}}(\lnot s \, \mathcal {U}\, T)}{1-\sum _{s'\in S} \mathcal {P}(s,s') \cdot \text {Pr}_{s'}^{\mathcal {D}}(\lnot T \, \mathcal {U}\, s)}. \end{aligned}$$

The denominator can not be zero as T is reachable from s. Since $\mathcal {D}$ is locally monotone, we have $\mathcal {P}(s,s') = \nicefrac {f_{s'}}{g_s}$ for $s' \in S$ and multilinear functions $f_{s'}, g_s \in \mathbb {Q}[p]$. We obtain:

$$\begin{aligned} \text {Pr}^{\mathcal {D}}(\lozenge T) = \frac{\sum _{s'\in S} f_{s'} \cdot \overbrace{\text {Pr}_{s'}^{\mathcal {D}}(\lnot s \, \mathcal {U}\, T)}^ constant }{g_s -\sum _{s'\in S} f_{s'} \cdot \underbrace{\text {Pr}_{s'}^{\mathcal {D}}(\lnot T \, \mathcal {U}\, s)}_ constant }. \end{aligned}$$

Hence, $\text {Pr}^{\mathcal {D}}(\lozenge T) = \nicefrac {f_1}{f_2}$ is a fraction of two multilinear functions $f_1,f_2 \in \mathbb {Q}[p]$ and therefore monotonic on R. $\square$

Proof of Theorem 4

We prove the statement by contraposition. Let $u\in \textsf {rel}_{}(R)$ with $\text {Pr}^{\textsf {rel}_{}(\mathcal {D})[u]}(\lozenge T) = \max _{v\in \textsf {rel}_{}(R)}\big ( \text {Pr}^{\textsf {rel}_{}(\mathcal {D})[v]}(\lozenge T) \big )$. For the contraposition, assume that there exists a parameter $p \in \textsf {rel}_{}(V)$ with $u(p) \in I_R(p) \setminus B_R(p)$ such that all instantiations $u' \in \textsf {rel}_{}(R)$ that set p to a value in $B_R(p)$ induce a smaller reachability probability, i.e. $u'(p) \in B_R(p)$ and $u'(q) = u(q)$ for $q \ne p$ implies

$$\begin{aligned} \text {Pr}^{\textsf {rel}_{}(\mathcal {D})[u']}(\lozenge T) < \text {Pr}^{\textsf {rel}_{}(\mathcal {D})[u]}(\lozenge T). \end{aligned}$$

Consider the pMC $\hat{\mathcal {D}} = (S, \{p\}, s, \hat{\mathcal {P}})$ with the single parameter p that arises from $\textsf {rel}_{}(\mathcal {D})$ by replacing all parameters $q \in \textsf {rel}_{}(V)\setminus \{p\}$ with u(q). We have $\hat{\mathcal {D}}[u] = \textsf {rel}_{}(\mathcal {D})[u]$. Moreover, $\text {Pr}^{\hat{\mathcal {D}}}(\lozenge T)$ is monotonic on I(p) according to Lemma 4. Thus, there is an instantiation $u' \in \textsf {rel}_{}(R)$ with $u'(p) \in B_R(p)$ and $u'(q) = u(q)$ for $q \ne p$ satisfying

$$\begin{aligned} \text {Pr}^{\hat{\mathcal {D}}[u]}(\lozenge T) \le \text {Pr}^{\hat{\mathcal {D}}[u']}(\lozenge T) = \text {Pr}^{\textsf {rel}_{}(\mathcal {D})[u']}(\lozenge T) . \end{aligned}$$

This contradicts our assumption for parameter p. $\square$

7.3 Replacing parameters by nondeterminism

In order to determine $\max _{u\in \textsf {rel}_{}(R) } \text {Pr}^{\textsf {rel}_{}(\mathcal {D})[u]}(\lozenge T)$, it suffices to make a discrete choice over instantiations $u :\textsf {rel}_{}(V) \rightarrow \mathbb {R}$ with $u(p_i^s) \in B(p_i)$. This choice can be made locally at every state, which brings us to the key idea of constructing a (non-parametric) MDP out of the pMC $\mathcal {D}$ and the region R, where nondeterministic choices represent all instantiations that have to be considered. In the following, it is convenient to refer to the parameters in a given state s by:

$$\begin{aligned} V_s=\{\, p \in V\mid p \text { occurs in } \mathcal {D}(s,s') \text { for some } s'\in S \, \}. \end{aligned}$$

Definition 27

(Substitution (pMCs)) For pMC $\mathcal {D}{}=(S{}, V{}, s_{ I }{}, \mathcal {P}{})$ and region R, let the MDP $\textsf {sub}_{R}(\mathcal {D}) = (S, s_{ I }, Act _{\textsf {sub}}, \mathcal {P}_{\textsf {sub}})$ with

• $Act _{\textsf {sub}} = \biguplus _{s \in S} Act _s$ where

  $$\begin{aligned} Act _s = \{u :V_s \rightarrow \mathbb {R}\mid \forall p \in V_s.\; u(p) \in B(p)\ \}, \text { and} \end{aligned}$$
• $$\begin{aligned} \mathcal {P}_{\textsf {sub}}(s,u, s') = {\left\{ \begin{array}{ll} \mathcal {P}(s,s')[u] &{}\text {if } u \in Act _s,\\ 0 &{}\text {otherwise.} \end{array}\right. } \end{aligned}$$

be the (parameter-)substitution of $\mathcal {D}$ and R.

Thus, choosing action u in s corresponds to assigning one of the extremal values $B(p_i)$ to the parameters $p_i^s$. The number of outgoing actions from state s is therefore $2^{|V_s|}$.

Example 39

Consider pMC $\mathcal {D}$—depicted in Fig. 12a—with $R = [\nicefrac {1}{10}, \nicefrac {4}{5}] \times [\nicefrac {2}{5}, \nicefrac {7}{10}]$ as before. The substitution of $\mathcal {D}$ and R is shown in Fig. 13a. In $\mathcal {D}$, each outgoing transition of states $s_0, s_1, s_2$ is replaced by a nondeterministic choice in MDP $\textsf {sub}_{R}(\mathcal {D})$. That is, we either pick the upper or lower bound for the corresponding variable. The solid (dashed) lines depict transitions that belong to the action for the upper (lower) bound. For the states $s_3$ and $s_4$, the choice is unique as their outgoing transitions in $\mathcal {D}$ are constant. Figure 13b depicts the MC $\textsf {sub}_{R}(\mathcal {D})^\sigma$ which is induced by the strategy $\sigma$ on MDP $\textsf {sub}_{\mathcal {D}}(R)$ that chooses the upper bounds at $s_0$ and $s_2$, and the lower bound at $s_1$. Notice that $\textsf {sub}_{R}(\mathcal {D})^\sigma$ coincides with $\textsf {rel}_{}(\mathcal {D})[v]$ for a suitable instantiation v, as depicted in Fig. 12c.

Fig. 13

Illustrating parameter-substitution

The substitution encodes the local choices for a relaxed pMC. That is, for an arbitrary pMC, there is a one-to-one correspondence between strategies $\sigma$ in the MDP $\textsf {sub}_{\textsf {rel}_{}(R)}(\textsf {rel}_{}(\mathcal {D}))$ and instantiations $u \in \textsf {rel}_{}(R)$ for $\textsf {rel}_{}(\mathcal {D})$ with $u(p_i^s) \in B(p_i)$. For better readability, we will omit the superscripts for sets of strategies $Str$. Combining these observations with Theorem 4, yields the following.

Corollary 5

For a pMC $\mathcal {D}$, a graph-preserving region R, and a set T of target states of $\mathcal {D}$:

Furthermore, the nondeterministic choices introduced by the substitution only depend on the values $B(p_i)$ of the parameters $p_i$ in R. Since the ranges of the parameters $p_i^s$ in $\textsf {rel}_{}(R)$ agree with the range of $p_i$ in R, we have

$$\begin{aligned} \textsf {sub}_{\textsf {rel}_{}(R)}(\textsf {rel}_{}(\mathcal {D})) = \textsf {sub}_{R}(\mathcal {D}) \quad \text {for all graph-preserving } R. \end{aligned}$$

(15)

A direct consequence of these statements yields:

Theorem 5

Let $\mathcal {D}$ be a pMC, R a graph-preserving region, $\varphi$ a reachability property, subject to Assumption 1. Then it holds:

$$\begin{aligned} \forall \sigma \in Str .\,~\textsf {sub}_{R}(\mathcal {D})^\sigma \models \varphi \implies&\mathcal {D}, R \models \varphi \quad \wedge \\ \forall \sigma \in Str .\,~ \textsf {sub}_{R}(\mathcal {D})^\sigma \models \lnot \varphi \implies&\mathcal {D}, R \models \lnot \varphi . \end{aligned}$$

Hence, we can deduce via Algorithm 4 whether $\mathcal {D}, R \models \varphi$ by applying standard techniques for MDP model checking to $\textsf {sub}_{R}(\mathcal {D})$, such as value- and policy iteration, cf. [13, 121]. We stress that while the relaxation is key for showing the correctness, equation (15) proves that this step does not actually need to be performed.

Example 40

Reconsider Example 39. From $\textsf {sub}_{R}(\mathcal {D})$ in Fig. 13a, we can derive $\max _{\sigma \in Str } \text {Pr}^{\textsf {sub}_{R}(\mathcal {D})^\sigma }(\lozenge T) = \nicefrac {47}{60}$ and, by Theorem 5, $\mathcal {D}, R \models \mathbb {P}_{\le \nicefrac {4}{5}}(\lozenge T)$ follows. Despite the large region R, we establish a non-trivial upper bound on the reachability probability over all instantiations in R.

If the over-approximation by region R is too coarse for a conclusive answer, region R can be refined, meaning that we split R into a set of smaller regions¹³ [30]. We discuss splitting strategies in Sect. 9. Intuitively, as more potential parameter values are excluded by reducing the region size, the actual choice of the parameter value has less impact on reachability probabilities. The smaller the region gets, the smaller the over-approximation: The optimal instantiation on the pMC $\mathcal {D}$ is over-approximated by some strategy on $\textsf {sub}_{R}(\mathcal {D})$. The approximation error originates from choices where an optimal strategy on $\textsf {sub}_{R}(\mathcal {D})$ chooses actions $u_1$ and $u_2$ at states $s_1$ and $s_2$, respectively, with $u_1(p_i^{s_1}) \ne u_2(p_i^{s_2})$ for some parameter $p_i$, and therefore intuitively disagree on its value. The probability mass that is affected by these choices decreases the smaller the region is. For infinitesimally small regions, the error from the over-approximation vanishes, as the actions for the upper and the lower bound of a parameter become equal up to an infinitesimal. More formally, the difference in reachability probability between two MCs corresponding to instantiations in a region tends is bounded and tends to zero if the region gets smaller [45, Lemma 9].

Algorithm 4

Parameter lifting

7.4 Expected reward properties

The reduction of bounding reachability probabilities on pMCs to off-the-shelf MDP model checking can also be applied to bound expected rewards. To see this, we have to extend the notion of locally monotone parametric Markov chains.

Definition 28

(Locally monotone reward pMC) A pMC $\mathcal {D}{}=(S{}, V{}, s_{ I }{}, \mathcal {P}{})$ with reward function $\text {rew}:S \rightarrow \mathbb {Q}(V)$ is locally monotone iff for all $s \in S$, there is a multilinear polynomial $g_s \in \mathbb {Q}[V]$ with

$$\begin{aligned} \{\text {rew}(s), \mathcal {P}(s, s') \mid s' \in S \} \subseteq \left\{ \nicefrac {f}{g_s} \mid f\in \mathbb {Q}[V] \text { multilinear} \right\} . \end{aligned}$$

We now generalise relaxation and substitution to the reward models, and obtain analogous results.

Definition 29

(Substitution for reward pMCs) Let $\mathcal {D}{}=(S{}, V{}, s_{ I }{}, \mathcal {P}{})$ be a pMC, $\text {rew}:S \rightarrow \mathbb {Q}(V)$ a reward function, $T \subseteq S$ a set of target states, and R a region. For $s \in S$, let

$$\begin{aligned} V^\text {rew}_s =V_s \cup \{p_i \in V\mid p_i \text { occurs in } \text {rew}(s)\}. \end{aligned}$$

The MDP $\textsf {sub}^\text {rew}_{R}(\mathcal {D}) = (S, s_{ I }, Act ^\text {rew}_\textsf {sub}, \mathcal {P}^\text {rew}_\textsf {sub})$ with reward function $\text {rew}_\textsf {sub}$ is the (parameter-)substitution of $\mathcal {D}, \text {rew}$ on R, where

• $Act ^\text {rew}_\textsf {sub}$ and $\mathcal {P}^\text {rew}_\textsf {sub}$ are analogous to Definition 27, but over $V^\text {rew}_s$.

• $\text {rew}_\textsf {sub}$ is given by:

  $$\begin{aligned} (s,u) \mapsto {\left\{ \begin{array}{ll} \text {rew}(s)[u] &{}\text {if } u \in Act ^\text {rew}_s,\\ 0 &{}\text {otherwise,} \end{array}\right. } \end{aligned}$$

  where $Act ^\text {rew}_s$ is defined analogously to $Act _s$ in Definition 27.

The reward approximation of a pMC can be used to identify regions as accepting or rejecting for expected reward properties.

Theorem 6

Let $\mathcal {D}$ be a pMC with locally monotone rewards $\text {rew}$, R a region, and $\varphi$ an expected reward property, subject to Assumption 1:

$$\begin{aligned} \forall \sigma \in Str .\,~ \textsf {sub}^\text {rew}_{R}(\mathcal {D}) \models \varphi \text { implies }&\mathcal {D}, R \models \varphi \text { and }\\ \forall \sigma \in Str .\,~ \textsf {sub}^\text {rew}_{R}(\mathcal {D}) \models \lnot \varphi \text { implies }&\mathcal {D}, R \models \lnot \varphi . \end{aligned}$$

The proof is analogous to the proof of Theorem 5.

8 Model-checking-based region verification of parametric MDPs

In the previous section, we approximated reachability probabilities in (locally-monotone) pMCs by considering the substitution MDP, see Definition 27. The non-determinism in the MDP encodes the finitely many parameter valuations that approximate the reachability probabilities in the pMC. By letting an adversary player resolve the non-determinism in the MDP, we obtain bounds on the reachability probabilities in the pMC. These bounds can efficiently be computed by standard MDP model checking.

In this section, we generalise the approach to pMDPs, which already contain non-determinism. The result naturally leads to a 2-player stochastic game: One player controls the nondeterminism inherent to the MDP, while the other player controls the (abstracted) parameter values. Letting the two players adequately minimise and/or maximise the reachability probabilities in the SG yields bounds on the minimal (and maximal) reachability probabilities in the pMDP. For example, if the player for the original non-determinism maximises and the parameter player minimises, we obtain a lower bound on the maximal probability. These bounds can efficiently be computed by standard SG model checking procedures.

In our presentation below, we discuss the interplay of the two sources of non-determinism. In particular, we show how the generalisation of the method yields an additional source of (over-)approximation. Then, we formalise the construction of the substitution with nondeterminism, analogous to the pMCs from the previous section. In particular, Definition 30 is analogous to Definition 27 and Theorem 7 is analogous to Theorem 5. We do not repeat relaxation, described in Sect. 7.2, as—as also discussed in the previous section—it is not a necessary ingredient for the correctness of the approach.

8.1 Two types of approximation

In the following, let $\mathcal {M}{}=(S{},V{},s_{ I }{}, Act {},\mathcal {P}{})$ be a pMDP and R a graph-preserving, rectangular, closed region.

Demonic strategies We analyse R with respect to the demonic relation $\models _d$. We have:

$$\begin{aligned} \mathcal {M}, R \models _d \varphi \iff \forall u \in R.~\forall \sigma \in Str ^{\mathcal {M}}.~\mathcal {M}[u]^\sigma \models \varphi . \end{aligned}$$

The two universal quantifiers can be reordered, and in addition $\mathcal {M}[u]^\sigma = \mathcal {M}^\sigma [u]$. We obtain:

$$\begin{aligned} \mathcal {M}, R \models _d \varphi \iff \forall \sigma \in Str ^{\mathcal {M}}. ~\forall u \in R.~\underbrace{\mathcal {M}^\sigma }_{\text {a pMC}}[u] \models \varphi \end{aligned}$$

Intuitively, the reformulation states that we have to apply pMC region verification on $\mathcal {M}^\sigma$ and R for all $\sigma \in Str ^\mathcal {M}$. We now want to employ parameter lifting for each strategy. Thus, we want to consider the verification of the substituted pMCs $\textsf {sub}_{R}(\mathcal {M}^\sigma )$. As these substituted pMCs share most of their structure, the set of all such substituted pMCs can be concisely represented as an SG, in which both players cooperate (as witnessed by the same quantifiers). In the scope of this paper, an SG with cooperating players can be concisely represented as an MDP. Consequently, for the demonic relation, pMDP verification can be approximated by MDP model checking.

Angelic strategies We now turn our attention to the angelic relation $\models _a$, cf. Definition 14.

$$\begin{aligned} \mathcal {M}, R \models _a \varphi \iff \forall u \in R.~\exists \sigma \in Str ^{\mathcal {M}}.~\mathcal {M}[u]^\sigma \models \varphi . \end{aligned}$$

Here, we cannot simply reorder the quantifiers. However:

$$\begin{aligned} \exists \sigma \in Str ^{\mathcal {M}}.~ \forall u \in R.~\mathcal {M}^\sigma [u] \models \varphi \implies \mathcal {M}, R \models _a \varphi . \end{aligned}$$

Now, the left-hand side expresses again that we want to do region verification for pMCs induced by a strategy, as in the demonic case, and that we likewise want to represent by a stochastic game. As witnessed by the quantifier alternation, this SG does not reduce to an MDP; the two players have opposing objectives. Nevertheless, we can efficiently analyse this SG (with a variant of value iteration), and thus the left-hand side of the implication above.

Observe that the over-approximation actually computes a robust strategy, as discussed in Remark 7. In particular, we now have two sources of approximation:

• The approximation that originates from dropping parameter dependencies (as also in the demonic case).

• The application of the substitution of parameters with non-determinism on robust strategies rather than of the actual angelic relation.

Both over-approximations vanish with declining region size.

8.2 Replacing parameters by nondeterminism

Fig. 14

Illustration of the substitution of a pMDP

Example 41

Consider the pMDP $\mathcal {M}$ in Fig. 14a, where the state s has two enabled actions $\alpha$ and $\beta$. The strategy $\sigma$ given by $\{s \mapsto \alpha \}$ applied to $\mathcal {M}$ yields a pMC, which is subject to substitution, cf. Fig. 14b.

The parameter substitution of a pMDP (cf. Fig. 14a) yields an SG—as in Fig. 14d. It represents, for all strategies of the pMDP, the parameter-substitution (as in Definition 27) of each induced pMC. To ensure that in the SG each state can be assigned to a unique player, we split states in the pMDP which have both (parametric) probabilistic branching and non-determinism, such that states have either probabilistic branching or non-determinism, but not both. The reformulation is done as follows: After each choice of actions, auxiliary states are introduced, such that the outcome of the action becomes deterministic and the probabilistic choice is delayed to the auxiliary state. This construction is similar to the conversion of Segala’s probabilistic automata into Hansson’s alternating model [127]. More precisely, we

• split each state $s \in S$ into $\{s\} \uplus \{\langle s, \alpha \rangle \mid \alpha \in Act (s) \}$,

• add a transition with probability one for each $s \in S$ and $\alpha \in Act (s)$. The transition leads from s to $\langle s, \alpha \rangle$, and

• move the probabilistic choice at s w. r. t. $\alpha$ to $\langle s, \alpha \rangle$.

Applying this to the pMDP from Fig. 14a, we obtain the pMDP $\mathcal {M}'$ in Fig. 14c, where the state s has only nondeterministic choices leading to states of the form $\langle s, \alpha \rangle$ with only probabilistic choices. The subsequent substitution on the probabilistic states yields the SG $\textsf {sub}_{R}(\mathcal {M}')$, where one player represents the nondeterminism of the original pMDP $\mathcal {M}$, while the other player decides whether parameters should be set to their lower or upper bound in the region R. For the construction, we generalise $V_s$ to state-action pairs: For a pMDP, a state s and action $\alpha$, let

$$\begin{aligned} V_{s,\alpha } = \{\, p \in V\mid p \text { occurs in } \mathcal {P}(s,\alpha ,s') \text { for some } s'\in S \, \}. \end{aligned}$$

Definition 30

(Substitution (pMDPs)) For pMDP $\mathcal {M}{}=(S{},V{},s_{ I }{}, Act {},\mathcal {P}{})$ and region R, let SG

$$\begin{aligned}\textsf {sub}_{R}(\mathcal {M}) = (S_{\bigcirc }\uplus S_{{\Box }}, s_{ I }, Act _{\textsf {sub}}, \mathcal {P}_{\textsf {sub}}) \end{aligned}$$

with

• $S_{\bigcirc }= S$

• $S_{{\Box }} = \{\langle s, \alpha \rangle \mid \alpha \in Act (s) \}$,

• $Act _{\textsf {sub}} = Act \uplus \big (\biguplus _{\langle s, \alpha \rangle \in S_{\Box }} Act _s^\alpha \big )$ where

  $$\begin{aligned} Act _s^\alpha = \{ u :V_{s,\alpha } \rightarrow \mathbb {R}\mid u(p) \in B(p) \;\forall p \in V_{s,\alpha } \}, \end{aligned}$$

  and,

• $$\begin{aligned} \mathcal {P}_{\textsf {sub}}(t,\beta ,t') = {\left\{ \begin{array}{ll} 1 &{} \text {if } t \in S_{\bigcirc }, \beta \in Act (t), t' {=} \langle t,\beta \rangle \in S_{\Box },\\ \mathcal {P}(s,\alpha ,t')[\beta ] &{} \text {if } t {=} \langle s,\alpha \rangle \in S_{\Box }, \beta \in Act _s^\alpha , t' \in S_{\bigcirc },\\ 0 &{} \text {otherwise.} \end{array}\right. } \end{aligned}$$

be the (parameter-)substitution of $\mathcal {M}$ and R.

We relate the SG $\textsf {sub}_{R}(\mathcal {M})$ under different strategies for player $\bigcirc$ with the substitution in the strategy-induced pMCs of $\mathcal {M}$. We observe that the strategies for player $\bigcirc$ in $\textsf {sub}_{R}(\mathcal {M})$ coincide with strategies in $\mathcal {M}$. Consider the induced MDP $(\textsf {sub}_{R}(\mathcal {M}))^\sigma$ with a strategy $\sigma$ for player $\bigcirc$. The MDP $(\textsf {sub}_{R}(\mathcal {M}))^\sigma$ is obtained from $\textsf {sub}_{R}(\mathcal {M})$ by erasing transitions not agreeing with $\sigma$. In $(\textsf {sub}_{R}(\mathcal {M}))^\sigma$ player $\bigcirc$-state have a single enabled action, while player ${\Box }$-states have multiple available enabled actions.

Example 42

Continuing Example 41, applying strategy $\sigma$ to $\textsf {sub}_{R}(\mathcal {M})$ yields $(\textsf {sub}_{R}(\mathcal {M}))^\sigma$, see Fig. 14e. The MDP $(\textsf {sub}_{R}(\mathcal {M}))^\sigma$ matches the MDP $\textsf {sub}_{R}(\mathcal {M}^{\sigma })$ apart from intermediate states of the form $\langle s, \alpha \rangle$: The outgoing transitions of s in $\textsf {sub}_{R}(\mathcal {M}^\sigma )$ coincide with the outgoing transitions of $\langle s, \alpha \rangle$ in $(\textsf {sub}_{R}(\mathcal {M}))^\sigma$, where $\langle s, \alpha \rangle$ is the unique successor of s.

The following corollary formalises that $(\textsf {sub}_{R}(\mathcal {M}))^\sigma$ and $\textsf {sub}_{R}(\mathcal {M}^{\sigma })$ induce the same reachability probabilities.

Corollary 6

For pMDP $\mathcal {M}$, graph-preserving region R, target states $T\subseteq S$, and strategies $\sigma \in Str _{\bigcirc }^{\textsf {sub}_{R}(\mathcal {M})}$ and $\rho \in Str ^{\textsf {sub}_{R}(\mathcal {M}^\sigma )}$, it holds that

$$\begin{aligned} \text {Pr}^{(\textsf {sub}_{R}(\mathcal {M}^\sigma ))^\rho }(\lozenge T) = \text {Pr}^{\textsf {sub}_{R}(\mathcal {M})^{\sigma , \widehat{\rho }}}(\lozenge T) \end{aligned}$$

with $\widehat{\rho } \in Str _{\Box }^{\textsf {sub}_{R}(\mathcal {M})}$ satisfies $\widehat{\rho }(\langle s, \sigma (s) \rangle ) = \rho (s)$.

Instead of performing the substitution on the pMC induced by $\mathcal {M}$ and $\sigma$, we can perform the substitution on $\mathcal {M}$ directly and preserve the reachability probability.

Consequently, and analogously to the pMC case (cf. Theorem 5), we can derive whether $\mathcal {M}, R \models _\clubsuit \varphi$ by analysing a stochastic game. For this, we consider various standard variants of model checking on stochastic games.

Definition 31

(Model-relation on SGs) For an SG $\mathcal {G}$, property $\varphi$, and quantifiers ${\mathcal {Q}}_1, {\mathcal {Q}}_2$, we define $\mathcal {G}\models ^{{\mathcal {Q}}_1,{\mathcal {Q}}_2} \varphi$ as:

$$\begin{aligned} {\mathcal {Q}}_1 \sigma _{\bigcirc }\in Str _{\bigcirc }^{\textsf {sub}_{R}(\mathcal {M})}.~{\mathcal {Q}}_2 \sigma _{\Box }\in Str _{\Box }^{\textsf {sub}_{R}(\mathcal {M})}\quad \mathcal {G}^{\sigma _{\bigcirc }, \sigma _{\Box }} \models \varphi \end{aligned}$$

The order of players, for these games, does not influence the outcome [48, 128].

Theorem 7

Let $\mathcal {M}$ be a pMDP, R a region, and $\varphi$ a reachability property, subject to Assumption 1¹⁴. Then:

$$\begin{aligned}&\textsf {sub}_{R}(\mathcal {M}) \models ^{\forall ,\forall } \varphi \text { implies } \mathcal {M}, R \models _d \varphi \text {, and }\\&\textsf {sub}_{R}(\mathcal {M}) \models ^{\exists ,\forall } \varphi \text { implies } \mathcal {M}, R \models _a \varphi . \end{aligned}$$

Proof

We only prove the second statement using $\varphi = \mathbb {P}_{> \lambda }(\lozenge T)$, other reachability properties are similar. A proof for the (simpler) first statement can be derived in an analogous manner. We have that $\mathcal {M}, R \models _a \mathbb {P}_{> \lambda }(\lozenge T)$ iff for all $u \in R$ there is a strategy $\sigma$ of ${\mathcal {M}}$ for which the reachability probability in the MC $\mathcal {M}^\sigma [u]$ exceeds the threshold $\lambda$, i. e.,

$$\begin{aligned} \mathcal {M}, R \models _a \mathbb {P}_{> \lambda }(\lozenge T)\iff \min _{u\in R} \max _{\sigma \in Str ^{\mathcal {M}}} \text {Pr}^{\mathcal {M}^\sigma [u]}(\lozenge T) > \lambda . \end{aligned}$$

A lower bound for this probability is obtained as follows:

$$\begin{aligned}&\min _{u\in R}\max _{\sigma \in Str ^{\mathcal {M}}} \big ( \text {Pr}^{\mathcal {M}^\sigma [u]}(\lozenge T) \big )\\&\quad \ge \max _{\sigma \in Str ^{\mathcal {M}}} \min _{u\in R}\big ( \text {Pr}^{\mathcal {M}^\sigma [u]}(\lozenge T) \big )\\&\quad \overset{*}{\ge }\ \max _{\sigma \in Str ^{\mathcal {M}}} \min _{\rho \in Str ^{\textsf {sub}_{R}(\mathcal {M}^\sigma )}}\big ( \text {Pr}^{(\textsf {sub}_{R}(\mathcal {M}^\sigma ))^\rho }(\lozenge T) \big )\\&\quad \overset{**}{=}\ \max _{\sigma \in Str _{\bigcirc }^{\textsf {sub}_{R}(\mathcal {M})}} \min _{\rho \in Str _{\Box }^{\textsf {sub}_{R}(\mathcal {M})}} \big (\text {Pr}^{\textsf {sub}_{R}(\mathcal {M})^{\sigma , \rho }}(\lozenge T) \big ). \end{aligned}$$

The inequality $*$ is due to Corollary 5. The equality $**$ holds by Corollary 6. Then:

$$\begin{aligned}&\textsf {sub}_{R}(\mathcal {M}) \models ^{\exists ,\forall } \mathbb {P}_{> \lambda }(\lozenge T)\\&\quad \iff \exists \sigma \in Str _{\bigcirc }^{\textsf {sub}_{R}(\mathcal {M})}.~\forall \rho \in Str _{\Box }^{\textsf {sub}_{R}(\mathcal {M})} \\&\quad \qquad \mathcal {G}^{\sigma , \rho } \models \mathbb {P}_{> \lambda }(\lozenge T)\\&\quad \iff \max _{\sigma \in Str _{\bigcirc }^{\mathcal {G}}} \Big ( \min _{\rho \in Str _{\Box }^{\mathcal {G}}} \big (\text {Pr}^{\mathcal {G}^{\sigma , \rho }}(\lozenge T) \big ) \Big )> \lambda \\&\quad \implies \min _{u\in R}\max _{\sigma \in Str ^{\mathcal {M}}} \big ( \text {Pr}^{\mathcal {M}^\sigma [u]}(\lozenge T) \big )> \lambda \\&\quad \iff \mathcal {M}, R \models _a \mathbb {P}_{> \lambda }(\lozenge T). \end{aligned}$$

$\square$

9 Approximate synthesis by parameter space partitioning

Parameter space partitioning is our iterative approach to the approximate synthesis problem. It builds on top of region verification, discussed above, and is, conceptually, independent of the methods used for verification discussed later.

Parameter space partitioning is best viewed as a counter-example guided abstraction refinement (CEGAR)-like [47] approach to successively divide the parameter space into accepting and rejecting regions. The main idea is to compute a sequence $\left( R^i_a \right) _i$ of simple accepting regions that successively extend each other. Similarly, an increasing sequence $\left( R^i_r \right) _i$ of simple rejecting regions is computed. At the i-th iteration, $R^i = R^i_a \cup R^i_r$ is the covered fragment of the parameter space. The iterative approach halts when $R^i$ is at least c% of the entire parameter space. Termination is guaranteed: in the limit a solution to the exact synthesis problem is obtained as $\lim _{i \rightarrow \infty } R_a^i = R_a$ and $\lim _{i \rightarrow \infty } R_r^i = R_r$.

Let us describe the synthesis loop for the approximate synthesis as depicted in Fig. 4 in detail. In particular, we discuss how to generate candidate regions that can be dispatched to the verifier along with a hypothesis whether the candidate region is accepting or rejecting. We focus on rectangular regions for several reasons:

• the automated generation of rectangular regions is easier to generalise to multiple dimensions,

• earlier experiments [65] revealed that rectangular regions lead to a more efficient SMT-based verification of regions (described in Sect. 6), and

• model-checking based region verification (described in Sect. 7) requires rectangular regions.

A downside of rectangular regions is that they are neither well-suited to approximate a region partitioning given by a diagonal, nor to cover well-defined regions that are not rectangular themselves.

Remark 11

In the following, we assume that the parameter space is given by a rectangular well-defined region R. If the parameter space is not rectangular, we over-approximate R by a rectangular region $\hat{R} \supseteq R$. If the potential over-approximation of the parameter space $\hat{R}$ is not well-defined, then we iteratively approximate $\hat{R}$ by a sequence of well-defined and ill-defined¹⁵ regions. The regions in the sequence of well-defined regions are then subject to the synthesis problem. Constructing the sequence of regions is done analogously to the partitioning into accepting and rejecting regions.

Before we present the procedure in full detail, we first outline a naive refinement procedure by means of an example.

Example 43

(Naive refinement loop) Consider the parametric die from Example 5. Suppose we want to synthesise the partitioning as depicted in Fig. 2. We start by verifying the full parameter space R against $\varphi$. The verifier returns false, as R is not accepting. Since R (based on our knowledge at this point) might be rejecting, we invoke the verifier with R and $\lnot \varphi$, yielding false too. Thus, the full parameter space R is inconsistent. We now split R into four equally-sized regions, all of which are inconsistent. Only after splitting again, we find the first accepting and rejecting regions. After various iterations, the procedure leads to the partitioning in Fig. 15.

Fig. 15

Parameter space partitioning into safe and unsafe regions

Algorithm 5 describes this naive region partitioning procedure. It takes a pSG, a region R, a specification $\varphi$, and a (demonic or angelic) satisfaction relation as input. It first initialises a (priority) queue Q with R. In each iteration, a subregion $R'$ of R is taken from the queue, the counter i is incremented, and the sequence of accepted and rejected regions is updated. There are three possibilities. Either $R'$ is accepting (or rejecting), and $R_a^{i}$ ($R_r^{i}$) extends $R_a^{i-1}$ ($R_r^{i-1}$) with $R'$, or $R'$ is inconsistent. In the latter case, we split $R'$ into a finite set of subregions that are inserted into the queue Q. Regions that are not extended are unchanged.

Algorithm 5

Naive refinement loop

The algorithm only terminates if $R_a$ and $R_r$ are a finite union of hyper-rectangles. However, the algorithm can be terminated after any iteration yielding a sound approximation. The algorithm ensures $\lim _{i \rightarrow \infty } R^i = R$, if we order Q according to the size of the regions. We omit the technical proof here; the elementary property is that the regions are Lebesgue-measurable (and have a positive measure by construction).

The naive algorithm has a couple of structural weaknesses:

• It invokes the verification algorithm twice to determine that the full parameter space is inconsistent.

• It does not provide any (diagnostic) information from a verification invocation yielding false.

• It checks whether a region is accepting before it checks whether it is rejecting. This order is suboptimal if the region is rejecting.

• If the region is inconsistent, it splits the region into $2^n$ equally large regions. Instead, it might be beneficial to select a smaller number of regions (only split in one dimension).

• Uninformed splitting yields many inconsistent subregions. Splitting in only one dimension even increases the number of verification calls yielding false.

In the remainder of this section, we discuss ways to alleviate these weaknesses. The proposed improvements are based on empirical observations about the benchmarks and are in line with the implementation in our tool PROPhESY. In particular, we tailor the heuristics to “well-behaved” models and specifications, which reflect the benchmarks from various domains. The notion of being well-behaved refers to

• a limited number of connected accepting and rejecting regions with smooth (albeit highly non-linear) borders between these regions.

• a limited number of accepting (rejecting) instantiations that are close to a rejecting (accepting) instantiations. We call instantiations that form a border between $R_a$ and $R_r$ border instantiations.

The parameter space depicted in Fig. 15 is well-behaved. It features only two connected regions, with a smooth border between them. Furthermore, the regions have a considerable interior, or equivalently, many instantiations are not too close to the border. We remark that we do rely on these assumptions to hold, but PROPhESY will be slow on models that are not well-behaved.

9.1 Sampling

A simple but effective improvement is to verify an instantiated model $\mathcal {G}[u]$ for some instantiation (a sample) $u \in R$. The verification result either reveals that the region is not accepting, if $\mathcal {G}[u]\not \models _\clubsuit \varphi$, or not rejecting, if $\mathcal {G}[u]\models _\clubsuit \varphi$. Two samples within a region R may suffice to conclude that R is inconsistent. In order to quickly find inconsistent regions by sampling, it is beneficial to seek for border instantiations. To this end, a good strategy is to start with a coarse (random) sampling to get a first indication of border instantiations. We then select additional instantiations by intra-/extrapolation between these samples.

Fig. 16

Parameter space partitioning in progress: Images generated by PROPhESY

Example 44

We discuss how sampling may improve the naive refinement loop as discussed in Example 43. Fig. 16a shows a uniform sampling. Red crosses indicate that the instantiated pMC satisfies $\lnot \varphi$, while green dots indicate that the instantiation satisfies $\varphi$. The blue rectangle is a candidate region (with the hypothesis $\lnot \varphi$, indicated by the hatching), which is consistent with all samples.

9.2 Finding region candidates

Fig. 17

Creating region candidates based on samples

We use the sampling results to steer the selection of a candidate region that may either be accepting or rejecting. A simple strategy is to split regions that we found to be inconsistent via sampling.

Example 45

Consider the parameter space with six samples depicted in Fig. 17a. After verifying only six instantiated models, we conclude that the parameter space is inconsistent.

Algorithm 6

Sampling-based refinement loop

The use of samples allows to improve the naive refinement scheme as given in Algorithm 5. This improvement is given in Algorithm 6. For each region R, we have a finite set X of samples. For each sample $u \in X$, it is known whether $\mathcal {G}[u] \models _\clubsuit \varphi$. The queue Q now contains pairs (R, X).

In each iteration, a pair $(R',X')$ where $R'$ is (as before) a subregion of R is taken from the queue. Then, we distinguish (again) three possibilities. Only when all samples in $X'$ satisfy $\varphi$, it is verified whether $R'$ is accepting. If $R'$ is accepting, we proceed as before: $R_a^{i}$ is extended by $R'$ while $R_r^{i}$ remains unchanged. In the symmetric case that all samples in $X'$ refute $\varphi$, we proceed in a similar way by verifying whether $R'$ rejects $\varphi$. Otherwise, $R'$ is split into a finite set of subregions with corresponding subsets of $X'$, and added to the queue Q. In case the verification engine provides a counterexample, we can add this counterexample as a new sample. We thus ensure that for all $(R', X') \in Q$, $u \in X'$ implies $u \in R'$. The algorithm can be easily extended such that sampling is also done once a region without samples is obtained: rather than inserting $(R', \emptyset )$ into Q, we insert the entry $(R', \text {{\textbf {sample}}}(R'))$.

Example 46

After several more iterations, the refinement loop started in Example 44 has proceeded to the state in Fig. 16b. First, we see that the candidate region from Fig. 16a was not rejecting. The verification engine gave a counterexample in form of an accepting sample (around $p \mapsto 0.45, q\mapsto 0.52$). Further iterations with smaller regions had some successes, but some additional samples were generated as counterexamples. The current blue candidate is to be checked next. In Fig. 16c, we see a further continuation, with even smaller regions being verified. Note the white box on the right border: It has been checked, but the verification timed out without a conclusive answer. Therefore, we do not have a counterexample in this subregion.

It remains to discuss some methods to split a region, and how we may discard some of the constructed regions. We outline more details below.

9.2.1 How to split

Splitting of regions based on the available samples can be done using different strategies. We outline two basic approaches. These approaches can be easily mixed and extended, and their performance heavily depends on the concrete example at hand.

Equal splitting. This approach splits regions in equally-sized regions; the main rationale is that this generates small regions with concise bounds (the bounds are typically powers of two). Splitting in equally sized regions can be done recursively: One projects all samples down to a single dimension, and splits if both accepting and rejecting samples are in the region. The procedure halts if all samples in a region are either accepting or rejecting. The order in which parameters are considered plays a crucial role. Typically, it is a good idea to first split along the larger dimensions.

Example 47

A split in equally-sized regions is depicted in Fig. 17b, where first the left region candidate is created. The remaining region can be split either horizontally or vertically to immediately generate another region candidate. A horizontal split in the remaining region yields a region without any samples.

The downside of equal splitting is that the position of the splits are not adapted based on the samples. Therefore, the number of splits might be significantly larger than necessary, leading to an increased number of verification calls.

Growing rectangles. This approach attempts to gradually obtain a large region candidate¹⁶. The underlying rationale is to quickly cover vast amounts of the parameter space. This is illustrated in Fig. 17d (notice that we adapted the samples for a consistent but concise description) where from an initial sampling a large rectangle is obtained as region candidate.

Example 48

Consider the shaded regions in Fig. 17c. Starting from vertex $v=(1,1)$, the outer rectangle is maximised to not contain any accepting samples. Taking this outer rectangle as candidate region is very optimistic, it assumes that the accepting samples are on the border. A more pessimistic variant of growing rectangles is given by the inner shaded region. It takes a rejecting sample as vertex $v'$ such that the v and $v'$ span the largest region.

The growing rectangles algorithm iterates over a subset of the hyper-rectangle’s vertices: For each vertex (referred to as anchor), among all possible sub-hyper-rectangles containing the anchor and only accepting or only rejecting samples, the largest is constructed.

Example 49

The growing rectangles approach pessimistically takes anchor (0, 0) as anchor and yields the candidate region in Fig. 17d.

The verification fails more often on large regions (either due to time-outs or due to the over-approximation). Consequently, choosing large candidate regions comes at the risk of failed verification calls, and fragmentation of the parameter space in more subregions.

Furthermore, growing rectangles requires a fall-back splitting strategy: To see why, consider Fig. 15. The accepting (green) region does not contain any anchors of the full parameter space, therefore the hypothesis for any created subregion is always rejection. Thus, no subregion containing a (known) accepting sample is ever considered as a region candidate.

9.2.2 Neighbourhood analysis

Besides considering samples within a region, we would like to illustrate that analysis of a region R can and should take information from outside of R into account. First, take Fig. 17b, and assume that the left region is indeed accepting. The second generated region contains only rejecting samples, but it is only rejecting if all points, including all those on the border to the left region, are rejecting. In other words, the border between the accepting and rejecting regions needs to exactly follow the border between the generated region candidates. The latter case does not occur often, so it is reasonable to shrink or split the second generated region. Secondly, a sensible hypothesis for candidate regions without samples inside is helpful, especially for small regions or in high dimensions. Instead of spawning new samples, we take samples and decided regions outside of the candidate region into account to create a hypothesis. Concretely, we infer the hypothesis for regions without samples via the closest known region or sample.

9.3 Requirements on verification back-ends

In this section, we have described techniques for iteratively partitioning the parameter space into accepting and rejecting regions. The algorithms rely on verifying regions (and sets of samples) against the specification $\varphi$. The way in which verification is used in the iterative parameter space partitioning scheme imposes the following requirements on the verification back-end:

• (i) The verification should work incrementally. That is to say, verification results from previous iterations should be re-used in successive iterations. Verifying different regions share the same model (pMC or pMDP). A simple example of working incrementally is to reuse minimisation techniques for the model over several calls. If a subregion is checked, the problem is even incremental in a more narrow sense: any bounds etc. obtained for the super-region are also valid for the subregion.

• (ii) If the verification procedure fails, i.e. if the verifier returns false, obtaining additional diagnostic information in the form of a counterexample is beneficial. A counterexample here is a sample which refutes the verification problem at hand.

This wish list is very similar to the typical requirements that theory solvers in lazy SMT frameworks should fulfil [23]. Therefore, SMT-based verification approaches naturally match the wish-list. Parameter-lifting can work incrementally: it reuses the graph-structure to avoid rebuilding the MDP, and it may use previous model checking results to improve the time until the model checker converges. Parameter-lifting, due to its approximative nature, does provide only limited diagnostic information: In particular, it provides information which parameters would be assigned with the upper or lower bounds based on the strategy that optimizes the MDP/SG.

10 Implementation

All the algorithms and constructions in this paper have been implemented, and are publicly available via PROPhESY¹⁷. In particular, PROPhESY supports algorithms for:

• the exact synthesis problem: via computing the solution function, using either of the three variants of state elimination, discussed in Sect. 5.

• the verification problem: via an encoding to an SMT-solver as in Sect. 6 or by employing the parameter lifting method as in Sects. 7 and 8.

• the approximate synthesis problem: via parameter space partitioning, that iteratively generates verification calls as described in Sect. 9.

PROPhESY is implemented in python, and designed as a flexible toolbox for developing and experimenting with parameter synthesis. PROPhESY internally heavily relies on high-performance routines of the probabilistic model checker Storm [66] and the SMT Z3. PROPhESY is built in a modular way, such that it is easy to use different backend solvers. The computation of the solution function and the parameter lifting presented in the experiments have been implemented in Storm.

PROPhESY can be divided in three parts:

• (i) First and foremost, it presents a library consisting of: (a) data structures for parameter spaces and instantiations, solution functions, specifications, etc., built around the python bindings of the library carl¹⁸ (featuring computations with polynomials and rational functions), (b) algorithms such as guided sampling, various candidate region generation procedures, decomposition of regions, etc., methods that require tight integration with the model are realised via the python bindings of Storm¹⁹, (c) abstract interfaces to backend tools, in particular probabilistic model checkers, and SMT-checkers, together with some concrete adapters for the different solvers, see Fig. 18.

• (ii) An extensive command-line interface which provides simple access to the different core functionalities of the library, ranging from sampling to full parameter synthesis.

• (iii) A prototypical web-service running on top of the library, which allows users to interact with the parameter synthesis via a web-interface.

PROPhESY is constructed in a modular fashion: besides the python bindings for carl, all non-standard packages and tools (in particular model checkers and SMT solvers) are optional. Naturally, the full power of PROPhESY can only be used if these packages are available. Besides the methods presented in this paper, PROPhESY contains two further mature parameter synthesis methods: (i) particle-swarm optimisation inspired by [43], and (ii) convex optimisation from [57].

The information in the remainder details the implementation and the possibilities provided by PROPhESY. The section contains some notions from probabilistic model checking [13, 16, 98]. We refrain from providing detailed descriptions of these notions, as it would go beyond the scope of this paper.

Fig. 18

High-level architecture of PROPhESY and its backends

10.1 Model construction and preprocessing (Realised in Storm )

The model checker Storm supports the creation of pMCs and pMDPs from both PRISM-language model descriptions [102] and JANI-specifications [32]. The latter can be used as intermediate format to support, e.g., digital-clock PTAs with parameters written in Modest [80], or to support expected time properties of generalised stochastic Petri nets [109] with parametric rates and/or weights. Parametric models can be built using the matrix-based, explicit representation, as well as the symbolic, decision diagram (dd)-based engine built on top of sylvan [135]. Both engines support the computation of qualitative properties, an essential preprocessing step, and bisimulation minimisation on parametric models, as described in [78]. We advocate the use of the Storm-python API adapter: Its interactive nature avoids the repetition of expensive steps. In particular, it allows for the incremental usage of parameter lifting and sampling.

The support for rational functions is realised via the library carl²⁰. The rational function is stored as a tuple consisting of multivariate polynomials. These polynomials are by default stored in a partially factorised fashion, cf. [91]. Each factor (a polynomial) is stored as an ordered sparse sum of terms, each term consists of the coefficient and a sparse representation of variables with their non-zero exponents. For manipulating the (rational) coefficients, we exploit gmp²¹ or cln²². The former is thread-safe, while the latter performs slightly better with single-thread usage. Computation of GCDs in multivariate polynomials is done either via ginac [22] or cocoa [2].

10.2 Solution function computation (Realised in Storm )

The computation of solution functions for pMCs as discussed in Sect. 5 is implemented for a variety of specifications:

• reachability and reach-avoid probabilities,

• expected rewards, including expected time of continuous-time Markov chains,

• step-bounded reachability probabilities, and

• long-run average probabilities and rewards.

The computation is realised either via state elimination, or via Gaussian elimination. An implementation of set-based transition elimination is available for symbolic representations of the pMC.

10.2.1 State elimination

As the standard sparse matrix representation used by Storm is not suitable for fast removal and insertion of entries, a flexible sparse matrix with faster delete and insert operations is used.

The order in which states are eliminated has a severe impact on the performance [65]. Storm supports a variety of static (pre-computed) and dynamic orderings for the elimination:

• several static orders (forward (reversed), backward (reversed)) based on the order of state-generation by the model construction algorithms. This latter order is typically determined by a depth-first search through the high-level model description²³,

• orders based on the topology of the pMC, e.g., based on the decomposition in strongly connected components,

• orders (Regex) which take into account the in-degree (the number of incoming transitions at a state), inspired by [84, 125],

• orders (SPen, DPen) which take into account the complexity of the rational function corresponding to the transition probability. The complexity is defined by the degree and number of terms of the occurring polynomials.

The orders are computed as penalties for states, and the order prefers states with a low penalty. For dynamic orderings (Regex, DPen), the penalties are recomputed as the in-degree of states and complexity of transition probabilities change during state elimination.

10.2.2 Gaussian elimination

Storm supports Eigen [76] as a linear equation system solver over the field of rational functions. It uses the “supernodal” (supernodes) LU factorisation. The matrix is permuted by the column approximate minimum degree permutation (COLAMD) algorithm to reorder the matrix. One advantage is that this solver is based on sparse model-checking algorithm for parameter-free models. The solver therefore, in addition to the properties supported by state elimination, supports the construction in [15] for conditional probabilities and rewards.

10.2.3 Set-based transition elimination

This elimination method is targeted for symbolic representations of the Markov chain. Set-based transition elimination is implemented via matrix-matrix multiplications. In every multiplication, a copy of the dd-representation of a matrix over variables $(\vec {s},\vec {t})$ is made. The copy uses renamed dd-variables $(\vec {t}, \vec {t'})$. Then, a multiplication of the original matrix with the copy can be done on the dd level yielding a matrix $(\vec {s}, \vec {t'})$. Renaming $\vec {t'}$ to $\vec {t}$ yields a matrix on the original dd-variables.

10.3 Parameter lifting (Realised in Storm )

For parameter lifting (Sects. 7 and 8), the major effort beyond calling standard model-checking procedures is the construction of the substituted (lifted) model. As parameter lifting for different regions does not change the topology of the lifted model, it is beneficial to create a template of the lifted model once, and to substitute the values according to the region at hand. The substitution operation can be sped up by exploiting the following observation: Typically, transition probability functions coincide for many transitions. Thus, we evaluate each occurring function once and substitute the outcome directly at all occurrences. Moreover, for a growing number of regions to be checked, any one-time preprocessing of the lifted model eventually pays off. In particular, we apply minimisation techniques before construction of the lifted model. We use both bisimulation minimisation as well as state elimination of parameter-free transitions. These minimisations drastically reduce the run-time of checking a single region. We use numerical methods first: for regions that we want to classify as accepting (or rejecting) we resort to the analysis of MDPs using policy iteration with rational numbers. For that, we initialise the policy iteration with a guess based on the earlier numerical results.

10.4 SMT-based region verification (Realised in PROPhESY )

This complete region checking procedure is realised by constructing SMT queries, as elaborated in Sect. 6. When invoking the SMT solver, we use some features of the SMT-lib standard [18]. First of all, when checking several regions, we use backtrack-points to only partly reset the solver: More precisely, the problem description is given by a conjunction of subformulae, where the conjunction is represented by a stack. We first push the constraints for the problem to the stack, save a backtrack point, and then store the region. Once we have checked a particular region, we backtrack to the backtrack point, that is, we remove the constraints for the particular region from the problem description. This way, we reuse simplifications and data structures the solver constructed for the problem description covering the model (and not the region). To support both verifying the property and its negation, the problem description is slightly extended. We add two Boolean variables (accepting and rejecting). The following gives an example of the encoding together with checking whether a region $R_1$ is accepting, and a region $R_2$ is rejecting, using the notation of Sect. 6.

$$\begin{aligned}&x = f_{\mathcal {D},\varphi } \wedge \big ( \textit{accepting} \implies x \ge \lambda \big ) \wedge \big ( \textit{rejecting} \implies x < \lambda \big ) \\&(\textsf {push}) \\&\textit{ accepting} \wedge \Upphi (R_1) \\&(\textsf {pop}) \qquad (\textsf {push}) \\&\textit{ rejecting} \wedge \Upphi (R_2) \end{aligned}$$

10.5 Sampling (Realised in PROPhESY )

We accelerate the selection of regions by getting a rough picture through sampling, as discussed in Sect. 9. We support two engines for computing the samples: Either via model checking, or by instantiating the solution function. Sampling on the solution function should always be done exactly, as the evaluation of the typically highly-nonlinear solution functions is (again typically) numerically unstable. In each iteration, based on the current set of samples, a new set of sampling candidates is computed. The choice of the new samples can be modified in several ways. The standard used here is via linear interpolation between accepting and rejecting samples.

10.6 Partitioning (Realised in PROPhESY )

For the construction of region candidates, we split the initial regions according to our heuristic (quads or growing rectangles, cf. Sect. 9.2) until none of the regions is inconsistent. We sort the candidate regions based on their size in descending order. Furthermore, we prefer regions where we deem verification to be less costly: Candidate regions that are supposed to be accepting and are further away from samples or regions that are rejecting are preferred over those regions which have rejecting samples or regions in their neighbourhood.

11 Experimental evaluation

In this section, we review the scalability of the presented approaches based on a selection of benchmarks.

11.1 Set-up

11.1.1 Benchmarks

We consider five case studies from the literature. The selection represents various application domains.

NAND multiplexing. With integrated circuits being built at ever smaller scale, they are more prone to defects and/or to exhibit transient failures [85]. One way to overcome these deficiencies is the implementation of redundancy at gate-level. In particular, one aims to construct reliable devices from unreliable components. NAND multiplexing is such a technique, originally due to von Neumann [140]. Automated analysis of NAND multiplexing via Markov chain model checking was considered first in [115]. They also studied the influence of gate failures in either of the stages of the multiplexing by sampling various values. We use the pMC from [65], that replaced fixed probabilities in the original formulation with parameters. We analyse the effect of changing failure probabilities of the gates on the reliability of the multiplexed NAND.

Herman’s self-stabilising protocol. In distributed systems, tokens are used to grant privileges (e.g., access to shared memory) to processes. Randomisation is an essential technique to break the symmetry among several processes [7]. Herman’s probabilistic algorithm [88] is a token circulation algorithm for ring structures. In each step, every process possessing a token passes the token along with probability p and keeps the token with probability $1{-}p$. The algorithm is self-stabilising, i.e., started from any illegal configuration with more than one token the algorithm recovers to a legal configuration with a unique token. The recovery time crucially depends on the probability of passing the token, and an optimal value for p depends on the size of the system [105]. We investigate the expected recovery time by parameter synthesis, inspired by [3].

Mean-time-to-failure of a computer system. In reliability engineering, fault trees are a prominent model to describe how a system may fail based on faults of its various components [24, 123]. Dynamic fault trees (DFTs, [71]) extend these fault trees with a notion of a state, and allow to model spare management and temporal dependencies in the failure behaviour. State-of-the-art approaches for dynamic fault trees translate such fault trees into Markov chains [27, 50, 139]; evaluation of the mean-time-to-failure boils down to the analysis of the underlying Markov chain. Probabilities and rewards originate from the failure rate of the components in the described system. Such failure rates are often not known (precisely), especially during design time. Therefore, they may be represented by parameters. We take the HECS DFT [137] benchmark describing the failure of a computer system with an unknown failure rate for the software interface and the spare processor, as first described in [138]. We analyse how this failure rate affects the expected time until the failure (mean-time-to-failure) of the complete computer system.

Network scheduling. This benchmark [143] concerns the wireless downlink scheduling of traffic to different users, with hard deadlines and prioritised packets. The system is time-slotted: time is divided into periods and each period is divided into an equal number of slots. At the start of each time period, a new packet is generated for each user with a randomly assigned priority. The goal of scheduling is to, in each period, deliver the packets to each user before the period ends. Packets not delivered by the end of a period are dropped. Scheduling is non-trivial, as successful transmissions are not stochastically independent, i.e., channels have a (hidden) internal state. The system is described as a partially observable Markov decision process [124], a prominent formalism in the AI community. We take the Network model from [116], and consider the pMC that describes randomised finite memory controllers that solve this scheduling problem, based on a translation from [96]. Concretely, the parameters represent how the finite memory controller randomises. We evaluate the effect of the randomisation in the scheduling on the expected packet loss.

Bounded retransmission protocol. The bounded retransmission protocol (BRP, [61, 87]) is a variant of the alternating bit protocol. It can be used as part of an OSI data link layer, to implement retransmitting corrupted file chunks between a sender and a receiver. The system contains two channels; from sender to receiver and vice versa. BRP is a famous benchmark in (non-parametric) probabilistic model checking, based on a model in [62]. We consider the parametric version from [78]. The parameters naturally reflect the channel qualities. The model contains non-determinism as the arrival of files on the link layer cannot be influenced. This non-determinism hampers a manual analysis. The combination of parametric probabilities and non-determinism naturally yields a pMDP. We analyse the maximum probability that a sender eventually does not report a successful transmission.

Remark 12

Other benchmarks and a thorough performance evaluation have been presented before in [65] (for state elimination and parameter space partitioning) and [122] (for parameter lifting).

Table 1 Detailed information for models in the benchmark set

11.1.2 Benchmark statistics

Table 1 summarises relevant information about the concrete instances that we took from the benchmarks. The id is used for reference. The benchmark refers to the name of the benchmark-set, while the instance describes the particular instance from this benchmark set. We give the total number of parameters |V| both in the transition matrix as well as in the reward structure whenever applicable. For the remainder of the columns, we give two numbers per benchmark instance: The upper row describes the original model, the latter describes the (strong) bisimulation quotient. The columns give the number of states and transitions. The last row gives the time (in seconds) required for constructing the model (top) and constructing the bisimulation quotient (bottom). We remark that all benchmarks have a limited number of parameters: Systems with many parameters are beyond the reach of the methods discussed here, but can be analysed with respect to simpler synthesis questions (such as finding one suitable instantiation). We refer to the related work for a discussion of such methods.

11.1.3 Evaluation

We conducted the empirical evaluation on an HP BL685C G7 with Debian 9.6. Each evaluation run could use 8 cores with 2.1GHz each. However, unless specified otherwise, algorithms use a single core. We set the timeout to 1 hour and the memory limit to 16GB. We used PROPhESY version 2.0, together with the Storm-python bindings version 1.3.1, z3 version 4.8.4. All benchmark files are made available via PROPhESY²⁴.

11.2 Exact synthesis via the solution function

To evaluate the exact synthesis approach, we use state elimination with 7 different heuristics, set-based transition elimination, and Gaussian elimination. All configurations are evaluated with and without strong bisimulation.

Table 2 Empirical performance of computing the solution function

First, we show the sizes of the solution function: The results are summarised in Table 2. The id references the corresponding benchmark instance in Table 1. The BRP pMDP is not included. The set of all strategies prevents the computation of the solution function for all induced pMCs. The next four columns display properties of the resulting rational function. We give the degree of both the numerator (degree num) and denominator (degree denom), as well as the number of terms in both polynomials (# terms num, # terms denom). The next column gives the number of configurations (out of the 18) which successfully finished within the time limit. The last two columns indicate timings. We give the times (in seconds) to compute the solution function (time mc) and the total time including model building, (optional) bisimulation minimisation and computing the solution function. For these timings we give two numbers per benchmark instance: The upper row describes the median value over all successful configurations and the lower row describes the best result obtained. Thus, while functions often grow prohibitively large, medium-sized functions can still be computed. Contrary to model checking for parameter-free models, model building is typically not the bottleneck.

Furthermore, we see that the selected heuristic is indeed crucial. Consider instance 11: 11 heuristics successfully compute the solution function (and most of them within a second). However, 7 others yield a timeout. That leads us to compare some heuristics in Fig. 19. The plot depicts the cumulative solving times for selected configurations over all 18 benchmark instances (excluding BRP). Gaussian and set-based refer to these approaches, respectively, all other configurations are variants of state elimination, cf. Sect. 10.2.1, (bisim) denotes that bisimulation minimisation is used. The x-axis represents the number of solved instances and the (logarithmic) y-axis represents the time in seconds. A point (x, y) in the plot represents the x fastest instances which could be solved within a total time of y seconds. For 15 instances, one of the depicted configurations was the fastest overall. Regex based configurations were the fastest eight times, DPen based ones four times and three times configurations based on FwRev were fastest. From these numbers, we conclude that the selection of the heuristic is essential, and depending on the model to be analysed. From the graph, we further observe that although using a Gaussian elimination yields good performance, state-elimination approaches can (significantly) outperform the Gaussian elimination on some benchmarks. The DPen solves all instances (the only configuration to do so), but Regex is overall (slightly) faster. The uninformed FwRev with bisimulation works surprisingly well for these benchmarks (but that is mostly coincidence). The set-based elimination is clearly inferior on the benchmarks considered here, but allows to analyse some models with a very regular structure and a gigantic state space, e.g., a parametric Markov chain for the analysis of the bluetooth protocol [70].

Fig. 19

Cumulative solving times for solution function computation

11.3 Three types of region verification

We evaluate region verification using two SMT-based approaches (SF: based on first computing the Solution Function, or ETR: encoding the equations into Existential Theory of the Reals), and PLA. In particular, we present some results for the Herman benchmark: it features a single parameter, and therefore is well-suited for the illustration of some concepts. We visualised the results for instance 11 in Fig. 20.

Fig. 20

Plot for Herman model with seven processes and parameter p (Benchmark Id: 11)

The x-axis represents the probability p and the y-axis the expected recovery time. We indicate the solution function in blue. The threshold in the following is set to $\lambda =5$ and indicated by the orange horizontal line. The black columns depict six different regions²⁵ that are evaluated with region checking. For each region we want to verify whether the expected recovery time is at least 5. The results are summarised in (the upper part of) Table 3.The first column id references the benchmark instance and the second column gives the threshold $\lambda$. The next columns indicate the considered region and the technique. The last columns give the result of the region verification and the time (in seconds) needed for the computation. The timeout (TO) was set to 120 s.

Table 3 Empirical performance of region verification algorithms

For benchmark instance 11, Parameter lifting (PLA) computes a result within milliseconds and the computation time is independent of the considered region. The SMT-based techniques take longer and the SF technique in particular does not terminate within two minutes. However, the ETR technique could yield a result for region [0.28, 0.35] whereas PLA could not give a conclusive answer due to its inherent over-approximation.

Fig. 21

Plotting the solution function for NAND $K=2,N=2$ (Benchmark Id: 13) and parameters prob1 and perr

We now consider the region verification on the NAND model with two parameters. We visualised the solution function for instance 13 in Fig. 21. The considered threshold is $\lambda =0.3$. Green coloured parts indicate parameter instantiations leading to probabilities above $\lambda$ and red parts lie below $\lambda$. The results of the verification for different regions are given in (the lower part of) Table 3. PLA is again the fastest technique, but for larger regions close to the threshold PLA can often not provide a conclusive answer. Contrary to before, SF is superior to ETR.

The performance of the SMT-based techniques (again) greatly depends on the considered region. It is only natural that the size of the region, and the difference to the threshold have a significant influence on the performance of region verification. These observations are general and do hold on all other benchmarks. Furthermore, parameter lifting seems broadly applicable, and in the setting evaluated here, clearly faster than SMT-based approaches. Parameter lifting over-approximates and therefore might only give a decisive result in a refinement loop such as parameter space partitioning. The SMT-based approaches are a valuable fallback. When relying on the SMT techniques, it is heavily model-dependent which performs better. Table 4 at the end of the next section gives some additional results, indicating the performance of the different verification techniques.

11.4 Approximative synthesis via parameter space partitioning

We now evaluate the parameter space partitioning. We use the implementation in PROPhESY with the three verification procedures evaluated above. Therefore, we focus here on the actual parameter space partitioning.

First, consider again Herman for illustration purposes. Region verification is not applicable for instance 10 (with threshold 5), as neither all instantiations accept nor all reject the specification. Instead, parameter space partitioning delivers which of these instantiations accept, and which reject the specification. The resulting parameter space partitioning is visualised in Fig. 22.

Fig. 22

Parameter space partitioning for Herman $N=5$ (Benchmark Id: 10) with parameter p

Fig. 23

Covered areas for parameter space partitioning on different models and thresholds

Next, we compare the three verification techniques—each with two different methods for selecting candidate regions—in Fig. 23. Figure 23a depicts the computation on the Herman model with 5 processes and threshold $\lambda =5$. The plot depicts the covered area for all three techniques with both quads (straight lines) and rectangles (dashed lines) as regions. The x-axis represents the computation time (in seconds) on a logarithmic scale and the y-axis represents the percentage of covered area. A point (x, y) in the plot represents y percent of the parameter space which could be covered within x seconds.

For Herman, SMT-based techniques perform better than PLA. PLA was able to cover 64% of the parameter space within milliseconds. However, in the remaining hour only 2% more space was covered. The SMT-based techniques were able to cover at least 99% of the parameter space within 15 s. Moreover, the rectangles cover the parameter space faster than quads. We also perform the parameter space partitioning on the NAND model with two different thresholds: We compare the parameter space partitioning techniques for threshold $\lambda =0.1$ in Fig. 23b, and for threshold $\lambda =0.3$ in Fig. 23c. For NAND, the PLA technique performs better than the SMT-based techniques. For threshold $\lambda =0.1$, PLA could cover at least 99% of the parameter space within 1 s. The main reason is that the border is in a corner of the parameter space. Additionally, the SMT-based techniques with rectangles are significantly faster than the quads for this threshold. For threshold $\lambda =0.3$, more region verification steps were necessary. PLA still outperforms ETR and SF. However, the use of rectangles over quads does not lead to a better performance for this threshold. At any point in time, there can be very significant differences between the heuristics for candidate generation, especially in settings where single region verification calls become expensive.

Table 4 Empirical performance of parameter space partitioning variations

Finally, we summarise an overview of the performance in Table 4. For brevity, we pruned some rows, especially if the present approaches already struggle with smaller instances. The id is a reference to the benchmark instance. The technique is given in the next column. In the next three columns we give for each technique the time (in seconds) needed to cover at least 50%, 90% and 98% of the complete parameter space. The next two columns give the complete covered area—i.e. the sum of the sizes of all accepting or rejecting regions— when terminating the parameter space partitioning after 1h, together with the safe area, i.e. the sum of the sizes of all accepting regions. The last two columns indicate the percentage of the total time spent in generating the regions (time reg gen) and verifying the regions (time analysis). PLA is almost always superior, but not on all benchmarks (and not on all (sub)regions. Depending on the model, SF or ETR are the best SMT-based technique. There might be room for improvement by portfolios and machine-learned algorithm selection schemes.

12 Related work and discussion

We discuss related work with respect to various relevant topics.

Complexity. For graph-preserving pMCs, many complexity results are collected in [97], including results from [45]. In particular, the complement of the verification problem, i.e., the question whether there exists an instantiation in a region that satisfies a reachability property, is ETR-complete for both pMDPs and pMCs²⁶. For any fixed number of parameters, the problem can be solved in polynomial time [17]. This paper also considers a richer fragment of the logic PCTL.

Computing a solution function. This approach was pioneered by [64] and significantly improved by [78]. Both PRISM [102] and PARAM [77] support the computation of a solution function based on the latter method. It has been adapted in [91] to an elimination of SCCs and a more clever representation of rational functions. This representation has been adapted by Storm [66]. In [72], computing a solution function via a computer algebra system was considered. That method targets small, randomly generated pMCs with many parameters. Recently, [17] explored the use of one-step fraction-free Gaussian elimination to reduce the number of GCD computations. For pMDPs, [79] experimented with the introduction of discrete parameters to reflect strategy choices—this method, however, scales poorly. In [67] and [68], variants of value iteration with a dd-based representation of the solution function are presented. Fast sampling on (concise representations of) the solution function is considered in [73, 89].

Equation system formulation. Regarding pMDPs, instead of introducing a Boolean structure, one can lift the linear program formulation for MDPs to a nonlinear program (NLP). This lifting has been explored in [20], and shown to be not feasible in general. A string of results rely on convex programming approaches. For instance, although the general NLP does not lie in the class of convex problems, a variety of verification related problems can be expressed by a sequence of geometric programs, which is exploited in [56]. Alternatively, finding satisfying parameter instantiations in pMDPs under demonic non-determinism and with affine transition probabilities can be approached by iteratively solving a convex-concave program that approximates the original NLP [57]. A comprehensive overview of exploiting convex programming is presented in [60]. Alternatively, more efficient solvers can be used [42] for subclasses of pMDPs. An alternative parametric model with a finite set of parameter instantiations, but without the assumption that these instantiations are graph preserving is considered in [41].

Model repair. The problem of model repair is related to parameter synthesis. In particular, for a Markov model and a refuted specification the problem is to transform the model such that the specification is satisfied. In the special case where repair amounts to changing transition probabilities, the underlying model is parametric as in this paper: the parameters are addive factors to be added to the original transition probabilities. The problem was first defined and solved either by a nonlinear program or parameter synthesis in [20]. A greedy approach was given in [117] and efficient simulation-based methods are presented in [43]. In addition, parametric models are used to rank patches in the repair of software [107].

Interval Markov chains. Instead of parametric transitions, interval MCs or MDPs feature intervals at their transitions [10, 74, 92, 141]. These models do not allow for parameter dependencies, but verification is necessarily “robust” against all probabilities within the intervals, see for instance [120], where convex optimization is utilised, and [81, 82], where efficient verification of multiple-objectives is introduced. In [6, 19], these models are extended to so-called parametric interval MCs, where interval bounds themselves are parametric. Extensions to richer models such as partially observable MDPs are considered in [59, 132].

Derivatives and monotonicity. Many systems behave monotonically in some of their system parameters. For example, most network protocols become more reliable if the communication channel reliability increases. If the solution function is monotonic, then parameter space partitioning can be accelerated [129]. Assessing monotonicity can be tightly integrated in a loop that uses parameter lifting [130]. Finally, the derivative of the solution function can be used for gradient descent whenever the goal is to find a counterexample for region verification [86].

Sensitivity analysis. Besides analysing in which regions the system behaves correctly w. r. t. the specification, it is often desirable to perform a sensitivity analysis [44, 131], i. e., to determine in which regions of the parameter space a small perturbation of the system leads to a relatively large change in the considered measure. In our setting, such an analysis can be conducted with little additional effort. Given a rational function for a measure of interest, its derivations w. r. t. all parameters can be easily computed. Passing the derivations with user-specified thresholds to the SMT solver then allows for finding parameter regions in which the system behaves robustly. Adding the safety constraints described earlier, the SMT solver can find regions that are both safe and robust.

Parameters with distributions. Rather than a model in which the parameter values are chosen from a set, they can be equipped with a distribution. The verification outcome consists then of confidence intervals rather than absolute guarantees. In [111], simulation based methods are used, whereas [33, 34] use statistical methods on a solution function. pMDPs with a distribution over the parameters are considered in [9]. Sampling-based methods that rely on the so-called scenario-approach [36, 37] are presented in [11, 58].

Ensuring graph preservation. Checking graph-preservation is closely related to checking whether a well-defined point instantiation exists, which has an exponential runtime in the number of parameters [106]. For parametric interval Markov chains, the question whether there exists a well-defined instantiation is referred to as consistency and received attention in [6, 118].

Robust strategies. Robust strategies for pMDPs, as mentioned in Remark 7, are considered in, among others, [108, 141]. These and other variants of synthesis problems on pMDPs were compared in [8]. A variant where parameters are not non-deterministically chosen, but governed by a prior over these parameters, has recently been considered [9]. In [119], data-driven bounds on parameter ranges are obtained, and properties are validated using parameter synthesis techniques.

Continuous time. Parametric CTMCs were first considered by [83]. A method using relaxations similarly to parameter lifting has been proposed in [30]. The method was improved in [39] and implemented in PRISM-PSY [40]. A combination with sampling-based algorithms to find good parameter instantiations is explored in [35]. Parameter synthesis with statistical guarantees has been explored in [25, 26]. Moreover, a sampling-based approach for so-called uncertain parametric CTMCs that have a distribution over the parameter values obtains statistical guarantees on reachability probabilities [12]. Finally, in [75], finding good parameter instantiations is considered by identifying subsets of parameters that have a strictly positive or negative influence on the property at hand.

Connection to other models. Furthermore, [96] establishes connections to the computation of strategies in partially observable MDPs [124], a prominent model in AI. In [142], the connection to concurrent stochastic games is shown. pMCs can be used to accelerate solving hierarchical Markov models [95, 113] and for parameter synthesis in Bayesian networks [126]. Finally, in [53], a method that maintains a belief over parameter values is introduced in a robotics context.

13 Conclusion and future work

This paper gives an extensive account of parameter synthesis for discrete-time Markov chain models. In particular, we considered three different variants of parameter synthesis questions. For each problem variant, we give an account of the available algorithms from the literature, together with several extensions from our side. All algorithms are available in the open-source tool PROPhESY.

Future work Future work in various directions is possible. Many of the results here can be ported to the more general setting of weighted automata over the adequate semiring [69], which can be interesting from a theoretical perspective. Algorithmically, we would like to develop methods which identify and exploit structural properties that are common to standard benchmarks for Markov chains and Markov decision processes. First steps in this direction have been taken, e.g., by exploiting monotonicity [129]. While graph-preservation is common in many applications, this restriction is not always natural. The decomposition presented in this paper yields an exponential blow-up in the number of parameters that we would like to avoid whenever possible. However, algorithms that do not rely on graph-preservation have not yet been integrated. The techniques to cover the parameter space by sets of smaller and easy-to-verify regions are still rather naive: This is true both for region verification, where we split due to the approximation, and for parameter space partitioning. The above mentioned monotonicity is one possibility to accelerate the way we split. In general, we plan to exploit parametric models in a data-driven context, where the structure provided by parameter dependencies can be exploited to accelerate learning of probabilistic models [133, 134].