FiniteHorizon Bisimulation Minimisation for Probabilistic Systems
 1 Citations
 430 Downloads
Abstract
We present model reduction techniques to improve the efficiency and scalability of verifying probabilistic systems over a finite time horizon. We propose a finitehorizon variant of probabilistic bisimulation for discretetime Markov chains, which preserves a bounded fragment of the temporal logic PCTL. In addition to a standard partitionrefinement based minimisation algorithm, we present onthefly finitehorizon minimisation techniques, which are based on a backwards traversal of the Markov chain, directly from a highlevel model description. We investigate both symbolic and explicitstate implementations, using SMT solvers and hash functions, respectively, and implement them in the PRISM model checker. We show that finitehorizon reduction can provide significant reductions in model size, in some cases outperforming PRISM’s existing efficient implementations of probabilistic verification.
1 Introduction
Probabilistic verification is an automated technique for the formal analysis of quantitative properties of systems that exhibit stochastic behaviour. A probabilistic model, such as a Markov chain or a Markov decision process, is systematically constructed and then analysed against properties expressed in a formal specification language such as temporal logic. Mature tools for probabilistic verification such as PRISM [15] and MRMC [13] have been developed, and the techniques have been applied to a wide range of application domains, from biological reaction networks [11] to car airbag controllers [1].
A constant challenge in this area is the issue of scalability: probabilistic models, which are explored and constructed in an exhaustive fashion, are typically huge for reallife systems, which can limit the practical applicability of the techniques. A variety of approaches have been proposed to reduce the size of these models. One that is widely used is probabilistic bisimulation [18], an equivalence relation over the states of a probabilistic model which can be used to construct a smaller quotient model that is equivalent to the original one (in the sense that it preserves key properties of interest to be verified).
Typically, it preserves both infinitehorizon (longrun) properties, e.g., “the probability of eventually reaching an error state”, finitehorizon (transient, or timebounded) properties, e.g. “the probability of an error occurring within k timesteps”, and, more generally, any property expressible in an appropriate temporal logic such as PCTL [10]. It has been shown that, in contrast to nonprobabilistic verification, the effort required to perform bisimulation minimisation can pay off in terms of the total time required for verification [12].
In this paper, we consider model reduction techniques for finitehorizon properties of Markov chains. We propose a finitehorizon variant of probabilistic bisimulation, which preserves stepwise behaviour over a finite number of steps, rather than indefinitely, as in standard probabilistic bisimulation. This permits a more aggressive model reduction, but still preserves satisfaction of PCTL formulae of bounded depth (i.e., whose interpretation requires only a bounded exploration of the model). Timebounded properties are commonly used in probabilistic verification, e.g., for efficiency (“the probability of task completion within k steps”) or for reliabilty (“the probability of an error occurring within time k”).
We formalise finitehorizon probabilistic bisimulation, define the subset of PCTL that it preserves and then give a partitionrefinement based algorithm for computing the coarsest possible finitehorizon bisimulation relation, along with a corresponding quotient model. The basic algorithm is limited by the fact it requires the full Markov chain to be constructed before it is minimised, which can be a bottleneck. So, we then develop onthefly approaches, which construct the quotient model directly from a highlevel model description of the Markov chain, based on a backwards traversal of its state space. We propose two versions: one symbolic, based on SMT solvers, and one explicitstate.
We implemented all algorithms in PRISM and evaluated them on a range of examples. First, we apply the partitionrefinement based approach to some standard benchmarks to investigate the size of the reduction that can be obtained in a finitehorizon setting. Then, we apply the onthefly approach to a class of problems to which it is particularly well suited: models with a large number of possible initial configurations, on which we ask questions such as “from which initial states does the probability of an error occurring within 10 s exceed 0.01?”. We show that onthefly finitehorizon bisimulation can indeed provide significant gains in both verification time and scalability, demonstrated in each case by outperforming the existing efficient implementations in PRISM.
Related Work. For the standard notion of probabilistic bisimulation on Markov chains [18], various decision procedure and minimisation algorithms have been developed. Derisavi et al. [9] proposed an algorithm with optimal complexity, assuming the use of splay trees and, more recently, a simpler solution was put forward in [20]. Signaturebased approaches, which our first, partitionrefinement algorithm adapts, have been studied in, for example, [9, 22]. Also relevant is the SMTbased bisimulation minimisation technique of [6] which, like our onthefly algorithm, avoids construction of the full model when minimising. Our SMTbased algorithm has an additional benefit in that it works on model descriptions with statedependent probabilities. Other probabilistic verification methods have been developed based on backwards traversal of a model, for example for probabilistic timed automata [16], but this is for a different class of models and does not perform minimisation. Della Penna et al. considered finitehorizon verification of Markov chains [7], but using diskbased methods, not model reduction.
2 Preliminaries
We start with some background on probabilistic verification of Markov chains.
2.1 DiscreteTime Markov Chains
A discretetime Markov chain (DTMC) can be thought of as a state transition system where transitions between states are annotated with probabilities.
Definition 1

\(\mathcal {S}\) is a finite set of states and \(\mathcal {S}_{ init }\subseteq \mathcal {S}\) is a set of initial states;

\({\mathbf P} : \mathcal {S}\times \mathcal {S}\rightarrow [0, 1]\) is a transition probability matrix, where, for all states \(s\in \mathcal {S}\), we have \(\sum _{s' \in \mathcal {S}} {\mathbf P}(s, s')= 1\);

\({\mathcal {AP}}\) is a set of atomic propositions and \(\mathcal {L : S} \rightarrow 2^{{\mathcal {AP}}}\) is a labelling function giving the set of propositions from \({\mathcal {AP}}\) that are true in each state.
For each pair \(s,s'\) of states, \({\mathbf P}(s,s')\) represents the probability of going from s to \(s'\). If \({\mathbf P}(s, s') > 0\), then s is a predecessor of \(s'\) and \(s'\) is a successor of s. For a state s and set \(C\subseteq \mathcal {S}\), we will often use the notation \({\mathbf P}(s,C) := \sum _{s'\in C}{\mathbf P}(s,s')\).
A path \(\sigma \) of a DTMC \({\mathcal D}\) is a finite or infinite sequence of states \(\sigma = s_{0}s_{1}s_{2}\dots \) such that \(\forall i\ge 0,\;s_{i} \in \mathcal {S}\) and \({\mathbf P}(s_{i}, s_{i+1}) > 0\). The \(i^{th}\) state of the path \(\sigma \) is denoted by \(\sigma [i]\). We let \(Path^{{\mathcal D}}(s)\) denote the set of infinite paths of \({\mathcal D}\) that begin in a state s. To reason formally about the behaviour of a DTMC, we define a probability measure \(Pr_s\) over the set of infinite paths \(Path^{{\mathcal D}}(s)\) [14]. We usually consider the behaviour from some initial state \(s\in \mathcal {S}_{ init }\) of \({\mathcal D}\).
2.2 Probabilistic Computation Tree Logic
Properties of probabilistic models can be expressed using Probabilistic Computation Tree Logic (PCTL) [10] which extends Computation Tree Logic (CTL) with time and probabilities. In PCTL, state formulae \(\varPhi \) are interpreted over states of a DTMC and path formulae \(\phi \) are interpreted over paths.
Definition 2
The main operator in PCTL, in addition to those that are standard from propositional logic, is the probabilistic operator \({\mathtt {P}_{\bowtie p}}[\phi ]\), which means that the probability measure of paths that satisfy \(\phi \) is within the bound \(\bowtie p\). For path formulae \(\phi \), we allow the (bounded) until operator \(\varPhi _1{\,\mathtt {U}^{\le {k}}\,}\varPhi _2\). If \(\varPhi _2\) becomes true within k time steps and \(\varPhi _1\) is true until that point, then \(\varPhi _1{\,\mathtt {U}^{\le {k}}\,}\varPhi _2\) is true. In the case where k equals \(\infty \), the bounded until operator becomes the unbounded until operator and is denoted by \({\,\mathtt {U}\,}\!\). For simplicity of presentation, in this paper, we omit the next (\({\mathtt {X}\,}\varPhi \)) operator, but this could easily be added.
Definition 3
For example, a PCTL formula such as \({\mathtt {P}}_{<0.01}[ \lnot fail _1 {\,\mathtt {U}^{\le {k}}\,} fail _2 ]\) means that the probability of a failure of type 2 occurring within k timesteps, and before a failure of type 1 does, is less than 0.01. Common derived operators are \({\mathtt {F}\,}\varPhi \equiv {\mathrm {true}}{\,\mathtt {U}\,}\varPhi \), which means that \(\varPhi \) eventually becomes true, and \({\mathtt {F}^{\le {k}}\,}\varPhi \equiv {\mathrm {true}}{\,\mathtt {U}^{\le {k}}\,} \varPhi \), which means that \(\varPhi \) becomes true within k steps.
2.3 Probabilistic Bisimulation
Larsen and Skou [18] defined (strong) probabilistic bisimulation for discrete probabilistic transition systems, which is an equivalence relation used to identify states with identical labellings and (probabilistic) stepwise behaviour.
Definition 4
Two states that are probabilistically bisimilar will satisfy the same properties, including both infinitehorizon (longrun) and finitehorizon (transient) properties. Aziz et al. [3] proved that any property in the temporal logic PCTL is also preserved in this manner. Thanks to these results, the analysis of the original Markov chain, such as probabilistic model checking of PCTL, can be equivalently performed on the quotient Markov chain, in which equivalence classes of bisimilar states are lumped together into a single state.
Usually, we are interested in the coarsest possible probabilistic bisimulation for a DTMC \({\mathcal D}\) (or, in other words, the union of all possible bisimulation relations). We denote the coarsest possible probabilistic bisimulation by \(\sim \). The quotient model \({\mathcal D}/\!\!\sim \) derived using this relation is defined as follows.
Definition 5

\(\mathcal {S}' = \mathcal {S}/\!\!\sim \ = \{[s]_\sim \ \ s\in \mathcal {S}\}\)

\(\mathcal {S}'_ init = \{[s]_{\sim }\ \ s\in \mathcal {S}_{ init }\}\)

\({\mathbf P}'([s]_\sim , [s']_\sim ) = {\mathbf P}(s, [s']_\sim )\)

\(\mathcal {L}'([s]_\sim ) = \mathcal {L}(s)\)
3 FiniteHorizon Bisimulation
We now formalise the notion of finitehorizon bisimulation, a stepbounded variant of standard probabilistic bisimulation for Markov chains [18]. We fix, from this point on, a DTMC \({\mathcal D}= {(\mathcal {S},\mathcal {S}_{ init },{\mathbf P},{\mathcal {AP}},\mathcal {L})}\). Intuitively, a kstep finitehorizon bisimulation, for nonnegative integer k, preserves the stepwise behaviour of \({\mathcal D}\) over a finite horizon of k steps. We use the following inductive definition.
Definition 6
 (i)
\(\mathcal {L}(s_1) = \mathcal {L}(s_2)\);
 (ii)
\({\mathbf P}(s_1,C) = {\mathbf P}(s_2,C)\) for each equivalence class \(C \in \mathcal {S}/\mathcal {R}_{k1}\),
Definition 7
(FiniteHorizon Bisimulation Equivalent). We say states \(s_1,s_2\) are (kstep) finitehorizon bisimulation equivalent (bisimilar), denoted \(s_1\sim _k s_2\), if there exists a kstep finitehorizon bisimulation \(\mathcal {R}_k\) such that \((s_1,s_2)\in \mathcal {R}_k\).
Two states \(s_1\) and \(s_2\) satisfying \(s_1\sim _k s_2\) have the same stepwise behaviour over k steps. The following simple, but useful, properties hold.
Proposition 1
 (a)
if \(s_1\sim _k s_2\), then \(s_1\sim _j s_2\) for any \(0\le j\le k\).
 (b)
if \(s_1\sim s_2\), then \(s_1\sim _k s_2\) for any \(k\ge 0\).
 (c)
if \(s_1\sim _k s_2\) and \(s_1 \rightarrow s_1'\), then \(s_1'\sim _{k1} s_2'\) for some state \(s_2'\) such that \(s_2 \rightarrow s_2'\).
From a model checking perspective, if \(s_1\sim _k s_2\), then \(s_1\) and \(s_2\) satisfy the same PCTL formulae up to a bounded depth k. We formalise this as follows.
Definition 8

\({ d ({\mathrm {true}})} = { d (a)} = 0\) for atomic proposition a;

\({ d (\lnot \varPhi )} = { d (\varPhi )}\);

\({ d (\varPhi _1\wedge \varPhi _2)} = \max ({ d (\varPhi _1)},{ d (\varPhi _2)})\);

\({ d ({\mathtt {P}_{\bowtie p}}[\varPhi _1{\,\mathtt {U}^{\le {j}}\,}\varPhi _2])} = j + \max ({ d (\varPhi _1)}{}1,{ d (\varPhi _2)})\).
For example, if a and b are atomic propositions, we have \({ d ({\mathtt {P}_{\bowtie p}}[{\mathrm {true}}{\,\mathtt {U}^{\le {5}}\,}a])} = 5\), \({ d ({\mathtt {P}_{\bowtie p}}[{\mathrm {true}}{\,\mathtt {U}^{\le {5}}\,}a] \wedge {\mathtt {P}_{\bowtie p}}[{\mathrm {true}}{\,\mathtt {U}^{\le {6}}\,}a])} = 6\), and \({ d ({\mathtt {P}_{\bowtie p}}[{\mathrm {true}}{\,\mathtt {U}^{\le {5}}\,}{\mathtt {P}_{\bowtie p}}[a{\,\mathtt {U}^{\le {3}}\,}b]])} = 8\).
If states \(s_1\) and \(s_2\) are (kstep) finitehorizon bisimilar, then they satisfy exactly the same PCTL formulae of depth at most k.
Theorem 1
Let \(s_1\) and \(s_2\) be two states such that \(s_1\sim _k s_2\), and \(\varPhi \) be a PCTL formula with depth \(d(\varPhi )\le k\), then \(s_1 \models \varPhi \) if and only if \(s_2 \models \varPhi \).
Proof
We prove the result by induction over the structure (see Definition 2) of PCTL formula \(\varPhi \). Propositional operators are straightforward since \(s_1\) and \(s_2\) satisfy the same atomic propositions, by the definition of \(\sim _k\), and, for \(\varPhi =\lnot \varPhi _1\) or \(\varPhi =\varPhi _1\wedge \varPhi _2\), the subformulae \(\varPhi _1\) and \(\varPhi _2\) have depth at most k so, by induction, we can assume that \(s_1 \models \varPhi _i \Leftrightarrow s_2 \models \varPhi _i\) for \(i\in \{1,2\}\).
In similar fashion to the standard (nonfinitehorizon) case, we are typically interested in the coarsest possible kstep finitehorizon bisimulation relation for a given DTMC (labelled with atomic propositions) and time horizon k, which we denote by \(\sim _k\). We can also define this as the union of all possible kstep finitehorizon bisimulation relations. Furthermore, for \(\sim _k\) (or any other finitehorizon bisimulation relation), we can define a corresponding quotient DTMC, whose states are formed from the equivalence classes of \(\sim _k\), and whose kstep behaviour is identical to the original DTMC \({\mathcal D}\).
This is similar, but not identical, to the process of building the quotient Markov chain corresponding to a full minimisation (see Definition 5). We must take care since, unlike for full bisimulation, given a state \(B\in \mathcal {S}/\!\!\sim _k\) of the quotient model, the probabilities \({\mathbf P}(s,B')\) of moving to other equivalence classes \(B'\in \mathcal {S}/\!\!\sim _k\) can be different for each state \(s\in B\) (according to the definition of \(\sim _k\), probabilities are the same to go states with the same \((k{}1)\)step, not kstep, behaviour). However, when they do differ, it suffices to pick an arbitrary representative from B. We formalise the quotient DTMC construction below, and then present some examples.
Definition 9

\(\mathcal {S}' = \mathcal {S}/\!\!\sim _k \ = \{[s]_{\sim _k}\ \ s\in \mathcal {S}\}\)

\(\mathcal {S}'_ init = \{[s]_{\sim _k}\ \ s\in \mathcal {S}_{ init }\}\)

\({\mathbf P}'(B,B') = {\mathbf P}(rep(B), B')\) for any \(B,B'\in \mathcal {S}'\)

\(\mathcal {L}'(B) = \mathcal {L}(rep(B))\) for any \(B\in \mathcal {S}'\),
Example 1
Figure 1 illustrates finitehorizon bisimulation on an example DTMC, shown in part (a). Figure 1(b) and (c) show quotient DTMCs for 0step and 1step finitehorizon bisimulation minimisation, respectively, where quotient state names indicate their corresponding equivalence class (e.g., \(B_{23}\) corresponds to DTMC states \(s_2\) and \(s_3\)). For 2step minimisation (not shown), blocks \(B_{23}\) and \(B_{01}\) are both split in two, and only the states \(s_4\) and \(s_5\) remain bisimilar.
From the above, we see that \(s_2 \sim _1 s_3\), but \(s_2 \not \sim _2 s_3\). Consider the PCTL formula \(\varPhi ={\mathtt {P}_{\bowtie p}}[{\mathrm {true}}{\,\mathtt {U}^{\le {k}}\,}a]\), which has depth \({ d (\varPhi )}=k\). Satisfaction of \(\varPhi \) is equivalent in states \(s_2\) and \(s_3\) for \(k=1\), but not for \(k=2\). To give another example, for \(\varPhi '={\mathtt {P}}_{>0}[{\mathtt {P}}_{>0.5}[{\mathrm {true}}{\,\mathtt {U}^{\le {2}}\,}a]{\,\mathtt {U}^{\le {1}}\,}a]\), which has \({ d (\varPhi ')}=1+21=2\), we have \(s_3\models \varPhi '\), but \(s_2\not \models \varPhi '\).
In constructing the 1step quotient model (Fig. 1(c)), we used \(s_1\) as a representative of equivalence class \(B_{01}=\{s_0,s_1\}\), which is why there is a transition to \(B_{23}\). We could equally have used \(s_0\), which would yield a different quotient DTMC, but which still preserves 1step behaviour.
4 FiniteHorizon Bisimulation Minimisation
Bisimulation relations have a variety of uses, but our focus here is on using them to minimise a probabilistic model prior to verification, in order to improve the efficiency and scalability of the analysis. More precisely, we perform finitehorizon bisimulation minimisation, determining the coarsest possible finitehorizon bisimulation relation \(\sim _k\), for a given k, and then constructing the corresponding quotient Markov chain. Theorem 1 tells us that it is then safe to perform verification on the smaller quotient model instead.
We begin, in this section, by presenting a classical partitionrefinement based minimisation algorithm, which is based on an iterative splitting of an initially coarse partition of the state space until the required probabilistic bisimulation has been identified. In the next section, we will propose onthefly approaches which offer further gains in efficiency and scalability.
4.1 A PartitionRefinement Based Minimisation Algorithm
The standard approach to partition refinement is to use splitters [9, 19], individual blocks in the current partition which show that one or more other blocks contain states that should be split into distinct subblocks. An alternative approach is to use a socalled signaturebased method [8]. The basic structure of the algorithm remains the same, however the approach to splitting differs: rather than using splitters, a signature corresponding to the current partition is computed at each iteration for each state s. This signature comprises the probability of moving from s in one step to each block in the partition. In the next iteration, all states with different signatures are placed in different blocks.
Because each iteration of the signaturebased algorithm considers the onestep behaviour of every state in the model, it is relatively straightforward to adapt to finitehorizon bisimulation minimisation. Algorithm 1 shows the finitehorizon minimisation algorithm MinimiseFiniteHorizon. It takes a DTMC \({\mathcal D}\) and the time horizon k as input. The partition \(\varPi \) is first initialised to group states based on the different combinations of atomic propositions, i.e., states with identical labellings are placed in one block.^{1} The partition is then repeatedly split, each time by computing the signatures for each state and splitting accordingly. The loop terminates either when k iterations have been completed or no further splitting is possible. Finally, the quotient model is constructed, as described in the previous section.
5 OntheFly FiniteHorizon Minimisation
A key limitation of the partitionrefinement approach presented in the previous section is that it takes as input the full DTMC to be minimised, the construction of which can be expensive in terms of both time and space. This can remove any potential gains in terms of scalability that minimisation can provide.
To resolve this, we now propose methods to compute a finitehorizon bisimulation minimisation in an onthefly fashion, where the minimised model is constructed directly from a highlevel modelling language description of the original model, bypassing construction of the full, unreduced DTMC. In our case, the probabilistic models are described using the modelling language of the PRISM model checker [15], which is based on guarded commands.
Our approach works through a backwards traversal of the model, which allows us to perform bisimulation minimisation on the fly. For simplicity, we focus on preserving the subclass of PCTL properties comprising a single \({\mathtt {P}}\) operator, more precisely, those of the form \({\mathtt {P}_{\bowtie p}}[\,b_1{\,\mathtt {U}^{\le {k}}\,}b_2\,]\) for atomic propositions \(b_1\) and \(b_2\). This is the kind of property most commonly found in practice.
5.1 The OntheFly Minimisation Algorithm
The basic approach to performing finitehorizon minimisation on the fly is shown as FiniteHorizonOnTheFly, in Algorithm 2. This takes model, which is a description of the DTMC, \(B_1\) and \(B_2\), the sets of states satisfying \(b_1\) and \(b_2\), respectively, in the property \({\mathtt {P}_{\bowtie p}}[\,b_1{\,\mathtt {U}^{\le {k}}\,}b_2\,]\), and the time horizon k. The algorithm does not make any assumptions about how sets of states are represented or manipulated. Below, we will discuss two separate instantiations of it.
The main loop of the algorithm iterates backwards through the model: after the ith iteration, it has found all states that can reach the target set \(B_2\) within i steps with positive probability. The new predecessors for each iteration are stored in a set of blocks P. A separate set \(P'\) is used to store predecessors of blocks in P, which will then be considered in the next iteration.
More precisely, P (and \(P'\)) store, like in Algorithm 1, a list of pairs (B, D) where B is a block (a set of states) and D is a (partial) probability distribution storing probabilities of outgoing transitions (from B, to other blocks). The set \(\varPi \), which is used to construct the partition representing the finitehorizon bisimulation relation, is also stored as a list of pairs.
Algorithm 2 begins by finding all immediate predecessors of states in \(B_2\) that are also in \(B_1\) and putting them in P. In each iteration, it takes each blockdistribution pair (B, D) from P one by one: it will add this to the current partition \(\varPi \). But, before doing so, it checks whether B overlaps with any existing blocks \(B'\) in \(\varPi \). If so, \(B'\) is split in two, and the overlap is removed from B. At this point, the partition \(\varPi \) is refined to take account of the splitting of block \(B'\). We repeatedly recompute the probabilities associated with each block in \(\varPi \) and, if these are then different for states within that block, it is also split.
Each iteration of the main loop finishes when all pairs (B, D) from P have been dealt with. If \(i<k\), then newly found predecessors \(P'\) are copied to P and the process is repeated. If \(i=k\), then the time horizon k has been reached and the finitehorizon bisimulation has been computed.
Finally, the quotient model is built. The basic construction is as in Algorithm 1 but, since onthefly construction only partially explores the model, we need to add an extra sink state to complete the DTMC.
In the following sections, we describe two approaches to finding predecessors: one symbolic, which represents blocks (sets of states) as predicates and uses an SMT (satisfiability modulo theories) [5] based implementation; and one explicitstate, which explicitly enumerates the states in each block.
5.2 Symbolic (SMTBased) Minimisation
SMTbased methods for probabilistic bisimulation minimisation have been developed previously [6]. One key difference here is that our approach handles transition probabilities expressed as statedependent expressions, rather than fixed constants, which are needed for some of the models we later evaluate.
5.3 ExplicitState Minimisation
As an alternative to the symbolic approach using SMT, we developed an explicitstate implementation of finitehorizon minimisation in which the blocks of equivalent states are represented by explicitly listing the states that comprise them. As in the previous algorithm, the blocks are refined at each time step such that states residing in the same block have equal transition probabilities to the required blocks. To improve performance and store states compactly, we hash them based on the valuation of variables that define them. This is done in such a way that the hash values are bidirectional (onetoone).
The algorithm explicitly computes the predecessor state for each update and each state in the set \( target \), the transition probability is then computed for each predecessor state and these are collected in order to group states into sets. The set \( restrict \) is not stored explicitly, but rather as a symbolic expression which is then evaluated against each state’s variable values to compute the intersection.
6 Experimental Results
We have implemented the bisimulation minimisation techniques presented in this paper as an extension of the PRISM model checker [15], and applied them to a range of benchmark models. For both the partitionrefinement based minimisation of Sect. 4, and the onthefly methods in Sect. 5, we build on PRISM’s “explicit” model checking engine. For the SMTbased variant, we use the Z3 solver [4], through the Z3 Java API. All our experiments were run on an Intel Core i7 2.8 GHz machine, using 2 GB of RAM.
Our investigation is in two parts. First, we apply the partitionrefinement algorithm to several DTMCs from the PRISM benchmark suite [17] to get an idea of the size of reductions that can be obtained on some standard models. We use: Crowds (an anonymity protocol), EGL (a contract signing protocol) and NAND (NAND multiplexing). Details of all models, parameters and properties used can be found at [24]. A common feature of these models is that they have a single initial state, from which properties are verified. Since onthefly approaches explore backwards from a target set, we would usually need to consider time horizons k high enough such that the whole model was explored.
So, to explore in more depth the benefits of the onthefly algorithms, we consider another common class of models in probabilistic verification: those in which we need to exhaustively check whether a property is true over a large set of possible configurations. We use Approximate majority [2], a population protocol for computing a majority value amongst a set of K agents, and two simple models of genetic algorithms [21] in which a population of K agents evolves over time, competing to exist according to a fitness value in the range \(0,\dots ,N{}1\). In the first variant, tournament, the agent with the highest value wins; in the second, modulo, the sum of the two scores is used modulo N. Again, details of all models, parameters and properties used can be found at [24].
6.1 The PartitionRefinement Algorithm
The bottom row of plots shows, for the same examples, the time required to perform bisimulation minimisation and then verify a kstep finitehorizon property (details at [24]). The black lines show the time for finitehorizon minimisation, the grey lines for full minimisation. The latter are relatively flat, indicating that the time for verification (which is linear in k) is very small compared to the time needed for minimisation. However, we see significant gains in the total time required for finitehorizon minimisation compared to full minimisation.
However, despite these gains, the times to minimise and verify the quotient model are still larger than to simply build and verify the full model. This is primarily because the partition refinement algorithm requires construction of the complete model first, the time for which eclipses any gains from minimisation. This was the motivation for the onthefly algorithms, which we evaluate next.
6.2 OntheFly Algorithms
Experimental results for onthefly bisimulation minimisation.
First, we note that finitehorizon minimisation yields useful reductions in model size in all cases, both with respect to the full model and to normal (nonfinite horizon) minimisation. Bisimulation reduces models by a factor of roughly 2 and 5, for the Approximate majority and Modulus examples, respectively. For Tournament, a very large reduction is obtained since, for the property checked, the model ends up being abstracted to only distinguish two fitness values. Finitehorizon minimisation gives models that are smaller again, by a factor of between 2 and 10 on these examples, even for relatively large values of k on the Approximate majority models. Comparing columns 7 and 8 in Table 1 shows that much of the reduction is indeed due to merging of bisimilar states, not just to a kstep truncation of the state space from the backwards traversal.
Regarding performance and scalability, we first discuss results for the SMTbased implementation. We were only able to apply this to the Tournament example, where a very large reduction in state space is achieved. On a positive note, the SMTbased approach successfully performs minimisation here and gives a symbolic (Boolean expression) representation for each block. However, the process is slow, limiting applicability to DTMCs that can already be verified without minimisation. Our experiments showed that the slow performance was largely caused by testing for overlaps between partition blocks resulting in a very large number of calls to the SMT solver.
The explicitstate onthefly implementation performed much better and Table 1 shows results for all three models. In particular, for the Tournament example, finitehorizon minimisation and verification is much faster than verifying the full model using the fastest engine in PRISM. This is because we can bypass construction of the full models, which have up to 14 million states for this example. For the Modulus example, the model reductions obtained are much smaller and, as a result, PRISM is able to build and verify the model faster. However, for the Approximate Majority example, the minimisation approach can be applied to larger models than can be handled by PRISM. For this example, although the state spaces of the full model are manageable, the models prove poorly suited to PRISM’s model construction implementation (which is based on binary decision diagram data structures).
7 Conclusions
We have presented model reduction techniques for verifying finitehorizon properties on discretetime Markov chains. We formalised the notion of kstep finitehorizon bisimulation mininisation and clarified the subset of PCTL that it preserves. We have given both a partitionrefinement algorithm and an onthefly approach, implemented in both a symbolic (SMTbased) and explicitstate manner as an extension of PRISM. Experimental results demonstrated that significant model reductions can be obtained in this manner, resulting in improvements in both execution time and scalability with respect to the existing efficient implementations in PRISM.
Future work in this area will involve extending the techniques to other classes of probabilistic models, and adapting the onthefly approaches to preserve the full timebounded fragment of PCTL, including nested formulae.
Footnotes
 1.
In the algorithm, we store the signatures with the partition, so \(\varPi \) is a list of pairs of blocks (statesets) and signatures (distributions).
Notes
Acknowledgements
This work has been supported by the EUFP7funded project HIERATIC.
References
 1.Aljazzar, H., Fischer, M., Grunske, L., Kuntz, M., Leitner, F., Leue, S.: Safety analysis of an airbag system using probabilistic FMEA and probabilistic counterexamples. In: Proceedings of the QEST 2009 (2009)Google Scholar
 2.Angluin, D., Aspnes, J., Eisenstat, D.: A simple population protocol for fast robust approximate majority. Distrib. Comput. 21(2), 87–102 (2008)CrossRefzbMATHGoogle Scholar
 3.Aziz, A., Singhal, V., Balarin, F., Brayton, R.K., SangiovanniVincentelli, A.L.: It usually works: the temporal logic of stochastic systems. In: Wolper, P. (ed.) CAV 1995. LNCS, vol. 939, pp. 155–165. Springer, Heidelberg (1995)CrossRefGoogle Scholar
 4.de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008)CrossRefGoogle Scholar
 5.De Moura, L., Bjørner, N.: Satisfiability modulo theories: introduction and applications. Commun. ACM 54(9), 69–77 (2011)CrossRefGoogle Scholar
 6.Dehnert, C., Katoen, J.P., Parker, D.: SMTbased bisimulation minimisation of Markov models. In: Giacobazzi, R., Berdine, J., Mastroeni, I. (eds.) VMCAI 2013. LNCS, vol. 7737, pp. 28–47. Springer, Heidelberg (2013)CrossRefGoogle Scholar
 7.Della Penna, G., Intrigila, B., Melatti, I., Tronci, E., Zilli, M.V.: Finite horizon analysis of Markov chains with the mur\(\phi \) verifier. STTT 8(4–5), 397–409 (2006)CrossRefzbMATHGoogle Scholar
 8.Derisavi, S.: Signaturebased symbolic algorithm for optimal Markov chain lumping. In: Proceedings of the QEST 2007, pp. 141–150. IEEE Computer Society (2007)Google Scholar
 9.Derisavi, S., Hermanns, H., Sanders, W.H.: Optimal statespace lumping in Markov chains. Inf. Process. Lett. 87(6), 309–315 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
 10.Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. FAC 6(5), 512–535 (1994)zbMATHGoogle Scholar
 11.Heath, J., Kwiatkowska, M., Norman, G., Parker, D., Tymchyshyn, O.: Probabilistic model checking of complex biological pathways. In: Priami, C. (ed.) CMSB 2006. LNCS (LNBI), vol. 4210, pp. 32–47. Springer, Heidelberg (2006)CrossRefGoogle Scholar
 12.Katoen, J.P., Kemna, T., Zapreev, I., Jansen, D.N.: Bisimulation minimisation mostly speeds up probabilistic model checking. In: Grumberg, O., Huth, M. (eds.) TACAS 2007. LNCS, vol. 4424, pp. 87–101. Springer, Heidelberg (2007)CrossRefGoogle Scholar
 13.Katoen, J.P., Zapreev, I.S., Hahn, E.M., Hermanns, H., Jansen, D.N.: The ins and outs of the probabilistic model checker MRMC. Perform. Eval. 68(2), 90–104 (2011)CrossRefGoogle Scholar
 14.Kemeny, J., Snell, J., Knapp, A.: Denumerable Markov Chains, 2nd edn. Springer, Heidelberg (1976)CrossRefzbMATHGoogle Scholar
 15.Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic realtime systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011)CrossRefGoogle Scholar
 16.Kwiatkowska, M., Norman, G., Sproston, J., Wang, F.: Symbolic model checking for probabilistic timed automata. Inf. Comput. 205(7), 1027–1077 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
 17.Kwiatkowska, M., Norman, G., Parker, D.: The PRISM benchmark suite. In: Proceedings of the QEST 2012, pp. 203–204 (2012)Google Scholar
 18.Larsen, K.G., Skou, A.: Bisimulation through probabilistic testing. Inf. Comput. 94(1), 1–28 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
 19.Paige, R., Tarjan, R.E.: Three partition refinement algorithms. SIAM J. Comput. 16(6), 973–989 (1987)MathSciNetCrossRefzbMATHGoogle Scholar
 20.Valmari, A., Franceschinis, G.: Simple O(m logn) time Markov chain lumping. In: Esparza, J., Majumdar, R. (eds.) TACAS 2010. LNCS, vol. 6015, pp. 38–52. Springer, Heidelberg (2010)CrossRefGoogle Scholar
 21.Vose, M.: The Simple Genetic Algorithm: Foundations and Theory. MIT Press, Cambridge (1999)zbMATHGoogle Scholar
 22.Wimmer, R., Becker, B.: Correctness issues of symbolic bisimulation computationfor Markov chains. In: MüllerClostermann, B., Echtle, K., Rathgeb, E.P. (eds.) MMB & DFT 2010. LNCS, vol. 5987, pp. 287–301. Springer, Heidelberg (2010)Google Scholar
 23.
 24.