Abstract
Saturation is an efficient exploration order for computing the set of reachable states symbolically. Attempts to parallelize saturation have so far resulted in limited speedup. We demonstrate for the first time that onthefly symbolic saturation can be successfully parallelized at a large scale. To this end, we implemented saturation in Sylvan’s multicore decision diagrams used by the LTSmin model checker.
We report extensive experiments, measuring the speedup of parallel symbolic saturation on a 48core machine, and compare it with the speedup of parallel symbolic BFS and chaining. We find that the parallel scalability varies from quite modest to excellent. We also compared the speedup of onthefly saturation and saturation for prelearned transition relations. Finally, we compared our implementation of saturation with the existing sequential implementation based on Meddly.
The empirical evaluation uses Petri nets from the model checking contest, but thanks to the architecture of LTSmin, parallel onthefly saturation is now available to multiple specification languages. Data or code related to this paper is available at: [34].
T. van Dijk—Supported by FWF, NFN Grant S11408N23 (RiSE).
J. Meijer—Supported by STW SUMBAT Grant 13859.
Download conference paper PDF
1 Introduction
Model checking is an exhaustive algorithm to verify that a finite model of a concurrent system satisfies certain temporal properties. The main challenge is to handle the large state space, resulting from the combination of parallel components. Symbolic model checking exploits regularities in the set of reachable states, by storing this set concisely in a decision diagram. In asynchronous systems, transitions have locality, i.e. they affect only a small part of the state vector. This locality is exploited in the saturation strategy, which is probably the most efficient strategy to compute the set of reachable states.
In this paper, we investigate the efficiency and speedup of a new parallel implementation of saturation, aiming at a multicore, sharedmemory implementation. The implementation is carried out in the parallel decision diagram framework Sylvan [16], in the languageindependent model checker LTSmin [22]. We empirically evaluate the speedup of parallel saturation on Petri nets from the Model Checking Contest [24], running the algorithm on up to 48 cores.
1.1 Related Work
The saturation strategy has been developed and improved by Ciardo et al. We refer to [13] for an extensive description of the algorithm. Saturation derives its efficiency from firing all local transitions that apply at a certain level of the decision diagram, before proceeding to the next higher level. An important step in the development of the saturation algorithm allows onthefly generation of the transition relations, without knowing the cardinality of the state variable domains in advance [12]. This is essential to implement saturation in LTSmin, which is based on the PINS interface to discover transitions onthefly.
Since saturation obtains its efficiency from a restrictive firing order, it seems inherently sequential. Yet the problem of parallelising saturation has been studied intensively. The first attempt, Saturation NOW [9], used a network of PCs. This version could exploit the collective memory of all PCs, but due to the sequential procedure, no speedup was achieved. By firing local transitions speculatively (but with care to avoid memory waste), some speedup has been achieved [10]. More relevant to our work is the parallelisation of saturation for a shared memory architecture [20]. The authors used CILK to schedule parallel work originating from firing multiple transitions at the same level. They reported some speedup on a dualcore machine, at the expense of a serious memory increase. Their method also required to precompute the transition relation. An improvement of the parallel synchronisation mechanism was provided in [31]. They reported a parallel speedup of 2\(\times \) on 4 CPUs. Moreover, their implementation supports learning the transition relation onthefly. Still, the successful parallelisation of saturation remained widely open, as indicated by Ciardo [14]: “Parallel symbolic statespace exploration is difficult, but what is the alternative?”
For an extensive overview of parallel decision diagrams on various hardware architectures, see [15]. Here we mention some other approaches to parallel symbolic model checking, different from saturation for reachability analysis. First, Grumberg and her team [21] designed a parallel BDD package based on vertical partitioning. Each worker maintains its own subBDD. Workers exchange BDD nodes over the network. They reported some speedup on 32 PCs for BDD based model checking under the BFS strategy. The Sylvan [16] multicore decision diagram package supports symbolic onthefly reachability analysis, as well as bisimulation minimisation [17]. Oortwijn [28] experimented with a heterogeneous distributed/multicore architecture, by porting Sylvan’s architecture to RDMA over MPI, running symbolic reachability on 480 cores spread over 32 PCs and reporting speedups of BFS symbolic reachability up to 50. Finally, we mention some applications of saturation beyond reachability, such as model checking CTL [32] and detecting strongly connected components to detect fair cycles [33].
1.2 Contribution
Here we show that implementing saturation on top of the multicore decision diagram framework Sylvan [16] yields a considerable speedup in a sharedmemory setting of up to 32.5\(\times \) on 48 cores with prelearned transition relations, and 52.2\(\times \) with onthefly transition learning.
By design decision, our implementation reuses several features provided by Sylvan, such as: its own finegrained, workstealing framework Lace [18], its implementation of both BDDs (Binary Decision Diagrams) and LDDs (a Listimplementation of Multiway Decision Diagrams), its concurrent unique table and operations cache, and finally, its parallel operations like set union and relational product. As a consequence, the pseudocode of the algorithm and additional code for saturation is quite small, and orthogonal to other BDD features. To improve orthogonality with the existing decision diagrams, we deviated from the standard presentation of saturation [13]: we never update BDD nodes in situ, and we eliminated the mutual recursion between saturation and the BDD operations for relational product to fire transitions.
The implementation is available in the opensource highperformance model checking tool LTSmin [22], with its languageagnostic interface, Partitioned NextState Interface (PINS) [5, 22, 25]. Here, a specification basically provides a nextstate function equipped with dependency information, from which LTSmin can derive locality information. We fully support the flexible method of learning the transition relation onthefly during saturation [12]. As a consequence, our contribution extends the tool LTSmin with saturation for various specification languages, like Promela, DVE, Petri nets, mCRL2, and languages supported by the ProB model checker. See Sect. 4 on how to use saturation in LTSmin.
The experiments with saturation in Sylvan are carried out in LTSmin as well. We used Petri nets from the MCC competition. Our experimental design has been carefully set up in order to facilitate fair comparisons. Besides learning the transition relation onthefly, we also prelearned them in order to measure the overhead of learning, and eliminating its effect in comparisons. It is well known that the variable ordering has a large effect on the BDD sizes [29]. Hence, our experiments are based on two of the best static variable orderings known, Sloan [26] and Force [1]. In particular, our experiments measure and compare:

The performance of our parallel algorithm with one worker, compared to a stateofthe art sequential implementation of saturation in Meddly [4].

The parallel speedup of our algorithm on 16 cores, and for specific examples up to 48 cores.

The efficiency and speedup of saturation compared to the BFS and chaining strategies for reachability analysis.

The effect of choosing Binary Decision Diagrams or List Decision Diagrams.

The effect of choosing Sloan or Force to compute static variable orders.
2 Preliminaries
This paper proposes an algorithm for decision diagrams to perform the fixed point application of multiple transition relations according to the saturation strategy, combined with onthefly transition learning as implemented in LTSmin. We briefly review these concepts in the following.
2.1 Partitioned Transition Systems
A transition system (TS) is a tuple \((S, \rightarrow , s^0)\), where S is a set of states, \(\rightarrow \subseteq S \times S\) is a transition relation and \(s^0 \in S\) is the initial state. We define \(\rightarrow ^*\) to be the reflexive and transitive closure of \(\rightarrow \). The set of reachable states is \(R = \{s \in S \mid s^0 \rightarrow ^* s\}\). The goal of this work is to compute R via a novel multicore saturation strategy.
In this paper, we evaluate multicore saturation using Petri nets. Figure 1 shows an example of a (safe) Petri net. We show its initial marking, which is the initial state. A Petri net transition can fire if there is a token in each of its source places. On firing, these tokens are consumed and tokens in each target place are generated. For example, \(t_1\) will produce one token in both \(p_2\) and \(p_5\), if there is a token in \(p_4\). Transition \(t_6\) requires a token in both \(p_3\) and \(p_1\) to fire. The markings of this Petri net form the states of the corresponding TS, so here \(\vert S \vert = 2^5 = 32\). From the initial marking shown, four more markings are reachable, connected by 10 enabled transition firings. This means \(\vert R \vert = 5\), and \(\vert {\rightarrow }\vert = 10\).
Notice that transitions in Petri nets are quite local; transitions consume from, and produce into relatively few places. The firing of a Petri net transition is called an event and the number of involved places is known as the degree of event locality. This notion is easily defined for other asynchronous specification languages and can be computed by a simple control flow graph analysis.
To exploit event locality, saturation requires a disjunctive partitioning of the transition relation \(\rightarrow \), giving rise to a Partitioned Transition System (PTS). In a PTS, states are vectors of length N, and \(\rightarrow \) is partitioned as a union of M transition groups. A natural way to partition a Petri net is by viewing each transition as a transition group. For Fig. 1 this means we have \(N=5\) and \(M=6\). After disjunctive partitioning, each transition group depends on very few entries of the state vector. This allows for efficiently computing the reachable state space for the large class of asynchronous specification languages. LTSmin supports commonly used specification languages, like DVE, mCRL2, Promela, PNML for Petri nets, and languages supported by ProB.
2.2 Decision Diagrams
Binary decision diagrams (BDDs) are a concise and canonical representation of Boolean functions \(\mathbb {B}^N\rightarrow \mathbb {B}\) [7]. A BDD is a rooted directed acyclic graph with leaves 0 and 1. Each internal node v has a variable label \(x_i\), denoted by \(\mathrm {var}(v)\), and two outgoing edges labeled 0 and 1, denoted by \(\mathrm {low}(v)\) and \(\mathrm {high}(v)\). The efficiency of reduced, ordered BDDs is achieved by minimizing the structure with some invariants: The BDD may neither contain equivalent nodes, with the same \(\mathrm {var}(v)\), \(\mathrm {low}(v)\) and \(\mathrm {high}(v)\), nor redundant nodes, with \(\mathrm {low}(v)=high(v)\). Also, the variables must occur according to a fixed ordering along each path.
Multivalued or multiway decision diagrams (MDDs) generalize BDDs to finite domains (\(\mathbb {N}^N\rightarrow \mathbb {B}\)). Each internal MDD node with variable \(x_i\) now has \(n_i\) outgoing edges, labeled 0 to \(n_i1\). We use quasireduced MDDs with sparse nodes. In the sparse representation, values with edges to leaf 0 are skipped from MDD nodes, so outgoing edges must be explicitly labeled with remaining domain values. Contrary to BDDs, MDDs are usually “quasireduced”, meaning that variables are never skipped. In that case, the variable \(x_i\) can be derived from the depth of the MDD, so it is not stored.
A variation of MDDs are list decision diagrams (LDDs) [5, 16], where sparse MDD nodes are represented as a linked list. See Fig. 2 for two visual representations of the same LDD. Each LDD node contains a value, a “down” edge for the corresponding child, and a “right” edge pointing to the next element in the list. Each list ends with the leaf 0 and each path from the root downwards ends with the leaf 1. The values in an LDD are strictly ordered, i.e., the values must increase to the “right”.
LDD nodes have the advantage that common suffixes can be shared: The MDD for Fig. 2a requires two more nodes, one for [2, 4] and one for [1], because edges can only point to an entire MDD node. LDDs suffer from an increased memory footprint and inferior memory locality, but their memory management is simpler, since each LDD node has a fixed small size.
2.3 Variable Orders and Event Locality
Good variable orders are crucial for efficient operations on decision diagrams. The syntactic variable order from the specification is often inadequate for the saturation algorithm to perform well. Hence, finding a good variable order is necessary. Variable reordering algorithms use heuristics based on event locality. The locality of events can be illustrated with dependency matrices. The size of those matrices is \(M \times N\), where M is the number of transition groups, and N is the length of the state vector. The order of columns in dependency matrices determines the order of variables in the DD. Figure 3a shows the natural order on places in Fig. 1. A measure of event locality is called event span [29]. Lower event span is correlated to a lower number of nodes in decision diagrams. This can be seen in LDDs in Figs. 4a and b that are ordered according to columns in Figs. 3a and b respectively.
Event span is defined as the sum over all rows of the distance from the leftmost nonzero column to the rightmost nonzero column. The event span of Fig. 3a is 22 (\(= 4 + 2 + 2 + 5 + 5 + 4\)); the event span of Fig. 3b is 16, which is better. Optimizing the event span and thus variable order of DDs is NPcomplete [6], yet there are heuristic approaches that run in subquadratic time and provide good enough orders. Commonly used algorithms are Noack [27], Force [1] and Sloan [30]. Noack creates a permutation of variables by iteratively minimizing some objective function. The Force algorithm acts as if there are springs in between nonzeros in the dependency matrix, and tries to minimize the average tension among them. Sloan tries to minimize the profile of matrices. In short, profile is the symmetric counterpart to event span. For a more detailed overview of these algorithms see [3]. In our empirical evaluation we use both Sloan and Force, because these have been shown to give the best results [2, 26].
2.4 The Saturation Strategy
The saturation strategy for reachability analysis, i.e., the transitive closure of transition relations applied to some set of states, was first proposed by Ciardo et al. See for an overview [11, 13]. Saturation was combined with onthefly transition learning in [12]. Besides reachability, saturation has also been applied to CTL model checking [32] and in checking fairness constraints with strongly connected components [33].
Saturation is wellstudied. The core idea is to always fire enabled transitions at the lower levels in the decision diagram, before proceeding to the next level. This tends to keep the intermediate BDD sizes much smaller than for instance the breadthfirst exploration strategy. This is in particular the case for asynchronous systems, where transitions exhibit locality. There is also a major influence from the variable reordering: if the variables involved in a transition are grouped together, then this transition only affects adjacent levels in the decision diagram.
We refer to [13] for a precise description of saturation. Our implementation deviates from the standard presentation in three ways. First, we implemented saturation for LDDs and BDDs, instead of MDDs. Next, we never update nodes in the LDD forest in situ; instead, we always create new nodes. Finally, the standard representation has a mutual recursion between saturation and firing transitions. Instead, we fire transition using the existing function for relational product, which is called from our saturation algorithm. As a consequence, the extension with saturation becomes more orthogonal to the specific decision diagram implementation. We refer to Sect. 3 for a detailed description of our algorithm. We show in Sect. 5 that these design decisions do not introduce computational overhead.
3 Multicore Saturation Algorithm
To access the three elements of an LDD node x, Sylvan [16] provides the functions \(\texttt {value}(x)\), \(\texttt {down}(x)\), \(\texttt {right}(x)\). To create or retrieve a unique LDD node using the hash table, Sylvan provides \(\texttt {LookupLDDNode}(\textit{value},\textit{down},\textit{right})\).
Furthermore, Sylvan provides several operations on LDDs that we use to implement reachability algorithms, such as \(\texttt {union}(A,B)\) to compute the set union \(A\cup B\) and \(\texttt {minus}(A,B)\) to compute the set difference \(A\setminus B\). For transition relations, Sylvan provides an operation \(\texttt {relprod}(S,R)\) to compute the successors of S with transition relation R, and an operation \(\texttt {relprodunion}(S,R)\) that computes \(\texttt {union}(S,\texttt {relprod}(S,R))\), i.e., computing the successors and adding them to the given set of states, in one operation. All these operations are internally parallelized, as described in [16].
We implement multicore saturation as in Algorithm 1. We have a transition relation disjunctively partitioned into M relations \(R_0\dots R_{M1}\). These relations are sorted by the level (depth) of the decision diagram where they are applied, which is the first level touched by the relation. We say that relation \(R_i\) is applied at depth \(d_i\). We identify the current next relation with a number k, \(0\le k\le M\), where \(k=M\) denotes “no next relation”. Decision diagram levels are sequentially numbered with 0 for the root level.
The saturate algorithm is given the initial set of states S and the initial next transition relation \(k=0\) and the initial decision diagram level \(d=0\). The algorithm is a straightforward implementation of saturation. First we check the easy cases where we reach either the end of an LDD list, where \(S=0\), or the bottom of the decision diagram, where \(S=1\). If there are no more transition relations to apply, then \(k=M\) and we can simply return S. When we arrive at line 4, the operation is not trivial and we consult the operation cache.
If the result of this operation was not already in the cache, then we check whether we have relations at the current level. Since the relations are sorted by the level where they must be applied, we compare the current level d with the level \(d_k\) of the next relation k. If we have relations at the current level, then we perform the fixed point computation where we first saturate S for the remaining relations, starting at relation \(k'\), which is the first relation that must be applied on a deeper level than d, and then apply the relations of the current level, that is, all \(R_i\) where \(k\le i<k'\). If no relations match the current level, then we compute in parallel the results of the suboperations for the LDD of successor “right” and for the LDD of successor “down”. After obtaining these sub results, we use LookupLDDNode to compute the final result for this LDD node. Finally, we store this result in the operation cache and return it.
The do in parallel keyword is implemented with the workstealing framework Lace [18], which is embedded in Sylvan [16] and offers the primitives spawn and sync to create subtasks and wait for their completion. The implementation using spawn and sync of lines 12–14 is as follows.
The implementation of multicore saturation for BDDs is identical, except that we parallelize on the “then” and “else” successors of a BDD node, instead of on the “down” and “right” successors of an LDD node.
To add onthefly transition relation learning to this algorithm, we simply modify the loop at line 9 as follows:
The learntransitions function provided by LTSmin updates relation i given a set of states S. The function first restricts S to socalled short states \(S^i\), which is the projection of S on the state variables that are touched by relation i. Then it calls the nextstate function of the PINS interface for each new short state and it updates \(R_i\) with the new transitions.
Updating transition relations from multiple threads is not completely trivial. LTSmin solves this using lockfree programming with the compareandswap operation. After collecting all new transitions, LTSmin computes the union with the known transitions and uses compareandswap to update the global relation; if this fails, the union is repeated with the new known transitions.
4 Contributed Tools
We present several new tools and extensions to existing tools produced in this work. The new tools support experiments and comparisons between various DD formats. The extension to Sylvan and LTSmin provides endusers with multicore saturation for reachability analysis.
4.1 Tools for Experimental Purposes
For the empirical evaluation, we need to isolate the reachability analysis of a given LDD (or BDD or MDD). To that end, we implemented three small tools that only compute the set of reachable states, namely for LDDs, for BDDs and for MDDs using the library Meddly. These tools are given an input file representing the model, compute the set of reachable states, and report the number of states and the required time to compute all reachable states. Additionally we provide the tools and that convert an LDD file to a BDD file and to an MDD file. The LDD input files are generated using LTSmin (see below). These tools can all be found online^{Footnote 1}.
4.2 Tools for OnTheFly Multicore Saturation
Onthefly multicore saturation is implemented in the LTSmin toolset, which can be found online^{Footnote 2}. The examples in this section are also online^{Footnote 3}. Onthefly multicore saturation for Petri nets is available in LTSmin’s tool . This tool computes all reachable markings with parallel saturation. The command line to run it on Fig. 1 is . The tool reports: . Additionally, it appears the final LDD has 16 nodes.
Here the syntactic variable order of the places in is used. To use a better variable order, the option is added to the command line. For instance adding runs Force, while runs Sloan’s algorithm (as implemented in the wellknown Boost library). Running reports that the final LDD has only 12 nodes.
The naming convention of LTSmin’s binaries follows the Partitioned NextState Interface (PINS) architecture [5, 22, 25]. PINS forms a bridge between several language frontends and algorithmic backends. Consequently, besides , LTSmin also provides and several other combinations. These binaries generate the state space for the languages PNML, DVE and Promela, by means of distributed explicitstate, multicore explicitstate and multicore symbolic algorithms, respectively. Additionally, LTSmin supports checking for deadlocks and invariants, and verifying LTL properties and \(\mu \)calculus formulas. In this work we focus on state space generation with the symbolic backend only.
We now demonstrate multicore saturation for Promela models. Consider the file which is an implementation of the GARP protocol [23]. To compute the reachable state space with the proposed algorithm and Force order, run: . On a consumer laptop with 8 hardware threads, LTSmin reports 385,000,995,634 reachable states within 1 min. To run the example with a single worker, run . On the same laptop, the algorithm runs in 4 min with 1 worker. We thus have a speedup of 4\(\times \) with 8 workers for symbolic saturation on a Promela model.
5 Empirical Evaluation
Our goal with the empirical study is fivefold. First, we compare our parallel implementation with only 1 core to the purely sequential implementation of the MDD library Meddly [4], in order to determine whether our implementation is competitive with the stateoftheart. Second, we study parallel scalability up to 16 cores for all models and up to 48 cores with a small selection of models. Third, we compare parallel saturation with LDDs to parallel saturation with ordinary BDDs, to see if we get similar results with BDDs. Fourth, we compare parallel saturation without onthefly transition learning to onthefly parallel saturation, to see the effects of onthefly transition learning on the performance of the algorithm. Fifth, we compare parallel saturation with other reachability strategies, namely chaining and BFS, to confirm whether saturation is indeed a better strategy than chaining and BFS.
To perform this evaluation, we use the P/T Petri net benchmarks obtained from the Model Checking Contest 2016 [24]. These are 491 models in total, stored in PNML files. We use parallel onthefly saturation (in LTSmin) with a generous timeout of 1 hour to obtain LDD files of the models, using the Force variable ordering and using the Sloan variable ordering. In total, 413 of potentially 982 LDD files were generated. These LDD files simply store the list decision diagrams of the initial states and of all transition relations. We convert the LDD files to BDD files (binary decision diagrams) with an optimal number of binary variables. We also convert the LDD files to MDD files for the experiments using Meddly. This ensures that all solvers have the same input model with the same variable order.
See Table 1 for the list of solving methods. As described in Sect. 4, we implement the tools lddmc, bddmc and medmc to isolate reachability computation for the purposes of this comparison, using respectively the LDDs and BDDs of Sylvan and the MDDs of Meddly. The onthefly parallel saturation using LDDs is performed with the pnml2ltssym tool of LTSmin. We use the command line pnml2ltssym ORDER laceworkers=WORKERS saturation=sat FILE, where ORDER is rf for Force and rbs for Sloan and WORKERS is a number from the set \(\{1,2,4,8,16\}\).
All experimental scripts, input files and log files are available online (see footnote 3). The experiments are performed on a cluster of Dell PowerEdge M610 servers with two Xeon E5520 processors and 24 GB internal memory each. The tools are compiled with gcc 5.4.0 on Ubuntu 16.04. The experiments for up to 48 cores are performed on a single computer with 4 AMD Opteron 6168 processors with 12 cores each and 128 GB internal memory.
When reporting on parallel executions, we use the number of workers for how many hardware threads (cores) were used.
Overview. After running all experiments, we obtain the results for 413 models in total, of which 196 models with the Force variable ordering and 217 models with the Sloan variable ordering. In the remainder of this section, we study these 413 benchmarks. See Table 2, which shows the number of models for which each method could compute the set of reachable states within 20 min.
To correctly compare all runtimes, we restrict the set of models to those where all methods finish within 20 min with any number of workers. We retain in total 301 models where no solver hit the timeout. See Table 3 for the cumulative times for each method and number of workers and the parallel speedup. Notice that this is the speedup for the entire set of 301 models and not for individual models.
Comparing LDD saturation with Meddly’s saturation. We evaluate how lddsat with just 1 worker compares to the sequential saturation of Meddly. The goal is not to directly measure whether there is a parallel overhead from using parallelism in Sylvan, as the algorithm in lddmc is fundamentally different because it uses LDDs instead of MDDs and the algorithm does not inplace saturate nodes, as also explained in Sect. 3. The low parallel overheads of Sylvan are already demonstrated elsewhere [15, 16, 18]. Rather, the goal is to see how our version of saturation compares to the stateoftheart.
Table 2 shows that Meddly’s implementation (mddsat) and our implementation (lddsat 1) are quite similar in the number of solved models. Meddly solves 375 benchmarks and our implementation solves 388 within 20 min.
See Table 3 for a comparison of runtimes. Meddly solves the 150 models with Sloan almost 2\(\times \) as fast as our implementation in Sylvan, but is slower than our implementation for the 151 models with Force. We observe for individual models that the difference between the two solvers is within an order of magnitude for most models, although there are some exceptions. Our implementation quickly overtakes Meddly with additional workers.
Parallel Scalability. As shown in Table 3, using 16 workers, we obtain a modest parallel speedup for saturation of 6.2\(\times \) (with Sloan) and 4.7\(\times \) (with Force). On individual models, the differences are large. The average speedup of the individual benchmarks is only 1.8\(\times \) with 16 workers, but there are many slowdowns for models that take less than a second with 1 worker. We take an arbitrary selection of models with a high parallel speedup and run these on a dedicated 48core machine. Table 4 shows that even up to 48 cores, parallel speedup keeps improving. We even see a speedup of 52.2\(\times \). For this superlinear speedup we have two possible explanations. One is that there is some nondeterminism inherent in any parallel computation; another is already noted in [20] and is related to the “chaining” in saturation, see further [20].
Comparing LDD saturation with BDD saturation. As Table 3 shows, the lddsat and bddsat method have a similar performance and similar parallel speedups.
Onthefly LDD saturation. Comparing the performance of offline saturation with onthefly saturation, we observe the same scalability with the Sloan variable order, but onthefly saturation requires roughly 2\(\times \) as much time. With the Force variable order, onthefly saturation is slower but has a higher parallel speedup of 7.9\(\times \).
Comparing saturation, chaining and BFS. We also compare the saturation algorithm with other popular strategies to compute the set of reachable states, namely standard (parallelized) BFS and chaining, given in Fig. 5. As Tables 2 and 3 show, chaining is significantly faster than BFS and saturation is again significantly faster than chaining. In terms of parallel scalability, we see that parallelized BFS scales better than the others, because it can already parallelize in the main loop by computing successors for all relations in parallel, which chaining and saturation cannot do. For the entire set of benchmarks, saturation is the superior method, however there are individual differences and for some models, saturation is not the fastest method.
6 Conclusion
We presented a multicore implementation of saturation for the efficient computation of the set of reachable states. Based on Sylvan’s multicore decision diagram framework, the design of the saturation algorithm is mostly orthogonal to the type of decision diagram. We showed the implementation for BDDs and LDDs; the translation relation can be learned onthefly. The functionality is accessible through the LTSmin highperformance model checker. This makes parallel saturation available for a whole collection of asynchronous specification languages. We demonstrated multicore saturation for Promela and for Petri nets in PNML representation.
We carried out extensive experiments on a benchmark of Petri nets from the Model Checking Contest. The total speedup of onthefly saturation is 5.9\(\times \) on 16 cores with the Sloan variable ordering and 7.9\(\times \) with the Force variable ordering. However, there are many small models (computed in less than a second) in this benchmark. For some larger models we showed an impressive 52\(\times \) speedup on a 48core machine. From our measurements, we further conclude that the efficiency and parallel speedup for the BDD variant is just as good as the speedup for LDDs. We compared efficiency and speedup of saturation versus other popular exploration strategies, BFS and chaining. As expected, saturation is significantly faster than chaining, which is faster than BFS; this trend is maintained in the parallel setting. Our measurements show that the variable ordering (Sloan versus Force), and the model representation (precomputed transition relations versus learned onthefly) do have an impact on efficiency and speedup. Parallel speedup should not come at the price of reduced efficiency. To this end, we compared our parallel saturation algorithm for one worker to saturation in Meddly. Meddly solves fewer models within the timeout, but is slightly faster in other cases, but parallel saturation quickly overtakes Meddly with multiple workers.
Future work could include the study of parallel saturation on exciting new BDD types, like tagged BDDs and chained BDDs [8, 19]. The results on tagged BDDs showed a significant speedup compared to ordinary BDDs on experiments in LTSmin with the BEEM benchmark database. Another direction would be to investigate the efficiency and speedup of parallel saturation in other applications, like CTL model checking, SCC decomposition, and bisimulation reduction.
References
Aloul, F.A., Markov, I.L., Sakallah, K.A.: FORCE: a fast and easytoimplement variableordering heuristic. In: VLSI 2003, pp. 116–119. ACM (2003)
Amparore, E.G., Beccuti, M., Donatelli, S.: Gradientbased variable ordering of decision diagrams for systems with structural units. In: D’Souza, D., Narayan Kumar, K. (eds.) ATVA 2017. LNCS, vol. 10482, pp. 184–200. Springer, Cham (2017). https://doi.org/10.1007/9783319681672_13
Amparore, E.G., Donatelli, S., Beccuti, M., Garbi, G., Miner, A.S.: Decision diagrams for Petri nets: which variable ordering? In: PNSE @ Petri Nets. CEUR Workshop Proceedings, vol. 1846, pp. 31–50. CEURWS.org (2017)
Babar, J., Miner, A.S.: Meddly: multiterminal and edgevalued decision diagram library. In: QEST, pp. 195–196. IEEE Computer Society (2010)
Blom, S., van de Pol, J., Weber, M.: LTSmin: distributed and symbolic reachability. In: Touili, T., Cook, B., Jackson, P. (eds.) CAV 2010. LNCS, vol. 6174, pp. 354–359. Springer, Heidelberg (2010). https://doi.org/10.1007/9783642142956_31
Bollig, B., Wegener, I.: Improving the variable ordering of OBDDs is NPcomplete. IEEE Trans. Comput. 45(9), 993–1002 (1996)
Bryant, R.E.: Graphbased algorithms for Boolean function manipulation. IEEE Trans. Comput. C–35(8), 677–691 (1986)
Bryant, R.E.: Chain reduction for binary and zerosuppressed decision diagrams. In: Beyer, D., Huisman, M. (eds.) TACAS 2018. LNCS, vol. 10805, pp. 81–98. Springer, Cham (2018). https://doi.org/10.1007/9783319899602_5
Chung, M., Ciardo, G.: Saturation NOW. In: QEST, pp. 272–281. IEEE Computer Society (2004)
Chung, M., Ciardo, G.: Speculative image computation for distributed symbolic reachability analysis. J. Logic Comput. 21(1), 63–83 (2011)
Ciardo, G., Lüttgen, G., Siminiceanu, R.: Saturation: an efficient iteration strategy for symbolic state—space generation. In: Margaria, T., Yi, W. (eds.) TACAS 2001. LNCS, vol. 2031, pp. 328–342. Springer, Heidelberg (2001). https://doi.org/10.1007/3540453199_23
Ciardo, G., Marmorstein, R.M., Siminiceanu, R.: Saturation unbound. In: Garavel, H., Hatcliff, J. (eds.) TACAS 2003. LNCS, vol. 2619, pp. 379–393. Springer, Heidelberg (2003). https://doi.org/10.1007/354036577X_27
Ciardo, G., Marmorstein, R.M., Siminiceanu, R.: The saturation algorithm for symbolic statespace exploration. STTT 8(1), 4–25 (2006)
Ciardo, G., Zhao, Y., Jin, X.: Parallel symbolic statespace exploration is difficult, but what is the alternative? In: PDMC, EPTCS, vol. 14, pp. 1–17 (2009)
van Dijk, T.: Sylvan: multicore decision diagrams. Ph.D. thesis, University of Twente, July 2016
van Dijk, T., van de Pol, J.: Sylvan: multicore framework for decision diagrams. STTT 19(6), 675–696 (2017)
van Dijk, T., van de Pol, J.: Multicore symbolic bisimulation minimisation. STTT 20(2), 157–177 (2018)
van Dijk, T., van de Pol, J.C.: Lace: nonblocking split deque for workstealing. In: Lopes, L., et al. (eds.) EuroPar 2014. LNCS, vol. 8806, pp. 206–217. Springer, Cham (2014). https://doi.org/10.1007/9783319143132_18
van Dijk, T., Wille, R., Meolic, R.: Tagged BDDs: combining reduction rules from different decision diagram types. In: FMCAD, pp. 108–115. IEEE (2017)
Ezekiel, J., Lüttgen, G., Ciardo, G.: Parallelising symbolic statespace generators. In: Damm, W., Hermanns, H. (eds.) CAV 2007. LNCS, vol. 4590, pp. 268–280. Springer, Heidelberg (2007). https://doi.org/10.1007/9783540733683_31
Heyman, T., Geist, D., Grumberg, O., Schuster, A.: A scalable parallel algorithm for reachability analysis of very large circuits. Formal Methods Syst. Des. 21(3), 317–338 (2002)
Kant, G., Laarman, A., Meijer, J., van de Pol, J., Blom, S., van Dijk, T.: LTSmin: highperformance languageindependent model checking. In: Baier, C., Tinelli, C. (eds.) TACAS 2015. LNCS, vol. 9035, pp. 692–707. Springer, Heidelberg (2015). https://doi.org/10.1007/9783662466810_61
Konnov, I., Letichevsky, O.: Model checking GARP protocol using Spin and VRS. In: IW on Automata, Algorithms, and Information Technology (2010)
Kordon, F., et al.: Complete Results for the 2016 Edition of the Model Checking Contest, June 2016. http://mcc.lip6.fr/2016/results.php
Meijer, J., Kant, G., Blom, S., van de Pol, J.: Read, write and copy dependencies for symbolic model checking. In: Yahav, E. (ed.) HVC 2014. LNCS, vol. 8855, pp. 204–219. Springer, Cham (2014). https://doi.org/10.1007/9783319133386_16
Meijer, J., van de Pol, J.: Bandwidth and wavefront reduction for static variable ordering in symbolic reachability analysis. In: Rayadurgam, S., Tkachuk, O. (eds.) NFM 2016. LNCS, vol. 9690, pp. 255–271. Springer, Cham (2016). https://doi.org/10.1007/9783319406480_20
Noack, A.: A ZBDD package for efficient model checking of Petri nets. Forschungsbericht, Branderburgische Technische Uinversität Cottbus (1999)
Oortwijn, W., van Dijk, T., van de Pol, J.: Distributed binary decision diagrams for symbolic reachability. In: 24th ACM SIGSOFT International SPIN Symposium on Model Checking of Software, pp. 21–30 (2017)
Siminiceanu, R., Ciardo, G.: New metrics for static variable ordering in decision diagrams. In: Hermanns, H., Palsberg, J. (eds.) TACAS 2006. LNCS, vol. 3920, pp. 90–104. Springer, Heidelberg (2006). https://doi.org/10.1007/11691372_6
Sloan, S.W.: A FORTRAN program for profile and wavefront reduction. Int. J. Numer. Methods Eng. 28(11), 2651–2679 (1989)
Vörös, A., Szabó, T., Jámbor, A., Darvas, D., Horváth, Á., Bartha, T.: Parallel saturation based model checking. In: ISPDC, pp. 94–101. IEEE Computer Society (2011)
Zhao, Y., Ciardo, G.: Symbolic CTL model checking of asynchronous systems using constrained saturation. In: Liu, Z., Ravn, A.P. (eds.) ATVA 2009. LNCS, vol. 5799, pp. 368–381. Springer, Heidelberg (2009). https://doi.org/10.1007/9783642047619_27
Zhao, Y., Ciardo, G.: Symbolic computation of strongly connected components and fair cycles using saturation. ISSE 7(2), 141–150 (2011)
van Dijk, T., van de Pol, J., Meijer, J.: Artifact and instructions to generate experimental results for TACAS 2019 paper: Multicore OnTheFly Saturation (artifact). Figshare (2019). https://doi.org/10.6084/m9.figshare.7825406.v1
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2019 The Author(s)
About this paper
Cite this paper
van Dijk, T., Meijer, J., van de Pol, J. (2019). Multicore OnTheFly Saturation. In: Vojnar, T., Zhang, L. (eds) Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2019. Lecture Notes in Computer Science(), vol 11428. Springer, Cham. https://doi.org/10.1007/9783030174651_4
Download citation
DOI: https://doi.org/10.1007/9783030174651_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 9783030174644
Online ISBN: 9783030174651
eBook Packages: Computer ScienceComputer Science (R0)