More Scalable LTL Model Checking via Discovering DesignSpace Dependencies (\(D^{3}\))
 6 Citations
 5.7k Downloads
Abstract
Modern system design often requires comparing several models over a large design space. Different models arise out of a need to weigh different design choices, to check core capabilities of versions with varying features, or to analyze a future version against previous ones. Model checking can compare different models; however, applying model checking offtheshelf may not scale due to the large size of the design space for today’s complex systems. We exploit relationships between different models of the same (or related) systems to optimize the modelchecking search. Our algorithm, \(D^{3}\), preprocesses the design space and checks fewer modelchecking instances, e.g., using nuXmv. It automatically prunes the search space by reducing both the number of models to check, and the number of LTL properties that need to be checked for each model in order to provide the complete modelchecking verdict for every individual modelproperty pair. We formalize heuristics that improve the performance of \(D^{3}\). We demonstrate the scalability of \(D^{3}\) by extensive experimental evaluation, e.g., by checking 1,620 reallife models for NASA’s NextGen air traffic control system. Compared to checking each modelproperty pair individually, \(D^{3}\) is up to 9.4\(\times \) faster.
Keywords
Model Checking Large Design Space Wheel Brake System (WBS) General Configuration Parameters (GenPC) Workﬂow Veriﬁcation1 Introduction
In the early phases of design, there are frequently many different models of the system under development [2, 23, 29] constituting a design space. We may need to evaluate different design choices, to check core capabilities of system versions with varying featurelevels, or to analyze a future version against previous ones in the product line. The models may differ in their assumptions, implementations, and configurations. We can use model checking to aid system development via a thorough comparison of the set of system models against a set of properties representing requirements. Model checking, in combination with related techniques like faulttree analysis, can provide an effective comparative analysis [23, 29]. The classical approach checks each model onebyone, as a set of independent modelchecking runs. For large and complex design spaces, performance can be inefficient or even fail to scale to handle the combinatorial size of the design space. Nevertheless, the classical approach remains the most widely used method in industry [3, 23, 25, 29, 30]. Algorithms for familybased model checking [11, 13] mitigate this problem but their efficiency and applicability still depends on the use of custom model checkers to deal with model families.
We assume that each model in the design space can be parameterized over a finite set of parametric inputs that enable/disable individual assumptions, implementations, or behaviors. It might be the case that for any pair of models the assumptions are dependent, their implementations contradict each other, or they have the same behavior. Since the different models of the same system are related, it is possible to exploit the known relationships between them, if they exist, to optimize the model checking search. These relationships can exist in two ways: relationships between the models, and relationships between the properties checked for each model.
We present an algorithm that automatically prunes and dynamically orders the modelchecking search space by exploiting intermodel relationships. The algorithm, Discover DesignSpace Dependencies (\(D^{3}\)), reduces both the number of models to check, and the number of LTL properties that need to be checked for each model. Rather than using a custom model checker, \(D^{3}\) works with any offtheshelf checker. This allows practitioners to use stateoftheart, optimized modelchecking algorithms, and to choose their preferred model checker, which enables adoption of our method by practitioners who already use model checking with minimum change in their verification workflow. We reason about a set of system models by introducing the notion of a Combinatorial Transition System (CTS). Each individual model, or instance, can be derived from the CTS by configuring it with a set of parameters. Each transition in the CTS is enabled/disabled by the parameters. We model check each instance of the CTS against sets of properties. We assume the properties are in Linear Temporal Logic (LTL) and are independent of the choice of parameters, though not all properties may apply to all instances. \(D^{3}\) preprocesses the CTS to find relationships between parameters and minimizes the number of instances that need to be checked to produce results for the whole set. It uses LTL satisfiability checking [33] to determine the dependencies between pairs of LTL properties, then reduces the number of properties that are checked for each instance. \(D^{3}\) returns results for every modelproperty pair in the design space, aiming to compose these results from a reduced series of modelchecking runs compared to the classical approach of checking every modelproperty pair. We demonstrate the industrial scalability of \(D^{3}\) using a set of 1,620 reallife, publiclyavailable SMVlanguage benchmark models with LTL specifications; these model NASA’s NextGen air traffic control system [8, 23, 29]. We also evaluate the propertydependence analysis separately on reallife models of Boeing AIR 6110 Wheel Braking System [3] to evaluate \(D^{3}\) in multiproperty verification workflows.
Related Work. One striking contrast between \(D^{3}\) and related work is that \(D^{3}\) is a preprocessing algorithm, does not require custom modeling, and works with any offtheshelf LTL model checker. Parameter synthesis [9] can generate the many models in a design space that can be analyzed by \(D^{3}\); however existing parameter synthesis techniques require custom modeling of a system. We take the easier path of reasoning over an alreadyrestricted set of models of interest to system designers. \(D^{3}\) efficiently compares any set of models rather than finding all models that meet the requirements. Several parameter synthesis approaches designed for parametric Markov models [15, 16, 24, 31] use PRISM and compute the region of parameters for which the model satisfies a given probabilistic property (PCTL or PLTL); \(D^{3}\) is an LTLbased algorithm. Parameter synthesis of a parametric Markov model with nonprobabilistic transitions can generate the many models that \(D^{3}\) can analyze. In multiobjective model checking [1, 21, 22, 28], given a Markov decision process and a set of LTL properties, the algorithms find a controller strategy such that the Markov process satisfies all properties with some set probability. Differently from multiobjective model checking, which generates “tradeoff” Pareto curves, \(D^{3}\) gives a boolean result. The parameterized model checking problem (PCMP) [20] deals with infinite families of homogeneous processes in a system; in our case, the models are finite and heterogeneous. Specialized modelset checking algorithms [18] can check the reduced set of \(D^{3}\) processed models.
In multiproperty model checking, multiple properties are checked on the same system. Existing approaches simplify the task by algorithm modifications [4, 7], SATsolver modifications [26, 27], and property grouping [5, 6]. The interproperty dependence analysis of \(D^{3}\) can be used in multiproperty checking. We compare \(D^{3}\) against the affinity [6] based approach to property grouping.
Product line verification techniques, e.g., with Software Product Lines (SPL), also verify parametric models describing large design spaces. We borrow the notion of an instance, from SPL literature [32, 34]. An extension to NuSMV in [13] performs symbolic model checking of featureoriented CTL. The symbolic analysis is extended to the explicit case and support for featureoriented LTL in [11, 12]. The work most closely related to ours is [17] where product line verification is done without a familybased model checker. \(D^{3}\) outputs modelchecking results for every modelproperty pair in the design space (e.g. all parameter configurations) without dependence on any feature whereas in SPL verification using an offtheshelf checker, if a property fails then it isn’t possible to know which models do satisfy the property [14, 17].
 1.
A fully automated, general, and scalable algorithm for checking design spaces; it can be applied to LTL model checking problems without major modifications to the system designers’ verification workflow.
 2.
Modification to the general modelchecking procedure of sequentially checking properties against a model to a dynamic procedure; the next property to check is chosen to maximize the number of yettobechecked properties for which the result can be determined from interproperty dependencies.
 3.
Comparison of our novel interproperty dependence analysis to existing work in multiproperty verification workflows [6].
 4.
Extensive experimental analysis using reallife benchmarks; all reproducibility artifacts and source code are publicly available.
2 Preliminaries
Definition 1
 1.
\({ \Sigma }\) is a finite alphabet, or set of atomic propositions,
 2.
S is a finite set of states,
 3.
\(s_0 \in S\) is an initial state,
 4.
\(L : S \rightarrow 2^{ \Sigma }\) is a labeling function that maps each state to the set of atomic propositions that hold in it, and
 5.
\(\delta : S \rightarrow S\) is the transition function.
A computation path, or run of LTS M is a sequence of states \(\pi = s_0 {\rightarrow } s_1 {\rightarrow } \ldots {\rightarrow } s_n\) over the word \(w = L(s_0), L(s_1), \ldots , L(s_n)\) such that \(s_i \in S\) for \(0 \le i \le n\), and \((s_i, s_{i+1}) \in \delta \) for \(0 \le i < n\). Given a LTL property \(\varphi \) and a LTS M, M models \(\varphi \), denoted \(M \models \varphi \), iff \(\varphi \) holds in all possible computation paths of M.
Definition 2
A parameter \(P_i\) is a variable with the following properties.
 1.
The domain of \(P_i\), denoted \(\llbracket P_i \rrbracket \), is a finite set of possible assignments to \(P_i\).
 2.
Parameter \(P_i\) is set by assigning a single value from \(\llbracket P_i \rrbracket \), i.e. \(P_i = d_{P_i} \in \llbracket P_i \rrbracket \). A nonassigned parameter is considered unset.
 3.
Parameter setting is static, i.e., it does not change during a run of the system.
Let P be a finite set of parameters. P denotes the number of parameters. For each \(P_i \in P\), \(P_i\) denotes the size of the domain of \(P_i\). Let Form(P) denote the set of all Boolean formulas over P generated using the BNF grammar \({\varphi }\, {:}{:=}~\top \mid P_i {\text { { == }}} \text {D} \) and \( \text {D}\, {:}{:=}~P_{i_1} \mid P_{i_2} \mid \ldots \mid P_{i_n};\) for each \(P_i \in P\), \(n = P_i\), and \(\llbracket P_i \rrbracket ~{=}~\{P_{i_1}, P_{i_2}, \ldots , P_{i_n}\}\). Therefore, Form(P) contains \(\top \) and equality constraints over parameters in P.
Definition 3
 1.
P is a finite set of parameters to the system, and
 2.
\(L_P : \delta \rightarrow Form(P)\) is function labeling transitions with a guard condition.
We limit the guard condition over a transition to \(\top \) or an equality constraint over a single parameter for simpler expressiveness and formalization. However, there can be multiple transitions between any two states with different guards. A transition is enabled if its guard condition evaluates to true, otherwise, it is disabled. A label of \(\top \) implies the transition is always enabled. A possible run of a CTS is a sequence of states \(\pi _P = s_0 {\mathrel {\mathop {\rightarrow }\limits ^{\nu _1}}} s_1 {\mathrel {\mathop {\rightarrow }\limits ^{\nu _2}}} \ldots {\mathrel {\mathop {\rightarrow }\limits ^{\nu _n}}} s_n\) over the word \(w = L(s_0), L(s_1), \ldots , L(s_n)\) such that \(s_i \in S\) for \(0 \le i \le n\), \(\nu _i \in Form(P)\) for \(0 < i \le n\), and \((s_i, s_{i+1}) \in \delta \) and \((s_i, s_{i+1}, \nu _{i+1}) \in L_P\) for \(0 \le i < n\), i.e., there is transition from \(s_i\) to \(s_{i+1}\) with guard condition \(\nu _{i+1}\). A prefix \(\alpha \) of a possible run \(\pi _P = \alpha {\mathrel {\mathop {\rightarrow }\limits ^{\nu _i}}} \ldots {\mathrel {\mathop {\rightarrow }\limits ^{\nu _n}}}s_n\) is also a possible run.
Example 1. A Boolean parameter has domain \(\{true,\) \(false\}\). Figure 1 shows a CTS with Boolean parameters \(P = \{P_1, P_2, P_3\}\). For brevity, guard condition \(P_i == true\) is written as \(P_i\), while \(P_i == false\) is written as \(\lnot P_i\). A transition with label \(P_1\) is enabled if \(P_1\) is set to true. Similarly, a label of \(\lnot P_3\) implies the transition is enabled if \(P_3\) is set to false.
Example 1
A Boolean parameter has domain \(\{true,\) \(false\}\). Figure 1 shows a CTS with Boolean parameters \(P = \{P_1, P_2, P_3\}\). For brevity, guard condition \(P_i == true\) is written as \(P_i\), while \(P_i == false\) is written as \(\lnot P_i\). A transition with label \(P_1\) is enabled if \(P_1\) is set to true. Similarly, a label of \(\lnot P_3\) implies the transition is enabled if \(P_3\) is set to false.
Definition 4
A parameter configuration c for a set of parameters P is a ktuple \((d_{P_1},\) \(d_{P_2}, \ldots , d_{P_k})\), for \(k = P\), that sets each parameter in P, i.e., for every \(1 \le i \le k\), \(P_i = d_{P_i}\) and \(d_{P_i} \in \llbracket P_i \rrbracket \) is a setting. The set of all possible configurations \(\mathbb {C}\) over P is equal to \(P_1 \times P_2 \times \ldots \times P_k\) where \(\times \) denotes the cross product. The setting for \(P_i\) in configuration c is denoted by \(c(P_i)\).
A configured run of a CTS \(M_P\) over a configuration c, or crun, is a sequence of states \(\pi _{P(c)} = s_0 \xrightarrow {\nu _1} s_1 \xrightarrow {\nu _2} \ldots \xrightarrow {\nu _n} s_n\) such that \(\pi _{P(c)}\) is a possible run, and \(c \vdash \nu _i\) for \(0 < i \le n\), where \(\vdash \) denotes propositional logic satisfaction of the guard condition \(\nu _i\) under parameter configuration c. Given a CTS \(M_P\) and a parameter configuration c, a state t is reachable iff there exists a crun such that \(s_n = t\), denoted \(s_0 \xrightarrow [c]{*} t\), i.e., t can be reached in zero or more transitions. A transition with guard \(\nu \) is reachable iff \((s_j, s_{j+1}, \nu ) \in L_P\), \((s_j, s_{j+1}) \in \delta \), and \(s_0 \xrightarrow [c]{*} s_j\).
Definition 5
An instance of a CTS \(M_P = (\Sigma , S, s_0, L, \delta , P, L_P)\) for parameter configuration c is a LTS \(M_{P(c)} = (\Sigma , S, s_0, L, \delta ')\) where \(\delta ' = \{t \in \delta \mid c \vdash L_P(t)\}\).
Given a LTL property \(\varphi \) and a CTS \(M_P = (\Sigma , S, s_0, L, \delta , P, L_P)\), the model checking problem for \(M_P\) is to find all parameter configurations \(c \in \mathbb {C}\) over P such that \(\varphi \) holds in all cruns of \(M_P\), or all computation paths of LTS \(M_{P(c)}\).
Definition 6

In all possible runs with a transition guard over \(P_j\), a transition with guard over \(P_i\) appears before a transition with guard over \(P_j\), and

In all configured runs, the setting for \(P_i\) in c makes transitions with guard conditions over \(P_j\) unreachable.
Example 2 In Fig. 1, if \(P_1\) is set to false, execution never reaches the transition labeled \(\lnot P_3\). Therefore, if configuration \(c = (false, true, true)\) then \(P_3 \leadsto _c P_1\).
Definition 7
A universal model U is a LTS that generates all possible computations paths over its atomic propositions.
Theorem 1 (LTL Satisfiability)
[33] Given a LTL property \(\varphi \) and a universal model U, \(\varphi \) is satisfiable if and only if Open image in new window .
This theorem reduces LTL satisfiability checking to LTL model checking. Therefore, \(\varphi \) is satisfiable when the model checker finds a counterexample.^{1}
3 Discovering DesignSpace Dependencies
In this section we describe \(D^{3}\). Our approach speeds up model checking of combinatorial transitions systems by preprocessing of the input instances; it therefore increases efficiency of both BDDbased and SATbased model checkers. The problem reduction is along two dimensions: number of instances, and number of properties.
3.1 Reduction Along the Number of Instances
Given a set of parameters P, a combinatorial transition system \(M_P\), and a property \(\varphi \), \(M_P\) is model checked by sending, for all parameter configuration \(c \in \mathbb {C}\), instance \(M_{P(c)}\) to the LTS model checker, along with the property \(\varphi \). The output is aggregated for \(\mathbb {C}\) runs of the model checker, and all parameter configurations c, such that \(M_{P(c)} \models \varphi \) are returned. In principle, parameters can be encoded as state variables, and the parametric model can be posed as one big modelchecking obligation, however there are caveats.
 1.
State space explosion before any useful results are obtained.
 2.
Counterexample generated from one run of the model checker gives a single undesirable configuration.
Our goal is to make the classical approach of individualmodel checking more scalable as the design space grows by intelligently integrating possible dependencies between parameter configurations.
Lemma 1
Given a CTS \(M_P = (\Sigma , S, s_0, L, \delta , P, L_P)\) with parameters \(A, B \in P\), if \(B \leadsto _c A\) for some parameter configuration c, then there does not exist any possible run of \(M_P\) with prefix \(\alpha = s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _B}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\rightarrow }\limits ^{\nu _A}}} s_l\), where \(\nu _A\) and \(\nu _B\) are guards over A and B, resp., and \(s_i, s_j, s_k, s_l \in S\), i.e., a transition with guard over B does not appear before a transition with guard over A.
 1.
\(s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _A}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _B}}} s_{l}\) (guard over A before guard over B)
 2.
\(s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _A}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _A}}} s_{l}\) (guards only over A)
 3.
\(s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{*}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\longrightarrow }\limits ^{*}}} s_{l}\) (guards neither over A nor B)
 1.
\(s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _B}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _A}}} s_{l}\) (guard over B before guard over A)
 2.
\(s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _B}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _B}}} s_{l}\) (guards only over B)
 3.
\(s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{*}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\longrightarrow }\limits ^{*}}} s_{l}\) (guards neither over A nor B)
Therefore, when A and B are not dependent, there is no possible run with transition guards over both A and B. Note that for a CTS \(M_P\) with \(A, B \in P\), if A and B are dependent, then either \(A \leadsto _c B\) or \(B \leadsto _c A\) but not both for any configuration c. We only show formalization for \(B \leadsto _c A\); \(A \leadsto _c B\) follows directly.
Theorem 2 (Redundant Instance)

\(c_i(A) = c(A)\) for \(0 < i \le k\), and

\(c_i(B) = d_{B_i} \in \llbracket B \rrbracket \) for \(0 < i \le k\) and \(\llbracket B \rrbracket = \{d_{B_1}, d_{B_2}, \ldots , d_{B_k}\}\)
For such configurations \(M_{P(c_1)} \models \varphi \equiv M_{P(c_2)} \models \varphi \equiv \ldots \equiv M_{P(c_k)} \models \varphi \).

defined, if its corresponding parameter is set in \(\hat{c}\), and

undefined, otherwise.
A defined guard evaluates to true when \(\hat{c} \vdash L_P(t)\), or false when Open image in new window . Algorithm FindUP (Find Unset Parameters) in Fig. 3(a) solves the dual problem of finding independent parameters. It takes as input a CTS \(M_P\) and a partial parameter configuration \(\hat{c}\), and returns unset parameters for which guard conditions are undefined and their corresponding transitions are reachable. It traverses (depthfirst) the CTS starting from a node for the initial state \(s_0\). During traversal, an edge (transition) \(t = (s_i, s_j)\) connects two nodes (states) \(s_i, s_j \in S\) if \(t \in \delta \) and \(\hat{c} \vdash L_P(t)\). The edge is disconnected if \(t \not \in \delta \) or Open image in new window . Since \(M_P\) is defined relationally in the annotated SMV language with preprocessor directives (Sect. 2), in the worst case, FindUP takes polynomial time in the number of symbolic states and transitions. From an implementation point of view, FindUP invokes the cpp for parameter settings in \(\hat{c}\) on the input model, and parses the output for unset parameters.
Lemma 2
FindUP returns unset parameters \(P_i \in P\) for all reachable transitions \(t \in \delta \) such that guard \(L_P(t)\) is a guard over \(P_i\), and is undefined.
Algorithm GenPC (Generate Parameter Configurations) in Fig. 3(b) uses FindUP as a subroutine to recursively find parameter configurations that need to be checked. It takes as input a CTS \(M_P\), queue of unset parameters \(P_u\), and a partial parameter configuration \(\hat{c}\). Initially, \(\hat{c}\) contains no set parameters and \(P_u = \) FindUP(\(M_P\), \(\hat{c}\)). Upon termination of GenPC, \(\hat{\mathbb {C}}\) contains the set of partial parameter configurations that need to be checked. On every iteration, GenPC picks a parameter p from \(P_u\), assigns it a value from its domain \(\llbracket p \rrbracket \) in \(\hat{c}\), and uses FindUP to find unset parameters in CTS \(M_{P}\). If the returned unset parameter queue is empty, \(\hat{c}\) added to \(\hat{\mathbb {C}}\). Otherwise, \(\textsc {GenPC}\) is called again with the new unset parameter queue.
Theorem 3 (GenPC is sound)

\(c_i(A) = \hat{c}(A)\) for \(0 < i \le k\), and

\(c_i(B) = d_{B_i} \in \llbracket B \rrbracket \) for \(0 < i \le k\) and \(\llbracket B \rrbracket = \{d_{B_1}, d_{B_2}, \ldots , d_{B_k}\}\)
Theorem 4 (GenPC is complete)

\(c_i(A) = d_{A_n}\) for \(0 < i \le k\) and \(d_{A_n} \in \llbracket A \rrbracket \), and

\(c_i(B) = d_{B_i} \in \llbracket B \rrbracket \) for \(0 < i \le k\) and \(\llbracket B \rrbracket = \{d_{B_1}, d_{B_2}, \ldots , d_{B_k}\}\)
GenPC returns partial configurations \(\hat{c} \in \hat{\mathbb {C}}\) over parameters. A partial configuration \(\hat{c}\) is converted to a parameter configuration c by setting the unset parameters in \(\hat{c}\) to an arbitrary value from their domain. Note that this operation is safe since the arbitrarily set parameters are not reachable in the instance \(M_{P(c)}\). As a result of this operation, \(\hat{\mathbb {C}}\) contains configurations c that have all parameters set to a value from their domain.
Theorem 5 (Minimality)
The minimal set of parameter configurations is \(\hat{\mathbb {C}}\).
3.2 Reduction Along the Number of Properties
In model checking, properties describe the intended behavior of the system. Usually, properties are iteratively refined to express the designer’s intentions. For small systems, it can be manually determined if two properties are dependent on one another. However, practically determining property dependence for large and complex systems requires automation. Given a set of properties \(\mathcal {P}\), and LTS M, an offtheshelf model checker is called \(N = \mathcal {P}\) times.
 1.
Statespace explosion due to orthogonal coneofinfluences of properties.
 2.
Need for additional analysis of individual properties onebyone in order to discriminate failed ones and generate individual counterexamples.
 3.
Computational cost of verifying grouped properties in one run can be significantly higher than verifying individual properties in a series of runs.
Our goal is to minimize the number of properties checked by intelligently using dependencies between LTL properties. For two LTL properties \(\varphi _1\) and \(\varphi _2\) dependence can be characterized in four ways: \((\varphi _1 \rightarrow \varphi _2)\), \((\varphi _1 \rightarrow \lnot \varphi _2)\), \((\lnot \varphi _1 \rightarrow \varphi _2)\), and \((\lnot \varphi _1 \rightarrow \lnot \varphi _2)\). Theorem 6 allows us to find dependencies automatically.
Theorem 6 (Property Dependence)
For two LTL properties \(\varphi _1\) and \(\varphi _2\) dependence can be established by model checking with universal model U.
4 Experimental Analysis
Our revised model checking procedure \(D^{3}\) is shown in Fig. 6. \(D^{3}\) takes as input a CTS \(M_P\) and a set of LTL properties \(\mathcal {P}\). It uses GenPC to find the parameter configurations that need to be checked. It then generates a property table to store dependencies between LTL properties. Lastly, CheckRP checks each instance against properties in \(\mathcal {P}\). Results are collated for every modelproperty pair.
4.1 Benchmarks
We evaluated \(D^{3}\) on two benchmarks derived from realworld case studies.
 (1)
Air Traffic Controller (ATC) Models: are a set of 1,620 realworld models representing different possible designs for NASA’s NextGen air traffic control (ATC) system. In previous work, this set of models were generated from a contractbased, parameterized nuXmv model; individualmodel checking enabled their comparative analysis with respect to a set of requirements for the system [23]. In the formulation of [23], the checking problem for each model is split in to five phases.^{2} In each phase, all 1,620 models are checked. For our analysis and to gain better understanding of the experimental results, we categories the phases based on the property verification results (unsat if property holds for the model, and sat if it does not). Each of the 1,620 models can be seen as instances of a CTS with seven parameters. Each of the 1620 instances is checked against a total of 191 LTL properties. The original nuXmv code additionally uses OCRA [10] for compositional modeling, though we do not rely on its features when using the generated modelset.
 (2)
Boeing Wheel Braking System (WBS) Models: are a set of seven realworld nuXmv models representing possible designs for the Boeing AIR 6110 wheel braking system [3]. Each model in the set is checked against \(\sim \)200 LTL properties. However, the seven models are not generated from a CTS. We evaluate \(D^{3}\) against this benchmark to evaluate performance on multiproperty verification workflows, and compare with existing work on property grouping [6].
4.2 Experiment Setup
Timing results of 1,620 models for each phase using individualmodel checking, and \(D^{3}\). For individualmodel checking, Time indicates model checking time, whereas, for \(D^{3}\), Time indicates preprocessing time + model checking time.
Phase  Property mix  Properties  Model checking time (in hours)  Speedup  Overall speedup  

Total (median)  Individual  \(D^{3}\)  
I  unsat  25 (24)  6.02  4.02  1.5\(\times \)  4.5\(\times \) 
II  unsat  29 (19)  12.76  5.17  2.5\(\times \)  
III  unsat  29 (1)  139.79  14.80  9.4\(\times \)  
IV  sat+unsat  54 (43)  24.81  14.25  1.7\(\times \)  1.8\(\times \) 
V  sat+unsat  54 (44)  31.15  16.03  1.9\(\times \)  
Total  191  214.53  54.27  4.0\(\times \)   
4.3 Experimental Results
(1) Air Traffic Controller (ATC) Models. All possible models are generated by running the C preprocessor (cpp) on the annotated composite SMV model representing the CTS. Table 1 summarizes the results for complete verification of the ATC design space: 191 LTL properties for each of 1,620 models.
Compared to individual model checking, wherein every modelproperty pair is checked onebyone, verification of the ATC design space using \(D^{3}\) is 4.0\(\times \) faster. It reduces the 1,620 models in the design space to 1,028 models. \(D^{3}\) takes roughly three hours to find dependencies between LTL properties for all phases. Dependencies established are local to each modelchecking phase and are computed only once per phase. The number of reduced LTL properties checked for each model in a phase vary; we use CheckRP with the Maximum Dependence heuristic (H1). Although the logical dependencies are global for each phase, the property verification results vary for different models. In phases containing unsat properties, speedup achieved by \(D^{3}\) varies between 1.5\(\times \) to 9.4\(\times \); since all properties are true for the model, only \((\varphi _1 : T \rightarrow \varphi _2 : T)\) dependencies in the property table are used. A median of one property is checked per model in phase III. For phases IV and V, \(D^{3}\)’s performance is consistent as shown in Fig. 7.
Interesting Observation. \(D^{3}\) requires a minimum number of models to be faster than individualmodel checking. When the design space is small, individually checking the models is faster than verifying using \(D^{3}\). This is due to the fact that \(D^{3}\) requires an initial setup time. The number of models after which \(D^{3}\) is faster is called the “crossover point”. For the benchmark, the crossover happens after \(\sim \)120 models. As the number of models, and the relationships between them increase, the time speedup due to \(D^{3}\) also increases.
 i.
Single: properties are checked onebyone against the model,
 ii.
CheckRP: properties are checked using interproperty dependencies,
 iii.
CheckRP + Maximum Dependence (H1): unchecked property with the maximum dependent properties as per interproperty dependencies is checked,
 iv.
CheckRP + Property Affinity (H2): properties are pairwise grouped and the unchecked pair with the maximum dependent properties is checked.
Figure 8 summarizes the results. On every call to the model checker, a single or grouped LTL property is checked. CheckRP is successful in reducing the number of checker runs by using interproperty dependencies. The Maximal Dependences (H1) and Property Grouping (H2) heuristics improve the performance of CheckRP, the former more than the latter. The timing results for each algorithm is shown in Table 2.
Timing results (in seconds) for performance of \(D^{3}\)’s interproperty dependence analysis. A property: single or grouped, is verified on each checker run. Overall time indicates the total time to verify all properties for a model.
Model  Single  CheckRP  CheckRP+H1  CheckRP+H2  

Overall time  Checker  Overall time  Checker runs  Overall time  Checker runs  Overall time  Checker runs  
1  17.81  179  2.92  23  1.28  10  2.05  11 
2  64.37  236  9.35  23  3.94  11  5.67  13 
3  54.22  234  7.11  20  3.40  11  4.97  14 
4  53.18  227  9.71  25  3.41  11  5.89  12 
5  61.02  227  6.86  16  4.01  11  5.58  12 
6  68.24  248  8.34  21  3.93  11  5.34  14 
7  58.40  248  7.74  21  3.39  11  5.98  15 
5 Conclusions and Future Work
We present an algorithm, \(D^{3}\), to increase the efficiency of LTL model checking for large design spaces. It is successful in reducing the number of models that need to be verified, and also the properties verified for each model. In contrast to software product line model checking techniques using an offtheshelf checker, \(D^{3}\) returns the modelchecking results for all models, and for all properties. \(D^{3}\) is general and extensible; it can be combined with optimized checking algorithms implemented in offtheshelf model checkers. We demonstrate the practical scalability of \(D^{3}\) on a reallife benchmark models. We calculate a crossover point as a crucial measure of when \(D^{3}\) can be used to speed up checking. \(D^{3}\) is fully automated and requires no special inputlanguage modifications; it can easily be introduced in a verification workflow with minimal effort. Heuristics for predicting the crossover point for other model sets are a promising topic for future work. We plan to examine extending \(D^{3}\) to other logics besides LTL, and its applicability to other types of transition systems, like families of Markov processes. We also plan to investigate further reduction in the search space by extending \(D^{3}\) to reuse intermediate model checking results across several models. In a nutshell, \(D^{3}\) is a frontend preprocessing algorithm, and future work involves tying in an improved model checking backend and utilizing available information to reduce the overall amortized performance. Finally, since checking families of models is becoming commonplace, we plan to develop more industrialsized SMV model sets and make them publicly available as research benchmarks.
6 Supporting Artifact
The artifact supports the following usage scenarios.
 1.
Verify the benchmarks using both individualmodel checking and model checking with \(D^3\), or run the complete experimental analysis to reproduce the results reported in Tables 1 and 2.
 2.
Study and evaluate the benchmarks and source code for \(D^3\), subalgorithms (GenPC and CheckRP), and heuristics (H1 and H2).
 3.
Introduce extensions to \(D^3\) and experiment with new heuristics.
Please refer to the README files in the artifact for further information. Every README inside a directory details the directory structure, usage of contained files with respect to the evaluation, and stepbystep instructions on how to the use the contained scripts to regenerate the experimental analysis.
Footnotes
Notes
Data Availability Statement
The benchmarks evaluated, source code, and datasets generated during our experimental analysis are available in the Springer/ Figshare repository: https://doi.org/10.6084/m9.figshare.5913013.v1. Theorem proofs and extended results are available on the paper’s accompanying website: http://temporallogic.org/research/TACAS18/.
References
 1.Baier, C., Dubslaff, C., Klüppelholz, S., Daum, M., Klein, J., Märcker, S., Wunderlich, S.: Probabilistic model checking and nonstandard multiobjective reasoning. In: Gnesi, S., Rensink, A. (eds.) FASE 2014. LNCS, vol. 8411, pp. 1–16. Springer, Heidelberg (2014). https://doi.org/10.1007/9783642548048_1CrossRefGoogle Scholar
 2.Bauer, C., Lagadec, K., Bès, C., Mongeau, M.: Flight control system architecture optimization for flybywire airliners. J. Guidance, Control Dyn. 30(4), 1023–1029 (2007)CrossRefGoogle Scholar
 3.Bozzano, M., Cimatti, A., Fernandes Pires, A., Jones, D., Kimberly, G., Petri, T., Robinson, R., Tonetta, S.: Formal design and safety analysis of AIR6110 wheel brake system. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015, Part I. LNCS, vol. 9206, pp. 518–535. Springer, Cham (2015). https://doi.org/10.1007/9783319216904_36CrossRefGoogle Scholar
 4.Cabodi, G., Camurati, P., Garcia, L., Murciano, M., Nocco, S., Quer, S.: Speeding up model checking by exploiting explicit and hidden verification constraints. In: DATE (2009)Google Scholar
 5.Cabodi, G., Camurati, P.E., Loiacono, C., Palena, M., Pasini, P., Patti, D., Quer, S.: To split or to group: from divideandconquer to subtask sharing for verifying multiple properties in model checking. Int. J. Softw. Tools Technol. Transfer (2017). https://doi.org/10.1007/s1000901704518
 6.Cabodi, G., Nocco, S.: Optimized model checking of multiple properties. In: DATE (2011)Google Scholar
 7.Cabodi, G., Garcia, L.A., Murciano, M., Nocco, S., Quer, S.: Partitioning interpolantbased verification for effective unbounded model checking. TCAD 29(3), 382–395 (2010)Google Scholar
 8.Cavada, R., et al.: The nuXmv symbolic model checker. In: Biere, A., Bloem, R. (eds.) CAV 2014. LNCS, vol. 8559, pp. 334–342. Springer, Cham (2014). https://doi.org/10.1007/9783319088679_22CrossRefGoogle Scholar
 9.Cimatti, A., Griggio, A., Mover, S., Tonetta, S.: Parameter synthesis with IC3. In: FMCAD (2013)Google Scholar
 10.Cimatti, A., Dorigatti, M., Tonetta, S.: OCRA: A tool for checking the refinement of temporal contracts. In: ASE (2013)Google Scholar
 11.Classen, A., Cordy, M., Heymans, P., Legay, A., Schobbens, P.Y.: Model checking software product lines with SNIP. JSTTT 14(5), 589–612 (2012)Google Scholar
 12.Classen, A., Cordy, M., Schobbens, P.Y., Heymans, P., Legay, A., Raskin, J.F.: Featured transition systems: foundations for verifying variabilityintensive systems and their application to LTL model checking. TSE 39(8), 1069–1089 (2013)Google Scholar
 13.Classen, A., Heymans, P., Schobbens, P.Y., Legay, A.: Symbolic model checking of software product lines. In: ICSE (2011)Google Scholar
 14.Classen, A., Heymans, P., Schobbens, P.Y., Legay, A., Raskin, J.F.: Model checking lots of systems: efficient verification of temporal properties in software product lines. In: ICSE (2010)Google Scholar
 15.Dehnert, C., Junges, S., Jansen, N., Corzilius, F., Volk, M., Bruintjes, H., Katoen, J.P., Ábrahám, E.: PROPhESY: a PRObabilistic ParamEter SYnthesis tool. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015, Part I. LNCS, vol. 9206, pp. 214–231. Springer, Cham (2015). https://doi.org/10.1007/9783319216904_13CrossRefGoogle Scholar
 16.Dehnert, C., Junges, S., Jansen, N., Corzilius, F., Volk, M., Bruintjes, H., Katoen, J.P., Ábrahám, E.: Parameter synthesis for probabilistic systems. In: MBMV (2016)Google Scholar
 17.Dimovski, A.S., AlSibahi, A.S., Brabrand, C., Wąsowski, A.: Familybased model checking without a familybased model checker. In: Fischer, B., Geldenhuys, J. (eds.) SPIN 2015. LNCS, vol. 9232, pp. 282–299. Springer, Cham (2015). https://doi.org/10.1007/9783319234045_18CrossRefGoogle Scholar
 18.Dureja, R., Rozier, K.Y.: FuseIC3: an algorithm for checking large design spaces. In: FMCAD (2017)Google Scholar
 19.Dureja, R., Rozier, K.Y.: More Scalable LTL Model Checking via Discovering DesignSpace Dependencies (Artifact) (2018). https://doi.org/10.6084/m9.figshare.5913013.v1
 20.Emerson, E.A., Kahlon, V.: Reducing model checking of the many to the few. In: McAllester, D. (ed.) CADE 2000. LNCS (LNAI), vol. 1831, pp. 236–254. Springer, Heidelberg (2000). https://doi.org/10.1007/10721959_19CrossRefGoogle Scholar
 21.Etessami, K., Kwiatkowska, M., Vardi, M.Y., Yannakakis, M.: Multiobjective model checking of markov decision processes. In: Grumberg, O., Huth, M. (eds.) TACAS 2007. LNCS, vol. 4424, pp. 50–65. Springer, Heidelberg (2007). https://doi.org/10.1007/9783540712091_6CrossRefzbMATHGoogle Scholar
 22.Forejt, V., Kwiatkowska, M., Norman, G., Parker, D., Qu, H.: Quantitative multiobjective verification for probabilistic systems. In: Abdulla, P.A., Leino, K.R.M. (eds.) TACAS 2011. LNCS, vol. 6605, pp. 112–127. Springer, Heidelberg (2011). https://doi.org/10.1007/9783642198359_11CrossRefzbMATHGoogle Scholar
 23.Gario, M., Cimatti, A., Mattarei, C., Tonetta, S., Rozier, K.Y.: Model checking at scale: automated air traffic control design space exploration. In: Chaudhuri, S., Farzan, A. (eds.) CAV 2016, Part II. LNCS, vol. 9780, pp. 3–22. Springer, Cham (2016). https://doi.org/10.1007/9783319415406_1CrossRefGoogle Scholar
 24.Hahn, E.M., Han, T., Zhang, L.: Synthesis for PCTL in parametric markov decision processes. In: Bobaru, M., Havelund, K., Holzmann, G.J., Joshi, R. (eds.) NFM 2011. LNCS, vol. 6617, pp. 146–161. Springer, Heidelberg (2011). https://doi.org/10.1007/9783642203985_12CrossRefGoogle Scholar
 25.James, P., Moller, F., Nguyen, H.N., Roggenbach, M., Schneider, S., Treharne, H.: On modelling and verifying railway interlockings: tracking train lengths. Sci. Comput. Program. 96(3), 315–336 (2014)CrossRefGoogle Scholar
 26.Khasidashvili, Z., Nadel, A.: Implicative simultaneous satisfiability and applications. In: Eder, K., Lourenço, J., Shehory, O. (eds.) HVC 2011. LNCS, vol. 7261, pp. 66–79. Springer, Heidelberg (2012). https://doi.org/10.1007/9783642341885_9CrossRefGoogle Scholar
 27.Khasidashvili, Z., Nadel, A., Palti, A., Hanna, Z.: Simultaneous SATbased model checking of safety properties. In: Ur, S., Bin, E., Wolfsthal, Y. (eds.) HVC 2005. LNCS, vol. 3875, pp. 56–75. Springer, Heidelberg (2006). https://doi.org/10.1007/11678779_5CrossRefGoogle Scholar
 28.Kwiatkowska, M., Norman, G., Parker, D., Qu, H.: Compositional probabilistic verification through multiobjective model checking. Inf. Comput. 232, 38–65 (2013)MathSciNetCrossRefGoogle Scholar
 29.Mattarei, C., Cimatti, A., Gario, M., Tonetta, S., Rozier, K.Y.: Comparing different functional allocations in automated air traffic control design. In: FMCAD (2015)Google Scholar
 30.Moller, F., Nguyen, H.N., Roggenbach, M., Schneider, S., Treharne, H.: Defining and model checking abstractions of complex railway models using CSP\(\)B. In: Biere, A., Nahir, A., Vos, T. (eds.) HVC 2012. LNCS, vol. 7857, pp. 193–208. Springer, Heidelberg (2013). https://doi.org/10.1007/9783642396113_20CrossRefGoogle Scholar
 31.Quatmann, T., Dehnert, C., Jansen, N., Junges, S., Katoen, J.P.: Parameter synthesis for Markov models: faster than ever. In: Artho, C., Legay, A., Peled, D. (eds.) ATVA 2016. LNCS, vol. 9938, pp. 50–67. Springer, Cham (2016). https://doi.org/10.1007/9783319465203_4CrossRefGoogle Scholar
 32.Rosenmüller, M., Siegmund, N.: Automating the configuration of multi software product lines. VaMoS 10, 123–130 (2010)Google Scholar
 33.Rozier, K.Y., Vardi, M.Y.: LTL satisfiability checking. In: Bošnački, D., Edelkamp, S. (eds.) SPIN 2007. LNCS, vol. 4595, pp. 149–167. Springer, Heidelberg (2007). https://doi.org/10.1007/9783540733706_11CrossRefGoogle Scholar
 34.Schirmeier, H., Spinczyk, O.: Challenges in software product line composition. In: HICSS. IEEE (2009)Google Scholar
Copyright information
<SimplePara><Emphasis Type="Bold">Open Access</Emphasis> This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.</SimplePara> <SimplePara>The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.</SimplePara>