Abstract
Modern system design often requires comparing several models over a large design space. Different models arise out of a need to weigh different design choices, to check core capabilities of versions with varying features, or to analyze a future version against previous ones. Model checking can compare different models; however, applying model checking offtheshelf may not scale due to the large size of the design space for today’s complex systems. We exploit relationships between different models of the same (or related) systems to optimize the modelchecking search. Our algorithm, \(D^{3}\), preprocesses the design space and checks fewer modelchecking instances, e.g., using nuXmv. It automatically prunes the search space by reducing both the number of models to check, and the number of LTL properties that need to be checked for each model in order to provide the complete modelchecking verdict for every individual modelproperty pair. We formalize heuristics that improve the performance of \(D^{3}\). We demonstrate the scalability of \(D^{3}\) by extensive experimental evaluation, e.g., by checking 1,620 reallife models for NASA’s NextGen air traffic control system. Compared to checking each modelproperty pair individually, \(D^{3}\) is up to 9.4\(\times \) faster.
Keywords
 Model Checking
 Large Design Space
 Wheel Brake System (WBS)
 General Configuration Parameters (GenPC)
 Workﬂow Veriﬁcation
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Thanks to NSF CAREER Award CNS1552934 for supporting this work.
Download conference paper PDF
1 Introduction
In the early phases of design, there are frequently many different models of the system under development [2, 23, 29] constituting a design space. We may need to evaluate different design choices, to check core capabilities of system versions with varying featurelevels, or to analyze a future version against previous ones in the product line. The models may differ in their assumptions, implementations, and configurations. We can use model checking to aid system development via a thorough comparison of the set of system models against a set of properties representing requirements. Model checking, in combination with related techniques like faulttree analysis, can provide an effective comparative analysis [23, 29]. The classical approach checks each model onebyone, as a set of independent modelchecking runs. For large and complex design spaces, performance can be inefficient or even fail to scale to handle the combinatorial size of the design space. Nevertheless, the classical approach remains the most widely used method in industry [3, 23, 25, 29, 30]. Algorithms for familybased model checking [11, 13] mitigate this problem but their efficiency and applicability still depends on the use of custom model checkers to deal with model families.
We assume that each model in the design space can be parameterized over a finite set of parametric inputs that enable/disable individual assumptions, implementations, or behaviors. It might be the case that for any pair of models the assumptions are dependent, their implementations contradict each other, or they have the same behavior. Since the different models of the same system are related, it is possible to exploit the known relationships between them, if they exist, to optimize the model checking search. These relationships can exist in two ways: relationships between the models, and relationships between the properties checked for each model.
We present an algorithm that automatically prunes and dynamically orders the modelchecking search space by exploiting intermodel relationships. The algorithm, Discover DesignSpace Dependencies (\(D^{3}\)), reduces both the number of models to check, and the number of LTL properties that need to be checked for each model. Rather than using a custom model checker, \(D^{3}\) works with any offtheshelf checker. This allows practitioners to use stateoftheart, optimized modelchecking algorithms, and to choose their preferred model checker, which enables adoption of our method by practitioners who already use model checking with minimum change in their verification workflow. We reason about a set of system models by introducing the notion of a Combinatorial Transition System (CTS). Each individual model, or instance, can be derived from the CTS by configuring it with a set of parameters. Each transition in the CTS is enabled/disabled by the parameters. We model check each instance of the CTS against sets of properties. We assume the properties are in Linear Temporal Logic (LTL) and are independent of the choice of parameters, though not all properties may apply to all instances. \(D^{3}\) preprocesses the CTS to find relationships between parameters and minimizes the number of instances that need to be checked to produce results for the whole set. It uses LTL satisfiability checking [33] to determine the dependencies between pairs of LTL properties, then reduces the number of properties that are checked for each instance. \(D^{3}\) returns results for every modelproperty pair in the design space, aiming to compose these results from a reduced series of modelchecking runs compared to the classical approach of checking every modelproperty pair. We demonstrate the industrial scalability of \(D^{3}\) using a set of 1,620 reallife, publiclyavailable SMVlanguage benchmark models with LTL specifications; these model NASA’s NextGen air traffic control system [8, 23, 29]. We also evaluate the propertydependence analysis separately on reallife models of Boeing AIR 6110 Wheel Braking System [3] to evaluate \(D^{3}\) in multiproperty verification workflows.
Related Work. One striking contrast between \(D^{3}\) and related work is that \(D^{3}\) is a preprocessing algorithm, does not require custom modeling, and works with any offtheshelf LTL model checker. Parameter synthesis [9] can generate the many models in a design space that can be analyzed by \(D^{3}\); however existing parameter synthesis techniques require custom modeling of a system. We take the easier path of reasoning over an alreadyrestricted set of models of interest to system designers. \(D^{3}\) efficiently compares any set of models rather than finding all models that meet the requirements. Several parameter synthesis approaches designed for parametric Markov models [15, 16, 24, 31] use PRISM and compute the region of parameters for which the model satisfies a given probabilistic property (PCTL or PLTL); \(D^{3}\) is an LTLbased algorithm. Parameter synthesis of a parametric Markov model with nonprobabilistic transitions can generate the many models that \(D^{3}\) can analyze. In multiobjective model checking [1, 21, 22, 28], given a Markov decision process and a set of LTL properties, the algorithms find a controller strategy such that the Markov process satisfies all properties with some set probability. Differently from multiobjective model checking, which generates “tradeoff” Pareto curves, \(D^{3}\) gives a boolean result. The parameterized model checking problem (PCMP) [20] deals with infinite families of homogeneous processes in a system; in our case, the models are finite and heterogeneous. Specialized modelset checking algorithms [18] can check the reduced set of \(D^{3}\) processed models.
In multiproperty model checking, multiple properties are checked on the same system. Existing approaches simplify the task by algorithm modifications [4, 7], SATsolver modifications [26, 27], and property grouping [5, 6]. The interproperty dependence analysis of \(D^{3}\) can be used in multiproperty checking. We compare \(D^{3}\) against the affinity [6] based approach to property grouping.
Product line verification techniques, e.g., with Software Product Lines (SPL), also verify parametric models describing large design spaces. We borrow the notion of an instance, from SPL literature [32, 34]. An extension to NuSMV in [13] performs symbolic model checking of featureoriented CTL. The symbolic analysis is extended to the explicit case and support for featureoriented LTL in [11, 12]. The work most closely related to ours is [17] where product line verification is done without a familybased model checker. \(D^{3}\) outputs modelchecking results for every modelproperty pair in the design space (e.g. all parameter configurations) without dependence on any feature whereas in SPL verification using an offtheshelf checker, if a property fails then it isn’t possible to know which models do satisfy the property [14, 17].
Contributions. The preprocessing algorithm presented is an important stepping stone to smarter algorithms for checking large design spaces. Our contributions are summarized as follows:

1.
A fully automated, general, and scalable algorithm for checking design spaces; it can be applied to LTL model checking problems without major modifications to the system designers’ verification workflow.

2.
Modification to the general modelchecking procedure of sequentially checking properties against a model to a dynamic procedure; the next property to check is chosen to maximize the number of yettobechecked properties for which the result can be determined from interproperty dependencies.

3.
Comparison of our novel interproperty dependence analysis to existing work in multiproperty verification workflows [6].

4.
Extensive experimental analysis using reallife benchmarks; all reproducibility artifacts and source code are publicly available.
2 Preliminaries
Definition 1
A labeled transition system (LTS) is a system model of the form \(M = ({ \Sigma }, S, s_0, L, \delta )\) where,

1.
\({ \Sigma }\) is a finite alphabet, or set of atomic propositions,

2.
S is a finite set of states,

3.
\(s_0 \in S\) is an initial state,

4.
\(L : S \rightarrow 2^{ \Sigma }\) is a labeling function that maps each state to the set of atomic propositions that hold in it, and

5.
\(\delta : S \rightarrow S\) is the transition function.
A computation path, or run of LTS M is a sequence of states \(\pi = s_0 {\rightarrow } s_1 {\rightarrow } \ldots {\rightarrow } s_n\) over the word \(w = L(s_0), L(s_1), \ldots , L(s_n)\) such that \(s_i \in S\) for \(0 \le i \le n\), and \((s_i, s_{i+1}) \in \delta \) for \(0 \le i < n\). Given a LTL property \(\varphi \) and a LTS M, M models \(\varphi \), denoted \(M \models \varphi \), iff \(\varphi \) holds in all possible computation paths of M.
Definition 2
A parameter \(P_i\) is a variable with the following properties.

1.
The domain of \(P_i\), denoted \(\llbracket P_i \rrbracket \), is a finite set of possible assignments to \(P_i\).

2.
Parameter \(P_i\) is set by assigning a single value from \(\llbracket P_i \rrbracket \), i.e. \(P_i = d_{P_i} \in \llbracket P_i \rrbracket \). A nonassigned parameter is considered unset.

3.
Parameter setting is static, i.e., it does not change during a run of the system.
Let P be a finite set of parameters. P denotes the number of parameters. For each \(P_i \in P\), \(P_i\) denotes the size of the domain of \(P_i\). Let Form(P) denote the set of all Boolean formulas over P generated using the BNF grammar \({\varphi }\, {:}{:=}~\top \mid P_i {\text { { == }}} \text {D} \) and \( \text {D}\, {:}{:=}~P_{i_1} \mid P_{i_2} \mid \ldots \mid P_{i_n};\) for each \(P_i \in P\), \(n = P_i\), and \(\llbracket P_i \rrbracket ~{=}~\{P_{i_1}, P_{i_2}, \ldots , P_{i_n}\}\). Therefore, Form(P) contains \(\top \) and equality constraints over parameters in P.
Definition 3
A combinatorial transition system (CTS) is a combinatorial system model \(M_P = ({ \Sigma }, S, s_0, L, \) \(\delta , P, L_P)\), such that \(({ \Sigma }, S, s_0, L, \delta )\) is a LTS and

1.
P is a finite set of parameters to the system, and

2.
\(L_P : \delta \rightarrow Form(P)\) is function labeling transitions with a guard condition.
We limit the guard condition over a transition to \(\top \) or an equality constraint over a single parameter for simpler expressiveness and formalization. However, there can be multiple transitions between any two states with different guards. A transition is enabled if its guard condition evaluates to true, otherwise, it is disabled. A label of \(\top \) implies the transition is always enabled. A possible run of a CTS is a sequence of states \(\pi _P = s_0 {\mathrel {\mathop {\rightarrow }\limits ^{\nu _1}}} s_1 {\mathrel {\mathop {\rightarrow }\limits ^{\nu _2}}} \ldots {\mathrel {\mathop {\rightarrow }\limits ^{\nu _n}}} s_n\) over the word \(w = L(s_0), L(s_1), \ldots , L(s_n)\) such that \(s_i \in S\) for \(0 \le i \le n\), \(\nu _i \in Form(P)\) for \(0 < i \le n\), and \((s_i, s_{i+1}) \in \delta \) and \((s_i, s_{i+1}, \nu _{i+1}) \in L_P\) for \(0 \le i < n\), i.e., there is transition from \(s_i\) to \(s_{i+1}\) with guard condition \(\nu _{i+1}\). A prefix \(\alpha \) of a possible run \(\pi _P = \alpha {\mathrel {\mathop {\rightarrow }\limits ^{\nu _i}}} \ldots {\mathrel {\mathop {\rightarrow }\limits ^{\nu _n}}}s_n\) is also a possible run.
Example 1. A Boolean parameter has domain \(\{true,\) \(false\}\). Figure 1 shows a CTS with Boolean parameters \(P = \{P_1, P_2, P_3\}\). For brevity, guard condition \(P_i == true\) is written as \(P_i\), while \(P_i == false\) is written as \(\lnot P_i\). A transition with label \(P_1\) is enabled if \(P_1\) is set to true. Similarly, a label of \(\lnot P_3\) implies the transition is enabled if \(P_3\) is set to false.
Example 1
A Boolean parameter has domain \(\{true,\) \(false\}\). Figure 1 shows a CTS with Boolean parameters \(P = \{P_1, P_2, P_3\}\). For brevity, guard condition \(P_i == true\) is written as \(P_i\), while \(P_i == false\) is written as \(\lnot P_i\). A transition with label \(P_1\) is enabled if \(P_1\) is set to true. Similarly, a label of \(\lnot P_3\) implies the transition is enabled if \(P_3\) is set to false.
Definition 4
A parameter configuration c for a set of parameters P is a ktuple \((d_{P_1},\) \(d_{P_2}, \ldots , d_{P_k})\), for \(k = P\), that sets each parameter in P, i.e., for every \(1 \le i \le k\), \(P_i = d_{P_i}\) and \(d_{P_i} \in \llbracket P_i \rrbracket \) is a setting. The set of all possible configurations \(\mathbb {C}\) over P is equal to \(P_1 \times P_2 \times \ldots \times P_k\) where \(\times \) denotes the cross product. The setting for \(P_i\) in configuration c is denoted by \(c(P_i)\).
A configured run of a CTS \(M_P\) over a configuration c, or crun, is a sequence of states \(\pi _{P(c)} = s_0 \xrightarrow {\nu _1} s_1 \xrightarrow {\nu _2} \ldots \xrightarrow {\nu _n} s_n\) such that \(\pi _{P(c)}\) is a possible run, and \(c \vdash \nu _i\) for \(0 < i \le n\), where \(\vdash \) denotes propositional logic satisfaction of the guard condition \(\nu _i\) under parameter configuration c. Given a CTS \(M_P\) and a parameter configuration c, a state t is reachable iff there exists a crun such that \(s_n = t\), denoted \(s_0 \xrightarrow [c]{*} t\), i.e., t can be reached in zero or more transitions. A transition with guard \(\nu \) is reachable iff \((s_j, s_{j+1}, \nu ) \in L_P\), \((s_j, s_{j+1}) \in \delta \), and \(s_0 \xrightarrow [c]{*} s_j\).
Definition 5
An instance of a CTS \(M_P = (\Sigma , S, s_0, L, \delta , P, L_P)\) for parameter configuration c is a LTS \(M_{P(c)} = (\Sigma , S, s_0, L, \delta ')\) where \(\delta ' = \{t \in \delta \mid c \vdash L_P(t)\}\).
Given a LTL property \(\varphi \) and a CTS \(M_P = (\Sigma , S, s_0, L, \delta , P, L_P)\), the model checking problem for \(M_P\) is to find all parameter configurations \(c \in \mathbb {C}\) over P such that \(\varphi \) holds in all cruns of \(M_P\), or all computation paths of LTS \(M_{P(c)}\).
Definition 6
Given a CTS \(M_P\) with parameters \(P_i, P_j\), and a parameter configuration c, \(P_j\) is dependent on \(P_i\), denoted \(P_j \leadsto _c P_i\), iff

In all possible runs with a transition guard over \(P_j\), a transition with guard over \(P_i\) appears before a transition with guard over \(P_j\), and

In all configured runs, the setting for \(P_i\) in c makes transitions with guard conditions over \(P_j\) unreachable.
Example 2 In Fig. 1, if \(P_1\) is set to false, execution never reaches the transition labeled \(\lnot P_3\). Therefore, if configuration \(c = (false, true, true)\) then \(P_3 \leadsto _c P_1\).
Definition 7
A universal model U is a LTS that generates all possible computations paths over its atomic propositions.
Theorem 1 (LTL Satisfiability)
[33] Given a LTL property \(\varphi \) and a universal model U, \(\varphi \) is satisfiable if and only if .
This theorem reduces LTL satisfiability checking to LTL model checking. Therefore, \(\varphi \) is satisfiable when the model checker finds a counterexample.^{Footnote 1}
Modeling a Combinatorial Transition System. Efficient modeling of a CTS requires language constructs to deal with parameters. Since our goal is to use an existing model checker, language extensions are outside the scope of this work. An alternative way to add parameters to any system description is by utilizing the C preprocessor (cpp). Given a set of parameters P, and a combinatorial model \(M_P\), each run of the preprocessor with a configuration \(c \in \mathbb {C}\) generates an instance \(M_{P(c)}\). Figure 2 demonstrates generating a CTS from two related SMV models. Model 1 and Model 2 differ in the initial configuration of the parameter. The corresponding CTS replaces the parameter initiation with the PARAMETER_CONF preprocessor directive. The cpp is run on the CTS model with #define PARAMETER_CONF 0, and #define PARAMETER_CONF 1 to generate the two models.
3 Discovering DesignSpace Dependencies
In this section we describe \(D^{3}\). Our approach speeds up model checking of combinatorial transitions systems by preprocessing of the input instances; it therefore increases efficiency of both BDDbased and SATbased model checkers. The problem reduction is along two dimensions: number of instances, and number of properties.
3.1 Reduction Along the Number of Instances
Given a set of parameters P, a combinatorial transition system \(M_P\), and a property \(\varphi \), \(M_P\) is model checked by sending, for all parameter configuration \(c \in \mathbb {C}\), instance \(M_{P(c)}\) to the LTS model checker, along with the property \(\varphi \). The output is aggregated for \(\mathbb {C}\) runs of the model checker, and all parameter configurations c, such that \(M_{P(c)} \models \varphi \) are returned. In principle, parameters can be encoded as state variables, and the parametric model can be posed as one big modelchecking obligation, however there are caveats.

1.
State space explosion before any useful results are obtained.

2.
Counterexample generated from one run of the model checker gives a single undesirable configuration.
Our goal is to make the classical approach of individualmodel checking more scalable as the design space grows by intelligently integrating possible dependencies between parameter configurations.
Lemma 1
Given a CTS \(M_P = (\Sigma , S, s_0, L, \delta , P, L_P)\) with parameters \(A, B \in P\), if \(B \leadsto _c A\) for some parameter configuration c, then there does not exist any possible run of \(M_P\) with prefix \(\alpha = s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _B}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\rightarrow }\limits ^{\nu _A}}} s_l\), where \(\nu _A\) and \(\nu _B\) are guards over A and B, resp., and \(s_i, s_j, s_k, s_l \in S\), i.e., a transition with guard over B does not appear before a transition with guard over A.
As a corollary to Lemma 1, there also do not exist possible runs with transition guards only over B (and no other \(P_i \in P\)). Therefore, given a CTS \(M_P\) with states \(s_i, s_j, s_k, s_l \in S\) and parameters \(A, B \in P\), if \(B \leadsto _c A\) for some parameter configuration c, then all possible runs of \(M_P\) have one of the following prefixes:

1.
\(s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _A}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _B}}} s_{l}\) (guard over A before guard over B)

2.
\(s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _A}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _A}}} s_{l}\) (guards only over A)

3.
\(s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{*}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\longrightarrow }\limits ^{*}}} s_{l}\) (guards neither over A nor B)
Similarly, if \(A \leadsto _c B\) for some parameter configuration c, then all possible runs of \(M_P\) have one of the following prefixes:

1.
\(s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _B}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _A}}} s_{l}\) (guard over B before guard over A)

2.
\(s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _B}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\longrightarrow }\limits ^{\nu _B}}} s_{l}\) (guards only over B)

3.
\(s_0 {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_i {\mathrel {\mathop {\longrightarrow }\limits ^{*}}} s_{j} {\mathrel {\mathop {\rightarrow }\limits ^{*}}} s_k {\mathrel {\mathop {\longrightarrow }\limits ^{*}}} s_{l}\) (guards neither over A nor B)
Therefore, when A and B are not dependent, there is no possible run with transition guards over both A and B. Note that for a CTS \(M_P\) with \(A, B \in P\), if A and B are dependent, then either \(A \leadsto _c B\) or \(B \leadsto _c A\) but not both for any configuration c. We only show formalization for \(B \leadsto _c A\); \(A \leadsto _c B\) follows directly.
Theorem 2 (Redundant Instance)
Given a CTS \(M_P = (\Sigma , S, s_0, L, \delta , P, L_P)\) with parameters \(A, B \in P\) such that \(B \leadsto _c A\) for some configuration c, and a LTL property \(\varphi \), there exist configurations \(c_1, c_2, \ldots c_k \in \mathbb {C}\) for \(k = B\) such that

\(c_i(A) = c(A)\) for \(0 < i \le k\), and

\(c_i(B) = d_{B_i} \in \llbracket B \rrbracket \) for \(0 < i \le k\) and \(\llbracket B \rrbracket = \{d_{B_1}, d_{B_2}, \ldots , d_{B_k}\}\)
For such configurations \(M_{P(c_1)} \models \varphi \equiv M_{P(c_2)} \models \varphi \equiv \ldots \equiv M_{P(c_k)} \models \varphi \).
Theorem 2 allows us to reduce the number of model checker runs by exploiting redundancy between instances. The question that needs to be answered is how to find dependent parameters? A partial parameter configuration, \(\hat{c}\), is a parameter configuration in which not all parameters have been set. Given a CTS \(M_P = (\Sigma , S, s_0, L, \delta , P, L_P)\), for a transition \(t \in \delta \), such that \(L_P(t) = \nu \), the guard \(\nu \) is

defined, if its corresponding parameter is set in \(\hat{c}\), and

undefined, otherwise.
A defined guard evaluates to true when \(\hat{c} \vdash L_P(t)\), or false when . Algorithm FindUP (Find Unset Parameters) in Fig. 3(a) solves the dual problem of finding independent parameters. It takes as input a CTS \(M_P\) and a partial parameter configuration \(\hat{c}\), and returns unset parameters for which guard conditions are undefined and their corresponding transitions are reachable. It traverses (depthfirst) the CTS starting from a node for the initial state \(s_0\). During traversal, an edge (transition) \(t = (s_i, s_j)\) connects two nodes (states) \(s_i, s_j \in S\) if \(t \in \delta \) and \(\hat{c} \vdash L_P(t)\). The edge is disconnected if \(t \not \in \delta \) or . Since \(M_P\) is defined relationally in the annotated SMV language with preprocessor directives (Sect. 2), in the worst case, FindUP takes polynomial time in the number of symbolic states and transitions. From an implementation point of view, FindUP invokes the cpp for parameter settings in \(\hat{c}\) on the input model, and parses the output for unset parameters.
Lemma 2
FindUP returns unset parameters \(P_i \in P\) for all reachable transitions \(t \in \delta \) such that guard \(L_P(t)\) is a guard over \(P_i\), and is undefined.
Algorithm GenPC (Generate Parameter Configurations) in Fig. 3(b) uses FindUP as a subroutine to recursively find parameter configurations that need to be checked. It takes as input a CTS \(M_P\), queue of unset parameters \(P_u\), and a partial parameter configuration \(\hat{c}\). Initially, \(\hat{c}\) contains no set parameters and \(P_u = \) FindUP(\(M_P\), \(\hat{c}\)). Upon termination of GenPC, \(\hat{\mathbb {C}}\) contains the set of partial parameter configurations that need to be checked. On every iteration, GenPC picks a parameter p from \(P_u\), assigns it a value from its domain \(\llbracket p \rrbracket \) in \(\hat{c}\), and uses FindUP to find unset parameters in CTS \(M_{P}\). If the returned unset parameter queue is empty, \(\hat{c}\) added to \(\hat{\mathbb {C}}\). Otherwise, \(\textsc {GenPC}\) is called again with the new unset parameter queue.
Theorem 3 (GenPC is sound)
Given a CTS \(M_P\) with parameters \(A,B \in P\), if there exists a partial configuration \(\hat{c} \in \hat{\mathbb {C}}\) with \(\hat{c}(A) = d_{A_n} \in \llbracket A \rrbracket \) and B unset, then there exist configurations \(c_1, c_2, \ldots c_k \in \mathbb {C}\) for \(k = B\) such that

\(c_i(A) = \hat{c}(A)\) for \(0 < i \le k\), and

\(c_i(B) = d_{B_i} \in \llbracket B \rrbracket \) for \(0 < i \le k\) and \(\llbracket B \rrbracket = \{d_{B_1}, d_{B_2}, \ldots , d_{B_k}\}\)
for which \(B \leadsto _{c_i} A\).
Theorem 4 (GenPC is complete)
Given a CTS \(M_P\) with parameters \(A,B \in P\), if there exist configurations \(c_1, c_2, \ldots c_k \in \mathbb {C}\) for \(k = B\) such that

\(c_i(A) = d_{A_n}\) for \(0 < i \le k\) and \(d_{A_n} \in \llbracket A \rrbracket \), and

\(c_i(B) = d_{B_i} \in \llbracket B \rrbracket \) for \(0 < i \le k\) and \(\llbracket B \rrbracket = \{d_{B_1}, d_{B_2}, \ldots , d_{B_k}\}\)
for which \(B \leadsto _{c_i} A\), then there exists a partial configuration \(\hat{c} \in \hat{\mathbb {C}}\) with \(\hat{c}(A) = d_{A_n}\) and B unset.
GenPC returns partial configurations \(\hat{c} \in \hat{\mathbb {C}}\) over parameters. A partial configuration \(\hat{c}\) is converted to a parameter configuration c by setting the unset parameters in \(\hat{c}\) to an arbitrary value from their domain. Note that this operation is safe since the arbitrarily set parameters are not reachable in the instance \(M_{P(c)}\). As a result of this operation, \(\hat{\mathbb {C}}\) contains configurations c that have all parameters set to a value from their domain.
Theorem 5 (Minimality)
The minimal set of parameter configurations is \(\hat{\mathbb {C}}\).
3.2 Reduction Along the Number of Properties
In model checking, properties describe the intended behavior of the system. Usually, properties are iteratively refined to express the designer’s intentions. For small systems, it can be manually determined if two properties are dependent on one another. However, practically determining property dependence for large and complex systems requires automation. Given a set of properties \(\mathcal {P}\), and LTS M, an offtheshelf model checker is called \(N = \mathcal {P}\) times.
In order to check all properties in \(\mathcal {P}\), a straightforward possibility is to generate a grouped property \(\varphi _g\) given by the conjunction of all properties \(\varphi _i \in \mathcal {P}\), i.e., \(\varphi _g = \bigwedge _i \varphi _i\). However, the straightforward approach may not scale [6] due to

1.
Statespace explosion due to orthogonal coneofinfluences of properties.

2.
Need for additional analysis of individual properties onebyone in order to discriminate failed ones and generate individual counterexamples.

3.
Computational cost of verifying grouped properties in one run can be significantly higher than verifying individual properties in a series of runs.
Our goal is to minimize the number of properties checked by intelligently using dependencies between LTL properties. For two LTL properties \(\varphi _1\) and \(\varphi _2\) dependence can be characterized in four ways: \((\varphi _1 \rightarrow \varphi _2)\), \((\varphi _1 \rightarrow \lnot \varphi _2)\), \((\lnot \varphi _1 \rightarrow \varphi _2)\), and \((\lnot \varphi _1 \rightarrow \lnot \varphi _2)\). Theorem 6 allows us to find dependencies automatically.
Theorem 6 (Property Dependence)
For two LTL properties \(\varphi _1\) and \(\varphi _2\) dependence can be established by model checking with universal model U.
The dependencies learned as a result of Theorem 6 have implications on the verification workflow. For instance, if \(\varphi _1 \rightarrow \varphi _2\) is valid, then for a model M, if \(M \models \varphi _1\) then \(M \models \varphi _2\). Of particular interest are \((\varphi _1 \rightarrow \varphi _2)\), \((\lnot \varphi _1 \rightarrow \varphi _2)\), and \((\lnot \varphi _1 \rightarrow \lnot \varphi _2)\) because they allow use of previous counterexamples (for \((\varphi _1 \rightarrow \lnot \varphi _2)\), even if \(\varphi _1\) is true, there is no counterexample to prove that \(\varphi _2\) is false).
The pairwise property dependencies are stored in a property table as shown in Fig. 4(a). Each row in the table is a (key, value) pair. For LTL properties \(\varphi _1\), \(\varphi _2\), and \(\varphi _3\) in \(\mathcal {P}\), if \((\varphi _1 \rightarrow \varphi _2)\) is valid, then the table contains a row \((\varphi _1:T, \varphi _2:T)\) implying that if \(\varphi _1\) holds for a model M then \(\varphi _2\) also holds. Similarly, for \((\lnot \varphi _3 \rightarrow \lnot \varphi _2)\) the table entry \((\varphi _3:F, \varphi _2:F)\) implies that if \(\varphi _3\) doesn’t hold for M then \(\varphi _2\) doesn’t hold. Algorithm CheckRP (Check Reduced Properties) in Fig. 5 takes as input a LTS M, a set of LTL properties \(\mathcal {P}\), and a property table T over \(\mathcal {P}\). CheckRP selects an unchecked LTL property \(\varphi \), checks whether \(\varphi \) holds in M, and stores the outcome. Based on the outcome, it uses the property table to determine checking results for all dependent properties and stores them. For example, in Fig. 4(b), if , then , , and \(M \models \varphi _6\). The LTL property to check is selected using two heuristics.
H1: Maximum Dependence. The tabular layout of property dependencies is used to calculate the number of dependencies for each property. The unchecked LTL property with the most righthand side entries is selected. If \(\mathcal {U} \subseteq \mathcal {P}\) are unchecked properties in table D, the next LTL property to check is then
where \(count(x) = D[x:T] \cup D[x:F]\) returns the number of dependencies for a LTL property in table D, and max(S) returns the largest element from S.
H2: Property Grouping. Most modelchecking techniques are computationally sensitive to the coneofinfluence (COI) size. Grouping properties based on overlap between their COI can speed up checking. Property affinity [5, 6] based on Jaccard Index can compare the similarity between COI. For two LTL properties \(\varphi _i\) and \(\varphi _j\), let \(V_i\) and \(V_j\), respectively, denote the variables in their COI with respect to a model M. The affinity \(\alpha _{ij}\) for \(\varphi _i\) and \(\varphi _j\) is given by
If \(\alpha _{ij}\) is larger than a given threshold, then properties \(\varphi _i\) and \(\varphi _j\) are grouped together. The model M is then checked against \(\varphi _i \wedge \varphi _j\). If verification fails, then \(\varphi _i\) and \(\varphi _j\) are checked individually against model M.
4 Experimental Analysis
Our revised model checking procedure \(D^{3}\) is shown in Fig. 6. \(D^{3}\) takes as input a CTS \(M_P\) and a set of LTL properties \(\mathcal {P}\). It uses GenPC to find the parameter configurations that need to be checked. It then generates a property table to store dependencies between LTL properties. Lastly, CheckRP checks each instance against properties in \(\mathcal {P}\). Results are collated for every modelproperty pair.
4.1 Benchmarks
We evaluated \(D^{3}\) on two benchmarks derived from realworld case studies.

(1)
Air Traffic Controller (ATC) Models: are a set of 1,620 realworld models representing different possible designs for NASA’s NextGen air traffic control (ATC) system. In previous work, this set of models were generated from a contractbased, parameterized nuXmv model; individualmodel checking enabled their comparative analysis with respect to a set of requirements for the system [23]. In the formulation of [23], the checking problem for each model is split in to five phases.^{Footnote 2} In each phase, all 1,620 models are checked. For our analysis and to gain better understanding of the experimental results, we categories the phases based on the property verification results (unsat if property holds for the model, and sat if it does not). Each of the 1,620 models can be seen as instances of a CTS with seven parameters. Each of the 1620 instances is checked against a total of 191 LTL properties. The original nuXmv code additionally uses OCRA [10] for compositional modeling, though we do not rely on its features when using the generated modelset.

(2)
Boeing Wheel Braking System (WBS) Models: are a set of seven realworld nuXmv models representing possible designs for the Boeing AIR 6110 wheel braking system [3]. Each model in the set is checked against \(\sim \)200 LTL properties. However, the seven models are not generated from a CTS. We evaluate \(D^{3}\) against this benchmark to evaluate performance on multiproperty verification workflows, and compare with existing work on property grouping [6].
4.2 Experiment Setup
\(D^{3}\) is implemented as a preprocessing script in \(\sim \)2,000 lines of Python code. We model check using nuXmv 1.1.1 with the IC3based backend. All experiments were performed on Iowa State University’s Condo Cluster comprising of nodes having two 2.6Ghz 8core Intel E52640 processors, 128 GB memory, and running Enterprise Linux 7.3. Each model checking run has dedicated access to a node, which guarantees that no resource conflict with other jobs will occur.
4.3 Experimental Results
(1) Air Traffic Controller (ATC) Models. All possible models are generated by running the C preprocessor (cpp) on the annotated composite SMV model representing the CTS. Table 1 summarizes the results for complete verification of the ATC design space: 191 LTL properties for each of 1,620 models.
Compared to individual model checking, wherein every modelproperty pair is checked onebyone, verification of the ATC design space using \(D^{3}\) is 4.0\(\times \) faster. It reduces the 1,620 models in the design space to 1,028 models. \(D^{3}\) takes roughly three hours to find dependencies between LTL properties for all phases. Dependencies established are local to each modelchecking phase and are computed only once per phase. The number of reduced LTL properties checked for each model in a phase vary; we use CheckRP with the Maximum Dependence heuristic (H1). Although the logical dependencies are global for each phase, the property verification results vary for different models. In phases containing unsat properties, speedup achieved by \(D^{3}\) varies between 1.5\(\times \) to 9.4\(\times \); since all properties are true for the model, only \((\varphi _1 : T \rightarrow \varphi _2 : T)\) dependencies in the property table are used. A median of one property is checked per model in phase III. For phases IV and V, \(D^{3}\)’s performance is consistent as shown in Fig. 7.
Interesting Observation. \(D^{3}\) requires a minimum number of models to be faster than individualmodel checking. When the design space is small, individually checking the models is faster than verifying using \(D^{3}\). This is due to the fact that \(D^{3}\) requires an initial setup time. The number of models after which \(D^{3}\) is faster is called the “crossover point”. For the benchmark, the crossover happens after \(\sim \)120 models. As the number of models, and the relationships between them increase, the time speedup due to \(D^{3}\) also increases.
Overall. From the initial problem of checking 1,620 models against 191 LTL properties, \(D^{3}\) checks 1,028 models with a median of 129 properties per model (45% reduction of design space). Once \(D^{3}\) terminates, the modelchecking results for each model are compared using the data analysis technique of [23].
(2) Boeing Wheel Braking System (WBS) Models. LTL Properties for each of the seven models are checked using four algorithms:

i.
Single: properties are checked onebyone against the model,

ii.
CheckRP: properties are checked using interproperty dependencies,

iii.
CheckRP + Maximum Dependence (H1): unchecked property with the maximum dependent properties as per interproperty dependencies is checked,

iv.
CheckRP + Property Affinity (H2): properties are pairwise grouped and the unchecked pair with the maximum dependent properties is checked.
Figure 8 summarizes the results. On every call to the model checker, a single or grouped LTL property is checked. CheckRP is successful in reducing the number of checker runs by using interproperty dependencies. The Maximal Dependences (H1) and Property Grouping (H2) heuristics improve the performance of CheckRP, the former more than the latter. The timing results for each algorithm is shown in Table 2.
Analysis. For H2, we limited our experiments to pairwise groupings, however, larger groupings may be possible (tradeoff required between property interdependencies and groupings). It took \(\sim \)50 min to establish dependence between properties for a model, which is much higher than checking them onebyone without using CheckRP. This brings us back to the question of estimating a crossover point. However, as the number of models increase for the same set of properties, CheckRP will start reaping benefits. Nevertheless, CheckRP is suited for multiproperty verification in large design spaces.
5 Conclusions and Future Work
We present an algorithm, \(D^{3}\), to increase the efficiency of LTL model checking for large design spaces. It is successful in reducing the number of models that need to be verified, and also the properties verified for each model. In contrast to software product line model checking techniques using an offtheshelf checker, \(D^{3}\) returns the modelchecking results for all models, and for all properties. \(D^{3}\) is general and extensible; it can be combined with optimized checking algorithms implemented in offtheshelf model checkers. We demonstrate the practical scalability of \(D^{3}\) on a reallife benchmark models. We calculate a crossover point as a crucial measure of when \(D^{3}\) can be used to speed up checking. \(D^{3}\) is fully automated and requires no special inputlanguage modifications; it can easily be introduced in a verification workflow with minimal effort. Heuristics for predicting the crossover point for other model sets are a promising topic for future work. We plan to examine extending \(D^{3}\) to other logics besides LTL, and its applicability to other types of transition systems, like families of Markov processes. We also plan to investigate further reduction in the search space by extending \(D^{3}\) to reuse intermediate model checking results across several models. In a nutshell, \(D^{3}\) is a frontend preprocessing algorithm, and future work involves tying in an improved model checking backend and utilizing available information to reduce the overall amortized performance. Finally, since checking families of models is becoming commonplace, we plan to develop more industrialsized SMV model sets and make them publicly available as research benchmarks.
6 Supporting Artifact
The artifact for reproducibility of our experiments [19] is publicly available under the MIT License, and supports all reported results of Sect. 4. It includes

1.
Benchmarks: NASA’s NextGen Air Traffic Control System [23] and Boeing’s Wheel Braking System [3] (Sect. 4.1).

2.
Scripts: Python scripts to run \(D^3\) on the two benchmarks (Fig. 6).

3.
Datasets: Readytouse datasets generated during our analysis (Sect. 4.3)
The artifact supports the following usage scenarios.

1.
Verify the benchmarks using both individualmodel checking and model checking with \(D^3\), or run the complete experimental analysis to reproduce the results reported in Tables 1 and 2.

2.
Study and evaluate the benchmarks and source code for \(D^3\), subalgorithms (GenPC and CheckRP), and heuristics (H1 and H2).

3.
Introduce extensions to \(D^3\) and experiment with new heuristics.
Please refer to the README files in the artifact for further information. Every README inside a directory details the directory structure, usage of contained files with respect to the evaluation, and stepbystep instructions on how to the use the contained scripts to regenerate the experimental analysis.
Notes
 1.
This is why we do not consider CTL; CTL satisfiability is EXPTIMEcomplete and cannot be accomplished via linear time CTL model checking.
 2.
For a detailed explanation we refer the reader to [23].
References
Baier, C., Dubslaff, C., Klüppelholz, S., Daum, M., Klein, J., Märcker, S., Wunderlich, S.: Probabilistic model checking and nonstandard multiobjective reasoning. In: Gnesi, S., Rensink, A. (eds.) FASE 2014. LNCS, vol. 8411, pp. 1–16. Springer, Heidelberg (2014). https://doi.org/10.1007/9783642548048_1
Bauer, C., Lagadec, K., Bès, C., Mongeau, M.: Flight control system architecture optimization for flybywire airliners. J. Guidance, Control Dyn. 30(4), 1023–1029 (2007)
Bozzano, M., Cimatti, A., Fernandes Pires, A., Jones, D., Kimberly, G., Petri, T., Robinson, R., Tonetta, S.: Formal design and safety analysis of AIR6110 wheel brake system. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015, Part I. LNCS, vol. 9206, pp. 518–535. Springer, Cham (2015). https://doi.org/10.1007/9783319216904_36
Cabodi, G., Camurati, P., Garcia, L., Murciano, M., Nocco, S., Quer, S.: Speeding up model checking by exploiting explicit and hidden verification constraints. In: DATE (2009)
Cabodi, G., Camurati, P.E., Loiacono, C., Palena, M., Pasini, P., Patti, D., Quer, S.: To split or to group: from divideandconquer to subtask sharing for verifying multiple properties in model checking. Int. J. Softw. Tools Technol. Transfer (2017). https://doi.org/10.1007/s1000901704518
Cabodi, G., Nocco, S.: Optimized model checking of multiple properties. In: DATE (2011)
Cabodi, G., Garcia, L.A., Murciano, M., Nocco, S., Quer, S.: Partitioning interpolantbased verification for effective unbounded model checking. TCAD 29(3), 382–395 (2010)
Cavada, R., et al.: The nuXmv symbolic model checker. In: Biere, A., Bloem, R. (eds.) CAV 2014. LNCS, vol. 8559, pp. 334–342. Springer, Cham (2014). https://doi.org/10.1007/9783319088679_22
Cimatti, A., Griggio, A., Mover, S., Tonetta, S.: Parameter synthesis with IC3. In: FMCAD (2013)
Cimatti, A., Dorigatti, M., Tonetta, S.: OCRA: A tool for checking the refinement of temporal contracts. In: ASE (2013)
Classen, A., Cordy, M., Heymans, P., Legay, A., Schobbens, P.Y.: Model checking software product lines with SNIP. JSTTT 14(5), 589–612 (2012)
Classen, A., Cordy, M., Schobbens, P.Y., Heymans, P., Legay, A., Raskin, J.F.: Featured transition systems: foundations for verifying variabilityintensive systems and their application to LTL model checking. TSE 39(8), 1069–1089 (2013)
Classen, A., Heymans, P., Schobbens, P.Y., Legay, A.: Symbolic model checking of software product lines. In: ICSE (2011)
Classen, A., Heymans, P., Schobbens, P.Y., Legay, A., Raskin, J.F.: Model checking lots of systems: efficient verification of temporal properties in software product lines. In: ICSE (2010)
Dehnert, C., Junges, S., Jansen, N., Corzilius, F., Volk, M., Bruintjes, H., Katoen, J.P., Ábrahám, E.: PROPhESY: a PRObabilistic ParamEter SYnthesis tool. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015, Part I. LNCS, vol. 9206, pp. 214–231. Springer, Cham (2015). https://doi.org/10.1007/9783319216904_13
Dehnert, C., Junges, S., Jansen, N., Corzilius, F., Volk, M., Bruintjes, H., Katoen, J.P., Ábrahám, E.: Parameter synthesis for probabilistic systems. In: MBMV (2016)
Dimovski, A.S., AlSibahi, A.S., Brabrand, C., Wąsowski, A.: Familybased model checking without a familybased model checker. In: Fischer, B., Geldenhuys, J. (eds.) SPIN 2015. LNCS, vol. 9232, pp. 282–299. Springer, Cham (2015). https://doi.org/10.1007/9783319234045_18
Dureja, R., Rozier, K.Y.: FuseIC3: an algorithm for checking large design spaces. In: FMCAD (2017)
Dureja, R., Rozier, K.Y.: More Scalable LTL Model Checking via Discovering DesignSpace Dependencies (Artifact) (2018). https://doi.org/10.6084/m9.figshare.5913013.v1
Emerson, E.A., Kahlon, V.: Reducing model checking of the many to the few. In: McAllester, D. (ed.) CADE 2000. LNCS (LNAI), vol. 1831, pp. 236–254. Springer, Heidelberg (2000). https://doi.org/10.1007/10721959_19
Etessami, K., Kwiatkowska, M., Vardi, M.Y., Yannakakis, M.: Multiobjective model checking of markov decision processes. In: Grumberg, O., Huth, M. (eds.) TACAS 2007. LNCS, vol. 4424, pp. 50–65. Springer, Heidelberg (2007). https://doi.org/10.1007/9783540712091_6
Forejt, V., Kwiatkowska, M., Norman, G., Parker, D., Qu, H.: Quantitative multiobjective verification for probabilistic systems. In: Abdulla, P.A., Leino, K.R.M. (eds.) TACAS 2011. LNCS, vol. 6605, pp. 112–127. Springer, Heidelberg (2011). https://doi.org/10.1007/9783642198359_11
Gario, M., Cimatti, A., Mattarei, C., Tonetta, S., Rozier, K.Y.: Model checking at scale: automated air traffic control design space exploration. In: Chaudhuri, S., Farzan, A. (eds.) CAV 2016, Part II. LNCS, vol. 9780, pp. 3–22. Springer, Cham (2016). https://doi.org/10.1007/9783319415406_1
Hahn, E.M., Han, T., Zhang, L.: Synthesis for PCTL in parametric markov decision processes. In: Bobaru, M., Havelund, K., Holzmann, G.J., Joshi, R. (eds.) NFM 2011. LNCS, vol. 6617, pp. 146–161. Springer, Heidelberg (2011). https://doi.org/10.1007/9783642203985_12
James, P., Moller, F., Nguyen, H.N., Roggenbach, M., Schneider, S., Treharne, H.: On modelling and verifying railway interlockings: tracking train lengths. Sci. Comput. Program. 96(3), 315–336 (2014)
Khasidashvili, Z., Nadel, A.: Implicative simultaneous satisfiability and applications. In: Eder, K., Lourenço, J., Shehory, O. (eds.) HVC 2011. LNCS, vol. 7261, pp. 66–79. Springer, Heidelberg (2012). https://doi.org/10.1007/9783642341885_9
Khasidashvili, Z., Nadel, A., Palti, A., Hanna, Z.: Simultaneous SATbased model checking of safety properties. In: Ur, S., Bin, E., Wolfsthal, Y. (eds.) HVC 2005. LNCS, vol. 3875, pp. 56–75. Springer, Heidelberg (2006). https://doi.org/10.1007/11678779_5
Kwiatkowska, M., Norman, G., Parker, D., Qu, H.: Compositional probabilistic verification through multiobjective model checking. Inf. Comput. 232, 38–65 (2013)
Mattarei, C., Cimatti, A., Gario, M., Tonetta, S., Rozier, K.Y.: Comparing different functional allocations in automated air traffic control design. In: FMCAD (2015)
Moller, F., Nguyen, H.N., Roggenbach, M., Schneider, S., Treharne, H.: Defining and model checking abstractions of complex railway models using CSP\(\)B. In: Biere, A., Nahir, A., Vos, T. (eds.) HVC 2012. LNCS, vol. 7857, pp. 193–208. Springer, Heidelberg (2013). https://doi.org/10.1007/9783642396113_20
Quatmann, T., Dehnert, C., Jansen, N., Junges, S., Katoen, J.P.: Parameter synthesis for Markov models: faster than ever. In: Artho, C., Legay, A., Peled, D. (eds.) ATVA 2016. LNCS, vol. 9938, pp. 50–67. Springer, Cham (2016). https://doi.org/10.1007/9783319465203_4
Rosenmüller, M., Siegmund, N.: Automating the configuration of multi software product lines. VaMoS 10, 123–130 (2010)
Rozier, K.Y., Vardi, M.Y.: LTL satisfiability checking. In: Bošnački, D., Edelkamp, S. (eds.) SPIN 2007. LNCS, vol. 4595, pp. 149–167. Springer, Heidelberg (2007). https://doi.org/10.1007/9783540733706_11
Schirmeier, H., Spinczyk, O.: Challenges in software product line composition. In: HICSS. IEEE (2009)
Data Availability Statement
The benchmarks evaluated, source code, and datasets generated during our experimental analysis are available in the Springer/ Figshare repository: https://doi.org/10.6084/m9.figshare.5913013.v1. Theorem proofs and extended results are available on the paper’s accompanying website: http://temporallogic.org/research/TACAS18/.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
This chapter is published under an open access license. Please check the 'Copyright Information' section either on this page or in the PDF for details of this license and what reuse is permitted. If your intended use exceeds what is permitted by the license or if you are unable to locate the licence and reuse information, please contact the Rights and Permissions team.
Copyright information
© 2018 The Author(s)
About this paper
Cite this paper
Dureja, R., Rozier, K.Y. (2018). More Scalable LTL Model Checking via Discovering DesignSpace Dependencies (\(D^{3}\)). In: Beyer, D., Huisman, M. (eds) Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2018. Lecture Notes in Computer Science(), vol 10805. Springer, Cham. https://doi.org/10.1007/9783319899602_17
Download citation
DOI: https://doi.org/10.1007/9783319899602_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 9783319899596
Online ISBN: 9783319899602
eBook Packages: Computer ScienceComputer Science (R0)