Learning-Based Assume-Guarantee Regression Verification

Abstract

Due to enormous resource consumption, model checking each revision of evolving systems repeatedly is impractical. To reduce cost in checking every revision, contextual assumptions are reused from assume-guarantee reasoning. However, contextual assumptions are not always reusable. We propose a fine-grained learning technique to maximize the reuse of contextual assumptions. Based on fine-grained learning, we develop a regressional assume-guarantee verification approach for evolving systems. We have implemented a prototype of our approach and conducted extensive experiments (with 1018 verification tasks). The results suggest promising outlooks for our incremental technique.

This work was supported in part by the Chinese National 973 Plan (2010CB328003) and the NSF of China (61272001, 91218302).

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Software systems evolve throughout their life cycles. In order to add new features, many revisions are released over time. Since errors may be introduced with new releases, each revision needs to be formally verified. Formal verification however is still very time-consuming. Verifying every revision of an evolving system is impractical. A more effective technique to ensure correctness of evolving software systems is desired.

Model checking is a formal verification technique [4, 17]. In model checking, lots of internal information is computed during a verification run. Note that two consecutive revisions share many behaviors. When a revision is verified, internal information from model checking may still be useful to verifying the next revision. Regression verification expands this idea by reusing internal information to speed up the verification of later revisions [3, 6, 7, 10, 22, 27–29]. Various internal information has been proposed for reuse, including state space graphs [22, 29], constraint solving results [28], function summaries [3, 25], and abstract precisions [6].

Assume-guarantee reasoning [18] is a compositional technique to improve the scalability of model checking. In the compositional technique, contextual assumptions decompose verification tasks by summarizing component behaviors. Depending on compositional proof rules, contextual assumptions are required to fulfill different criteria for sound verification. Although they used to be constructed manually, contextual assumptions can be generated automatically by machine learning algorithms [13, 14, 18, 21].

Like internal information from model checking, contextual assumptions for the current revision may be reused for the next revision as well. Since contextual assumptions contains the most important information for verifying the current revision, they may immediately conclude the verification of the next revision. Contextual assumptions may be more suitable for regression verification. Compared to internal information from model checking, contextual assumptions are external information. They can be stored and reused without modifying model checking algorithms. In [26], contextual assumptions are exploited in regression verification. When the component summarized by contextual assumptions is not changed, the contextual assumptions are reused and modified to verify revised composed systems. If a system evolves into a new version, components may all be revised. Contextual assumptions thus can not be reused in regression verification. This can be a severe limitation.

Recall that system models are often represented by logic formulas in symbolic verification algorithms. A component may be represented by several logic formulas. Moreover, such logic formulas are further decomposed into more subformulas to attain the best performance. When a system with few components is updated, it is unlikely that all subformulas are revised. The chance of information reuse can be greatly improved if systems are decomposed into finer constituents. In our fine-grained learning framework, an instance of the learning algorithm [8, 19, 23] is deployed for each logic subformulas. When all instances infer their conjectures, a contextual assumption can be built from these conjectures and sent for assume-guarantee reasoning. We call this the fine-grained learning-based verification.

Using our fine-grained technique, we improve regression verification by incremental assume-guarantee reasoning. The word incremental means the previously-computed results are reused in later verification runs. Given a new revision of the system model represented as a number of logical formulas. We compare the previous revision and the new revision for each subformula. If they remain the same, the inferred conjecture in the previous verification for this subformula can be safely reused. Otherwise the conjecture is re-constructed. Since two revisions have similar behaviors, many of their subformulas remain unchanged. Previously inferred conjectures is likely to be reused.

We have implemented a prototype on top of NuSMV. We performed extensive experiments (with 1018 verification tasks) to evaluate the efficiency of our technique. Experimental results are very promising. If properties are satisfied before and after revisions, our new technique is about four times faster than conventional assume-guarantee reasoning. A similar speedup is also observed for unsatisfied properties before and after revisions. If properties are satisfied before but unsatisfied after revisions, incremental assume-guarantee reasoning also outperforms but less significantly. Overall, we report more than three times speedup on more than a thousand verification tasks.

The remainder of this paper is organized as follows. Section 2 introduces necessary background. Section 3 explains our motivation. Fine-grained learning is discussed in Sect. 4. Our regression verification framework is presented in Sect. 5. Experimental results are reported in Sect. 6. Related work are discussed in Sect. 7. Finally Sect. 8 concludes this paper.

2 Background

Let $\mathbb {B}$ be the Boolean domain and $X$ a finite set of Boolean variables. A valuation $s : X\rightarrow \mathbb {B}$ of $X$ is a mapping from $X$ to $\mathbb {B}$. A predicate $\phi (X)$ over $X$ maps a valuation of $X$ to $\mathbb {B}$. We may write $\phi$ if its variables are clear from the context.

Definition 1

A transition system $M = (X, \varLambda , \varGamma )$ consists of a finite set of variables $X$, an initial condition $\varLambda$ over $X$, and a transition relation $\varGamma$ which is a predicate over $X$ and $X' = \{ x' : x \in X\}$.

Definition 2

Let $M_i = \langle X_i, \varLambda _i, \varGamma _i \rangle$ be transition systems for $i = 0, 1$ ($X_i$’s are not necessarily disjoint), the composition $M_0 \Vert M_1 = \langle X, \varLambda , \varGamma \rangle$ is a transition system where $X= X_0 \cup X_1$, $\varLambda (X) = \varLambda _0(X_0) \wedge \varLambda _1(X_1)$, and $\varGamma (X) = \varGamma _0(X_0) \wedge \varGamma _1(X_1)$.

Let $M=(X, \varLambda , \varGamma )$ be a transition system. A state s of M is a valuation over $X$. A trace $\sigma$ of M is a sequence of states $s_0, s_1, \cdots , s_n$, such that $s_0$ is an initial state, and there is a transition from $s_i$ to $s_{i+1}$ for $i = 0, \ldots , n - 1$. For any predicate $\phi$, a sequence $\sigma$ of states $s_0, s_1, \ldots , s_n$ satisfies $\phi$ (written $\sigma \models \phi$) if $s_i \models \phi$ for $i = 0, \ldots , n$. We say M satisfies $\phi$ (written $M \models \phi$) if $\sigma \models \phi$ for all traces of M. Given a transition system M and a predicate $\phi$, the invariant checking problem is to decide whether M satisfies $\phi$.

2.1 Learning-Based Assume-Guarantee Verification

Assume-guarantee reasoning aims to mitigate the state explosion problem by divide-and-conquer strategy. It uses assumptions to summarize components. Since details of components can be ignored in assumptions, the compositional technique can be more effective than monolithic verification.

Definition 3

Let $M_i = \langle X, \varLambda _i, \varGamma _i \rangle$ be transition systems for $i = 0, 1$, $M_1$ simulates $M_0$ (written $M_0 \preceq M_1$) if $\varLambda _0 \Rightarrow \varLambda _1$ and $\varGamma _0 \Rightarrow \varGamma _1$.

Note that the above simulation relation is defined over first-order representation of models. Informally, $M_0\preceq M_1$ if $M_1$ simulates all behaviors of $M_0$.

Theorem 1

[14]. Let $M_i = \langle X_i, \varLambda _i, \varGamma _i \rangle$ be transition systems for $i = 0, 1$, $X= X_0 \cup X_1$, and $\phi (X)$ a predicate, the following assume-guarantee reasoning rule is sound and invertible:

$$\begin{aligned} \frac{M_0 \preceq A\qquad A \Vert M_1 \models \phi }{M_0 \Vert M_1 \models \phi } \end{aligned}$$

(1)

A rule is sound if its conclusion holds when its premises are fulfilled. A rule is invertible if its premises can be fulfilled when its conclusion holds. In the proof rule (1), the transition system A is called a contextual assumption (for short, assumption) of $M_0$. A contextual assumption is valid if it either satisfies both premises of above rule, or is able to reveal a counterexample to $M_0\Vert M_1\models \phi$.

Active learning algorithms have been deployed to automatically learn the assumptions for compositional verification [1, 13, 14, 18, 20, 21]. Let $U$ be an unknown predicate. A learning algorithm infers a Boolean formula characterizing $U$ by making queries. It assumes a teacher who knows the target predicate $U$ and answers the following two types of queries:

• On a membership query MQ(s) with a valuation s, the teacher answers $YES$ if $U(s)$ holds, and $NO$ otherwise.

• On a equivalence query $EQ(H)$ with a hypothesis Boolean formula $H$, the teacher answers $YES$ if $H$ is semantically equal to $U$. Otherwise, she returns a valuation t on which $H$ and $U$ evaluate to different Boolean values as a counterexample.

Fig. 1.

The learning-based verification framework

Figure 1 shows the learning-based verification framework [13, 14, 21]. In the framework, a mechanical teacher is designed to answer queries from the learner. For simplicity of illustration, the mechanical teacher in the figure is divided into two parts, each answering one type of queries. Let $M_0 = \langle X_0, \varLambda _0, \varGamma _0 \rangle$ be a transition system. The mechanical teacher knows $\varLambda _0$ and $\varGamma _0$, and guides $Learner$ to infer an assumption $A = \langle X_0, \varLambda _A, \varGamma _A\rangle$ fulfilling the premises of the proof rule (1). Two learning algorithms are instantiated: one for the initial condition $\varLambda _A$, the other for the transition relation $\varGamma _A$. For instance, consider the learning algorithm for $\varGamma _A$. For a membership query $MQ_{\varGamma }(s,t)$ from $Learner _{\varGamma }$, the mechanical teacher checks if $\langle s,t\rangle$ satisfies $\varGamma _0$. If so, the mechanical teacher answers $YES$. Otherwise, she answers $NO$. Conceptually, the mechanical teacher uses $\varGamma _0$ as the target predicate. In the worst case, the mechanical teacher infers $\varGamma _0$ as $\varGamma _A$.

The equivalence queries of the two learning algorithms need be synchronized. Let $\overline{\varLambda }_A$ and $\overline{\varGamma }_A$ be the current purported representations of $\varLambda _A$ and $\varGamma _A$, respectively, the mechanical teacher first constructs $\overline{A} = \langle X_0, \overline{\varLambda }_A, \overline{\varGamma }_A \rangle$, then it checks if the purported conjecture of $\overline{A}$ satisfies both premises of the assume-guarantee reasoning rule. If it does, the verification terminates and returns “safe”. Otherwise, the premises checker returns a counterexample. The teacher then proceeds to check whether this counterexample is real or not. If it is a real counterexample, the verification algorithm terminates and reports “unsafe”. Otherwise, the teacher returns this counterexample to $Learner$. $Learner$ will use this counterexample to refine its purported formulas. This process repeats until a valid assumption is inferred.

2.2 Regression Verification

Computer systems evolve during their life time. Since the current version of a system has different behaviors from its previous versions, properties must be re-verified against the current version. In regression verification, we consider the invariant checking problem on two versions of a system. We would like to exploit any information from the previous verification in the current verification.

Definition 4

Let $M = (X, \varLambda , \varGamma )$ and $M' = (X, \varLambda ', \varGamma ')$ be transition systems and $\phi (X)$ a specification. The regression verification problem is to check whether $M' \models \phi$ after the verification of $M \models \phi$.

Note that Definition 4 does not assume whether the previous version M satisfies the property $\phi$ or not. We would like to re-use any information from the previous verification regardless of whether $M \models \phi$ holds or not.

3 Motivation

Let $M_0$ and $M_1$ be two components of a system, and $A^*$ a valid contextual assumption. To perform regression verification on updated components $M'_0$ and $M'_1$, a natural idea is to reuse the contextual assumption $A^*$. However, it is shown in [26] that $A^*$ as a whole can only be reused if $M'_0 = M_0$ and $M'_1$ simulates $M_1$. This can be a severe limitation.

3.1 An Example

Consider an email system composed of two clients ${c_i}$ ($i=0,1$). The client $c_i$ is shown in Fig. 2(a). Each $c_i$ is associated with a data variable ${msg_i}$, whose value being ${\mathsf {true}}$ indicates that $c_i$ is sending a message. When $c_i$ sends a message, $c_{1-i}$ will be informed and vice versa. The client $c_i$ has four states: the idle state (“${\texttt {idle}}$”), the receiving state (“${\texttt {recv}}$”), the outgoing state (“${\texttt {otgo}}$”), and the sent state (“${\texttt {sent}}$”). Initially, $c_i$ is at the $\texttt {{idle}}$ state. If a message arrives (that is, ${msg_{1-i}}={\mathsf {true}}$), $c_i$ transits to the receiving state $\texttt {{recv}}$. Otherwise, it non-deterministically transits to the outgoing state $\texttt {otgo}$ and sets ${msg_i}$ to ${\mathsf {true}}$, or remains at the idle state $\texttt {{idle}}$. After the message is sent, the client transits to its $\texttt {sent}$ state. Denote $M_{c_i}$ the model of $c_i$.

Fig. 2.

The email client

The requirement $\phi _{es}$ is that all sent emails are well received. Formally, $\phi _{es} := (state_0 =\texttt {sent}) \leftrightarrow (state_1=\texttt {recv})$. Apparently, $\phi _{es}$ is not satisfied by the model. Assume that both clients transit from their idle states to their otgo states simultaneously, representing both are going to send a message. The only next state for both of them is the sent state, which means that both clients have sent their messages, but none of them was well received.

The original model needs be revised to satisfy the requirement. Let $c'_i (i=0,1)$ be the updated client, shown in Fig. 2(b). In the new model, sending out a message is granted for a client if another client does not require sending at the same time. If both clients simultaneously want to send their messages, a new variable, called “$turn$”, is introduced to assign priority to one of them.

Let us consider the regression verification of $M_{c'_0} \Vert M_{c'_1}$. Apparently, $M_{c'_0} \ne M_{c_0}$ and $M_{c'_1} \ne M_{c_1}$. According to [26], the contextual assumptions inferred in the verification of $M_{c_0}\Vert M_{c_1}$ cannot be reused in the verification of $M_{c'_0}\Vert M_{c'_1}$. However, if we take a look at the symbolic representations of these two revisions of the system, many commonalities can be identified.

Denote $M_{c_i} = \langle X_{c_i}, \varLambda _{c_i}, \varGamma _{c_i}\rangle$ for $i=0,1$, where $X_{c_i}$ is a set including a state variable $state_i$ $\in \{\texttt {idle}$, $\texttt {recv}$, $\texttt {otgo}$, $\texttt {sent}\}$ and a data variable $msg_i$. The model $M_{c_i}$ can be specified in a way that specifies for each variable x its initial values $init (x)$ and its next-state values $next (x)$. (for example, in NuSMV language [16]):

$$\begin{aligned}&init (state_i) := \texttt {idle}, init (msg_i) := \mathsf {false}, \\&next (state_i) := \\&\quad case \\&\quad \quad (state_i = \texttt {idle}) \wedge msg_{1-i} : \texttt {recv}; \\&\quad \quad (state_i = \texttt {idle}) \wedge \lnot msg_{1-i} : \{\texttt {idle, otgo}\}; \\&\quad \quad (state_i = \texttt {otgo}) : \texttt {sent}; \\&\quad \quad (state_i = \texttt {recv}) : \texttt {idle}; \\&\quad \quad (state_i = \texttt {sent}) : \texttt {idle}; \\&\quad esac \\&next (msg_i) := \\&\quad case \\&\quad \quad (state_i = \texttt {idle}) \wedge ( next (state_i) = \texttt {otgo}) : {\mathsf {true}}; \\&\quad \quad (state_i = \texttt {otgo}) \wedge ( next (state_i) = \texttt {sent}) : \mathsf {false}; \\&\quad \quad {\mathsf {true}}: msg_i;\\&\quad esac \end{aligned}$$

The “$case \dots esac$” expression in above formulas returns the first expression on the right hand side of “:”, such that the corresponding condition on the left hand side evaluates to ${\mathsf {true}}$ [16]. For short, we write $\lambda _x$ for the logic formula $x = init (x)$ and $\gamma _x$ for the formula $x' = tran (x)$. Then $\varLambda _{c_i}$ and $\varGamma _{c_i}$ can be represented as:

$$\begin{aligned}&\varLambda _{c_i} = \lambda _{state_i} \wedge \lambda _{msg_i},&\varGamma _{c_i} = \gamma _{state_i} \wedge \gamma _{msg_i}. \end{aligned}$$

The formulas $init (state_i)$, $init (msg_i)$ and $next (msg_i)$ in the new model are identical to those in the old model. The only difference lies in the formula $next (state_i)$, which in the new model is:

$$\begin{aligned} \begin{aligned}&next (state_i) := \\&\quad case \\&\quad \quad \quad (state_i = \texttt {idle}) \wedge msg_{1-i} : \texttt {recv}; \\&\quad \quad \quad (state_i = \texttt {idle}) \wedge \lnot msg_{1-i} : \{\texttt {idle, otgo}\}; \\&\quad \quad \quad (state_i = \texttt {otgo}) \wedge msg_{1-i} \wedge \lnot turn: \texttt {recv}; \\&\quad \quad \quad (state_i = \texttt {otgo}) \wedge (\lnot msg_{1-i} \vee turn): \texttt {sent}; \\&\quad \quad \quad (state_i = \texttt {recv}) \wedge msg_i: \texttt {otgo}; \\&\quad \quad \quad (state_i = \texttt {recv}) \wedge \lnot msg_i: \texttt {idle}; \\&\quad \quad \quad (state_i = \texttt {sent}) : \texttt {idle}; \\&\quad esac \end{aligned} \end{aligned}$$

3.2 Our Solutions

To take full advantage of commonalities between revisions, we propose to learn the contextual assumptions in a fine-grained fashion. Recall that $M_{c_i}$ in the email system is represented using four predicate formulas, i.e., $\lambda _{state_i}$, $\gamma _{state_i}$, $\lambda _{msg_i}$ and $\gamma _{msg_i}$. Instead of inferring the contextual assumption as a whole model [26], we suggest to learn it as these four formulas. Note that the former three formulas are identical in the updated model, the inferred conjectures for these three formulas can be safely reused. In this way, the chance of assumption reuse is improved.

We intend to learn the contextual assumptions also in a symbolic fashion. In [26], the contextual assumptions are represented as deterministic finite automata (DFA’s). However, the DFA is not a compact representation of a model. A Boolean formula representable by a BDD having n nodes may need mn nodes even in its most compact DFA representation [23], where m is the number of variables in the formula. Learning models via their DFA representations is thus not an efficient approach. We utilize the learning technique in [21] to learn the BDD representation of contextual assumptions. The benefits are multiple folds. Firstly, the symbolic representation of a model is more compact. Recording and reusing the contextual assumption in its symbolic representation is thus more memory-efficient. Secondly, symbolic assumptions can be better adapted to the symbolic model checking. Finally, with the symbolic representations, the equivalence checking of models can be performed in a much more efficient way.

4 Fine-Grained Learning Technique

In this section, we propose a fine-grained learning technique for assume-guarantee verification. Let $M_{U} = \langle X, \varLambda , \varGamma \rangle$ be the unknown target model. Its initial condition $\varLambda$ and transition relation $\varGamma$ can oftentimes be represented as a set of logical formulas. Instead of inferring $M_{U}$ as a DFA, or as two big logical formulas (i.e. $\varLambda$ and $\varGamma$), we propose to infer it as a set of small logical formulas. Fine-grained learning technique will give us more chances to reuse the inferred results.

Fig. 3.

The fine-grained learning framework

Without loss of generality, we assume $\varLambda$ and $\varGamma$ are decomposed into n predicate formulas: $\varphi _1, \varphi _2, \cdots , \varphi _n$. Define templates to be constructed inductively by logical operators and subscripted square parentheses ($[\bullet ]_k$). Let $\zeta _{\varLambda }$ and $\zeta _{\varGamma }$ be two templates. With $\zeta _{\varLambda }$ and $\zeta _{\varGamma }$, we can construct a contextual assumption from the purported formulas. For example, consider the templates $\zeta _{\varLambda }[\bullet ]_1[\bullet ]_2 = [\bullet ]_1 \wedge [\bullet ]_2$ and $\zeta _{\varGamma }[\bullet ]_1[\bullet ]_2 = [\bullet ]_1 \wedge [\bullet ]_2$ in the email system. Suppose $\overline{\lambda }_{state_0}$, $\overline{\lambda }_{msg_0}$, $\overline{\gamma }_{state_0}$ and $\overline{\gamma }_{msg_0}$ are the current purported formulas. The initial condition and transition relation of the contextual assumption can be constructed as $\zeta _{\varLambda }[\overline{\lambda }_{state_0}]_1[\overline{\lambda }_{msg_0}]_2 = \overline{\lambda }_{state_0} \wedge \overline{\lambda }_{msg_0}$, and $\zeta _{\varGamma }[\overline{\gamma }_{state_0}]_1[\overline{\gamma }_{msg_0}]_2 = \overline{\gamma }_{state_0} \wedge \overline{\gamma }_{msg_0}$, respectively.

The fine-grained learning model is shown in Fig. 3. For each subformula $\varphi _i$ ($1\le i\le n$), one instance of the learning algorithm is deployed. All learners make membership and equivalence queries to a mechanical teacher. Similar to the learning-based framework in Sect. 2.1, equivalence queries need be synchronized. When all learners get a conjecture, the mechanical teacher constructs a contextual assumption (using $\zeta _{\varLambda }$ and $\zeta _{\varGamma }$). If the constructed assumption fulfills both premises of the assume-guarantee reasoning rule (1), the verification is finished. Otherwise, the mechanical teacher helps the learners refine their conjectures by providing counterexamples.

Note that our fine-grained technique is not limited to the NuSMV language, and the target model is not necessary to be decomposed by variables (as in the email system example). To see an example, consider the ELTS (extended labelled transition systems with variables) model that is usually specified by transitions. Let k be the number of transitions in an ELTS model. Encoding each transition as a logical formula, the transition relation of the ELTS model is the disjunction of all transition formulas, and the template $\zeta _{\varGamma }= [\bullet ]_1 \vee [\bullet ]_2 \vee \cdots \vee [\bullet ]_k$. Generally, we follow the syntactic structure to decompose the symbolic representation of the target model.

5 Assume-Guarantee Regression Verification

In this section, we discuss the data structures of contextual assumptions, propose our regression verification framework, and finally prove the correctness of our technique.

5.1 Data Structures of Contextual Assumptions

Our framework employs Nakamura’s algorithm [23] to infer the BDD representation of contextual assumptions. Nakamura’s algorithm is an instance of the active learning algorithm. Its basic procedure follows that discussed in Sect. 2.1. When we say assumption reusing, we actually mean reusing the data structure of the learning algorithm. We thus discuss in the following the data structures used in Nakamura’s algorithm.

Let $D$ be the target (reduced and ordered) BDD with m variables. A BDD is a directed acyclic graph with one root node and two sink nodes. Each sink node is labeled with 0 or 1, and each non-sink node is labeled with a variable. A BDD can be regarded as a DFA. For any node of $D$, an access string u is a string that leads the BDD from its initial node to that node. Each node of $D$ can be represented by its access string. In the following, we abuse the notation of u (and v) to represent both a node and its access string. For any two distinct nodes u and v, a distinguishing string w is a string such that uw reaches the terminal 1 and vw reach the terminal 0, or vice versa. Denote nodes(D) the set of strings $a_1a_2\cdots a_k$ such that $k=m$ or the assignment of $x_1\leftarrow a_1, x_2\leftarrow a_2, \cdots , x_k \leftarrow a_k$ leads to a node labeled $x_{k+1}$ in D. Let v be a string of length m, denote $D(v)$ the sink label that v reaches in D.

Two data structures are maintained in the BDD learning algorithm: a BDD with access strings (for short, BDDAS) $S$, and a set $T = \{T_1, T_2, \cdots , T_m\}$ of classification trees. A BDDAS is different from an ordinary BDD mainly in the following points: it may have a dummy root node; each of its nodes has an access string; each of its edges is labeled with a binary string. Denote $nodes ^S_i(S)$ the set of access strings possessed by the non-dummy nodes in S whose length is i. Let $nodes ^S(S) = \bigcup ^m_{i=0} nodes ^S_i(S)$. Let v be a string of length m, denote $S(v)$ the sink label that v reaches in $S$.

A classification tree $T_i$ $(1\le i\le m)$ decides which node in $S$ a given string of length i will reach. It is composed of internal nodes and leaf nodes. Each internal node is labeled with a distinguishing string of length $m-i$, and each leaf node is labeled with either a special symbol $\mu$, or an access string of length i that is possessed by a node of $S$. Any string $\alpha$ of length i is classified by $T_i$ into one of its leaf nodes. Denote $T_i(\alpha )$ the leaf label into which $\alpha$ is classified. A string classified into a leaf node labeled with $\mu$ means that this string cannot reach any node in the corresponding OBDDAS.

A BDD can be obtained from a BDDAS. The obtained BDD is sent to the teacher for equivalence checking. If it passes the equivalence checking, we are done. Otherwise, a string is returned by the teacher as a counterexample. With this counterexample, the learner updates its BDDAS and classification trees. During the updating, the teacher’s answers to membership queries are stored in classification trees. After updating, the cardinality of $S$ (i.e. the number of nodes in $S$) increases by one. The target BDD is restored when the cardinality of $S$ equals the number of nodes in the target BDD [23].

5.2 Regression Verification Framework

Our assume-guarantee regression verification algorithm is depicted in Algorithm 1. Before the new round of verification starts, an initialization step is performed, which attempts to reuse the contextual assumption inferred in the previous round of verification.

Let $M_0$ and $M_1$ be two components of a system. Recall that $M_0$ is the learning target. Assume $M_0$ is represented as n logical formulas: $\varphi _1$, $\varphi _2$, $\cdots$, $\varphi _n$. Let $\varphi '_i$ ($1\le i\le n$) be the updated form of $\varphi _i$ in $M'_0$. In the regression verification, the algorithm checks for each i ($1\le i\le n$) if $\varphi '_i$ is equivalent to $\varphi _i$. If it is, the data structures (the BDDAS and classification trees) of the previous learner $Learner _{\varphi _i}$ is restored and used to initialize $Learner _{\varphi '_i}$. Otherwise, $Learner _{\varphi '_i}$ starts with empty data structures.

5.3 Correctness

We prove the correctness of our assume-guarantee regression verification framework in this subsection.

Let $\alpha _1, \alpha _2$ be two binary strings, we use $|\alpha _1|$ to denote the length of $\alpha _1$, $pre(\alpha _1,i)$ the prefix string of $\alpha _1$ with length i, and $\alpha _1\cdot \alpha _2$ the concatenation of $\alpha _1$ and $\alpha _2$.

Definition 5

A BDDAS S and a set $T=\{T_1,\cdots ,T_m\}$ of classification trees are said valid for the target BDD $D$, if the following conditions are satisfied [23]:

• 1. $nodes ^S(S)\subseteq nodes(D)$;

• 2. $\forall v \in nodes ^S_m(S)$, $S(v) = D(v)$;

• 3. $\forall v_1, v_2 \in nodes ^S(S)$, if $v_1$ and $v_2$ lead to the same node in $D$, there must be $v_1 = v_2$;

• 4. $\forall v \in nodes ^S(S)$, $T_{|v|}(v) = v$;

• 5. for any binary string $\alpha$ of length $i (1\le i\le m)$, $\alpha \not \in nodes(D)\Rightarrow T_i(\alpha )=\mu$;

• 6. for any edge in S that is from u to v and labeled with l,

  • $T_{|v|}(u\cdot l)=v$, and

  • $|u|< \forall j<|v|$, $T_{j} (u \cdot pre(l, j-|u|)) = \mu$.

Lemma 1

[23]. The Nakamura’s learning algorithm terminates with a correct result starting from any BDDAS S and classification trees $T_i$ for $i = 1, \cdots , m$ that are valid for the target BDD $D$.

Theorem 2

Given two BDD’s $D_1$ and $D_2$, if $D_1\equiv D_2$, the BDDAS S and classification trees $T_i$ for $i =1, \cdots , m$ generated by the learner of $D_1$ are valid for $D_2$.

Recall that in our verification framwork, only results of equivalent formulas are reused. Theorem 2 is thus applicable. The correctness of our assume-guarantee regression verification framework (Algorithm 1) follows from Lemma 1 and Theorem 2.

Theorem 3

(Correctness). The assume-guarantee regression verification algorithm (Algorithm 1) always terminates with a correct result.

Note that our regression verification framework is not limited to Nakamura’s algorithm [23]. Conceptually, any active learning algorithm can apply, such as the $L^*$ algorithm for regular languages [2]. However, to be better suited for the fine-grained learning technique, an implicit learning algorithm is preferred. Alternatively, one can also use the CDNF learning algorithm [8] that infers Boolean functions.

6 Evaluation

A prototype of our regressional assume-guarantee verification technique was implemented on top of NuSMV 2.4.3 [16]. We have performed extensive experiments (in total, 1018 verification tasks from 108 revisions of 7 examples) to evaluate the efficiency of our technique. All experiments were conducted on a machine with 3.06 GHz CPU and 2G RAM, running the Ubuntu 12.04 operation system.

A verification task is specified by a base model, an update to the base model, and a specification. It consists of two rounds of verifications. The contextual assumption inferred in the first round of verification (on the base model) can be optionally reused in the second round of verification (on the updated model). We compare the performance of the second round of verification with and without assumption reuse. The maximal run time is set to 3 h.

The experiments are performed on seven examples, where Gigamax models a cache coherence protocol for the Gigamax multiprocess, MSI models a cache coherence protocol for consistence ensuring between processors and main memory, Guidance models the Shuttle Digital Autopilot engines out (3E/O) contingency guidance requirements, SyncArb models a synchronous bus arbiter, Philo models the dining philosophers problem [12], Phone models a simple phone system with four terminals [24], and Lift models the lift system in [5]. The former four examples are obtained from the NuSMV website¹, while the latter two are obtained from literatures. Each example model contains a number of interacting components. Our tool selects one component as $M_0$ and the composition of others as $M_1$.

We consider different degrees of changing a model: small changes (using mutations) and significant changes (with significant difference in the functionalities).

Two performance metrics are used in our experiments: (a) the run time (Time) for each verification run; (b) the number of membership queries ($| MQ |$) and the number of equivalence queries ($| EQ |$) raised in each verification run. Recall that answering learners’ queries is the most costly operation in the learning-based verification framework, these two metrics are related to each other.

6.1 Results for Small Changes

Model changes are often small. We realized a program to randomly produce a number of mutations to a model either by introducing new variables or by changing the initial condition or transition relation of an existing variable.

This experiment was performed on five examples: Gigamax, MSI, Guidance, SyncArb and Philo. Results are shown in the upper part of Table 1. The columns |Update|, |Spec.| and |Task| list for each example the numbers of updates, specifications, and verification tasks, respectively. The following two column show the performance of the regression verification with and without assumption reuse respectively. All performance results (including the number of membership queries $| MQ |$, the number of equivalence queries $| EQ |$, and the run time) are given in average values over all tasks per example. The last column compares these two approaches. More experiment details of the highlighted example SyncArb will be discussed in Sect. 6.3. The experiment analysis is deferred to the next subsection. We will combine other examples’ results and give a combined analysis.

6.2 Results for Significant Changes

During the evolution of a system, new features can be added to improve the original design. This kind of updates involves significant changes to the original model.

The second experiment was performed on two examples: Phone and Lift. These two examples were obtained from the software product-line engineering community [5, 24]. For each example, there are a base model and a set of features. Each feature is considered as a significant change to the base model. Results of this experiment are shown in the bottom part of Table 1. The last Total row gives the average of respective values over all examples, including the examples mentioned in the former experiment and those in this experiment.

From Table 1, we observe an impressive improvement of our incremental approach with assumption reuse. Depending on examples, the average speed up of assumption reuse is between 1.26 to 3.79. Over all examples (with 1018 verification tasks in total), the average speed up is 3.47. We also find that the number of queries made by the incremental approach is greatly reduced compared to those without reuse. Over all examples, the average number of membership queries $| MQ |$ is reduced by a ratio of 2.89, and the average number of equivalence queries $| EQ |$ is reduced by a ratio of 3.44. Recall that answering learners’ queries is the most costly operation in the learning-based assume-guarantee verification, these results conforms to those about run time.

There is no significant difference for the performance improvement of our incremental approach between the examples with small changes and others with significant changes. This observation supports that our incremental approach is applicable to both degrees of model changes.

Table 1. Results for all examples: time in seconds

6.3 Results for a Single Example

Detailed results for SyncArb example are shown in Table 2. The Sat. column shows a pair of Boolean values (“T” for ${\mathsf {true}}$, “F” for $\mathsf {false}$), representing the satisfiability of the specification on the base model and the updated model, respectively. The term “max” in the last column denotes a divided-by-zero value. The bottom two rows report the sum and the average of the respective values over all verification tasks.

With assumption reuse, the numbers of membership queries $| MQ |$ and the number of equivalence queries $| EQ |$ are 0’s in 15 out of 19 tasks. In other words, the reused assumptions immediately conclude the second round of verification in these tasks. This observation further witnesses the usability of assumption reuse to regression verification.

Table 2. Results for SyncArb: time in seconds

6.4 Impact of the Satisfiability Results to the Performance

Recall that when models change, the previously established (or falsified) specifications may become unsatisfied (or satisfied). We test in this experiment the impact of the satisifiability results to the efficiency of our incremental approach.

We group verification tasks of each example by their satisfiability results. In total, there are four types of groups: both ${\mathsf {true}}$ (denoted as (T, T)), ${\mathsf {true}}$ on the base model and $\mathsf {false}$ on the updated model (denoted as (T, F)), $\mathsf {false}$ on the base model and ${\mathsf {true}}$ on the updated model (denoted as (F, T)), and both $\mathsf {false}$ (denoted as (F, F)). Results are shown in Table 3, where |Task| column lists the number of verification tasks in each group. Empty groups (with $|Task| = 0$) are omitted from the table.

Table 3. Results grouped by the satisfiability results on the base and the updated models: time in seconds

We got very interesting findings from these results. The last column of Table 3 shows that the regression verification is most likely to be improved by the assumption reuse if the specification was previously satisfied. There are two (F, T) groups (Gigamax, Phone) and two (F, F) groups (Guidance, Philo) on which the assumption reuse leads to notably performance degeneration. In contrast, the performance of the regression verification is always improved (or nearly improved) by assumption reuse in all (T, T) and (T, F) groups. We speculate the reasons as follows. Recall the assume-guarantee reasoning rule (1). If the specification is satisfied by the system, we need to find a contextual assumption to prove both premises in the rule. In contrast, if the specification is dissatisfied by the system, we need only an assumption that reveals a counterexample to the specification. Finding a counterexample is always much easier than proving the correctness. From the viewpoint of reuse, the assumption revealing a counterexample is certainly less useful than the one proving the correctness of the model.

The Total row in Table 3 gives that the average speedup of the incremental technique over all examples for (T, T), (T, F), (F, T) and (F, F) groups are 4.12, 1.75, 0.59, and 4.29, respectively. It further shows that the incremental technique tends to gets the best performance when the staisfiability of the specification are the same on both models. This phenomenon is also reasonable. Given that many behaviors are shared between these two models, the previously found proof (or counterexample) is very likely to be a valid proof (or a valid counterexample) for the updated model.

7 Related Work

The first technique on learning-based assume-guarantee reasoning was proposed in [18], where the $L^*$ algorithm [2] was adopted to learn the DFA representation of contextual assumptions. The $L^*$-based assume-guarantee reasoning was further optimized in different directions by many researchers, including [1, 11, 15, 30]. An implicit learning framework for assume-guarantee reasoning was proposed in [14], where contextual assumptions are inferred in their symbolic representations. Both the BDD learning algorithm [23] and $CDNF$ learning algorithm [8] have been adapted to this framework. Moreover, the techinque in [21] improves the implicit learning framework by a progressive witness analysis algorithm. In [20], the learning-based assume-guarantee reasoning was fruther applied to probabilistic model checking. Our technique contributes in assume-guarantee reasoning by providing a new fine-grained learning technique.

Regression verification was investigated mainly in two directions, the equivalence analysis, and the reuse of previously computed results. In the latter direction, a variety of information have been proposed for reuse in regression verification. In [22, 29], the state-space graphs are recorded for reuse in latter verification runs. In [28], the intermediate results of a constraint solver are stored and reused. In [7], the abstraction precision used for performing predicate abstraction on previous program is reused. Note that the precision reuse technique is orthogonal to ours. Our technique contributes in this area by integrating regression verification and automated assume-guarantee reasoning.

The most relevant work to ours are [9, 26]. They used the idea of assumption reuse to solve the dynamic component substitutability problem. Their technique requires $M_0'=M_0$ and $M'_1$ simulates $M_1$. This is surely a severe limitation. We removed this limitation by fine-grained learning technique. With our technique, the assume-guarantee regression verification is enabled.

8 Conclusions and Future Work

We presented in this paper a learning-based assume-guarantee regression verification technique. With this technique, contextual assumptions of the previous round of verification can be efficiently reused in the current verification. Correctness of this techniques is established. Experimental results (with 1018 verification tasks) show significant improvements of our technique.

Currently, we implemented a prototype of our technique on top of NuSMV. We are considering to extend this technique to a component-based modeling language that allows hierarchical components and sophisticated interactions. We are also planning to integrate our technique with predicate abstraction, and then apply it to program verification.