Scalable Algorithms for Abduction via Enumerative Syntax-Guided Synthesis

Abstract

The abduction problem in logic asks whether there exists a formula that is consistent with a given set of axioms and, together with these axioms, suffices to entail a given goal. We propose an approach for solving this problem that is based on syntax-guided enumeration. For scalability, we use a novel procedure that incrementally constructs a solution in disjunctive normal form that is built from enumerated formulas. The procedure can be configured to generate progressively weaker and simpler solutions over the course of a run of the procedure. Our approach is fully general and can be applied over any background logic that is handled by the underlying SMT solver in our approach. Our experiments show our approach compares favorably with other tools for abductive reasoning.

This work was partially supported by NSF grant #1656926 and DARPA grants #N66001-18-C-4006 and #N66001-18-C-4012.

1 Introduction

The abduction problem for theory T, a set of axioms $\mathsf {A}$ and goal $\mathsf {G}$ asks whether there exists a formula $\varphi$ such that: (i) $\mathsf {A}\,\wedge \,\varphi$ is T-satisfiable and (ii) $\mathsf {A}\,\wedge \,\varphi \,\models _{T}\,\mathsf {G}$. In other words, it asks for a formula $\varphi$ that is consistent with the axioms and when added to it allows the goal to be proven. Ideally, $\varphi$ should be as weak as possible and typically, it is expected to satisfy additional syntactic restrictions, such as, for instance, on its quantifier prefix. Abductive reasoning has gained a variety of applications recently, including extending knowledge bases for failed verification conditions [16] and invariant generation [17, 20]. Despite the usefulness of abductive reasoning, and the recent development of a few abductive reasoners, such as GPiD  [19] and Explain [15], general tools for automatic abductive inference are not yet mainstream.

Independently from the research on abduction, many high-performance general-purpose solvers for syntax-guided synthesis (SyGuS) have also been developed in the past decade. These solvers have been applied successfully in a number of domains, including the implementation of network protocols [36], data processing [22], and code optimization [29]. The performance and scalability of SyGuS solvers has made considerable progress in recently years, as demonstrated by an annual competition [4].

In this paper, we investigate scalable approaches to solving the abduction problem using (enumerative) syntax-guided synthesis techniques. We impose no requirements on the background theory T other than it must be supported by an existing SMT solver and amenable to syntax-guided synthesis, as we explain in more detail later. Our immediate goal is to leverage the power of syntax-guided synthesis solvers. Our longer term goal is to standardize the interface for these solvers for abduction problems and make them available to users of program analysis and automated reasoning who would benefit from high performance automated reasoning systems for abduction.

Contributions

• We introduce a novel procedure for solving abduction problems using enumerative syntax-guided synthesis.

• We give an extension of the procedure that is capable of generating progressively weaker solutions to a given abduction problem.

• We provide an implementation of these techniques in cvc4sy  [31], a state-of-the-art SyGuS solver implemented within the SMT solver cvc4  [8], and discuss several experiments we designed to test its effectiveness. We show that it has compelling advantages with respect to to other approaches for abduction including those implemented in Explain  [15] and GPiD  [19].

2 Preliminaries

We work in the context of many-sorted first-order logic with equality ($\simeq$) and assume the reader is familiar with the notions of signature, terms, and so on (see, e.g., [21]). A theory is a pair $T=(\varSigma ,I)$ where $\varSigma$ is a signature and I is a non-empty class of $\varSigma$-interpretations, the models of T, that is closed under variable reassignment (i.e., every $\varSigma$-interpretation that differs from one in I only in how it interprets the variables is also in I) and isomorphism. A $\varSigma$-formula $\varphi$ is T -satisfiable (respectively, T -unsatisfiable) if it is satisfied by some (resp., no) interpretation in I. A satisfying interpretation for $\varphi$ is a model $\varphi$. A formula $\varphi$ is valid in T (or T -valid), written $\models _{T}\varphi$, if every model of T is a model of $\varphi$. We write $\varphi [\varvec{x}]$ for a tuple $\varvec{x}$ of distinct variables to indicate that the free variables of $\varphi$ occur in $\varvec{x}$. Given $\varphi [\varvec{x}]$, we write $\varphi [\varvec{t}]$ to denote the result of replacing every occurrence of every variable of $\varvec{x}$ in $\varphi$ with the corresponding term in the tuple $\varvec{t}$. We write conjunctions of formulas as sets.

Syntax-Guided Synthesis (SyGuS). Syntax-guided synthesis [2] is a recent paradigm for automated synthesis that combines semantic and syntactic restrictions on the space of solutions. Specifically, a SyGuS problem for a function f in a theory T consists of

• 1. semantic restrictions, a specification given by a (second-order) T-formula of the form $\exists f.\, \forall \varvec{x}.\, \varphi [ f, \varvec{x} ]$, and

• 2. syntactic restrictions on the solutions for f, given by a context-free grammar $\mathcal {R}$.

The grammar $\mathcal {R}$ is a triple $( s_0, S, R )$ where $s_0$ is an initial symbol, S is a set of symbols with $s_0 \in S$, and R is a set of production rules of the form $s \rightarrow t$, where $s \in S$ and t is a term built from the symbols in the signature of theory T, free variables, and symbols from S. The rules define a rewrite relation over such terms, also denoted by $\rightarrow$, as expected. We say a term t is generated by $\mathcal {R}$ if $s_0 \rightarrow ^*t$ where $\rightarrow ^*$ is the reflexive-transitive closure of $\rightarrow$ and t does not contain symbols from S. For example, the terms x, $( x + x )$ and $( ( 1 + x ) + 1)$ are all generated by the grammar $\mathcal {R}= ( \mathsf {I}, \{ \mathsf {I}\}, \{ \mathsf {I}\rightarrow x, \mathsf {I}\rightarrow 1, \mathsf {I}\rightarrow (\mathsf {I}+ \mathsf {I}) \} )$. A solution for the SyGuS problem for f is a lambda term $\lambda \varvec{x}.e$ of the same type as f such that (i) $\forall \varvec{x}.\,\varphi [\lambda \varvec{x}.e,\, \varvec{x}]$ is T-valid and (ii) e is generated by $\mathcal {R}$.

A number of recent approaches for the syntax-guided synthesis problem exist that target specific classes of semantic and syntactic restrictions, including programming-by-examples [22], single invocation conjectures [32], and pointwise specifications [5, 27]. General purpose methods for solving the syntax-guided synthesis problem are generally based on enumerative counterexample-guided inductive synthesis (CEGIS) [34, 35]. Enumerative approach uses a grammar to generate candidate solutions systematically based on some term ordering, typically term size (e.g., the number of non-nullary function applications in the term). The generated candidate solutions are then tested for correctness using a verification oracle (typically an SMT solver). This process is accelerated by the use of counterexamples for previously discarded candidates, i.e., valuations for the input variables $\varvec{x}$, or points, that witness the failure of those candidates to satisfy the specification. Despite its simplicity, enumerative CEGIS is the de-facto approach for solving the general class of SyGuS problems, as implemented in a several recent tools, notably cvc4sy  [31] and the enumerative solver ESolver  [3]. Its main downside remains scalability to cases where the required solution is very large. As we will show in Sect. 4, we present a more scalable procedure for the abduction problem that builds on top of enumerative CEGIS and is capable of quickly finding (conjunctive) solutions.

3 The Abduction Problem

In general, the abduction problem for a set $\mathsf {A}$ of axioms and a goal $\mathsf {G}$ is the problem of finding a formula $\mathsf {S}$ that is consistent with $\mathsf {A}$ and, together with $\mathsf {A}$, entails the goal. We refine the problem by restricting it to first-order logic and to a given background theory T, and also considering syntactic restrictions on the solution $\mathsf {S}$. We refer to this as the syntax-restricted abduction problem, which we formalize in the following definition.

Definition 1 (Abduction Problem)

The (syntax-restricted) abduction problem for a theory T, a conjunction $\mathsf {A}[\varvec{x}]$ of axioms, a goal $\mathsf {G}[\varvec{x}]$ and a grammar $\mathsf {R}$, where axioms and goal are first-order formulas, is that of finding a first-order formula $\mathsf {S}[\varvec{x}]$ such that:

• 1. $\mathsf {A}\wedge \mathsf {S}\,\models _{T}\,\mathsf {G}$,

• 2. $\mathsf {A}\wedge \mathsf {S}$ is T-satisfiable, and

• 3. $\mathsf {S}$ is generated by grammar $\mathsf {R}$.

In practice, as in SyGuS, syntactic restrictions on the solution space may be used to capture user-requirements on the desired shape of a solution. They can also be used as a mechanism for narrowing the search space to one where one believes the solver is likely to find a solution. Observe that the formulation of the problem includes the case with no syntactic restriction as a trivial case of a grammar that accepts all formulas in the signature of the theory T. In the abduction solver we have developed for this work, the syntax restriction is optional. When it is missing, a grammar generating the full language is constructed internally automatically.

Syntax-restricted abduction bears a strong similarity to SyGuS.¹ We exploit this similarity by leveraging much of the technology we developed for SyGuS, with the goal of achieving generality and scalability.

Normally, an abduction problem admits many solutions. Thus, it may be useful to look for solutions that optimize certain criteria, such as generality with respect to entailment in T, or minimality with respect to size or number of free variables. Our evaluation contains several case studies where we explore this aspect in detail.

Recent Applications. Abduction has a long history in logic and automatic reasoning (see, e.g., [24]). More recently, it has found many useful applications in program analysis. It has been used for identifying the possible facts a verification tool is missing to either discharge or validate a verification condition [16], inferring library specifications that are needed for verifying a client program [37], and synthesizing specifications for multiple unknown procedures called from a main program [1]. Other applications of abduction includes loop invariant generation [17, 20], where it is used to iteratively strengthen candidate solutions until they are inductive and strong enough to verify a program, and compositional program verification [25], where it is used for inferring not only loop invariants but also preconditions required for the invariants to hold. Abductive inference has also been applied to modular heap reasoning [12], and the synthesis of missing guards for memory safety [18].

4 Abduction via Enumerative Syntax-Guided Synthesis

In this section, we fix a theory T and describe our approach for solving the abduction problem in T using enumerative syntax-guided synthesis. We first present a basic procedure for abduction in the following section, and then extend it to generate (conjunctive) solutions in a highly scalable manner. We then describe how either approach can be extended to be incremental so that it constructs progressively logically weaker solutions over time. For simplicity, we restrict ourselves to abduction problems where axioms, goals, and solutions are quantifier-free. Note, however, that the procedure can be used for abduction problems where these components are quantified, as long the restrictions below (lifted to quantified formulas) are satisfied.

Requirements on $\varvec{T}$. We assume that the T-satisfiability of quantifier-free formulas is decidable. For each sort of T, we also assume a distinguished set of variable-free terms of that sort which we call values (e.g., numerals and negated numerals in the case of integer arithmetic) such that every T-satisfiable formula is satisfied by a valuation of its free variables to sort elements denoted by values. Finally, we require the availability of a computable function $\mathsf {Eval}$ that takes a first-order formula $\varphi [\varvec{x}]$ and a tuple $\varvec{p}$ of values of the same length as $\varvec{x}$, and returns $\top$ if $\varphi [\varvec{p}]$ is T-satisfiable and $\bot$ otherwise. These restrictions are met by most theories used in Satisfiability Modulo Theories (SMT).

Fig. 1.

Basic procedure for the abduction problem for axioms $\mathsf {A}$, goal $\mathsf {G}$ and grammar $\mathsf {R}$.

4.1 Enumerative Counterexample-Guided Inductive Synthesis for Abduction

We start with a basic CEGIS-style synthesis procedure for solving the syntax-restriction abduction problem where points that represent counterexamples for candidate solutions are cached and used to discard subsequent candidates. The procedure is presented in Fig. 1. It takes as input: axioms $\mathsf {A}$, goal $\mathsf {G}$ and grammar $\mathsf {R}$, and maintains an internally set $\mathsf {P}$ of points that satisfy the axioms and falsify the goal. On line 3, it invokes the stateful sub-procedure $\mathsf {NextEnum}(\mathsf {R})$ which enumerates the formulas generated by grammar $\mathsf {R}$ based on enumerative techniques used in SyGuS solvers. We will refer to the return formula c as the current candidate solution. Then, using the (fast) evaluation function $\mathsf {Eval}$, it checks at line 4 that c is satisfied by none of the counterexample points in $\mathsf {P}$. If the check fails, the procedure discards c and loops back to line 3 because adding c to $\mathsf {A}$ would definitely be not enough to entail $\mathsf {G}$. If the check succeeds, it also checks, at line 5, whether $c \wedge \mathsf {A}\wedge \lnot \mathsf {G}$ is T-satisfiable. If so, it obtains a witness point $\varvec{p}$ for the satisfiability, adds it to current set of points $\mathsf {P}$ on line 6, and discards c; otherwise, it checks that c is consistent with $\mathsf {A}$ before returning it as a possible solution.

Example 1

Let T be the theory of linear integer arithmetic with the usual signature. Let $\mathsf {A}$ be the set $\{ y \geqslant 0 \}$, let $\mathsf {G}$ be the set $\{ x + y + z \geqslant 0 \}$, and assume $\mathsf {R}$ is a grammar generating all linear arithmetic atomic formulas over the variables x, y, z. The results of the procedure are summarized in the table below. We provide, for each iteration, the candidate c generated by syntax-guided enumeration on line 3, the Boolean value of the conditions on lines 4, 5 and 7 of the procedure when applicable, and the point (x, y, z) added to $\mathsf {P}$ in when the condition on line 5 evaluates to true. The last column specifies the solution returned on that iteration if any.

$$ \begin{array}{lcccccl} \\ \hline \# &{} c &{} \text { line 4 } &{} \text { line 5 } &{} p \in \mathsf {P}&{} \text { line 7 } &{} \text {return} \\ \hline 1 &{} x \geqslant 0 &{} \mathrm {true}&{} \mathrm {true}&{} (0,0,-1) \\ 2 &{} x< 0 &{} \mathrm {true}&{} \mathrm {true}&{} (-1,0,0) \\ 3 &{} y \geqslant 0 &{} \mathrm {false}\\ 4 &{} y< 0 &{} \mathrm {true}&{} \mathrm {false}&{} &{} \mathrm {false}\\ 5 &{} z \geqslant 0 &{} \mathrm {false}\\ 6 &{} z< 0 &{} \mathrm {false}\\ 7 &{} x+y \geqslant 0 &{} \mathrm {false}\\ 8 &{} x+y < 0 &{} \mathrm {false}\\ 9 &{} x+z \geqslant 0 &{} \mathrm {true}&{} \mathrm {false}&{} &{} \mathrm {true}&{} x+z \geqslant 0 \\ \hline \end{array} $$

On the first iteration, the syntax-guided enumeration generates the formula $x \geqslant 0$ as the candidate solution c. This fails to imply the goal, specifically, with $(x,y,z) = (0,0,-1)$ the axioms and c are satisfied and the goal is falsified. The second candidate fails for similar reasons for point $(-1,0,0)$. The check on line 4 fails for five of the next six candidates, with the exception of the candidate $y < 0$. This candidate is falsified by both points in $\mathsf {P}$ but it must be discarded since it is inconsistent with the axioms (line 7). Finally, the candidate $x+z \geqslant 0$ generated on the ninth iteration passes all the tests and is returned as a solution for this abduction problem.    $\square$

4.2 A Procedure for Abduction Based on Unsat Core Learning

This section extends the procedure from Fig. 1 with techniques that make it scalable when the intended solution to the abduction problem is a conjunction of formulas. The procedure is applicable to cases where the language generated by grammar $\mathsf {R}$ is closed under conjunction. In essence, the procedure in this section applies when $s_0 \rightarrow s_0 \wedge s_0$ is a production rule in $\mathsf {R}$ where $s_0$ is the start symbol of $\mathsf {R}$. However, it avoids enumerating conjunctive formulas directly, preferring instead to generate them as sets of (non-conjunctive) formulas.

Fig. 2.

Procedure for the abduction problem for $\mathsf {A}$, $\mathsf {G}$ and $\mathsf {R}$ based on unsat core learning.

This procedure is presented in Fig. 2. Similarly to the basic procedure from the previous section, it maintains a set of points $\mathsf {P}$ that satisfy the axioms and falsify the goal. Additionally, the new procedure maintains a set $\mathsf {E}$ of enumerated formulas, and a set $\mathsf {U}$ of subsets of $\mathsf {E}$ that are inconsistent with the axioms. The procedure modifies to each of these three sets during the course of its run. Each loop iteration attempts to construct a set $\mathsf {C}$ of formulas whose conjunction is a solution to the abduction problem. This is in contrast to the basic procedure from Fig. 1 which considers only individual formulas as candidate solutions.

To construct the candidate set $\mathsf {C}$, the procedure uses a helper function $\mathsf {EnsureCexFalsify}$ which ensures that (i) $\mathsf {C}$ is non-empty, (ii) the conjunction of the formulas in $\mathsf {C}$ is falsified by each point in $\mathsf {P}$ and (iii) no subset of $\mathsf {C}$ occurs in $\mathsf {U}$. The first condition is to ensure that the candidate is generated by the grammar. The second condition ensures that $\mathsf {C}$ along with our axioms suffices to prove the goal. The third condition ensures that $\mathsf {C}$ is consistent with the axioms. If we are able to successfully construct a candidate solution set $\mathsf {C}$, then line 6 checks whether that candidate indeed suffices when added to the axioms to show the goal. If it does not, we add a counterexample point to $\mathsf {P}$; otherwise, we construct a (ideally minimal) subset of $\mathsf {C}_{\textit{min}}$ of $\mathsf {C}$ that also suffices to show the goal. This information can be readily computed by an SMT solver [10] with support respectively for model generation and for unsatisfiable core generation [13], two features common to most modern solvers, including cvc4. We then check whether $\mathsf {C}_{\textit{min}}$ is consistent with our axioms. If it is consistent, we return it as a solution to the abduction problem; if it is not, we add some subset of it to $\mathsf {U}$ that is also inconsistent with the axioms, where again the subset can be computed by an SMT solver with support for unsatisfiable cores. Adding such subset amounts to learning that subset should never be included in future candidate solutions. To maintain the invariant that no subset of $\mathsf {C}$ occurs in $\mathsf {U}$, we remove one enumerated formula $e \in u$ from $\mathsf {C}$ on line 12. In the case where a point is added to $\mathsf {P}$ (line 15) or when an unsat core is added to $\mathsf {U}$ (line 12), we run the method $\mathsf {EnsureCexFalsify}$ starting from the current resultant set $\mathsf {C}$. This will force the procedure to try to construct a new candidate solution based on the set $\mathsf {E}$. When this strategy fails to construct a candidate, the inner loop terminates and the next formula is added to $\mathsf {E}$ based on syntax-guided enumeration.

We now revisit Example 1. As demonstrated in this example, $\mathsf {GetAbductUCL}$ is often capable of generating solutions to the abduction problem faster than the one from Fig. 1, albeit those solutions may be logically stronger.

Example 2

We revisit Example 1, where $\mathsf {A}$ is the set $\{ y \geqslant 0 \}$ and $\mathsf {G}$ is $\{ x + y + z \geqslant 0 \}$. A run of the procedure from Fig. 2 is summarized in the table below. We list iterations of the outer loop of the procedure (lines 2–18) in the first column of this table. For each iteration, we provide the formula that is added to our pool $\mathsf {E}$ (line 3), and the considered candidate set $\mathsf {C}$ upon a successful call to $\mathsf {EnsureCexFalsify}$. Notice that the inner loop of the procedure may consider multiple candidates $\mathsf {C}$ for a single iteration of the outer loop. For each candidate, when applicable, we give the result of the evaluation of the condition on line 6, the point p added to $\mathsf {P}$ if that condition is false (line 15), the minimal candidate set $\mathsf {C}_{\textit{min}}$ constructed on line 7, the evaluation of the condition on line 8, the set of formulas added to our set of unsatisfiable cores if that condition is false (line 12), and finally the formula (if any) returned as a solution (line 9).

$$ \begin{array}{lcccccccl} \\ \hline \# &{} e \in \mathsf {E}&{} \mathsf {C}&{} \text { line 6 } &{} p \in \mathsf {P}&{} \mathsf {C}_{\textit{min}}&{} \text { line 8 } &{} u \in \mathsf {U}&{} \text { return } \\ \hline 1 &{} x \geqslant 0 &{} \{ x \geqslant 0 \} &{} \mathrm {false}&{} ( 0,0,-1) \\ 2 &{} x< 0 &{} \{ x< 0 \} &{} \mathrm {false}&{} (-1,0,0) \\ &{} &{} \{ x< 0, x \geqslant 0 \} &{} \mathrm {true}&{} &{} \mathsf {C}&{} \mathrm {false}&{} \{ x< 0, x \geqslant 0 \} \\ 3 &{} y \geqslant 0 &{} \\ 4 &{} y< 0 &{} \{ y< 0 \} &{} \mathrm {true}&{} &{} \mathsf {C}&{} \mathrm {false}&{} \{ y < 0 \} \\ 5 &{} z \geqslant 0 &{} \{ x \geqslant 0, z \geqslant 0 \} &{} \mathrm {true}&{} &{} \mathsf {C}&{} \mathrm {true}&{} &{} x \geqslant 0 \wedge z \geqslant 0 \\ \hline \end{array} $$

We assume the same ordered list of formulas enumerated from Fig. 1. On the first iteration, we add $x \geqslant 0$ to our pool of enumerated formulas $\mathsf {E}$. The helper function $\mathsf {EnsureCexFalsify}$ constructs the set $\mathsf {C}= \{ x \geqslant 0 \}$ since (vacuously) it is true for all points in $\mathsf {P}$. Similar to the first iteration of Fig. 1, on line 6 we learn that $x \geqslant 0$ does not suffice with our axioms to show the goal; a counterexample point is $(x,y,z) = ( 0,0,-1)$ which is added to $\mathsf {P}$. Afterwards, $\mathsf {EnsureCexFalsify}$ is not capable of constructing another $\mathsf {C}$ since there are no other formulas in $\mathsf {E}$. In contrast to Fig. 1 which discards the formula $x \geqslant 0$ at this point, here it remains in $\mathsf {E}$ and can be added as part of $\mathsf {C}$ in future iterations.

On the second iteration, we add $x < 0$ to our pool. We check the candidate set $\mathsf {C}= \{ x < 0 \}$, which fails to imply the goal for counterexample point $(x,y,z) = ( -1,0,0)$. To construct the next candidate set $\mathsf {C}$, we must find an additional formula from $\mathsf {E}$ that evaluates to false on this point (or otherwise we again would fail to imply our goal). Indeed, $x \geqslant 0 \in \mathsf {E}$ evaluates to false on this point, and thus $\mathsf {EnsureCexFalsify}$ returns the set $\{ x < 0, x \geqslant 0 \}$. This set suffices to prove the goal given the axioms, that is, the condition on line 6 succeeds; the unsatisfiable core $\mathsf {C}_{\textit{min}}$ computed for this query is the same as $\mathsf {C}$. However, on line 8, we learn that this set is inconsistent with our axioms (in fact, the set by itself is equivalent to false). On line 12, we add $\{ x < 0, x \geqslant 0 \}$ to $\mathsf {U}$. In other words, we learn that any solution that contains both these formulas is inconsistent with our axioms. Learning this subset will help prune later candidate solutions. The procedure on this iteration proceeds by removing one of these formulas from our candidate solution set $\mathsf {C}$. Subsequently the helper function $\mathsf {EnsureCexFalsify}$ cannot construct a new candidate subset due to $\{ x < 0, x \geqslant 0 \} \in \mathsf {U}$ and since no other formulas occur in $\mathsf {E}$.

On the third iteration, $y \geqslant 0$ is added to our pool. However, no candidate solution can be constructed, where notice that $y \geqslant 0$ evaluates to $\top$ on both points in $\mathsf {P}$. On the fourth iteration, $y < 0$ is added to our pool and the candidate solution set $\{ y < 0 \}$ is constructed, where notice that this formula evaluates to $\bot$ on both points in $\mathsf {P}$. This formula suffices to show the goal from the axioms, but is however inconsistent with our axioms. Thus, $\{ y < 0 \}$ is added to our set of unsatisfiable cores $\mathsf {U}$. In other words, we have learned that no solution $\mathsf {C}$ should include the formula $y < 0$ since it is alone inconsistent with our axioms.

On the fifth iteration, $z \geqslant 0$ is added to our pool. The only viable candidate that falsifies all points in $\mathsf {P}$ and does not contain a subset from $\mathsf {U}$ is $\{ x \geqslant 0, z \geqslant 0 \}$. This set is a solution to the abduction problem and so the formula $x \geqslant 0\,\wedge \,z \geqslant 0$ is returned. Due to our assumption that $\mathsf {R}$ admits conjunctions, this formula meets the syntax restrictions of our grammar. A run of this procedure required the enumeration of only 5 formulas before finding a solution whereas the basic one in Fig. 1 required 9.    $\square$

While the solution in the previous example $x \geqslant 0 \wedge z \geqslant 0$ was found in fewer iterations, notice that it is logically stronger than the solution $x + z \geqslant 0$ produced in Example 1, since $x \geqslant 0 \wedge z \geqslant 0$ entails $x + z \geqslant 0$ but not vice versa. We remark that the main advantage of procedure Fig. 2 is that is typically capable of generating any feasible solution to the abduction problem faster than the procedure from Fig. 1. This is especially the case if the only solutions to the abduction problem consist of a large conjunction of literals of small term size $\ell _1 \wedge \ldots \,\wedge \,\ell _n$. The basic procedure does not scale to this case, if its enumeration is by formula size, since it will have to wait until the conjunction above is enumerated as an individual formula.

Furthermore, we remark that procedure in this section can be configured to have the same solution completeness guarantees as the basic procedure from Fig. 1. In particular, our choice of e in the $\mathsf {EnsureCexFalsify}$ method chooses the most recently enumerated formula when the candidate pool $\mathsf {C}$ is empty. Since a single loop of the procedure is terminating and due to the above policy for selection, the procedure will terminate in the worst case when the enumerated pool $\mathsf {E}$ contains a formula that by itself is the solution to the synthesis conjecture.

4.3 Incremental Weakening for Abduction

The user may be interested in obtaining an abduction problem solution that maximizes some criteria and is not necessarily the first one discovered by (either of) the procedures we have described so far. In this section, we describe an extension to our approach for abduction that maintains the advantage of returning solutions quickly while still seeking to generate the best solution in the long run according to metric such as logical weakness.

We observe that it is straightforward to extend our enumerative syntax-guided approach to generate multiple solutions. We are interested, however, in generating increasingly better solutions over time. We briefly give an overview of how the procedures of Fig. 1 and Fig. 2 can be extended in this way and discuss a few relevant details of the extension. We focus on the problem of generating the logically weakest solution to the abduction problem in this section.

Figure 3 presents an incremental procedure for generating (multiple) solutions to a given abduction problem. The procedure requires that the language restriction $\mathsf {R}$ admit disjunctive formulas which is the case, for instance, if $s_0 \rightarrow s_0 \vee s_0$ is a production rule in $\mathsf {R}$ where $s_0$ is again the start symbol. It maintains a formula $\mathsf {S}$ that, when not $\bot$, represents the logically weakest solution to the abduction problem known so far. In its main loop, on line 3, the procedure calls one of the previous procedures for generating single solutions to the abduction problem (written $\mathsf {GetAbduct}^{*}$). Line 4 then checks whether a new solution can be constructed that is logically weaker with respect to the axioms than the current one. In particular, this is the case if $\mathsf {C}\wedge \mathsf {A}\wedge \lnot \mathsf {S}$ is T-satisfiable, which means that there is at least one point that satisfies the current candidate but not the current solution $\mathsf {S}$. In that case, the current solution $\mathsf {S}$ is updated to $\mathsf {S}\vee \mathsf {C}$, which is by construction guaranteed to also be a solution to the abduction problem. If no such point can be found, then $\mathsf {C}$ is redundant with respect to the current candidate solution since it does not generalize it. Optionally, the procedure may learn a subset u of $\mathsf {C}$ that is also redundant with respect to the current candidate solution. This subset can be learned as an unsatisfiable core when using the procedure $\mathsf {GetAbductUCL}$ as the sub-procedure on line 3.

4.4 Implementation Details

We implemented the procedures above in the state-of-the-art SMT solver cvc4  [8]. cvc4 incorporates a SyGuS solver, cvc4sy, implementing several strategies for enumerative syntax-guided synthesis [31]. It accepts as input both SMT problems written in the SMT-LIB version 2.6 format [9], and synthesis problems written in the SyGuS version 2.0 format [30]. SMT-LIB version 2.6 is a scripting language that allows one to assert a formula F to the solver with a command of the form (assert F ). The solver checks the satisfiability the formulas asserted so far in response to the command (check-sat). We extended cvc4 ’s SMT-LIB parser to support also commands of the form (get-abduct p G R ) where p is a symbol, the identifier of the expected solution formula; G is a formula, the goal of the abduction problem; and the optional R is a grammar expressed in the SyGuS version 2.0 format. This command asks the solver to find a formula that is a solution to the abduction problem (A, G), where A, standing for the set of axioms, consists of the conjunction of the currently asserted formulas. The expected response from the solver is a definition of the form (define-fun p () Bool S ) where p is the identifier provided in the first argument of get-abduct and S is a formula that solves the abduction problem.

Fig. 3.

Incremental abduction procedure for axioms $\mathsf {A}$, goal $\mathsf {G}$ and grammar $\mathsf {R}$.

Internally, invoking a get-abduct command causes a synthesis conjecture to be constructed and passed to cvc4sy. The latter normally accepts conjectures of the form $\exists f.\, \forall \varvec{x}.\, \varphi [f,\varvec{x}]$ where $\varphi$ is quantifier-free. Thus, we must pass the abduction problem in two parts: (i) the synthesis conjecture $\exists P.\, \forall \varvec{x}.\, \lnot ( P( \varvec{x} ) \wedge A \wedge \lnot G )$ where $\varvec{x}$ collects the free variables of A and of G,² stating that the expected solution P along with the axioms A must entail the goal G, and (ii) a side condition $\exists \varvec{x}.\, P( \varvec{x} ) \wedge A$ stating that P must be consistent with the axioms. The synthesis conjecture is of a form that can be readily handled by cvc4sy and processed using its current techniques. We have modified it so that it considers the side condition as well during solving, as described in Figs. 1 and 2.

The procedure in Fig. 2 is implemented as a strategy on top of the basic enumerative CEGIS loop of cvc4sy. We give some noteworthy implementation details here. Firstly, we use a data structure for efficiently checking whether any subset of $\mathsf {C}$ occurs in our set of unsatisfiable cores $\mathsf {U}$, which keeps the sets in $\mathsf {U}$ in an index and is traversed dynamically as formulas are added to $\mathsf {C}$. We chose enumerated formulas on line 2 of $\mathsf {EnsureCexFalsify}$ by selecting first the most recently generated formula, and then a random one amongst those that meet the criteria to be included in $\mathsf {C}$. Finally, since the number of candidate solutions can be exponential in the worst case for a given iteration of the inner loop of this procedure, we use a heuristic where formulas cannot be added to $\mathsf {C}$ more than once in the same iteration of the loop, making the number of candidate sets tried on a given iteration linear in the size of $\mathsf {E}$ in the worst case.

5 Evaluation

We evaluated our approach³ in comparison with cvc4sy ’s enumerative CEGIS, a general purpose synthesis approach, as well as with GPiD  [19] and Explain  [15], state-of-the-art solvers for similar abduction problems as the one defined here. In the comparison below, we refer to the basic procedure from Fig. 1 as cvc4sy +b and the one from Fig. 2 as cvc4sy +u. Experiments ran on a cluster with Intel E5-2637 v4 CPUs, Ubuntu 16.04. Each execution of a solver on a benchmark was provisioned one core, 300 s and 8 GB RAM.

5.1 Benchmarks

Since abduction tools are generally focused on specific application domains, there is no standard language or benchmark library for evaluation. Moreover, these tools use abduction as part of a larger verification toolchain. As here we did not target a specific application but rather the abduction problem as a whole, an evaluation with their benchmarks would require integrating our solver in the tools as an alternative abduction engine. This was not feasible due to either the source code not being available or the verification and abduction engines being too tightly coupled for us to use our solver as an alternative. Thus we had to generate our own abduction benchmark sets. We did so using benchmarks relevant for verification from SMT-LIB [9], the standard test suite for SMT solvers. We chose as a basis the SMT-LIB logics QF_LIA, QF_NIA, and QF_SLIA due to their relevance for verification. For QF_NIA, we focus on the benchmark family VeryMax and on kaluza for QF_SLIA. In QF_LIA we excluded benchmark families whose benchmarks explode in size without the let operator. This was necessary to allow a comparison with Explain, whose parser does not fully support let, on let-free benchmarks. We considered both benchmarks that were (annotated as) satisfiable and unsatisfiable for generating abduction problems, according to the following methodology.

Given a satisfiable SMT-LIB problem ⁴ in the theory T, we see it as an encoding of a validity problem that could not be proven. We consider the abduction problem where $\mathsf {G}$ is $\lnot \psi _n$, $\mathsf {A}$ is , and $\mathsf {R}$ is a grammar that generates any quantifier-free formula in the language of T over the free variables of $\mathsf {G}$ and $\mathsf {A}$. A solution $\mathsf {S}$ to this problem allows the validity of $\varphi$ to be proven, since $\varphi \wedge \mathsf {S}$ is unsatisfiable.

Given an unsatisfiable SMT-LIB problem $\varphi$, let be a minimal unsatisfiable core for this formula, i.e. any conjunctive set , for some $\psi \in U$, is satisfiable. Let $\psi _{\mathrm {max}}$ be U’s component with maximal size. We will call $\psi _{\mathrm {max}}$ the reference to the abduction problem. We consider the abduction problem whose $\mathsf {G}$ is $\lnot \psi _{\mathsf {G}}$, for some $\psi _{\mathsf {G}}\in U$ and $\psi _{\mathsf {G}}\ne \psi _{\mathrm {max}}$, whose axioms $\mathsf {A}$ are and $\mathsf {R}$ as before is a grammar that generates any formula in the language of T over the free variables of $\mathsf {G}$ and $\mathsf {A}$. A solution $\mathsf {S}$ to this problem allows proving the validity of , since is unsatisfiable. Solving this abduction problem amounts to “completing” the original unsatisfiable core with the further restriction that this completion is at least as weak as the reference, as well as consistent with all but one of the other core components, seen as axioms for the abduction problem.

From satisfiable SMT-LIB benchmarks we generated 2025 abduction problems in QF_LIA, 12214 in QF_NIA and 11954 in QF_SLIA. For unsatisfiable benchmarks we were limited not only by the benchmark annotations but also by being able to find minimal unsatisfiable cores. We used the Z3 SMT solver [14] to generate minimal unsatisfiable cores with a 120s timeout. Excluding benchmarks whose cores had less than three assertions (so we could have axioms, a goal and a reference), we ended up with 97 problems in QF_LIA, 781 in QF_NIA and 2546 in QF_SLIA. We chose the reference as the component of the unsatisfiable core with maximal size and the goal as the last formula in the core (viewed as a list) after the reference was removed.

Table 1. Comparison of abduction problems from originally SAT SMT-LIB benchmarks.

5.2 Finding Missing Assumptions in SAT Benchmarks

In this section we evaluate how effective cvc4sy +b and cvc4sy +u are in (i) finding any solution to the abduction problem and (ii) finding logically weak solutions. The evaluation is done on the abduction problems produced from satisfiable SMT-LIB benchmarks as above. Results are summarized in Table 1. The number of solved problems corresponds to the problems for which a given cvc4 configuration could find a solution within 300s. cvc4sy +u solves a significant number of problems more than cvc4sy +b in all logics but QF_LIA. In both QF_LIA and QF_NIA we can see a significant orthogonality between both approaches. We attribute these both to the fragility of integer arithmetic reasoning, where the underlying ground solver checking the consistency of candidate solutions is greatly impacted by the shape of the problems it is given. Overall, the procedure in cvc4sy +u leads to a better success rate than the basic procedure in cvc4sy +b. Solution strength was evaluated on commonly solved problems considering the solutions produced according to the incremental procedures shown in Sect. 4.3, in which the overall solution is a disjunction of individual solutions found over time. As expected, cvc4sy +u is able to solve more problems but at the cost of often producing stronger (and bigger) solutions than cvc4sy +b. This is particularly the case in QF_SLIA and QF_NIA, in which cvc4sy +u both solves many more problems and often finds stronger solutions.

5.3 Completing UNSAT Cores

Here we evaluate how effective cvc4sy +b and cvc4sy +u are in solving the abduction problem with the extra restriction of finding a solution that is at least as weak as a given reference formula. We use the abduction problems produced from unsatisfiable SMT-LIB benchmarks following the methodology of Sect. 5.1 as the basis for this evaluation.

Table 2. Comparison of abduction problems from originally UNSAT SMT-LIB benchmarks.

The results are summarized in Table 2. cvc4sy +b significantly outperforms cvc4sy +u in QF_SLIA, in which the references are very simple formulas (generally with size below 3), for which the specialized procedure of cvc4sy +u is not necessary. Overall, as in the previous section when checking who finds the weakest solution, cvc4sy +b has as advantage over cvc4sy +u for finding solutions as weak as the reference.

5.4 Comparison with Explain

Explain  [15] is a tool for abductive inference based on quantifier elimination. It accepts as input a subset of SMT-LIB and we extended it to support abduction problems as generated in Sect. 5.1. However, Explain imposes more restrictions to their solutions, only producing those with a minimal number of variables and for which every other solution with those variables is not stronger than it. Their rationale is finding “simple” solutions, according to the above criteria, which are more interesting to their applications. Since we do not apply these restrictions in cvc4, nor is in the scope of this paper incorporating them into our procedure, it should be noted that comparing cvc4 and Explain puts the latter at a disadvantage. We considered satisfiable SMT-LIB problems in the QF_LIA logic for our evaluation, as QF_LIA is better supported by Explain (Table 3).

Table 3. Comparison with Explain in 2025 abduction problems in QF_LIA

All problems solved by Explain are solved by cvc4sy +u. Of these 33 problems, cvc4sy +u, in incremental mode, finds a solution with the same minimal number of variables as Explain for 25 of them. Of the 8 problems to which it only finds solutions with more variables, in 4 of them the difference is of a single variable. All other 4 are in the slacks benchmark family, which contains crafted problems. A similar comparison occurs with cvc4sy +b. This shows that even though cvc4 is not optimized to minimize the number of variables it its solutions, it can still often finds solutions that are optimal (or close to optimal) according to Explain ’s criteria, while solving a much larger number of problems with a fully general approach.

5.5 Comparison with GPiD

We also compared cvc4 with GPiD  [19], a framework for generating implicates, i.e. logical consequences of formulas. As Echenim et al. say in their paper, negating the implicate of a satisfiable formula $\varphi$ yields the “missing hypothesis” for making $\varphi$ unsatisfiable. Therefore GPiD solves a similar problem to that of Sect. 5.2, differing by they always considering an empty set of axioms and the whole original formula as the goal. Given this similarity, we compare the performance of GPiD in generating implicates for satisfiable benchmarks and of cvc4sy +b and cvc4sy +u in solving abduction problems generated from those same benchmarks. We did not consider the benchmarks from the previous sections because we were not able to produce abduces, which are the syntactic components GPiD uses to find implicates, for other logics using the tools in GPiD public repository⁵. Thus we restricted our analysis to 400 abduction problems produced, as per the methodology of Sect. 5.1, from satisfiable QF_UFLIA benchmarks that were used in [19]. Note however that the cvc4 configurations will require solutions to be consistent with all but the last assertion in the problems (which are the axioms in the respective abduction problem). Since that, as far as we know, this is not a requirement in GPiD, effectively cvc4sy +b and cvc4sy +u are solving a harder problem than GPiD. We formulated the abduction problem this way, rather than as with all assertions as goals, to avoid trivializing the abduction problem, for which the negation of the goal would always be a solution. Also note that the presence of uninterpreted functions in the abduction problem requires solutions to be generated in a higher-order background logic, which cvc4 supports after a recent extension [7]. As in [19], we used GPiD ’s version with the Z3 backend. We present their results with (GPiD-1) and without (GPiD) the restriction to limit the set of abduces to size 1.

Table 4. Comparison with GPiD on 400 abduction problems in the QF_UFLIA logic.

Results are summarized in Table 4. cvc4sy +u significantly outperforms cvc4sy +b, both in the number of problems solved and in total time, besides being almost 20% faster on commonly solved problems. We also see that solution finding in GPiD is heavily dependent on which abduces are considered when building solutions, as it solves almost all benchmarks when limited to abduces of size 1 but barely half when unrestricted. It should also be noted that GPiD takes pre-computed abduces, whose production time is not accounted for in the evaluation. Despite this, cvc4sy +u is only on average 30% slower on commonly solved problems than GPiD-1 and solves many more problems than GPiD. The big variation of GPiD results in terms of what pre-determined set of candidates can be used in the computation is a severe limitation of their tool. Similarly, while the method proposed in [19] is theory agnostic, their tooling for producing abduces imposes strong limitations on the usage of GPiD for theories other than QF_UFLIA.

6 Related Work

The procedure introduced in Sect. 4.2 based on unsat core learning follows a recent trend in enumerative syntax-guided synthesis solving that aims to improve scalability by applying divide-and-conquer techniques, where candidate solutions are built from smaller enumerated pieces rather than being directly enumerated. While previous approaches, both for pointwise [5, 27] and for unrestricted specifications [6], have targeted general-purpose function synthesis, we specialize divide and conquer for solving the abduction problem with a lean (see Sect. 4.4) and effective (see Sect. 5) procedure.

Abductive inference tools for the propositional case include the AbHS and AbHS+ tools [26, 33], based on SAT solvers [11] and hitting set procedures, and the Hyper [23] tool, that includes a series of algorithmic improvements over the former, and uses a MaxSAT solver for computing the hitting set. Like AbHS, $\mathsf {GetAbductUCL}$ checks entailment and consistency using two separate calls to the underlying solver, and uses its failures for the selection of new candidates. In contrast, $\mathsf {GetAbductUCL}$ keeps this information in two dedicated data structures rather than encoding it with an implicit hitting set. Another significant difference is that the set of hypotheses in the propositional case is fixed and finite, whereas in our setting it is generated dynamically from a grammar. More general approaches, to which our work bears more resemblance and to which we provided an experimental comparison in Sect. 5, are GPiD [19] and Explain [15]. GPiD uses an off-the-shelf SMT solver as a black box to generate ground implicates. It can be used with any theory supported by the underlying SMT solver, similarly to our SyGuS-based approach. While we enumerate formulas that compose the solution for the abduction problem GPiD’s authors use abducibles, which are equalities and disequalities over the variables in the problem. They similarly build candidates in a refinement loop by combining abducibles according to consistency checks performed by an underlying SMT solver. They use an order on abducibles to guide the search, which is analogous to the enumeration order in enumerative synthesis. Explain on the other hand is built on top of an SMT solver for the theories of linear integer arithmetic and of equality with uninterpreted functions, although its abduction inference procedure in principle can work with any theory that admits quantifier elimination. The method implemented in Explain is based on first determining a subset of the variables in the abduction problem and trying to build the weakest solution over these variables via quantifier elimination, while computing minimal satisfying assignments to ensure that a found solution covers a minimal subset. This method, however, is not complete, as it can miss solutions. The tool also allows the user to specify costs for each variable, so that a given minimal set may be favored.

7 Conclusion

We have described approaches for solving the abduction problem using a modern enumerative solver for syntax-guided synthesis. Our evaluation shows that procedures based on enumerative CEGIS scale for several non-trivial abduction tasks, and have several compelling advantages with respect to other approaches like those used in Explain and GPiD. In several cases, it suffices to use a basic procedure for enumerative CEGIS to generate solutions to abduction problems that are optimal according to certain metrics. Moreover, the generation of feasible solutions can be complemented and accelerated via a procedure for generating conjunctions of enumerated formulas as shown in Fig. 2.

We believe that new abduction capabilities presented in this paper and implemented in cvc4 will be useful in all the applications of abduction we describe in Sect. 3. In addition, we see a number of promising applications in the context of SMT itself. For example, we plan to use abduction to generate useful conditional rewrite rules for SMT solvers. Many such rules are used internally by SMT solvers to simplify their input formulas by (equivalence-preserving) term rewriting. The manual identification and selection of good rewrite rules is a tedious and error-prone process. Abduction can be used to generalize a recent approach for the semi-automated development of rewrite rules [28] by synthesizing (most general) conditions under which two terms are equivalent. This in turn can be used to develop new solving strategies in the SMT solver based on those rewrite rules.