figure a

1 Introduction

Given a first-order logic formula with quantifiers, quantifier elimination (or QE) requires us to find a quantifier-free formula that is semantically equivalent to the given quantified formula. Not every first-order theory admits QE; however, several important ones do, and QE for several such theories are implemented in modern Satisfiability Modulo Theories (SMT) solvers (viz. [1, 9, 27, 31, 36]). QE in combinations of first-order theories is particularly challenging, and algorithms that achieve this for some theories used in practical applications have been reported in earlier works (e.g. [11, 38, 51]). However, QE (even approximate versions) in combinations of theories including non-linear real arithmetic (NRA) has proved more difficult. This is not surprising since QE over NRA is computationally challenging by itself [29]. In this paper, we add to the repertoire of practically efficient techniques for reasoning about NRA constraints by showing how NRA constraints over bounded variables can be approximated efficiently using a Boolean combination of real interval constraints. This yields a practical algorithm for approximately solving QE over NRA, and also allows us to bootstrap on existing QE techniques that work well for combinations of LRA and other theories (viz. [38]) to solve QE in combinations of theories including NRA.

At the heart of our approach lies a practically efficient technique for approximating a Boolean combination of polynomial inequalities over bounded reals with a Boolean combination of real interval constraints. This immediately yields a practically efficient approximate QE algorithm for NRA. This problem is also popularly called QE over reals (henceforth called QER). QER is a central problem in computer algebra and real algebraic geometry, with many practical applications, including control system design [35, 42, 46], program verification [14, 50, 62, 64, 66], analysis of hybrid systems [7, 79] and robot motion planning [53, 56, 77]. The study of QER has a long and storied history. Tarski first showed the decidability of QER in [71]. By the Tarski-Seidenberg theorem, the projection of a semi-algebraic set (i.e. solutions of a Boolean combination of polynomial inequalities) is always semi-algebraic [67, 71]. Hence, it suffices to eliminate existentially quantified variables from a conjunction of polynomial inequalities. A landmark result in this area was the development of the cylindrical algebraic decomposition (CAD) algorithm by Collins [28] in 1975. Over the past half century, CAD has remained one of the most important algorithms for QER, although several improvements have been proposed over the years. An excellent, albeit dated, survey of these algorithms can be found in [16, 29], while more recent works have been reported in [3, 12, 26, 49, 52, 57, 58, 65, 68]. The book by Basu, Pollack and Roy [10] is a definitive treatise on exact algorithms for QER and related problems. Over the years, practical scalability concerns have also motivated researchers to investigate versions of QER for special cases [32, 47, 48, 54, 59, 63, 75, 76]. Advances resulting from these efforts have been implemented in state-of-the-art tools, including open-source academic tools such as QEPCAD [13, 30], REDLOG [34], SMT-RAT [31] and SageMath [72], as well as commercial tools such as Mathematica [44, 68, 69] and Maple [25, 45].

The verification community has long been interested in QER, thanks to its many applications in problems related to automated reasoning. For example, QER for polynomial equalities and disequalities has been used to compute strongest post- and weakest pre-conditions of programs [14, 62], to compute abstract transformers for program statements [60], and for inductive assertion and program invariant generation [50, 66]. In hybrid systems verification, reach set computation has been shown to reduce to QER [7, 79]. In [78], QER has been used to find parametric optimal strategies for Markov decision processes. Quantifier elimination in mixed theories including the theory of linear real arithmetic (LRA) has been reported in several earlier works (see e.g. [11, 38, 51]). For example, [11] gives model based projection techniques for several combinations of theories and [38] gives e-graph based techniques for similar combinations. However, quantifier elimination (even approximate versions) in combinations of theories including NRA has remained elusive in practice, primarily because of the high-degree polynomials that result in general from QER.

Our algorithm provides strong guarantees of approximation and allows the user to trade off precision for performance. It builds upon the well-known theorem of Handelman [41] which characterizes positive polynomials over polytopes. This theorem has previously been used in developing static analysis methods for termination and runtime analysis [17,18,19, 43], cost analysis [15, 21, 23, 24, 70, 74], invariant generation [20], reachability [8, 73] and LTL verification [22], as well as program synthesis [4, 39]. See [6] for a comparison between the current work and [4]. Most of these approaches are template-based and use Handelman’s theorem to solve for unknown variables in their templates. In contrast, our approach is gridding-based and uses techniques similar to PROPhESY [33] but combines them with Handelman-based reasoning. The primary workhorse we use at the backend is a linear-programming (LP) solver, with occasional invocations of an SMT solver. This allows our method to scale well on many non-trivial examples. Our primary contributions are as follows:

  1. 1.

    We formalize two notions of approximation for QER, called \(\epsilon \)-approximation and \((\epsilon ,\delta )\)-approximation, that are motivated by practical applications and introduce union of (adaptively sized) hyperrectangles as a knowledge representation form for approximate QER. This allows us to compute \(\epsilon \)- and \((\epsilon ,\delta )\)-approximations of QER, for every \(\epsilon ,\delta > 0\), efficiently in practice.

  2. 2.

    We present an approach to over- and under-approximate NRA constraints with a Boolean combination of LRA constraints, where each dimension is bounded. Specifically, we use Handelman’s Theorem in combination with dynamic adaptive gridding to reduce the approximation problem to multiple linear programming (LP) instances, that are then discharged by a state-of-the-art LP solver.

  3. 3.

    We prove the soundness of our algorithm, and its completeness under two different settings. Assuming access to a sound and complete satisfiability oracle for polynomial inequalities (in practice, an SMT solver), we show that our algorithm produces an \(\epsilon \)-approximation of QER. Without access to the above oracle, and relying only on linear programming, we can obtain \((\epsilon ,\delta )\)-approximations of QER. Our notions of approximation for the original semi-algebraic set are closely related to those of [37]. Due to the special format of our approximation as a union of hyperrectangles, we obtain approximations of the projection set easily. Our approach extends the results of [55] which directly approximate the projection.

  4. 4.

    We apply this new algorithm to show how QE over theories involving Non-linear Real Arithmetic (NRA) can be reduced to QE over LRA and other theories, thereby making it possible to solve problems beyond the reach of state-of-the-art solvers.

  5. 5.

    We show the practical effectiveness of our algorithm through two sets of experiments with POQER – a tool that implements our algorithm. First, a comparison with state-of-the-art tools shows that POQER significantly outperforms available open-source tools that perform exact QER, even with small values of \(\epsilon \) and \(\delta \). Comparison with Mathematica, a commercial tool, shows that our tool almost always generates solutions (unions of hyperrectangles) that are easier to process subsequently than solutions generated by Mathematica. Second, we demonstrate how POQER can find approximate solutions for NRA+ADT benchmarks well beyond the reach of state-of-the-art SMT-solvers like Z3 and Z3EG.

2 Algorithm

In this section, we start by formalizing our quantifier elimination problem as computing a projection \(\pi (S)\) of a semialgebraic set \(S\). We then present the concept of \(\epsilon \)-inflations to overapproximate semialgebraic sets, in our case the projection \(\pi (S)\), to a desired level \(\epsilon \) of precision. This is followed by our algorithm which computes an \(\epsilon \)-approximation of \(\pi (S).\)

2.1 Problem Definition

Input Format. We are given a positive real number \(\epsilon \) and a finite set \(\mathcal {V}= \{v_1, v_2, \ldots , v_n\}\) of real-valued variables partitioned into two sets \(\mathcal {V}_1\) and \(\mathcal {V}_2\). Throughout this paper, we use the standard vector notation for valuations to variables and assume that \(\mathcal {V}_1\) comes before \(\mathcal {V}_2\) lexicographically. Our input also contains a formula \(\varphi \) from the grammar below:

$$ \begin{matrix} \varphi := \ell ~|~ \lnot \varphi ~|~ \varphi \wedge \varphi ~|~ \varphi \vee \varphi & \text {formulas}\\ \ell := f \ge 0 ~|~ f > 0 & \text {literals}\\ f \in \mathbb R[\mathcal {V}] & \text {polynomials} \end{matrix} $$

The input formula \(\varphi \) naturally defines the semialgebraic set

$$S:= \textit{SAT}(\varphi ) := \{ \textbf{x} \in \mathbb R^{n}~|~ \textbf{x} \models \varphi \}.$$

We assume the set \(S\) is bounded, i.e. there is a positive real number \(B\) given in the input such that for all \(\textbf{x} \in S,\) we have \(||\textbf{x}|| < B.\)

Projection. Given a set \(S\subseteq \mathbb R^{n},\) its projection \(\pi (S)\) onto \(\mathcal {V}_1\) is defined as

$$\pi (S) := \{ \mathbf {x_1} \in \mathbb R^{|\mathcal {V}_1|} ~|~ \exists \mathbf {x_2} \in \mathbb R^{|\mathcal {V}_2|} ~~(\mathbf {x_1}, \mathbf {x_2}) \in S\}.$$

Our goal is to approximate \(\pi (S).\) We now formalize this.

\(\epsilon \)-inflations and \(\epsilon \)-approximations. Given \(\epsilon > 0\) and a set \(T \subseteq \mathbb R^n,\) we define the \(\epsilon \)-inflation of T as

$$ \mathcal {I}_\epsilon (T) := \{ \textbf{x} \in \mathbb R^n ~|~ \exists \mathbf {x'} \in T~~||\textbf{x}-\mathbf {x'}|| < \epsilon \}. $$

In other words, \(\mathcal {I}_\epsilon (T)\) consists of all the points in T as well as points that are within a distance \(\epsilon \) to T. We say \(O \subseteq \mathbb R^n\) is an \(\epsilon \)-approximation of T iff \(T \subseteq O \subseteq \mathcal {I}_\epsilon (T).\) Intuitively, an \(\epsilon \)-approximation includes everything in the original set T and may also include some extra points, but these points are guaranteed to be within \(\epsilon \) distance to the boundary of T. In this work, we use the Euclidean norm, but our results are independent of the distance metric used and can be straightforwardly extended to other norms.

Output. Our algorithm outputs an \(\epsilon \)-approximation of \(\pi (S).\)

Fig. 1.
figure 1

A semi-algebraic set \(S\) (black) and its \(\epsilon \)-inflation (red) (Color figure online)

Full size image

Example. Figure 1 shows a semi-algebraic set in black and its \(\epsilon \)-inflation in red.

Hyperrectangles. A hyperrectangle \(H \subseteq \mathbb R^{n}\) is the set of points that satisfy the inequalities

$$ \psi _H := \left\{ \begin{matrix} \alpha _1 \le v_1 \le \beta _1\\ \alpha _2 \le v_2 \le \beta _2\\ \vdots \\ \alpha _n \le v_n \le \beta _n \end{matrix}\right. $$

where the \(\alpha _i\) and \(\beta _i\)’s are real constants and we have \(\beta _i > \alpha _i\) for every \(1 \le i \le n.\)

Literal Complements. Let \(\ell \) be a literal. We define its complement \(\overline{\ell }\) as follows:

$$ \overline{\ell } := \left\{ \begin{matrix} -f > 0 & \quad \quad \quad & \ell = (f \ge 0) \\ -f \ge 0 & & \ell = (f > 0) \\ \end{matrix}\right. $$

It is easy to see that \(\overline{\ell } \equiv \lnot \ell .\)

Ternary Evaluation. Let \(\varphi \) be a Boolean formula and L the set of literals appearing in \(\varphi .\) Consider a function \(\theta : L \rightarrow \{0, 1, ?\}\) that assigns a truth value to each literal. Here, ? models uncertainty. Based on the function \(\theta \), we define the evaluation of \(\varphi \) recursively as follows:

$$ \begin{matrix} [|\ell |]_\theta = \theta (\ell ) & ~~~~ & [|\varphi _1 \vee \varphi _2|]_\theta = \left\{ \begin{matrix} 1 & \quad \quad & [|\varphi _1|]_{\theta } = 1 \vee [|\varphi _2|]_\theta = 1 \\ 0 & & [|\varphi _1|]_\theta = 0 \wedge [|\varphi _2|]_\theta = 0 \\ ? & & \text {otherwise} \end{matrix}\right. \\ [|\lnot \varphi |]_\theta = \left\{ \begin{matrix} ? & & [|\varphi |]_\theta = ? \\ \lnot [|\varphi |]_\theta & \quad \quad & \text {otherwise} \\ \end{matrix}\right. & & [|\varphi _1 \wedge \varphi _2|]_\theta = \left\{ \begin{matrix} 1 & \quad \quad & [|\varphi _1|]_\theta = 1 \wedge [|\varphi _2|]_\theta = 1 \\ 0 & & [|\varphi _1|]_\theta = 0 \vee [|\varphi _2|]_\theta = 0 \\ ? & & \text {otherwise} \end{matrix}\right. \end{matrix} $$

Informally, we are going to use this kind of evaluation when we want to check whether a given \(\varphi \) holds over all points in a set (1), none of the points in the set (0) or potentially some of them (?). We say we are uncertain about \(\varphi \) when \([|\varphi |]_\theta = ?.\)

2.2 Our Overapproximation Algorithm

Oracles. Our algorithm is modular and relies on two oracles:

  • Implication Oracle: Given a hyperrectangle \(H \subseteq \mathbb R^{n}\) and a literal \(\ell ,\) this oracle checks whether \(\ell \) holds at every point in H. Equivalently, as \(\psi _H\) is the formula defining H,  it checks whether \(\forall \textbf{x} \in \mathbb R^{n}~~ \psi _H \Rightarrow \ell .\)

  • Satisfiability Oracle: This oracle decides whether a given semialgebraic set is non-empty, i.e., it checks the satisfiability of a given formula \(\varphi .\)

We say that an oracle is sound if whenever it returns true, the implication (resp. satisfiability) holds. Conversely, an oracle is complete if whenever the implication (resp. satisfiability) holds, it returns true.

In this section, we provide the main procedure of our algorithm, assuming that the two oracles above are available. In Sect. 2.3, we will provide an LP-based implication oracle. Thus, calls to the implication oracle are relatively cheap in practice. In contrast, we rely on SMT solvers as satisfiability oracles. Thus, for practical scalability, our approach calls this oracle as late as possible and only in \(\epsilon \)-diameter subsets of \(\mathbb R^{n}\). Finally, in Sect. 2.4 we show that our over-approximation remains sound even in the absence of a satisfiability oracle but can only provide a weaker guarantee of approximation quality.

Fig. 2.
figure 2

An example of our gridding algorithm with memoization

Full size image

Our Algorithm. We are now ready to present our algorithm that finds an \(\epsilon \)-approximation of \(\pi (S).\) See [6] for a discussion of the intuition. Our algorithm consists of three steps and is provided in Algorithm 1.

Step 1. Literal Extraction. In the first step, our algorithm generates a set L consisting of all literals \(\ell \) that appear in the formula \(\varphi .\) This is done by a standard parsing of \(\varphi .\)

Step 2. Dynamic Gridding. In this step, our initial goal is to produce an \(\epsilon \)-approximation O of \(S\) itself, rather than its projection. Given that \(S\) is bounded, we can apply the idea of gridding. However, we do this in a dynamic and recursive manner, creating smaller grid cells only when necessary. We keep a set A of hyperrectangles whose union forms our answer. Initially \(A = \emptyset .\) We start with a hyperrectangle \(H_0\) which covers all of \(S\) as the initial grid cell. For example, we can set \(H_0 = \{ (x_1, x_2, \ldots , x_{n}) ~|~ -B\le x_i \le B\}.\) When processing each grid cell H,  our algorithm does the following:

  1. (a)

    For every literal \(\ell \in L,\) use the implication oracle to decide whether \(\ell \) holds at every point in H,  i.e. check \(\forall \textbf{x} \in \mathbb R^{n}~~ \psi _H \Rightarrow \ell .\)

  2. (b)

    For every literal \(\ell \in L,\) use implication oracle to decide whether its complement \(\overline{\ell }\) holds at every point in H,  i.e. query the oracle for \(\forall \textbf{x} \in \mathbb R^{n}~~ \psi _H \Rightarrow \overline{\ell }.\)

  3. (c)

    Create a ternary valuation \(\theta : L \rightarrow \{0, 1, ?\}\) in which \(\theta (\ell ) = 1\) if the check in (a) passes, \(\theta (\ell ) = 0\) if the check in (b) passes and otherwise \(\theta (\ell ) = ?.\) Use this valuation to evaluate \([|\varphi |]_\theta ,\) thus deciding whether \(\varphi \) holds at every point in H.

  4. (d)

    If \([|\varphi |]_\theta = 1,\) then add the grid cell H to the answer A. Conversely, if \([|\varphi |]_\theta = 0,\) exclude H from A. The only remaining case is if we are uncertain about \(\varphi .\) We break this down into two further cases:

    1. (i)

      If the diameter of H is more than \(\epsilon ,\) cut H into two halves \(H'\) and \(H''\) by bisecting its longest edge. Apply the algorithm recursively on both.

    2. (ii)

      If the diameter of H is at most \(\epsilon ,\) then use the satisfiability oracle on \(\psi _H \wedge \varphi .\) This will tell us whether there exists at least one point in H that satisfies \(\varphi .\) If such a point exists, include H in A. Else, exclude H.

Memoization. If one of the checks in (a) or (b) above succeed, then the corresponding (complement) literal holds at every point in H. Thus, if the algorithm later divides H in (d), we do not need to check the same literals again in \(H'\) and \(H''.\) Hence, our algorithm memoizes the set of literals that are known to hold or not hold at every point in H. This is shown as \(L_1\) and \(L_0\) in the pseudocode.

Example. Figure 2 shows a simple example of our dynamic gridding. Our goal is to approximate the intersection \(\varphi = \ell _1 \wedge \ell _2\) of the two red circles, i.e. each circle corresponds to a literal \(\ell _i\). A grid cell is shown in blue in part (i). Initially, our algorithm finds out that \(\ell _1\) holds at every point in the cell, but \(\ell _2\) is uncertain. Thus, in part (ii), we divide our cell in two. At this point we already know that \(\ell _1\) holds in both halves. This is memoized (shown in green) and not recomputed. In the left half, \(\ell _2\) does not hold at any point. Thus, we have \(\theta (\ell _2) = 0\) and exclude this half from the solution. In part (iii), we cut the right half in two. The bottom part is excluded from the solution since no point in it satisfies \(\ell _2.\) In the top right part, \(\ell _1\) is known to hold everywhere (memoized) and \(\ell _2\) is uncertain. However, at this point the diameter of the cell is less than \(\epsilon .\) Thus, our approach makes an SMT call in part (iv) and realizes that there is a point in this cell that satisfies \(\varphi .\) Hence, the top right cell is included in the answer.

Step 3. Projection. Let \(O = \bigcup _{H \in A} H.\) We will prove further below that O is an \(\epsilon \)-approximation of \(S.\) However, we would like an \(\epsilon \)-approximation of \(\pi (S).\) In this step, the algorithm computes \(\pi (O) = \bigcup _{H \in A} \pi (H)\) and outputs it as the answer. We note that projecting each hyperrectangle \(H \in A\) is a simple matter of dropping some constraints. Specifically, we have:

$$ \begin{matrix} \psi _H = \left\{ \begin{matrix} \alpha _1 \le v_1 \le \beta _1\\ \vdots \\ \alpha _n \le v_n \le \beta _n \end{matrix}\right. & ~~~\Rightarrow ~~~ & \psi _{\pi (H)} = \left\{ \begin{matrix} \alpha _1 \le v_1 \le \beta _1\\ \vdots \\ \alpha _{|\mathcal {V}_1|} \le v_{|\mathcal {V}_1|} \le \beta _{|\mathcal {V}_1|} \end{matrix}\right. \end{matrix}. $$
Algorithm 1
figure b

POQER

Full size image

Theorem 1

(Correctness, Proof in [6]). Assume that we have a sound implication oracle and a sound and complete satisfiability oracle. Given \(\varphi , \epsilon , n, \mathcal {V}, \mathcal {V}_1, \mathcal {V}_2,\) and B as input, let \(S := \{\textbf{x} \in \mathbb R^{n}~~ x \models \varphi \}\) be bounded by a ball of radius B around the origin. Then, Algorithm 1 (POQER), outputs an \(\epsilon \)-approximation of \(\pi (S),\) i.e. the projection of S onto \(\mathcal {V}_1,\) as desired.

2.3 Our Implication Oracle

As mentioned in the previous section, our algorithm depends on a sound oracle to check whether a given polynomial inequality (literal) \(\ell \) of the form \(f \ge 0\) or \(f > 0\) holds over the entirety of a hyperrectangle H. In this section, we provide such an oracle. Specifically, given the inequalities \(\psi _H\) that define the hyperrectangle H,  our goal is to check whether \(\forall {\textbf {x}} \in \mathbb R^{n}~~ \psi _H \Rightarrow \ell \) holds. Our algorithm is sound and can also provide semi-completeness guarantees for strict literals, i.e. literals of the form \(f > 0.\)

Semi-group generated by \(\varPhi \) . Consider the set \(\mathcal {V}= \{v_1,\dots v_n \}\) of real-valued variables and the following system of linear inequalities over \(\mathcal {V}\):

$$\begin{aligned} \varPhi := {\left\{ \begin{array}{ll} a_{1,0} + a_{1,1} \cdot v_1 + \ldots + a_{1,n} \cdot v_n \bowtie _1 0\\ \qquad \qquad \qquad \quad \vdots \\ a_{m,0} + a_{m,1} \cdot v_1 + \ldots + a_{m,n} \cdot v_n \bowtie _m 0\\ \end{array}\right. } \end{aligned}$$

where \(\bowtie _i \in \{>, \ge \}\) for all \(1 \le i \le m\). Let \(g_i\) be the left hand side of the i-th inequality, i.e. \(g_i(v_1,\dots ,v_n) := a_{i,0} + a_{i, 1} \cdot v_1 + \dots a_{i,n} \cdot v_n\). The semi-group of \(\varPhi \) is defined as: \( \textstyle \textit{SG}(\varPhi ) := \left\{ \prod _{i=1}^m g_i^{k_i} \mid m \in \mathbb {N} ~\wedge ~\forall i ~\; k_i \in \mathbb {N} \cup \{0\} \right\} . \) In other words, this semi-group contains all polynomials that can be obtained as a multiplication of the \(g_i\)’s. Note that \(1 \in \textit{SG}(\varPhi ).\) We define \(\textit{SG}_d(\varPhi )\) as the subset of polynomials in \(\textit{SG}(\varPhi )\) of degree at most d.

Theorem 2

(Handelman’s Theorem [41]). Consider the following system of equations over \(\mathcal {V}\):

$$\begin{aligned} \varPhi := {\left\{ \begin{array}{ll} a_{1,0} + a_{1,1} \cdot v_1 + \ldots + a_{1,n} \cdot v_n \ge 0\\ \qquad \qquad \qquad \quad \vdots \\ a_{m,0} + a_{m,1} \cdot v_1 + \ldots + a_{m,n} \cdot v_n \ge 0\\ \end{array}\right. }. \end{aligned}$$

If \(\varPhi \) is satisfiable, its solution set is compact, \(f \in \mathbb {R}[\mathcal {V}]\) and we have \(\forall {\textbf {x}} \in \mathbb R^{n}~~ \varPhi \Rightarrow f > 0,\) then there exist non-negative real numbers \(\lambda _0,\dots \lambda _k\) and semigroup elements \(h_1,\dots ,h_k \in \textit{SG}(\varPhi )\) such that \( \textstyle f = \lambda _0 + \lambda _1 \cdot h_1 + \dots + \lambda _k \cdot h_k. \)

Basic Idea of Our Implication Oracle. Consider the hyperrectangle H and its defining inequalities \(\psi _H\) which can be rewritten in the following form:

$$ \psi _H = \left\{ \begin{matrix} g_1 := v_1 - \alpha _1 \ge 0\\ g_2 := \beta _1 - v_1 \ge 0\\ \vdots \\ g_{2\cdot n - 1} := v_n - \alpha _n \ge 0\\ g_{2 \cdot n} := \beta _n - v_n \ge 0 \end{matrix}\right. $$

It is clear by definition that every \(g_i\) is non-negative at every point in H. Thus, any multiplication \(h \in \textit{SG}_d(\psi _H)\) of the \(g_i\)’s will also be non-negative throughout H. Finally, we can take any linear combination of such polynomials with non-negative coefficients, i.e.

$$\begin{aligned} f = \lambda _0 + \lambda _1 \cdot h_1 + \dots + \lambda _k \cdot h_k \end{aligned}$$
(1)

\( h_i \in \textit{SG}_d(\psi _H) ~~~ \lambda _i \ge 0 \), and such an f will be non-negative at every point in H. Moreover, if \(\lambda _0 > 0,\) then f will be strictly positive at every point in H.

Fig. 3.
figure 3

The hyperrectangle H defined by \(\psi _H\) lying inside the region \(f > 0.\)

Full size image

The Oracle. Our implication oracle is provided in Algorithm 2. Given a fixed degree \(d \in \mathbb N,\) it first generates \(\textit{SG}_d(\psi _H).\) It then symbolically computes a linear combination of the polynomials in \(\textit{SG}_d(\psi _H)\) by creating fresh variables for each \(\lambda _i\) as in the RHS of Eq. (1). This allows us to write Eq. (1) symbolically. Note that both sides of this equation are polynomials in \(\mathbb {R}[\mathcal {V}],\) thus they are equal if and only if each monomial has the same coefficient on both sides. The algorithm computes the coefficient of each monomial on both sides and equates them. This leads to a linear programming instance over the \(\lambda _i\)’s, which is in turn handled by an external solver.

Example. Consider the literal \(\ell = (f > 0)\) where \(f = 4 - x^2 - y^2 .\) Let H be the hyperrectangle defined by the inequalities \(-1 \le x \le 1\) and \(-1 \le y \le 1.\) We have \(\psi _H = \{x+1 \ge 0, -x+1 \ge 0, y+1 \ge 0, -y+1 \ge 0\}.\) Let \(d = 2.\) Then, \(\textit{SG}_d(\psi _H)\) contains all polynomials of degree at most 2 that can be obtained as a multiplication of the \(g_i\)’s. This includes \(g_1^2, g_1 \cdot g_2, g_1 \cdot g_3, g_1 \cdot g_4, g_2^2, g_2 \cdot g_3, g_2 \cdot g_4, g_3^2, g_3 \cdot g_4, g_4^2, g_1,g_2,g_3,g_4,1.\) Since \(\ell \) holds at every point in the hyperrectangle H,  we can write f as a linear combination of these polynomials as follows: \(4 - x^2 - y^2 = 1\cdot (1 - x^2) + 1\cdot (1-y^2) + 2\cdot 1 = 1 \cdot g_1 \cdot g_2 + 1 \cdot g_3 \cdot g_4 + 2 \cdot 1\) (Fig. 3).

Algorithm 2
figure c

Our Implication Oracle

Full size image

Theorem 3

(Soundness and Semi-completeness, Proof in [6]). Given a hyperrectangle H and a literal \(\ell \) in the input, and a degree bound d,  Algorithm 2 is sound in deciding whether \(\forall {\textbf {x}} \in \mathbb R^{n}~~ \psi _H \Rightarrow \ell .\) Moreover, if \(\ell \) is of the form \(f > 0,\) then there exists a degree bound d,  depending on both H and \(\ell ,\) for which the algorithm is complete in deciding \(\forall {\textbf {x}} \in \mathbb R^{n}~~ \psi _H \Rightarrow \ell .\)

Runtime Analysis. In Algorithm 2, let d be the degree, n the number of variables, and m the number of linear inequalities in our hypothesis hyperrectangle H. Then, the size of \(|M| = \left( {\begin{array}{c}n+d\\ d\end{array}}\right) \) and \(|SG| = \left( {\begin{array}{c}m+d\\ d\end{array}}\right) \). For each element of SG, we add a \(\lambda _i\) to our linear programming instance. Similarly, for each monomial in M, we add a constraint equating its coefficients on the two sides. Thus, we have an LP instance with \(O\left( \left( {\begin{array}{c}m+d\\ d\end{array}}\right) \right) \) variables and \(O\left( \left( {\begin{array}{c}n+d\\ d\end{array}}\right) \right) \) constraints. We note that current state-of-the-art LP-solving algorithms work in polynomial-time \(O(N^\omega )\) where N is their input size and \(\omega \) is the matrix multiplication constant.

2.4 Removing the Satisfiability Oracle

Our approximate quantifier elimination algorithm in Sect. 2.2 requires two oracles: one for implication and another for satisfiability. As mentioned above, we use SMT calls for the satisfiability oracle, but the implication oracle (Sect. 2.3) is much more practical and relies only on linear programming. Moreover, it provides a semi-completeness guarantee (Theorem 3) which is not used in the main algorithm (Theorem 1). So, a natural question is whether we can remove the satisfiability oracle altogether. We first argue that this is unlikely to lead to an efficient algorithm with our notion of \(\epsilon \)-approximation, since it is an ETR-hard problem. However, we can provide a weaker guarantee for positive formulas.

ETR-Hardness. Let \(\psi := \left( \exists v_1, v_2, \ldots , v_n ~~\varphi \right) \) be a formula in the existential theory of the reals. \(\psi \) holds if and only if \(\textit{SAT}(\varphi ) \ne \emptyset ,\) but we have

$$ \textit{SAT}(\varphi ) \ne \emptyset \Leftrightarrow \pi (\textit{SAT}(\varphi )) \ne \emptyset \Leftrightarrow \mathcal {I}_\epsilon (\pi (\textit{SAT}(\varphi ))) \ne \emptyset . $$

Thus, to decide \(\psi ,\) we can simply find an \(\epsilon \)-approximation of \(\pi (\textit{SAT}(\varphi ))\) and check its non-emptiness.

Positive Formulas. A formula \(\varphi \) is called positive if it is generated from the grammar below:

$$ \begin{matrix} \varphi := \ell ~|~ \varphi \wedge \varphi ~|~ \varphi \vee \varphi & ~~ & \text {positive formulas}\\ \ell := f \ge 0 ~|~ f > 0 & & \text {literals}\\ f \in \mathbb R[\mathcal {V}] & & \text {polynomials} \end{matrix} $$

The only difference between this grammar and that of Sect. 2.1 is the absence of the negation operator. We note that any formula can be written as an equivalent positive formula since the complement of each literal is itself a literal. Thus, in the remainder of this section, we assume that the formula \(\varphi \) is positive.

\((\epsilon , \delta )\)-perturbation. Let \(\epsilon , \delta >0\) and \(\varphi \) be a positive formula. We define the \((\epsilon , \delta )\)-perturbation \(\textit{SAT}_{\epsilon , \delta }(\varphi )\) of \(\textit{SAT}(\varphi )\) recursively as follows:

  • For every literal \(\ell = (f > 0)\) or \(\ell = (f \ge 0)\) we have

    $$\textit{SAT}_{\epsilon , \delta }(\ell ) = \mathcal {I}_\epsilon (\textit{SAT}(f+\delta \ge 0)).$$

    Intuitively, we are overapproximating \(\textit{SAT}(\ell )\) in two ways: (i) we are allowing the value of f to decrease to \(-\delta \) instead of just 0,  and (ii) we are taking an \(\epsilon \)-inflation of the resulting solutions. In other words, we are considering that our evaluation of f might have a numerical error of up to \(\delta \) and that our approximation of the solution set might contain some extra points which are within \(\epsilon \) distance to the original set.

  • If \(\varphi = \varphi _1 \wedge \varphi _2\), then \(\textit{SAT}_{\epsilon ,\delta }(\varphi ) := \textit{SAT}_{\epsilon ,\delta }(\varphi _1) \cap \textit{SAT}_{\epsilon ,\delta }(\varphi _2)\).

  • If \(\varphi = \varphi _1 \vee \varphi _2\), then \(\textit{SAT}_{\epsilon ,\delta }(\varphi ) := \textit{SAT}_{\epsilon ,\delta }(\varphi _1) \cup \textit{SAT}_{\epsilon ,\delta }(\varphi _2)\).

We remark that we always have \(\textit{SAT}(\varphi ) \subseteq \mathcal {I}_\epsilon (\textit{SAT}(\varphi )) \subseteq \textit{SAT}_{\epsilon ,\delta }(\varphi ).\) We say that a set O is an \((\epsilon ,\delta )\)-approximation of \(\textit{SAT}(\varphi )\) if \(\textit{SAT}(\varphi ) \subseteq O \subseteq \textit{SAT}_{\epsilon , \delta }(\varphi ).\) We note that there are subtle yet important differences in the definitions of \(\epsilon \)-approximation and \((\epsilon , \delta )\)-approximation, thus an \((\epsilon , 0)\)-approximation is not the same as an \(\epsilon \)-approximation as defined in Sect. 2.1.

Modified Algorithm. We take the exact same algorithm as in Sect. 2.2 (Algorithm 1), but only change Step 2 (d)(ii) as follows:

  • If the diameter of H is at most \(\epsilon ,\) for every literal \(\ell \in L\) of the form \(f>0\) or \(f \ge 0,\) use the implication oracle to decide the following formula:

    • \(\forall {\textbf {x}} \in \mathbb R^{n}~~ \psi _H \Rightarrow -f-\delta > 0\)

    If the check passes, update \(\theta (\ell )\) to 0. Otherwise, update it to 1. Finally, compute \([|\varphi |]_\theta \) and if it is 1 then include H in the answer A.

Algorithm 3 in [6] provides a psuedocode of this variant. See [6] for a discussion of the intuition behind this approach.

Theorem 4

(Proof in [6]). Assume that we have a sound and complete implication oracle. Given \(\varphi , \epsilon , \delta , n, \mathcal {V}, \mathcal {V}_1, \mathcal {V}_2\) and B as input, let \(\textit{SAT}(\varphi )\) be bounded by a ball of radius B around the origin. Then, Algorithm 3 of [6] (Modified POQER), outputs a set X such that \(\pi (\textit{SAT}(\varphi )) \subseteq X \subseteq \pi (\textit{SAT}_{\epsilon , \delta }(\varphi )).\) In other words, it outputs an \((\epsilon ,\delta )\)-approximation of \(\textit{SAT}(\varphi ).\)

3 Experimental Results

We implemented our approach in a tool called POQER (Practical Overapproximate Quantifier Elimination for Reals) and performed two experiments:

  • Our first experiment considers QER over formulas in Non-linear Real Arithmetic (NRA). To the best of our knowledge, we are providing the first approximate solution for quantifier elimination over NRA. Thus, we had to compare our scalability with previous exact solutions. Note that under-approximating the result of applying QER to a polynomial constraint \(\varphi \) is equivalent to complementing the over-approximation of QER applied to \(\lnot \varphi \). Hence, we focus only on over-approximating QER in this experiment. Moreover, since every Boolean combination of polynomial constraints can be equivalently expressed in disjunctive normal form, and since existential quantification distributes over disjunction, we focus only on conjunctions of polynomial constraints.

  • In our second experiment, we considered the problem of satisfiability checking for mixed formulas in NRA+ADT, i.e. theories of Non-linear Real Arithmetic and Algebraic Data Types. We first used POQER to eliminate quantifiers in the NRA part of the formula, obtaining both over- and under-approximations, and writing it as a union of hyperrectangles. We then combined this approximation with the ADT part and passed it to a state-of-the-art tool for LRA+ADT, namely Z3EG. As baselines, we compared our performance with state-of-the-art SMT solvers Z3 and Z3EG.

Implementation and Environment Details. We implemented POQER (Algorithm 1) in C++, with Z3 as the satisfiability oracle (see Sect. 2). We used Gurobi [40] as our LP-solver. The results were obtained on a 3.5GHz Intel Core i5 1030NG7 Machine with 8 GB of RAM running MacOS. We will submit our tool (POQER) for artifact evaluation and make it publicly available as free and open-source software.

First Experiment (QER). Due to the lack of publicly-available tools performing approximate quantifier elimination in NRA (non-linear real arithmetic), we are unable to present an apples-to-apples comparison. However, we report comparisons with several tools that perform exact QER. There are several (academic and commercial) tools implementing CAD and its variants, but we observe that the result of QER given by them is often as high-degree polynomials or their radicals. This makes it practically impossible to use these results in downstream processing using modern SMT solvers. Hence, we study not only whether these tools are able to solve a QER problem within a time budget, but also the format in which they provide the answer. Although our algorithm is parallel, we only compare using a sequential variant to be as fair as possible.

CAD (and variant algorithms for QER) are reported to be implemented in publicly available SMT solvers such as SMT-RAT [31], Yices2 [36], Z3 [61] and cvc5 [9]. However, SMT solvers are decision procedures for checking satisfaction of (possibly quantified) formulas in a combination of theories. Hence, they do not provide the result of quantifying a subset of variables in a formula. While this suffices in applications where the goal is to check if a formula is satisfiable, it falls short of the requirements in other applications, viz. weakest pre-condition computation, where we genuinely require the result of quantifying a subset of variables from NRA constraints. SMT-RAT [31] appears to have had a soundness issue in the quantifier elimination for QER (as noted in [2]) which we were unable to circumvent. Therefore, we compare our approach to two state-of-the-art methods: (a) SageMath [72], a versatile open-source computer algebra system, that includes an implementation of QEPCAD [13, 30], and (b) Mathematica [44], a widely-used and highly-optimized commercial computer algebra system, that employs a portfolio of powerful algorithms and heuristics for QER. We aim to answer the following research questions through our first experiment:

  • RQ1: Given a time of 30 min, how many QER tasks from our benchmark suite are solved by SageMath, Mathematica and POQER? We use the Reduce function in Mathematica and qepcad in SageMath.

  • RQ2: For each of the above three tools, is the output of a QER problem free of further NRA constraints?

  • RQ3: Does using Handelman’s Theorem and linear programming in POQER help achieve better performance compared to the use of a state-of-the-art SMT solver (Z3)? To answer this, we performed an ablation study by removing our Handelman-based implication oracle and instead directly applying Z3 as both implication and satisfiability oracles.

Benchmarks. Given the lack of standard benchmarks for QER, we designed a suite of benchmarks, each of which is a conjunction of polynomial inequalities, with range constraints on each dimension. Our benchmarks (see [6] for details) have 2–8 variables, degrees 2–6, and between 2 to 10 polynomials each.

Results. Our results are summarized in Table 1. We computed \(\epsilon \)-approximations using POQER for three different values of \(\epsilon \) to understand how POQER’s performance scales with decreasing values of \(\epsilon \). We observe that SageMath failed to complete the QER task within the timeout in all but four cases, where it provided a solution in NRA, as indicated by the asterisks. Mathematica performed significantly better, generating solutions in NRA in most instances. In 4 instances, the solutions are generated in a form that would require NRA with quantifiers, if we were to encode them in SMT. We show these solutions in [6]. Clearly, these solutions are intractably complicated and pose serious challenges for downstream automated reasoning tasks. Since SageMath and Mathematica implement exact QER, it is not possible to circumvent these complicated solution forms in general. POQER with \(\epsilon = 0.1\) successfully solved all benchmarks within the 30-minute timeframe, showcasing the effectiveness of our tool. POQER with \(\epsilon = 0.05\) and 0.01, fell short only in two and three instances respectively. Thus, our experiments answer research question RQ1 in favor of POQER for all three values of \(\epsilon \) considered, when compared to SageMath or Mathematica. RQ2 is answered in the positive for POQER in all cases, while it is mostly in the negative for SageMath and Mathematica, since they compute exact solutions which are often non-linear (in NRA) and sometimes even quantified. Finally, for RQ3, from columns Z3dir.05 and PQ.05 of Table 1, we can conclude that using Handelman’s Theorem and LP-solving significantly improves the performance of POQER vis-a-vis using a state-of-the-art SMT solver (Z3) in all but one example. On Benchmark Ex11, the Z3 approach is unusually fast, which is presumably due to its internal heuristics. We also report the number of hyperrectangles that we generate as well as number of hyperrectangles after the projection of variables (shown in last two columns of Table 1). Finally, more details are reported in [6]. Looking deeper into the results, we observe that Mathematica tends to solve most benchmarks efficiently, but has difficulty in solving problems with a larger number of eliminations, as each quantifier elimination results in increasingly complex solutions. In contrast, our approach benefits from the simpler structure of our solutions, enabling us to deliver faster results even when multiple quantifiers must be eliminated. Furthermore, Mathematica produces solutions in a complicated format, i.e. as degree polynomial inequalities in multiple variables. See [6] for more details.

Table 1. Results of our First Experiment. SM refers to Sagemath, MA refers to Mathematica, PQ.1, PQ.05, PQ.01 refer respectively to POQER with \(\epsilon =0.1, 0.05, 0.01\). indicates that the method terminates within 30 min, indicates a timeout. indicates that the solution is in QF-NRA. indicates NRA (with quantifiers). The columns PQ.05 and Z3dir.05 respectively refer to time taken in seconds by POQER and POQER where the Implication Oracle is replaced by Z3. #H is the number of hyperrectangles computed by POQER while #PH is the number of hyperrectangles after the projection.
Full size table

Second Experiment: NRA+ADT. The last observation above enables the use of approximate quantifier elimination in a combination of NRA and other theories, such as ADT (theory of algebraic data types), by reducing it to LRA+ADT. NRA+ADT formulas are often highly intractable and beyond the reach of modern SMT solvers. To the best of our knowledge, there are no approximate solutions for NRA+ADT in the literature, either. In contrast, an effective tool for LRA+ADT, called Z3EG, has recently been developed in [38].

Benchmarks. We took the Z3EG benchmarks which are in LRA+ADT and added a single NRA constraint to each of them, thus obtaining NRA+ADT formulas. The added NRA constraint is \(\forall x ~~x \in [-10, 10] \Rightarrow \exists y \in [-10, 10]~~x^3+x \ge y^3+3\cdot y+4,\) in which x is a variable already present in the original ADT formula and y is a fresh variable. Thus, our formulas combine NRA and ADT and are particularly challenging for modern SMT solvers. To each NRA+ADT benchmark, we first applied POQER to obtain over- and under-approximations in LRA+ADT. We then passed the resulting approximate formulas to Z3EG. As baseline comparisons, we also passed the same NRA+ADT benchmarks to Z3 and Z3EG. We observed that POQER significantly outperforms other tools on these SMT benchmarks. The results are summarized in Table 2.

Table 2. Results of our Second Experiment. TO stands for timeout \(> 2\) mins.
Full size table

4 Conclusion

We presented an algorithm that computes \(\epsilon \)- and \((\epsilon ,\delta )\)-approximations of QER, for every \(\epsilon ,\delta > 0\). Our approach combines adaptive dynamic gridding with application of Handelman’s Theorem to solve the approximation problem via a sequence of linear programs (LP). We provide formal guarantees of soundness, and guarantee completeness under mild assumptions. Our approach also allows us to solve quantified SMT problems over mixed theories including NRA, such as NRA+ADT.