ZoPE: A Fast Optimizer for ReLU Networks with Low-Dimensional Inputs

Abstract

Deep neural networks often lack the safety and robustness guarantees needed to be deployed in safety critical systems. Formal verification techniques can be used to prove input-output safety properties of networks, but when properties are difficult to specify, we rely on the solution to various optimization problems. In this work, we present an algorithm called ZoPE that solves optimization problems over the output of feedforward ReLU networks with low-dimensional inputs. The algorithm eagerly splits the input space, bounding the objective using zonotope propagation at each step, and improves computational efficiency compared to existing mixed-integer programming approaches. We demonstrate how to formulate and solve three types of optimization problems: (i) minimization of any convex function over the output space, (ii) minimization of a convex function over the output of two networks in series with an adversarial perturbation in the layer between them, and (iii) maximization of the difference in output between two networks. Using ZoPE, we observe a $25\times$ speedup on property 1 of the ACAS Xu neural network verification benchmark compared to several state-of-the-art verifiers, and an $85\times$ speedup on a set of linear optimization problems compared to a mixed-integer programming baseline. We demonstrate the versatility of the optimizer in analyzing networks by projecting onto the range of a generative adversarial network and visualizing the differences between a compressed and uncompressed network.

A Appendix

1.1 A.1 Maximum Distance Between Points in Two Hyperrectangles

We would like to derive an analytical solution for the maximum distance given by a p-norm with $p \ge 1$ between two hyperrectangles $H_1$ and $H_2$. We will let $\mathbf {c}_1$ and $\mathbf {c}_2$ be the centers of $H_1$ and $H_2$, and $\mathbf {r}_1$ and $\mathbf {r}_2$ be the radii of $H_1$ and $H_2$. The maximum distance can be found by solving the following optimization problem

$$\begin{aligned} \begin{array}{ll} \underset{\mathbf {h}_1, \mathbf {h}_2}{\text {maximize}} &{}\quad \left\Vert \mathbf {h}_1 - \mathbf {h}_2\right\Vert _p \\ \text {subject to} &{}\;\;\; \mathbf {h}_1 \in H_1 \\ &{}\;\;\; \mathbf {h}_2 \in H_2 \end{array} \end{aligned}$$

The p-norm for finite p is defined as

$$ \left\Vert \mathbf {x}\right\Vert _p = (\sum _{i=1}^n |(\mathbf {x})_i|^p)^{\frac{1}{p}} $$

We expand the objective of our maximization problem to be

$$ (\sum _{i=1}^n (|(\mathbf {h}_1)_i - (\mathbf {h}_2)_i|^p))^{\frac{1}{p}} $$

and since $x^\frac{1}{p}$ is monotonically increasing on the non-negative reals for $p \ge 1$, we can remove the power of $\frac{1}{p}$ giving us the equivalent problem

$$\begin{aligned} \begin{array}{ll} \underset{\mathbf {h}_1, \mathbf {h}_2}{\text {maximize}} &{}\quad \sum _{i=1}^n (|(\mathbf {h}_1)_i - (\mathbf {h}_2)_i|^p) \\ \text {subject to} &{}\;\;\; \mathbf {h}_1 \in H_1 \\ &{}\;\;\; \mathbf {h}_2 \in H_2 \end{array} \end{aligned}$$

(10)

Now we see that the constraints $\mathbf {h}_1 \in H_1$ and $\mathbf {h}_2 \in H_2$ apply independent constraints to each dimension of $\mathbf {h}_1$ and $\mathbf {h}_2$. We also note that the objective can be decomposed coordinate-wise. As a result, in order to solve this optimization problem, we will need to solve n optimization problems of the form

$$\begin{aligned} \begin{array}{ll} \underset{(\mathbf {h}_1)_i, (\mathbf {h}_2)_i}{\text {maximize}} &{}\quad |(\mathbf {h}_1)_i - (\mathbf {h}_2)_i|^p \\ \text {subject to} &{}\;\;\; (\mathbf {c}_1)_i - (\mathbf {r}_1)_i \le (\mathbf {h}_1)_i \le (\mathbf {c}_1)_i + (\mathbf {r}_1)_i\\ &{}\;\;\; (\mathbf {c}_2)_i - (\mathbf {r}_2)_i \le (\mathbf {h}_2)_i \le (\mathbf {c}_2)_i + (\mathbf {r}_2)_i \end{array} \end{aligned}$$

(11)

Since $x^p$ is monotonically increasing for $p \ge 1$ we can equivalently maximize $|(\mathbf {h}_1)_i - (\mathbf {h}_2)_i|$. We show an analytic form for the maximum by checking cases. If $(\mathbf {c}_2)_i$ is larger than $(\mathbf {c}_1)_i$, the maximum will be found by pushing $(\mathbf {h}_2)_i$ to its upper bound and $(\mathbf {h}_1)_i$ to its lower bound. Conversely, if $(\mathbf {h}_1)_i$ is larger than $(\mathbf {h}_2)_i$, the maximum will be found by pushing $(\mathbf {h}_1)_i$ to its upper bound and $(\mathbf {h}_2)_i$ to its lower bound. If $(\mathbf {c}_1)_i$ is equal to $(\mathbf {c}_2)_i$, then we can arbitrarily choose one to push to its lower bound and the other to push to its upper bound—we select $(\mathbf {h}_1)_i$ to go to its upper bound and $(\mathbf {h}_2)_i$ to go to its lower bound. As a result we have the optimal inputs

$$\begin{aligned} (\mathbf {h}_1)_i^*&= (\mathbf {c}_1)_i + \text {sign}((\mathbf {c}_1)_i - (\mathbf {c}_1)_i) (\mathbf {r}_1)_i \\ (\mathbf {h}_2)_i^*&= (\mathbf {c}_2)_i + \text {sign}((\mathbf {c}_2)_i - (\mathbf {c}_2)_i) (\mathbf {r}_2)_i \end{aligned}$$

where the sign function is given by

$$ \text {sign}(x) = {\left\{ \begin{array}{ll} 1.0 &{} x \ge 0 \\ -1.0 &{} x < 0 \end{array}\right. } $$

Then, backtracking to our original problem and vectorizing gives us the analytical solution to this optimization problem with optimal value $d^*$

$$\begin{aligned} \mathbf {h}_1^*&= \mathbf {c}_1 + \text {sign}(\mathbf {c}_1 - \mathbf {c}_2) \odot \mathbf {r}_1 \\ \mathbf {h}_2^*&= \mathbf {c}_2 + \text {sign}(\mathbf {c}_2 - \mathbf {c}_1) \odot \mathbf {r}_2 \\ d^*&= \left\Vert \mathbf {h}_1^* - \mathbf {h}_2^*\right\Vert _p \end{aligned}$$

where the sign function is applied elementwise. This completes our derivation of the analytical solution for the maximum distance between two points contained in two hyperrectangles.

1.2 A.2 Verifier Configuration for the Collision Avoidance Benchmark

This section describes how each verifier was configured for the collision avoidance benchmark discussed in Sect. 5.1. Table 2 summarizes the non-default parameters for each solver and the location where the parameter was set. Both NNENUM and ERAN by default make use of parallelization, and Marabou has a parallel mode of operation, but for this experiment we restrict all tools to a single core. We ran the experiments on a single core to try to separate the aspects of how each solver was parallelized from what we viewed as the core of its algorithmic approach. We expect ZoPE would parallelize well, especially on more challenging problems. The hyperparameters we ran for ERAN may be better suited for multiple cores than a single core, so further comparison could explore these in more depth. Additionally, the timing results from the Verification of Neural Networks 2020 competition⁴ for several properties for ERAN were slower than we expected from the change in hardware and the restriction to a single core. Exploring the tool further, we observed that on several problem instances it would return back a failed status before reaching a timeout. On these same instances we saw that ERAN would find several inputs that were almost counter-examples, for example with a margin of $1 \times 10^{-6}$ from violating the property, flag these as potential counter-examples, then move on. It is possible that the root cause of the abnormalities we observed affected timing results. On problems where ERAN did return a status the results were consistent with the ground truth.

The parameters were chosen based off of a mix of recommendations from developers on their best configuration for the collision avoidance benchmark or existing documented settings for this benchmark. For example, ERAN’s parameters were based off of the VNN20 competition as found at https://github.com/GgnDpSngh/ERAN-VNN-COMP/blob/master/tf_verify/run_acasxu.sh. The code for for Marabou,⁵ NNENUM,⁶ ERAN,⁷ and our optimizer ZoPE⁸ is available for free online.

Table 2. Non-default verifier parameters