Keywords

figure a

1 Introduction

Over the last few years, neural networks have emerged as an increasingly classical choice for the control of autonomous systems, in particular due to their properties as universal function approximators. However, their adoption in safety-critical systems, the inherent uncertainties from the dynamic environment, and their sensitivity to adversarial examples make it crucial to establish their safety and robustness. This verification is challenging because of the complex non-linear characteristics of neural networks. Recent works come up with some approaches and tools to bound the output uncertainty of neural networks with respect to input perturbations. However, many of them are restricted to the analysis of networks with ReLU activation functions. Moreover, the approaches considering general differentiable activation functions and systems with general non linear dynamics provide over-approximations, which conservatism is difficult to estimate. RINO proposes a scalable and adaptive approach to compute both inner (or under) and outer (or over) approximations for the closed loop reachability problem of neural network controlled systems, with differentiable activation functions. The outer-approximation allows for property verification, while the inner-approximation allows for property refutation. Combined, the inner and outer-approximations allow to assess the conservatism of the approximations.

As the behavior of a neural network controlled closed-loop system relies on the interaction between the continuous dynamics and the neural network controller, a good precision requires to not only compute the output range but also describe the input-output mapping for the controller. In this work, we propose to use a zonotope-based abstraction to compute in a unified way both the reachable sets of neural networks and dynamical systems. This seamless integration of the reachability of neural networks and dynamical systems presents the advantage of a natural propagation of useful correlations through the different components of the closed loop system, resulting in an efficient and precise approach compared to many existing works which rely on external reachability tools.

Contributions

  • RINO implements all ideas presented in [8,9,10,11] for the joint computation of inner and outer approximations of robustly reachable sets of differentiable nonlinear discrete-time or continuous-time systems (without neural networks in the loop), possibly with constant delays. These previous works demonstrated the good scaling properties of our approach on different examples including a full nonlinear quadcoptor flight model but the tool was never presented as such.

  • Additionally, we demonstrate here that an application of these ideas to the case of neural networks enabled dynamical systems provides very competitive results for the over-approximation compared to the state of the art (at least similar precision and one order of magnitude faster) while also providing the first approach for inner-approximation of the reachable sets of such systems, which we use to falsify some safety properties.

  • Finally, RINO also computes approximations of output ranges that are reachable robustly or adversarially with respect to a subset of inputs: while these robust ranges are mostly used in this work to compute inner-approximations of joint ranges of state variables instead of projections, we believe this sensitivity information can be a useful tool in the future in particular to assess global robustness properties of neural networks.

Related Work. The safety verification for DNNs has received considerable attention recently, with several threads of work being developed. We draw below a non exhaustive panorama focusing on available tools for reachability analysis of neural network controlled systems with smooth activation functions.

Different approaches have been proposed to the reachability analysis closed-loop systems with neural network controllers, often by a transformation to a continuous or hybrid system reachability. Sherlock [6] targets both the open-loop and closed-loop problems with ReLU activation functions, in particular using the regressive polynomial rule inference approach [5] for the closed-loop, and Flow* [3] for the reachability of the dynamical system. NNV [24] also targets both the open loop and closed loop verification problems, with various activation functions and set representations such as polyhedra or star sets [23], and different reachability algorithms for dynamical systems relying on CORA [1] and the MPT toolbox [18]. ReachNN [13] and its successor ReachNN* [7] propose a reachability analysis based on Bernstein polynomials for closed-loop systems with general activation functions, also relying on Flow* [3] for the reachability of the dynamical system. Verisig [14] handles NNCS with nonlinear plants controlled by sigmoid-based networks, exploiting the fact that the sigmoid is the solution to a differential equation to transform the neural network into an equivalent hybrid system, which is then fed to Flow*. Verisig 2.0 [15] uses preconditioned Taylor Models to propagate reachable sets in neural networks, and also relies on Flow* for reachability of the hybrid system component.

The very recent works [21] and [12] implemented respectively over JuliaReach and in POLAR are also closely related to our work. In [21], the authors implement a bridge between zonotope abstractions and Taylor model abstractions in order to combine tools analyzing controllers (e.g. using zonotopes like deepZ [22]) with tools analyzing ordinary differential equations (e.g. Flow* [3]). In [12], the authors use a polynomial arithmetic made up of a combination of Berstein polynomials and Taylor models to iteratively overapproximate networks layers, according to whether the activation function is differentiable or not.

2 Problem Statement and Background

2.1 Robust Reachability of Closed-Loop Dynamical Systems

We consider in this work a closed-loop system consisting of a plant with states x, modeled as a discrete-time or continuous-time system with time-varying disturbances w and inputs u, where some components of the control inputs can be the output a neural network h taking x as input. For notation’s simplicity, we focus on continuous-time systems and define:

$$\begin{aligned} {\left\{ \begin{array}{ll} \dot{x}(t) = f(x(t),u(t),w(t)) &{} \text{ if } t \ge 0 \\ x(t) = x_0 &{} \text{ if } t = 0 \end{array}\right. } \end{aligned}$$
(1)

where f is a sufficiently smooth function and at least \(\mathcal{C}^1\), and controls u and disturbances w are also supposed to be sufficiently smooth \(C^k\) for some \(k\ge 0\) stepwise. This allows discontinuous controls and disturbances, where the discontinuities can only appear at discrete times \(t_j\).

The neural network h is a fully-connected feedforward NN with differentiable activation functions, defined as the composition \(h(x)=h_L \circ h_{L-1} \circ \ldots h_1(x)\) of L layers where each layer \(h_i(x) = \sigma (W_ix+b_i)\) performs a linear transform followed by a sigmoid or hyperbolic tangent activation \(\sigma \). We assume the control is decomposed as \(u(t)=(u_1(t),u_2(t))\) where \(u_2(t)\) is a control input defined in \(\mathbb {U}_2\) and \(u_1(t)\) is the output of the neural network controller. This controller is executed in a time-triggered fashion with control step T, so that \(u_1(t)=h(x(t_k))\), for \(t \in [t_k,t_k+T)\), where \(t_k=kT\) for positive integers k. System (1) can then be rewritten as

$$\begin{aligned} {\left\{ \begin{array}{ll} \dot{x}(t) = f(x(t),h(x(t_k)),u_2(t),w(t)) &{} \text{ if } t \in [t_k,t_k+T), \; t_k=kT, k\ge 0\\ x(t) = x_0 &{} \text{ if } t = 0 \end{array}\right. } \end{aligned}$$
(2)

Let \(\varphi ^f(t ; x_0,u_2,w)\) for time \(t \in \mathbb {T}\) denote the time trajectory of (2) with initial state \(x(0)=x_0\), for input signal \(u_2\) and disturbance w.

We consider the problem of computing inner and outer-approximations of robust reachable sets as introduced in [9], defined here as

$$ R^f_{\mathcal {A}\mathcal {E}}(t ; \mathbb {X}_0,\mathbb {U}_2,\mathbb {W}) = \{ x \, | \, \forall w \in \mathbb {W}, \exists u_2 \in \mathbb {U}_2, \, \exists x_0 \in \mathbb {X}_0, \, x = \varphi ^f(t ; x_0,u,w) \} $$

Note that this notion of robust reachability extends the classical notions of minimal and maximal reachability [20]. We use the subscript notation \(\mathcal {A}\mathcal {E}\) to indicate that the reachable set is minimal with respect to the disturbances w (universal quantification \(\mathcal {A}\)) and maximal with respect to the input \(u_2\) (existential quantification indicated by \({\mathcal {E}}\)), and that the universal quantification always precedes the existential quantification.

2.2 Mean-Value Inner and Outer-Approximating Robust Extensions

A classical but often overly conservative way to overapproximate the image of a set by a real-valued function \(f: \mathbb {R}^m \rightarrow \mathbb {R}\) is the natural interval extension \(\mathcal {F}: {\mathbb I \mathbb R}^m \rightarrow {\mathbb I \mathbb R}\), \({\mathbb I \mathbb R}\) being the set of intervals with real bounds, which consists in replacing real operations by their interval counterparts in the expression of the function.

A generally more accurate extension relies on a linearization by the mean-value theorem. Mean-value extensions can be generalized to compute ranges that are robust to disturbances, identified as a subset of the input components. Let f be a continuously differentiable function from \(\mathbb {R}^m\) to \(\mathbb {R}\) with input decomposed as \(x=(u,w) \in (\mathcal {U},\mathcal {W}) \subseteq {\mathbb I \mathbb R}^m\). We define the robust range of function f on \(\boldsymbol{x}\), robust with respect to component \(w \in \mathcal {W}\), as \( R^f_{\mathcal {A}\mathcal {E}}(\mathcal {U},\mathcal {W}) = \{ z \, | \, \forall w \in \mathcal {W}, \, \exists u \in \mathcal {U}, \, z = f(u,w) \}\).

For a continuously differentiable function \(f: \mathbb {R}^m \rightarrow \mathbb {R}^n\), we note \(\nabla f = (\nabla _j f_i)_{ij} = (\frac{\partial f_i}{\partial x_j})_{1\le i \le n, 1 \le j \le m}\) its Jacobian matrix. We note \(\langle x, y \rangle \) the scalar product of vectors x and y, and \(|x |\) the absolute value extended componentwise. For a vector of intervals \(\mathcal {X}=[\underline{\mathcal {X}},\overline{\mathcal {X}}]\), we note \(c(\mathcal {X})=(\overline{\mathcal {X}}+\underline{\mathcal {X}})/2.0\) and \(r(\mathcal {X})=(\overline{\mathcal {X}}-\underline{\mathcal {X}})/2.0\) its center and radius defined componentwise.

Theorem 1

([8], slightly simplified version of Thm. 2). Let f be a continuously differentiable function from \(\mathbb {R}^m\) to \(\mathbb {R}\) and \(\mathcal {X}=\mathcal {U} \times \mathcal {W} \subseteq {\mathbb I \mathbb R}^m\). Let \(\mathcal {F}^0\), \(\mathcal {\nabla }_{w}^{\mathcal {X}}\) and \(\mathcal {\nabla }_{u}^{\mathcal {X}}\) be vectors of intervals such that \( c(\mathcal {X}) \subseteq \mathcal {F}^0 \), \(\{ \left| \nabla _w f (u,w) \right| \, , \, (u,w) \in \mathcal {X} \} \subseteq \mathcal {\nabla }_{w}^{\mathcal {X}}\) and \(\{ \left| \nabla _u f (u,w) \right| , (u,w) \in \mathcal {X} \} \subseteq \mathcal {\nabla }_{u}^{\mathcal {X}}\). We have:

$$\begin{aligned}&\quad [ \overline{\mathcal {F}^0} - \langle \underline{\mathcal {\nabla }_{u}^{\mathcal {X}}} , r({\mathcal {U}}) \rangle + \langle \overline{\mathcal {\nabla }_{w}^{\mathcal {X}}}, r({\mathcal {W}}) \rangle , \underline{\mathcal {F}^0} + \langle \underline{\mathcal {\nabla }_{u}^{\mathcal {X}}}, r({\mathcal {U}}) \rangle - \langle \overline{\mathcal {\nabla }_{w}^{\mathcal {X}}}, r({\mathcal {W}}) \rangle ] \subseteq R^f_{\mathcal {A}\mathcal {E}}(\mathcal {U},\mathcal {W}) \\ {}&R^f_{\mathcal {A}\mathcal {E}}(\mathcal {U},\mathcal {W}) \subseteq [\underline{\mathcal {F}^0} - \langle \overline{\mathcal {\nabla }_{u}^{\mathcal {X}}}, r({\mathcal {U}}) \rangle + \langle \underline{\mathcal {\nabla }_{w}^{\mathcal {X}}}, r({\mathcal {W}})\rangle , \overline{\mathcal {F}^0} + \langle \overline{\mathcal {\nabla }_{u}^{\mathcal {X}}}, r(\mathcal {U}) \rangle - \langle \underline{\mathcal {\nabla }_{w}^{\mathcal {X}}}, r({\mathcal {W}})\rangle ] \end{aligned}$$

Theorem 1 provides inner and outer-approximations of the robust range (or of the classical range when there is no disturbance component w) of scalar-valued functions, or of the projections on each component of vector-valued functions, using bounds on the slopes on the input set. The result is useful to compute a projected range that is robustly reachable with respect to the disturbances w, or as a brick in computing an under-approximation of the image of a vector-valued function, as stated in Theorem 3 in [8].

Note that the accuracy of the mean-value AE extension can be improved with an evaluation by a quadrature formula ([10], Sect. 4.2). Alternatively, an order 2 Taylor-based extension ([10], Sect. 3) can be used.

2.3 Reachability of Neural Network Controlled Closed-Loop Systems

The inner and outer approximations defined in Sect. 2.2 can be computed for f being a simple function, possibly involving a neural network evaluation, or f being the function defined by the iterated values of a discrete systems, or finally f being the solution flow of closed-loop system (2).

In both discrete-time and the continuous-time cases, and whether some neural network controller is present or not, the evaluation of an outer-approximation of the image of the solution and its Jacobian with respect to inputs and disturbances over sets is needed in order to apply Theorem 1.

In our work and implementation, we advocate the use of a unique abstraction by affine forms (or zonotopes for the geometric view of a tuple of variables represented by affine forms) for these sets and these evaluations, including performing reachability of the neural network controller. This abstraction is very convenient and versatile to over-approximate any smooth function, providing a good trade-off between efficiency and precision in most cases (and for more precision, one can consider extensions with e.g. polynomial zonotopes [2]).

For continuous-time systems, we use Taylor expansions in time of the solution on a time grid. To build these Taylor expansions, we evaluate function f and its (Lie) derivatives over affine forms by a combination of automatic differentiation and numerical evaluation by affine arithmetic, as described in e.g. [9]. The neural network is seen as a nonlinear function h, composed with f to build function g for which we compute the solution flow. Theorem 1 is applied to this solution flow. We build the abstraction of h and thus g by a simple propagation of affine forms by affine arithmetic in the network: linear transformers are exact, and we propagate affine forms through the activation functions seen as standard nonlinear functions relying on the elementary exponential function, \(tanh(x)=2/(1+e^{-2x}) - 1\) and \(sig(x) = 1/(1+e^{-x})\). For differentiating the activation functions, we use \(tanh'(x)=1.0-tanh(x)^2\) and \(sig'(x)=sig(x) (1 - sig(x))\).

3 Implementation

As mentioned in the introduction, RINO implements all ideas presented in [8,9,10,11] for the joint computation of inner and outer approximations of robustly reachable sets of differentiable nonlinear discrete-time [8, 10] or continuous-time systems [8, 9], possibly with constant delays [11]. For experiments with systems without neural networks, we refer to the results presented in these works, obtained with a previous version of RINO.

RINO is written in C++. Intervals and zonotopes are used for set representation: the tool relies on the FILIB++ library [19] for interval computations and the aaflib libraryFootnote 1 for affine arithmetic [4]. Ole Stauning’s FADBAD++ libraryFootnote 2 is used for automatic differentiation: its implementation with template enables us to easily evaluate the differentiation in the set representation of our choice (affine forms or zonotopes mostly). The tool takes as inputs:

  • an open-loop or closed loop system, either discrete time or continuous-time, which for now is hard-coded in C++,

  • an optional neural network, provided to the tool in a format directly inspired from the format analyzed by Sherlock [6], which can be used as some inputs of the closed-loop system,

  • an optional configuration file to set initial values, input and disturbances ranges, and some parameter of the analysis (such as time step, order of Taylor expansion in time)

It computes inner and outer-approximations of the projection on each component of ranges, as well as joint 2D and 3D inner-approximations (provided as yaml file and Jupyter/python-produced figures). Additionally to the classical ranges, RINO computes approximations of output ranges that are reachable robustly or adversarially with respect to disturbances, specified as a subset of inputs. In the experiments presented herafter, we consider examples only of classical reachability, for which comparisons with existing work are available, but the extension to robust reachability based on our previous work is straightforward.

4 Experiments

For space reasons, we focus here on the main novelty which is the extension of this previous work to compute under and over-approximations of (robust) reachable sets of neural network controlled systems (2).

Choice of Tools and Benchmark Examples. We compare RINO against ReachNN* and Verisig 2.0 that are the most recent fully-fledged reachability analyzers for neural network based control systems, and for which comparisons with other tools on classical benchmarks are well documented in e.g. [15]. They both improve on previous versions, Verisig and ReachNN, and on state of the art tools Sherlock, also based on Flow*, and NNV. As noted in e.g. [15]: “Firstly, note that Verisig takes significantly more time to compute reachable sets (21 times slower in the case of the B5 benchmark). Furthermore, Verisig is unable to verify some properties due to increasing error. Note that NNV is unable to verify any of the properties considered in this paper due to high approximation error.”. Remark though that there has been some amelioration to the internal solvers used in NNV which should qualify the latter statement (see e.g. [16]). We do not compare with the implementation in JuliaReach [21] since, first, timings are difficult to compare with an interpreted framework, and second, because it would require mixing several tools together, with many potential combinations. We try to provide elements of comparison with POLAR [12], but in many ways the latter addresses a different problem, with the emphasis on being able to interpret e.g. ReLU activation functions.

Table 1. List of benchmarks (see [15])
Full size table

We use a large subset (7/10) of the examples from Verisig 2.0 [15], which are benchmarks used by most of the tools in the field, through e.g. the ARCH competition [17]. We also consider the same settings in terms of initial sets and the same time horizon. These are recalled in Table 1.

We indicate some of RINO’s reachability results on these benchmarks in Table 2, before comparing the tightness and computing times with other tools.

Table 2. RINO’s results for time step 0.05 (except Mountain Car, step 1.)
Full size table

Settings. All tools, Verisig 2.0 and ReachNN* and RINO, were run without GPU support, under Ubuntu 18.04 docker, on a Mac running Mac OS Big Sur 11.2.3 on a 2.3 GHz Intel Core i9 processor with 16Gb of memory. Verisig 2.0 and ReachNN* were run with the Reproductibility Package of Verisig 2.0 [15]. For fairness of timing results, we also run RINO with docker, and the running ratios given in Table 3 are those using these docker versions. RINO was also run natively on the same Mac. The performance degradation between the two versions of RINO can be estimated from the full data given in Table 2 from none to a 40% increase (with one exception at 80%), and most between 20 and 30%. This is higher than generally observed with docker, but due to the fact that docker on Macintosh is known to perform badly when it comes to IOs, using the underlying file system. Therefore, the performance degrades more when the system is of higher dimension and have more time steps to evaluate, since RINO logs all estimated ranges for all variables in separate files.

Comparisons Results. We compare in Table 3 the running times of Verisig 2.0, ReachNN* and RINO, and volumes of their final over-approximations, more precisely the widths of the projections of each component at final time horizon.

The three tools depend on some parameters, in particular integration time steps and order of approximation. RINO does not require tuning the integration time steps and order of Taylor models so much, so we use one fixed time step of 0.05 for all examples. We use for Verisig 2.0 and ReachNN* the settings of the CAV Reproductibility package, that we suppose give good results. Verisig 2.0 and ReachNN* actually perform poorly on the same examples with a fixed time steps of 0.05 s.

We experimented RINO with different time steps. The precision is relatively stable and does not necessarily improve when decreasing the time step. Indeed, as already noted [25], the improvement in approximation by Taylor models on smaller time steps is balanced by the loss of precision due to set-based abstraction being performed more often. Note also that the analysis time does not depend linearly on the time step: the control step, which rules the frequency at which the analysis of the neural net controller has to be performed, is fixed (see Table 1) and does not depend on the integration time step.

Column 2 in Table 3 describes the relative width of the intervals given by Verisig 2.0 for each variable at the final time and for each system, with respect to the one given by RINO. Column 4 is the same, but for ReachNN*. Columns 3 and 5 give the ratio of the analysis time of Verisig 2.0 (respectively ReachNN*), with respect to the analysis time of RINO.

In all cases, RINO is much faster than both Verisig 2.0 and ReachNN*, by factors ranging from 13 to 638.5. Moreover, this includes for RINO the time to compute the inner-approximations that Verisig 2.0 and ReachNN* do not compute. ReachNN* could not analyze TORA because of lack of memory on our platform, and timed out on ACC. Finally, interpolating the timings given in Table 1 of [12], e.g. for B1 (sig), Verisig 2.0 is reported to take 47 s whereas POLAR is reported to take 20 s on their platform. As Verisig 2.0 took 81.33 s on our platform, we can infer that RINO is most certainly much faster, with e.g. 3.62 s for B1, than POLAR.

RINO’s precision is of the same order as Verisig 2.0, and always better than ReachNN* by a factor of about 2 to 10. RINO is in fact even substantially more precise than Verisig 2.0 in some cases (B1 and B2 in particular).

Table 3. Precision and running time comparisons RINO [timestep=0.05] vs Verisig 2.0 [time steps of [15]] vs ReachNN* [time steps of [15]]
Full size table

Inner-Approximations. Let us take example B1 (with sigmoid-based controller), and suppose we have a safety property that the value of \(x_1\) should never be bigger than 1. Figure 1a represents in filled blue region the inner-approximation, as plain black lines the bounds of the outer-approximation, and as purple dots values actually reached, obtained by trajectories for sample initial conditions The over-approximation alone does raise a potential alarm with respect to the unsafe zone (in red), only the inner-approximation actually proves that the safety property is falsified. We also note on this picture that the over-approximation is very tight, given that samples give almost indistinguishable ranges. Figure 1b represents the inner and outer approximations of joint range \((x_1,x_2)\) as well as estimation by sampling. As shown by the samples, \((x_1,x_2)\) becomes almost a 1D curve after some time, making inner approximation extremely difficult to estimate. Indeed our inner-approximation in orange is fairly precise for the first time steps, and the corresponding inner skewed boxes are rotated to match the curvy, 1D, shape of the samples. The green boxes printed on the picture are the box enclosure of the actually computed outer-approximation. Note that the inner-approximation of the projections on each component can be non-empty while having an empty joint inner range, as some approximation is committed in the joint inner range computation (as a skewed box) from the projected ranges.

Fig. 1.
figure 1

B1: inner-approximation, outer-approximation and sampling (purple dots) (Color figure online)

Full size image

5 Conclusion and Future Work

We presented the RINO tool, dedicated to the reachability analysis of dynamical systems, possibly controlled by neural networks. While providing accurate results, RINO is significantly faster than other state-of-the-art tools, which is key in view to address real-life reachability problems, where the systems and neural networks can be of high dimension. Moreover, as far as we are aware, it is the only existing tool to propose inner-approximations of the reachable sets of such systems. We currently handle only differentiable activation functions. We are thinking of some abstractions to handle ReLU activations as well, even though the approach is less natural in that case as it will introduce conservatism. We also plan to improve the accuracy of our current results by further specializing this work to exploit the structure of neural network, such as monotonicity of activation functions. Finally, robustness is a crucial property for neural networks enabled systems, and we plan to explore the possibilities offered by the computation of robust reachable sets.