Verification of Deep Convolutional Neural Networks Using ImageStars

Convolutional Neural Networks (CNN) have redefined state-of-the-art in many real-world applications, such as facial recognition, image classification, human pose estimation, and semantic segmentation. Despite their success, CNNs are vulnerable to adversarial attacks, where slight changes to their inputs may lead to sharp changes in their output in even well-trained networks. Set-based analysis methods can detect or prove the absence of bounded adversarial attacks, which can then be used to evaluate the effectiveness of neural network training methodology. Unfortunately, existing verification approaches have limited scalability in terms of the size of networks that can be analyzed. In this paper, we describe a set-based framework that successfully deals with real-world CNNs, such as VGG16 and VGG19, that have high accuracy on ImageNet. Our approach is based on a new set representation called the ImageStar, which enables efficient exact and over-approximative analysis of CNNs. ImageStars perform efficient set-based analysis by combining operations on concrete images with linear programming (LP). Our approach is implemented in a tool called NNV, and can verify the robustness of VGG networks with respect to a small set of input states, derived from adversarial attacks, such as the DeepFool attack. The experimental results show that our approach is less conservative and faster than existing zonotope and polytope methods.


Introduction
Convolutional neural networks (CNN) have rapidly accelerated progress in computer vision with many practical applications such as face recognition [18], image classification [17], document analysis [20] and semantic segmentation.Recently, it has been shown that CNNs are vulnerable to adversarial attacks, where a well-trained CNN can be fooled into producing errant predictions due to tiny changes in their inputs [8].Many applications such as autonomous driving seek to leverage the power of CNNs.However due the opaque nature of these models there are reservations about using in safety-critical applications.Thus, there is an urgent need for formally evaluating the robustness of a trained CNN.
The formal verification of deep neural networks (DNNs) has recently become a hot topic.The majority of the existing approaches focus on verifying safety and robustness properties of feedforward neural networks (FNN) with the Rectified Linear Unit activation function (ReLU).These approaches include: mixedinteger linear programming (MILP) [5,16,22], satisfiability (SAT) and satisfiability modulo theory (SMT) techniques [14], optimization [6,10,21,39,41,48], and geometric reachability [28,29,33,34,38,40,[42][43][44]47].Adjacent to these methods are property inference techniques for DNNs, which are also an important and interesting research area investigated in [9].In a similar fashion, the problem of verifying the safety of cyber-physical systems (CPS) with learning-enabled neural network components with imperfect plant models and sensing information [1, 11-13, 23, 30-32, 45, 46] has recently attracted significant attention due to their real world applications.This research area views the safety verification problem in a more holistic manner by considering the safety of the whole system in which learning-enabled components interact with the physical world.
Although numerous tools have been proposed for neural network verification, only a handful of methods can deal with CNNs [2,15,16,26,28,29].Moreover, in the aforementioned techniques, only one [26] can deal with real-world CNNs, such as VGGNet [27].Their approach makes used of the concept of the L 0 distance between two images.Their optimization-based approach computes a tight bound on the number of pixels that may be changed in an image without affecting the classification result of the network.It can also efficiently generate adversarial examples that can be used to improve the robustness of network.In a similar manner, this paper seeks to verify the robustness of real-world deep CNNs.Thus, we propose a set-based analysis method through the use of the ImageStar, a new set representation that can represent an infinite number of images.As an example, this representation can be used to represent a set of images distorted by an adversarial attack.Using the ImageStar, we propose both exact and overapproximate reachability algorithms to construct reachable sets that contain all the possible outputs of a CNN under an adversarial attack.These reachable sets are then used to reason about the overall robustness of the network.When a CNN violates a robustness property, our exact reachability scheme can construct a set of concrete adversarial examples.Our approach differs from [26] in two ways.First, our method does not provide robustness guarantees for a network in terms of the number of pixels that are allowed to be changed (in terms of L 0 distance).Instead, we prove the robustness of the network on images that are attacked by disturbances bounded by arbitrary linear constraints.Second, our approach relies on reachable set computation of a network corresponding to a bounded input set, as opposed to a purely optimization-based approach.
We implement the proposed method in a tool called NNV and compare it with the zonotope method used in DeepZ [28] and the polytope method used in DeepPoly [29].The experimental results show that our method is less conservative and faster than any of these approaches when verifying the robustness of CNNs.The main contributions of the paper are as follows.
-The ImageStar set representation, which is an efficient representation for reachability analysis of CNNs.
-The provision of exact and over-approximate reachability algorithms for constructing reachable sets and verifying robustness of CNNs.-The implementation of the ImageStar representation and reachability algorithms in NNV [36].-Rigorous evaluation and comparison of proposed approaches, such as zonotope [28] and polytope [29] methods on different CNNs.

Problem formulation
The reachability problem for CNNs is the task of analyzing a trained CNN with respect to some perturbed input set in order to construct a set containing all possible outputs of the network.In this paper, we consider the reachability of a CNN N that consists of a series of layers L that may include convolutional layers, fully connected layers, max-pooling layers, average pooling layers, and ReLU activation layers.Mathematically, we define a CNN with n layers as N = {L i }, i = 1, 2, . . ., n.The reachability of the CNN N is defined based on the concept of reachable sets.
. ., n corresponding to a linear input set I is defined incrementally as: where L i (•) is a function representing the operation of the i th layer.
The definition shows that the reachable set of the CNN N can be constructed layer-by-layer.The core computation is constructing the reachable set of each layer L i defined by a specific operation, i.e., convolution, affine mapping, max pooling, average pooling, or ReLU.

ImageStar
is a set of m images in R h×w×nc called generator images, P : R m → { , ⊥} is a predicate, and h, w, nc are the height, width and number of channels of the images respectively.The generator images are arranged to form the ImageStar's h × w × nc × m basis array.The set of images represented by the ImageStar is given as:
Sometimes we will refer to both the tuple Θ and the set of states Θ as Θ.In this work, we restrict the predicates to be a conjunction of linear constraints, 2] applied on the pixel of the position (1, 2, 1) can be described as an ImageStar depicted in Figure 1.
Remark 1.An ImageStar is an extension of the generalized star set recently defined in [3,4,34,35].In a generalized star set, the anchor and the generators are vectors, while in an ImageStar, the anchor and generators are images with multiple channels.We will later show that the ImageStar is a very efficient representation for the reachability analysis of convolutional layers, fully connected layers, and average pooling layers.

Proposition 1 (Affine mapping of an ImageStar
).An affine mapping of an ImageStar Θ = c, V, P with a scale factor γ and an offset image β is another ImageStar Θ = c , V , P in which the new anchor, generators and predicate are as follows.
Note that, the scale factor γ can be a scalar or a vector containing scalar scale factors in which each factor is used to scale one channel in the ImageStar.

Reachability of CNN using ImageStars
In this section, we present the reachable set computation for the convolutional layer, average pooling layer, fully connected layer, batch normalization layer, max pooling layer, and the ReLU layer with respect to an input set consisting of an ImageStar.

Reachability of a convolutional layer
We consider a two-dimensional convolutional layer with following parameters: the weights W Conv2d ∈ R h f ×w f ×nc×nf , the bias b Conv2d ∈ R 1×1×nf , the padding size P , the stride S, and the dilation factor D where h f , w f , nc are the height, width, and the number of channels of the filters in the layer respectively.Additionally, nf is the number of filters.The reachability of a convolutional layer is given in the following lemma.
Lemma 1.The reachable set of a convolutional layer with an ImageStar input set I = c, V, P is another ImageStar I = c , V , P where c = Convol(c) is the convolution operation applied to the anchor image, V = {v 1 , . . ., v m }, v i = ConvolZeroBias(v i ) is the convolution operation with zero bias applied to the generator images, i.e., only using the weights of the layer.
Proof.Any image in the ImageStar input set is a linear combination of the center and basis images.For any filter in the layer, the convolution operation applied to the input image performs local element-wise multiplication of a local matrix (of all channels) containing the values of the local pixels of the image and the the weights of the filter and then combine the result with the bias to get the output for that local region.Due to the linearity of the input image, we can perform the convolution operation with the bias on the center and the convolution operation with zero bias on the basis images and then combine the result to get the output image.
Example 2 (Reachable set of a convolutional layer).The reachable set of a convolutional layer with single 2 × 2 filter and the ImageStar input set in Example 1 is described in Figure 2, where the weights and the bias of the filter are

Reachability of an average pooling layer
The reachability of an average pooling layer with pooling size P S, padding size P , and stride S is given below, with its proof similar to that of the convolutional layer.
Lemma 2. The reachable set of a average pooling layer with an ImageStar input set I = c, V, P is another ImageStar I = c , V , P where c = average(c), V = {v 1 , . . ., v m }, v i = average(v i ), average(•) is the average pooling operation applied to the anchor and generator images.
Example 3 (Reachable set of an average pooling layer).The reachable set of an 2 × 2 average pooling layer with padding size P = [0 0 0 0], stride S = [2 2], and an ImageStar input set given by Example 1 is shown in Figure 3.

Reachability of a fully connected layer
The reachability of a fully connected layer is stated in the following lemma.
Lemma 3. Given a two-dimensional fully connected layer with weight , and an ImageStar input set I = c, V, P , the reachable set of the layer is another ImageStar I = c , V , P where c ).Note that it is required for consistency between the ImageStar and the weight matrix that m f c = h × w × nc, where h, w, nc are the height, width and number of channels of the ImageStar.
Proof.Similar to the convolutional layer and the average pooling layer, for any image in the ImageStar input set, the fully connected layer performs an affine mapping of the input image which is a linear combination of the center and the basis images of the ImageStar.Due to the linearity, the affine mapping of the input image can be decomposed into the affine mapping of the center image and the affine mapping without the bias of the basis images.The final result is the sum of the individual affine maps.

Reachability of a batch normalization layer
In the prediction phase, a batch normalization layer normalizes each input channel x i using the mean µ and variance σ 2 over the full training set.Then the batch normalization layer further shifts and scales the activations using the offset β and the scale factor γ that are learnable parameters.The formula for normalization is as follows.
where is a used to prevent division by zero.The batch normalization layer can be described as a tuple B = µ, σ 2 , , γ, β .The reachability of a batch normalization layer with an ImageStar input set is given in the following lemma.
Lemma 4. The reachable set of a batch normalization layer B = µ, σ 2 , , γ, β with an ImageStar input set I = c, V, P is another ImageStar I = c , V , P where: Proof.The reachable set of a batch normalization layer can be obtained in a straightforward fashion using two affine mappings of the ImageStar input set.

Reachability of a max pooling layer
Reachability of max pooling layer with an ImageStar input set is challenging because the value of each pixel in an image in the ImageStar depends on the predicate variables α i .Therefore, the local max point when applying max-pooling operation may change with the values of the predicate variables.In this section, we investigate the exact reachability and over-approximate reachability of a max pooling layer with an ImageStar input set.The first obtains the exact reachable set while the second constructs an over-approximate reachable set.
Exact reachability of a max pooling layer The central idea in the exact analysis of the max-pooling layer is finding a set of local max point candidates when we apply the max pooling operation on the image.We consider the max pooling operation on the ImageStar in Example 1 with a pool size of 2 × 2, a padding size of P = [0 0 0 0], and a stride S = [2 2] to clarify the exact analysis step-by-step.First, the max-pooling operation is applied on 4 local regions I, II, III, IV , as shown in Figure 4.The local regions II, III, IV have only one max point candidate whic is the pixel that has the maximum value in the region.It is interesting to note that region I has two max point candidates at the positions (1, 2, 1) and (2, 2, 1) and these candidates correspond to different conditions of the predicate variable α.For example, the pixel at the position (1, 2, 1) is the max point if and only if 4 Since the local region I has two max point candidates, and other regions have only one, the exact reachable set of the max-pooling layer is the union of two new ImageStars Θ 1 and Θ 2 .In the first reachable set Θ 1 , the max point of the region I is (1, 2, 1) with an additional constraint on the predicate variable α ≥ −1.For the second reachable set Θ 2 , the max point of the region I is (2, 2, 1) with an additional constraint on the predicate variable α ≤ −1.One can see that from a single ImageStar input set, the output reachable set of the max-pooling layer is split into two new ImageStars.Therefore, the number of ImageStars in the reachable set of the max-pooling layer may grow quickly if each local region has more than one max point candidates.The worst-case complexity of the number of ImageStars in the exact reachable set of the max-pooling layer is given in Lemma 5.The exact reachability algorithm is presented in the Appendix A.1.3+α * 0 −2≤α≤2 1 ) −1 ) Proof.An image in the ImageStar output set has h × w pixels in each channel.
For each pixel, in the worst case, there are p 1 × p 2 candidates.Therefore, the number of ImageStars in the output set in the worst case is O(((p 1 × p 2 ) h×w ) nc ).
Finding a set of local max point candidates is the core computation in the exact reachability of max-pooling layer.To optimize this computation, we divide the search for the local max point candidates into two steps.The first one is to estimate the ranges of all pixels in the ImageStar input set.We can solve h I × w I ×nc linear programming optimizations to find the exact ranges of these pixels, where [h I , w I , nc] is the size of the input set.However, unfortunately this is a time-consuming computation.For example, if a single linear optimization can be done in 0.01 seconds, for an ImageStar of the size 224×224×32, we need about 10 hours to find the ranges of all pixels.To overcome this bottleneck, we quickly estimate the ranges using only the ranges of the predicate variables to get rid of a vast amount of non-max-point candidates.In the second step, we solve a much smaller number of LP optimizations to determine the exact set of the local max point candidates and then construct the ImageStar output set based on these candidates.
Lemma 5 shows that the number of ImageStars in the exact reachability analysis of a max-pooling layer may grow exponentially.To overcome this problem, we propose the following over-approximate reachability method.
Over-approximate reachability of a max pooling layer The central idea of the over-approximate analysis of the max-pooling layer is that if a local region has more than one max point candidates, we introduce a new predicate variable standing for the max point of that region.We revisit the example introduced earlier in the exact analysis to clarify this idea.Since the first local region I has two max point candidates, we introduce new predicate variable β to represent the max point of this region by adding three new constraints: 1) β ≥ 4+α * 1, i.e., β must be equal or larger than the value of the first candidate ; 2) β ≥ 3 + α * 0, i.e., β must be equal or larger than the value of the second candidate; 3) β ≤ 6, i.e., β must be equal or smaller than the upper bound of the pixels values in the region.With the new predicate variable, a single over-approximate reachable set Θ can be constructed in Figure 5.The approximate reachability algorithm is presented in the Appendix A.2. 3+α * 0 −2≤α≤2

Reachability of a ReLU layer
Similar to max-pooling layer, the reachability analysis of a ReLU layer is also challenging because the value of each pixel in an ImageStar may be smaller than zero or larger than zero depending on the values of the predicate variables (ReLU (x) = max(0, x)).In this section, we investigate the exact and overapproximate reachability algorithms for a ReLU layer with an ImageStar input set.The techniques we use in this section are adapted from in [34].
Exact reachability of a ReLU layer The central idea of the exact analysis of a ReLU layer with an ImageStar input set is performing a sequence of stepReLU operations over all pixels of the ImageStar input set.Mathematically, the exact reachable set of a ReLU layer L can be computed as follows.
where N is the total number of pixels in the ImageStar input set I. The stepReLU i operation determines whether or not a split occurs at the i th pixel.If the pixel value is larger than zero, then the output value of that pixel remains the same.If the pixel value is smaller than zero than the output value of that pixel is reset to be zero.The challenge is that the pixel value depends on the predicate variables.Therefore, there is the case that the pixel value may be negative or positive with an extra condition on the predicate variables.In this case, we split the input set into two intermediate ImageStar reachable sets and apply the ReLU law on each intermediate reach set.An example of the stepReLU operation on an ImageStar is illustrated in Figure 6.The value of the first pixel value −1 + α would be larger than zero if α ≤ 1, and in this case we have ReLU (−1 + α) = −1 + α.If α <= 1, then ReLU (−1 + α) = 0 + α × 0. Therefore, the first stepReLU operation produces two intermediate reachable sets Θ 1 and Θ 2 , as shown in the figure .The number of ImageStars in the exact reachable set of a ReLU layer increases quickly along with the number of splits in the analysis, as stated in the following lemma.
Lemma 7. The worst-case complexity of the number of ImageStars in the exact analysis of a ReLU layer is O(2 N ), where N is the number of pixels in the ImageStar input set.
Proof.There are h × w × nc local regions in the approximate analysis.In the worst case, we need to introduce a new variable for each region.Therefore, the worst case complexity of new predicate variables introduced is O(h × w × nc).
Similar to [34], to control the explosion in the number of ImageStars in the exact reachable set of a ReLU layer, we propose an over-approximate reachability algorithm in the following.Over-approximate reachability of a ReLU layer The idea behind the overapproximate reachability of ReLU layer is replacing the stepReLU operation at each pixel in the ImageStar input set by an approxStepReLU operation.At each pixel where a split occurs, we introduce a new predicate variable to overapproximate the result of the stepReLU operation at that pixel.An example of the overStepReLU operation on an ImageStar is depicted in Figure 7 in which the first pixel of the input set has the ranges of [l 1 = −3, u 1 = 1] indicating that a split occurs at this pixel.To avoid this split, we introduce a new predicate variable β to over-approximate the exact intermediate reachable set (i.e., two blue segments in the figure) by a triangle.This triangle is determined by three constraints: 1) β ≥ 0 (the ReLU (x) ≥ 0 for any x); 2) β ≥ −1+α (ReLU (x) ≥ x for any x); 3) β ≤ 0.5+0.25α(upper bound of the new predicate variable).Using this over-approximation, a single intermediate reachable set Θ is produced as shown in the figure.After performing a sequence of approxStepReLU operations, we obtain a single over-approximate ImageStar reachable set for the ReLU layer.However, the number of predicate variables and the number of constraints in the obtained reachable set increase.Lemma 8.The worst case complexity of the increment of predicate variables and constraints is O(N ) and O(3 × N ) respectively, where N is the number of pixels in the ImageStar input set.
Proof.In the worst case, splits occur at all N pixels in the ImageStar input set.In this case, we need to introduce N new predicate variables to over-approximate the exact intermediate reachable set.For each new predicate variable, we add 3 new constraints.
One can see that determining where splits occur is crucial in the exact and over-approximate analysis of a ReLU layer.To do this, we need to know the ranges of all pixels in the ImageStar input set.However, as mentioned earlier, the computation of the exact range is expensive.To reduce the computation cost, we first use the estimated ranges of all pixels to get rid of a vast amount Over-approximate set Exact set β Fig. 7: approxStepReLU operation on an ImageStar. of non-splitting pixels.Then we compute the exact ranges for the pixels where splits may occur to compute the exact or over-approximate reachable set of the layer.

Reachabilty algorithm and parallelization
We have presented the core ideas for reachability analysis of different types of layers in a CNN.The reachable set of a CNN is constructed layer-by-layer in which the output reachable set of the previous layer is the input for the next layer.For the convolutional layer, average pooling layer and fully connected layer, we always can compute efficiently the exact reachable set of each layer.For the max pooling layer and ReLU layer, we can compute both the exact and the over-approximate reachable sets.However, the number of ImageStars in the exact reachable set may grow quickly.Therefore, in the exact analysis, a layer may receive multiple input sets which can be handled in parallel to speed up the computation time.The reachability algorithm for a CNN is summarized in Algorithm 4.7.1.The detail implementation of the reachability algorithm for each layer can be found in NNV [36].The proposed reachability algorithms are implemented in NNV [36], a tool for verification of deep neural networks and learning-enabled autonomous CPS.NNV utilizes core functions in MatConvNet [37] for the analysis of the convolutional and average pooling layers.The evaluation of our approach consists of two parts.First, we evaluate our approach in comparison with the zonotope [28] and polytope methods [29] re-implemented in NNV via robustness verification of deep neural networks.Second, we evaluate the scalability of our approach and the DeepPoly polytope method using real-world image classifiers, VGG16, and VGG19 [27].The experiments are done on a computer with following configurations: Intel Core i7-6700 CPU @ 3.4GHz × 8 Processor, 62.8 GiB Memory, Ubuntu 18.04.1 LTS OS. 5 Finally, we present the comparison with ERAN-DeepZ method on their ConvM axP ool network trained on CIFAR-10 data set in the Appendix of this paper.

Robustness Verification of MNIST Classification Networks
We compare our approach with the zonotope and polytope methods in two aspects including verification time and conservativeness of the results.To do that, we train 3 CNNs a small, a medium, and a large CNN with 98%, 99.7% and 99.9% accuracy respectively using the MNIST data set consisting of 60000 images of handwritten digits with a resolution of 28×28 pixels [19].The network architectures are given in Figure 13 in the Appendix.The networks classify images into ten classes: 0, 1, . . ., 9. The classified output is the index of the dimension that has maximum value, i.e., the argmax across the 10 outputs.We evaluate the robustness of the network under the well-known brightening attack used in [7].The idea of a brightening attack is that we can change the value of some pixels independently in the image to make it brighter or darker to fool the network, to misclassify the image.In this case study, we darken a pixel of an image if its value x i (between 0 and 255) is larger than a threshold d, i.e., x i ≥ d.Mathematically, we reduce the value of that pixel x i to the new value x i such that 0 The robustness verification is done as follows.We select 100 images that are correctly classified by the networks and perform the brightening attack on these, which are then used to evaluate the robustness of the networks.A network is robust to an input set if, for any attacked image, this is correctly classified by the network.We note that the input set contains an infinite number of images.Therefore, to prove the robustness of the network to the input set, we first compute the output set containing all possible output vectors of the network using reachability analysis.Then, we prove that in the output set, the correctly classified output always has the maximum value compared with other outputs.Note that we can neglect the softmax and classoutput layers of the networks in the analysis since we only need to know the maximum output in the output set of the last fully connected layer in the networks to prove the robustness of the network.
We are interested in the percentage of the number of input sets that a network is provably robust and the verification times of different approaches under different values of d and θ.When d is small, the number of pixels in the image that are attacked is large and vice versa.For example, the average number of pixels attacked (computed on 100 cases) corresponding to d = 250, 245 and 240 are 15, 21 and 25 respectively.The value of δ dictates the size of the input set that can be created by a specific attack.Stated differently it dictates the range in which the value of a pixel can be changed.For example, if d = 250 and δ = 0.01, the value of an attacked pixel many range from 0 to 2.55.
The experiments show that using the zonotope method, we cannot prove the robustness of any network.The reason is that the zonotope method obtains very conservative reachable sets.Figure 8 illustrates the ranges of the outputs computed by our ImageStar (approximate scheme), the zonotope and polytope approaches when we attack a digit 0 image with brightening attack in which d = 250 and δ = 0.05.One can see that, using ImageStar and polytope method, we can prove that the output corresponding to the digit 0 is the one that has a maximum value, which means that the network is robust in this case.However, the zonotope method produces very large output ranges that cannot be used to prove the robustness of the network.The figure also shows that our ImageStar method produces tighter ranges than the polytope method, which means our result is less conservative than the one obtained by the polytope method.We note that the zonotope method is very time-consuming.It needs 93 seconds to compute the reachable set of the network in this case, while the polytope method only needs 0.3 seconds, and our approximate ImageStar method needs 0.74 seconds.The main reason is that the zonotope method introduces many new variables when constructing the reachable set of the network, which results in the increase in both computation time and conservativeness.
The comparison of the polytope and our ImageStar method is given in Tables 1, 2, and 3.The tables show that in all networks, our method is less conservative than the polytope approach since the number of cases that our approach can prove the robustness of the network is larger than the one proved by the polytope method.For example, for the small network, for d = 240 and δ = 0.015, we can prove 71 cases while the polytope method can prove 65 cases.Importantly, the number of cases proved by DeepPoly reduces quickly when the network becomes larger.For example, for the case that d = 240 and δ = 0.015, the polytope method is able to prove the robustness of the medium network for 38 cases while our approach can prove 88 cases.This is because the polytope method becomes more and more conservative when the network or the input set is large.The tables show that the polytope method is faster than our ImageStar method on the small network.However, it is slower than the ImageStar method on any larger networks in all cases.Notably, for the large network, the ImageStar approach is significantly faster than the polytope approach, 16.65 times faster in average.

Polytope
Fig. 8: An example of output ranges of the small MNIST classification networks using different approaches.
The results also show that the polytope approach may run into memory problem for some large input sets.
Robustness Results (in Percent) Table 1: Verification results of the small MNIST CNN.

Robustness Verification of VGG16 and VGG19
In this section, we evaluate the polytope and ImageStar methods on real-world CNNs, the VGG16 and VGG19 classification networks [27].We use Foolbox [25] to generate the well-known DeepFool adversarial attacks [24] on a set of 20 bell pepper images.From an original image ori im, Foolbox generates an adversarial image adv im that can fool the network.The difference between two images is defined by dif f im = adv im − ori im.We want to verify if we apply (l + δ) percent of the attack on the original image, whether or not the network classifies the disturbed images correctly.The set of disturbed images can be represented as an ImageStar as follows disb im = ori im + (l + δ) × dif f im, where l is the percentage of the attack at which we want to verify the robustness of the network, and δ is a small perturbation around l, i.e., 0 ≤ δ ≤ δ max .Intuitively, l describes how close we are to the attack, and the perturbation δ represents the size of the input set.Table 3: Verification results of the large MNIST CNN.
Table 4 shows the verification results of VGG16 and VGG19 with different levels of the DeepFool attack.The networks are robust if they classify correctly the set of disturbed images disb im as bell peppers.To guarantee the robustness of the networks, the output corresponding to the bell pepper label (index 946) needs to be the maximum output compared with others.The table shows that with a small input set, small δ, the polytope and ImageStar can prove the robustness of VGG16 and VGG19 with a reasonable amount of time.Notably, the verification times as well as the robustness results of the polytope and ImageStar methods are similar when they deal with small input sets except for two cases where ImageStar is faster than the polytope method.It is interesting to note that according to the verification results for the VGG and MNIST networks, deep networks seem to be more robust than shallow networks.

Exact Analysis vs. Approximate Analysis
We have compared our ImageStar approximate scheme with the zonotope and polytope approximation methods.It is interesting to investigate the performance of ImageStar exact scheme in comparison with the approximate one.To illustrate the advantages and disadvantages of the exact scheme and approximate scheme, we consider the robustness verification of VGG16 and VGG19 on a single Im-ageStar input set created by an adversarial attack on a bell pepper image.The verification results are presented in Table 5.The table shows that for a small perturbation δ, the exact and over-approximate analysis can prove the robustness of the VGG16 around some specific levels of attack in approximately one minute.We can intuitively verify the robustness of the VGG networks via visualization of their output ranges.An example of the output ranges of VGG19 for the case of l = 0.95%, δ max = 2 × 10 −7 is depicted in Figure 9.One can see from the figure that the output of the index 946 corresponding to the bell pepper label is always the maximum one compared with others, which proves that VGG19 is robust in this case.From the table, it is interesting that VGG19 is not robust if we apply ≥ 98% of the attack.Notably, the exact analysis can give us correct answers with a counter-example set in this case.However, the over-approximate analysis cannot prove that VGG19 is not robust since its obtained reachable set is an over-approximation of the exact one.Therefore, it may be the case that the over-approximate reachable set violates the robustness property because of its conservativeness.A counter-example generated by the exact analysis method is depicted in Figure 10 in which the disturbed image is classified as strawberry instead of bell pepper since the strawberry output is larger than the bell pepper output in this case.Table 5: Verification results of the VGG16 and VGG19 in which V T is the verification time (in seconds) using the ImageStar exact and approximate schemes.
To optimize the verification time, it is important to know the times consumed by each type of layers in the reachability analysis step. Figure 11

Discussion
When we apply our approach on real-world networks, it has been shown that the size of the input set is the most important factor that affects the performance of verification approaches.However, this important issue has not been emphasized in the existing literature.Most of the existing approaches focus on the size of the network that they can analyze.We believe that all methods (including the method we proposed in this paper) are scalable for large networks only for small input sets.When the input set is large, it causes three major problems in the analysis, which are the explosions in 1) computation time; 2) memory usage; and 3) conservativeness.In the exact analysis method, a large input set causes more splits in the max-pooling layer and the ReLU layer.A single ImageStar may split into many new ImageStars after these layers, which leads to the explosion in the number of ImageStars in the reachable set as shown in Figure 12.Therefore, it requires more memory to handling the new ImageStars and more time for the computation.One may think that the over-approximate method can overcome this challenge since it obtains only one ImageStar at each layer and the cost we need to pay is only the conservativeness of the result.The fact is, an over-approximate method usually helps reduce the computation time, as shown in the experimental results.However, it is not necessarily efficient in terms of memory consumption.the dimensions of the ImageStars obtained in the first several convolutional layers are usually large.Therefore, if splitting happens in these layers, we may need to deal with "out of memory" problem.We see that all existing approaches such as the zonotope [28] and polytope [29], all face the same challenges.Additionally, the conservativeness of an over-approximate reachable set is a crucial factor in evaluating an over-approximation approach.Therefore, the exact analysis still plays an essential role in the analysis of neural networks since it helps to evaluate the conservativeness of the over-approximation approaches.

Conclusion
We have proposed a new set-based method for robustness verification of deep CNNs using the concept of the ImageStar.The core of our method are the exact and over-approximate reachability algorithms for ImageStar input sets.The experiments show that our approach is less conservative than the recent zonotope [28] and polytope [29] approaches.It is also faster than these approaches when dealing with deep networks.Notably, our approach can be applied to the robustness of real-world CNNs with small perturbed input sets.It can also compute the exact reachable set and visualize the exact output range of deep CNNs, and the analysis can sped up significantly using parallel computing.We have found and shown the size of the input set to be an important factor that affects the reachability algorithms performance.Our future work is improving the proposed method to deal with larger input sets and optimizing the memory and time complexity of our computations.

Fig. 5 :Lemma 6 .
Fig.5: Over-approximate reachability of max pooling layer using ImageStar.Lemma 6.The worst-case complexity of the new predicate variables introduced in the over-approximate analysis is O(h × w × nc) where [h, w, nc] is the size of the ImageStar output set.

Fig. 10 :
Fig. 10: A counter-example shows that VGG19 misclassifies the input image as a strawberry instead of a bell pepper.

Fig. 11 :
Fig. 11: Total reachability time of each type of layers in the VGG19 in which the max pooling and ReLU layers dominate the total reachability time of the network.

Fig. 12 :
Fig. 12: Number of ImageStars in exact analysis increases with input size. 015

Table 2 :
Verification results of the medium MNIST CNN.

Table 4 :
Verification results of VGG networks.
Yes 63.67108 Yes 59.69298No 106.84492Unknown 101.04668 de- The reason is, if there is a split, it introduces a new predicate variable and new generator.If the number of generators and the dimensions of the ImageStar are large, it requires a massive amount of memory to store the over-approximate reachable set.For instance, if there are 100 splits happened in the first ReLU layer of the VGG19, the second convolutional layer will receive an ImageStar of size 224 × 224 × 64 with 100 generators.To store this ImageStar with double precision, we need approximately 2.4GB of memory.In practice,