Verifying Attention Robustness of Deep Neural Networks Against Semantic Perturbations

Abstract

It is known that deep neural networks (DNNs) classify an input image by paying particular attention to certain specific pixels; a graphical representation of the magnitude of attention to each pixel is called a saliency-map. Saliency-maps are used to check the validity of the classification decision basis, e.g., it is not a valid basis for classification if a DNN pays more attention to the background rather than the subject of an image. Semantic perturbations can significantly change the saliency-map. In this work, we propose the first verification method for attention robustness, i.e., the local robustness of the changes in the saliency-map against combinations of semantic perturbations. Specifically, our method determines the range of the perturbation parameters (e.g., the brightness change) that maintains the difference between the actual saliency-map change and the expected saliency-map change below a given threshold value. Our method is based on activation region traversals, focusing on the outermost robust boundary for scalability on larger DNNs. We empirically evaluate the effectiveness and performance of our method on DNNs trained on popular image classification datasets.

H Appendix

1.1 H.1 Linearity of Activation Regions

Given activation pattern $p \in AP^f$ as constant, within activation region $ar^f(p)$ each output of ReLU-FNN $f_j(x \in ar^f(p))$ is linear for x (cf. Fig. 8) because all ReLU operators have already resolved to 0 or x [14]. i.e., $f_j(x \in ar^f(p)) = A'_j x + b'_j$: where, $A'_j$ and $b'_j$ denote simplified weights and bias about activation pattern p and class j. That is, the gradient of each ReLU-FNN output $f_j(x)$ within activation region $ar^f(p)$ is constant, i.e., the following equation holds: where $C \in \mathbb {R}$ is a constant value.

Fig. 8.

An example of activation regions [14]. ReLU-FNN output is linear on each activation region, i.e., each output plane painted for each activation region is flat.

$$\begin{aligned} Feasible^f(p \in AP^f) \Rightarrow \frac{\partial f_j(x)}{\partial x_i} = C \;\; (x \in ar^f(p)) \end{aligned}$$

(1)

An activation region can be interpreted as the H-representation of a convex polytope on input space $\mathbb {R}^{N^f}$. Specifically, neuron activity $p_{l,n}$ and p have a one-to-one correspondence with a half-space and convex polytope defined by the intersection (conjunction) of all half-spaces, because $f_n^{(l)}(x)$ is also linear when $p \in AP^f$ is constant. Therefore, we interpret activation region $ar^f(p)$ and the following H-representation of convex polytope $HConvex^f(x;p)$ each other as needed: where, $A''$ and $b''$ denote simplified weights and bias about activation pattern p, and $A''_{l,n} x \le b''_{l,n}$ is the half-space corresponding to the n-th neuron activity $p_{l,n}$ in the l-th layer.

$$\begin{aligned} \begin{aligned} HConvex^f(x;p) {\mathop {=}\limits ^{\textrm{def}}}A'' x \le b'' \;\equiv \; \bigwedge _{l,n} A''_{l,n} x \le b''_{l,n} \end{aligned} \end{aligned}$$

(2)

1.2 H.2 Connectivity of Activation Regions

When feasible activation patterns $p,p' \in AP^f$ are in a relationship with each other that flips single neuron activity $p_{l,n} \in \{0,1\}$, they are connected regions because they share single face $HFace^f_{l,n}(x;p) {\mathop {=}\limits ^{\textrm{def}}}A''_{l,n} x = b''_{l,n}$ corresponding to flipped $p_{l,n}$ [18]. It is possible to flexibly traverse activation regions while ensuring connectivity by selecting a neuron activity to be flipped according to a prioritization; several traversing methods have been proposed [9, 18, 21]. However, there are generally rather many neuron activities that become infeasible when flipped [18]. For instance, half-spaces $h_{1,3}$ is a face of activation region $\eta$ in Fig. 6(1a); thus, flipping neuron activity $p_{1,3}$, GBS can traverse connected region $\eta$ in Fig. 6(1b). In contrast, half-space $h_{1,1}$ is not a face of activation region $\eta$ in Fig. 6(1a); thus, flipping neuron activity $p_{1,1}$, the corresponded activation region is infeasible (i.e., the intersection of flipped half-spaces has no area).

1.3 H.3 Hierarchy of Activation Regions

When feasible activation patterns $p,p' \in AP^f$ are in a relationship with each other that matches all of $L'^f$-th upstream activation pattern $p_{\le L'^f} {\mathop {=}\limits ^{\textrm{def}}}[ p_{l,n} \mid 1 \le l \le L'^f, 1 \le n \le N^f_l ] \;\; (1 \le L'^f \le L^f)$, they are included parent activation region $ar^f_{\le L'^f}(p)$ corresponding to convex polytope $HConvex^f_{\le L'^f}(x;p) {\mathop {=}\limits ^{\textrm{def}}}\bigwedge _{l \le L'^f,n} A''_{l,n} x \le b''_{l,n}$ [21]. That is, $\forall x \in ar^f(p).\; x \in ar^f_{\le L'^f}(p)$ and $\forall x \in \mathbb {R}^{N^f}.\; HConvex^f(x;p) \Rightarrow HConvex^f_{\le L'^f}(x;p)$.

Similarly, we define $L'^f$-th downstream activation pattern as $p_{\ge L'^f} {\mathop {=}\limits ^{\textrm{def}}}[ p_{l,n} \mid L'^f \le l \le L^f, 1 \le n \le N^f_l ] \;\; (1 \le L'^f \le L^f)$.

1.4 H.4 Linear Programming on an Activation Region

Based on the linearity of activation regions and ReLU-FNN outputs, we can use Linear Programming (LP) to compute (a) the feasibility of an activation region, (b) the flippability of a neuron activity, and (c) the minimum (maximum) of a ReLU-FNN output within an activation region. We show each LP encoding of the problems (a,b,c) in the SciPy LP form²: where, $p \in AP^f$ is a given activation pattern of ReLU-FNN f, and $p_{l,n}$ is a give neuron activity to be flipped.

$$\begin{aligned} \begin{aligned} \mathbf {(a)} \;\;&\exists x \in \mathbb {R}^{N^f}.\; HConvex^f(x;p) {\mathop {\longrightarrow }\limits ^{\textrm{encode}}}\min _{x}{\textbf{0} x} \;\; \mathbf {s.t.,} \; A'' x \le b'' \\ \mathbf {(b)} \;\;&\exists x \in \mathbb {R}^{N^f}.\; HConvex^f(x;p) \wedge HFace^f_{l,n}(x;p) \\&\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; {\mathop {\longrightarrow }\limits ^{\textrm{encode}}}\min _{x}{\textbf{0} x} \;\; \mathbf {s.t.,} \; A'' x \le b'' ,\; A''_{l,n} x = b''_{l,n} \\ \mathbf {(c)} \;\;&\min _{x}{f_j(x)} \;\; \mathbf {s.t.,} \; HConvex^f(x;p) {\mathop {\longrightarrow }\limits ^{\textrm{encode}}}\Bigl ( \min _{x}{A'_j x} \;\; \mathbf {s.t.,} \; A'' x \le b'' \Bigr ) + b'_j \\ \end{aligned} \end{aligned}$$

(3)

1.5 H.5 Full Encoding Semantic Perturbations

We focus here on the perturbations of brightness change (B), patch (P), and translation (T), and then describe how to encode the combination of them into ReLU-FNN $g^{x0}: \varTheta \rightarrow X$: where, $|\theta ^{(l)}| = \dim \theta ^{(l)}$, w is the width of image x0, px, py, pw, ph are the patch x-position, y-position, width, height, and tx is the amount of movement in x-axis direction. Here, perturbation parameter $\theta \in \varTheta$ consists of the amount of brightness change for (B), the density of the patch for (P), and the amount of translation for (T). In contrast, perturbation parameters not included in the dimensions of $\varTheta$, such as w, px, py, pw, ph, tx, are assumed to be given as constants before verification.

$$\begin{aligned} \begin{aligned}&A^{(B)} = \left[ a^{(B)}_{r,c}\right] , A^{(P)} = \left[ a^{(P)}_{r,c}\right] , A^{(T)} = \left[ a^{(T)}_{r,c}\right] \\ \end{aligned} \end{aligned}$$

Fig. 9.

MNIST images used for experiments.

Fig. 10.

Fashion-MNIST images used for experiments.

1.6 H.6 Images Used for Our Experiments

We used 10 images (i.e., Indexes 69990–69999) selected from the end of the MNIST dataset (cf. Fig. 9) and the Fashion-MNIST dataset (cf. Fig. 10), respectively. We did not use these images in the training of any ReLU-FNNs.

1.7 H.7 An Example of Lemma 1

Lemma 1 is reprinted below (Fig. 11).

$$\begin{aligned} \frac{\partial f_j(x)}{\partial x_i} = C \;\; (x \in \{ g^{x0}(\theta ) \mid \theta \in ar^{f \circ g}(p) \}) \end{aligned}$$

Fig. 11.

An image for a small example of Lemma 1.

1.8 H.8 Algorithm BFS

Algorithm BFS traverses entire activation regions in perturbation parameter space $\varTheta$, as shown in Fig. 12.

Fig. 12.

Examples of BFS results. (Near the edges, polygons may fail to render, resulting in blank regions.)

Algorithm BFS initializes Q with $ap^{f \circ g^{x0}}(\textbf{0})$ (Line 3). Then, for each activation pattern p in Q (Lines 5–6), it reconstructs the corresponding activation region $\eta$ (subroutine constructActivationRegion, Line 8) as the H-representation of p (cf. Eq. 2). Next, for each neuron in $f \circ g^{x0}$ (Line 12), it checks whether the neuron activity $p_{l,n}$ cannot flip within the perturbation parameter space $\varTheta$, i.e., one of the half-spaces has no feasible points within $\varTheta$ (subroutine isStable, Line 13). Otherwise, a new activation pattern $p'$ is constructed by flipping $p_{l,n}$ (subroutine flipped, Line 14) and added to the queue (Line 20) if $p'$ is feasible (subroutine calcInteriorPointOnFace, Lines 17–18). Finally, the activation region $\eta$ is simplified (Line 24) and used to verify CR (subroutine solveCR and solveVR, Lines 25–27, cf. Sect. 4.4) and VR (subroutine solveAR and solveIR, Lines 32–34, cf. Sect. 4.4).

1.9 H.9 Details of Experimental Results

Table 2 shows the breakdown of verification statuses in experimental results for each algorithm and each DNN size (cf. Sect. 5). In particular, for traversing AR boundaries, we can see the problem that the ratio of “Timeout” and “Failed (out-of-memory)” increases as the size of the DNN increases. This problem is because gbs-AR traverses more activation regions by the width of the hyperparameter $w^\delta$ than gbs-CR. It would be desirable in the future, for example, to traverse only the small number of activation regions near the AR boundary.

Table 2. Breakdown of verification statuses. “Robust” and “NotRobust” mean algorithm found “only robust regions” and “at least one not-robust region”, respectively. “Timeout” and “Failed” mean algorithm did not finish “within 2 h” and “due to out-of-memory”, respectively.