1 Introduction

Deep neural networks (DNNs) have achieved human-level performance in several tasks, and are increasingly being incorporated into various application domains such as autonomous driving [4] and medical diagnostics [53]. Modern DNNs usually contain a great many parameters which are typically stored as 32/64-bit floating-point numbers, and require a massive amount of floating-point operations to compute the output for a single input [60]. As a result, it is often challenging to deploy them on resource-constrained, embedded devices. To mitigate the issue, quantization, which quantizes 32/64-bit floating-points to low bit-width fixed-points (e.g., 4-bits) with little accuracy loss [23], emerges as a promising technique to reduce resource requirements. In particular, binarized neural networks (BNNs) [27] represent the case of 1-bit quantization using the bipolar binaries \(\pm 1\). BNNs can drastically reduce memory storage and execution time with bit-wise operations, hence substantially improve the time and energy efficiency. BNNs have been demonstrated to achieve a high accuracy for a wide variety of applications [34, 41, 52].

DNNs have been shown to lack robustness [11, 14, 36, 49, 59] and interpretability of the predictions they make [25, 43]. Various formal techniques and heuristics have been proposed to analyze DNNs and interpret their behaviors, most of which focus on real-numbered DNNs only. Verification of quantized DNNs has not been thoroughly explored so far, although recent results have highlighted its importance: it was shown that a quantized DNN does not necessarily preserve the properties satisfied by the real-numbered DNN before quantization [14, 22]. Indeed, the fixed-point number semantics effectively yields a discrete state space for the verification of quantized DNNs whereas real-numbered DNNs feature a continuous state space. The discrepancy could invalidate current verification techniques for real-numbered DNNs when they are directly applied to the quantized counterparts (e.g., both false negative and false positive could occur). Therefore, specialized techniques are required for rigorously verifying quantized DNNs.

Broadly speaking, the existing techniques for quantized DNNs make use of constraint solving which is based on either SAT/SMT or (reduced, ordered) binary decision diagrams (BDDs). A majority of work resorts to SAT/SMT solving. For the 1-bit quantization (i.e., BNNs), typically BNNs are transformed into Boolean formulas where SAT solving is harnessed [12, 33, 45, 46]. Some recent work also studies variants of BNNs [28, 48], i.e., BNNs with ternary weights. For quantized DNNs with multiple bits (i.e., fixed-points), it is natural to encode them as quantifier-free SMT formulas, e.g., using bit-vector and fixed-point theories [7, 22, 24], so that off-the-shelf SMT solvers can be leveraged. In another direction, BDD-based approaches currently can tackle BNNs only [54]. In a nutshell, they encode a BNN and an input region as a BDD, based on which various analyses can be performed via queries on the BDD. The crux of the approach is how to generate the BDD efficiently. In the work [54], the BDD is constructed by BDD learning [44], thus, currently limited to toy BNNs (e.g., 64 input size, 5 hidden neurons, and 2 output size) with relatively small input regions.

On the other hand, existing work mostly focuses on qualitative verification, which asks whether there exists an input x (in a specified region) for a neural network such that a property (e.g., local robustness) is violated. In many practical applications, checking only the existence is not sufficient. Indeed, for local robustness, such an (adversarial) input almost surely exists which makes a qualitative answer less meaningful. Instead, quantitative verification, which asks how often a property \(\phi \) is satisfied or violated, is far more useful yet more challenging as it could provide a probabilistic guarantee of the behavior of neural networks. Such a quantitative guarantee is essential to certify, for instance, certain implementations of neural network based perceptual components against safety standards of autonomous vehicles  [29, 32]. Quantitative analysis of general neural networks, however, is challenging, hence received little attention and for which the results are rather limited so far. DeepSRGR [69] presented an abstract interpretation based quantitative robustness verification approach for DNNs which is sound but incomplete. For BNNs, approximate SAT model-counting solvers (\(\sharp \)SAT) are leveraged [6, 47] based on the SAT encoding for the qualitative counterpart. Though probably approximately correct (PAC) style guarantees can be provided, verification cost is usually prohibitively high to achieve higher precision and confidence.

Main Contributions. We propose a BDD-based framework BDD4BNN to support quantitative analysis of BNNs. The main challenge is how to efficiently build BDDs from BNNs  [47]. In contrast to previous work [54] which is learning-based and largely treats the BNN as a blackbox, we directly encode a BNN and the associated input region into BDDs. In a nutshell, a BNN is a sequential composition of multiple internal blocks and one output block. Each block comprises 3 layers and captures a function \(f:\{+1,-1\}^{n}\rightarrow \{+1,-1\}^{m}\), where n (resp. m) denotes the number of inputs (resp. outputs) of the block. Technically, the function f can be alternatively rewritten as a function over the standard Boolean domain, i.e., \(f:\{0,1\}^{n}\rightarrow \{0,1\}^{m}\). A key stepping-stone of our encoding is the observation that the i-th output \(y_i\) of the block can be captured by a cardinality constraint of the form \(\sum _{j=1}^n \ell _j\ge k\) such that \(y_i=+1\Leftrightarrow \sum _{j=1}^n \ell _j\ge k\), where each literal \(\ell _j\) is either \(x_j\) or \(\lnot x_j\) for the input variable \(x_j\), and k is a constant. We then present an algorithm to encode a cardinality constraint \(\sum _{j=1}^n \ell _j\ge k\) as a BDD with \(O((n-k)\cdot k)\) nodes in \(O((n-k)\cdot k)\) time. As a result, the input-output relation of each block can be encoded as a BDD, the composition of which yields the BDD for the entire BNN. A distinguished advantage of our BDD encoding lies in its support of incremental encoding. In particular, when different input regions are of interest, there is no need to construct the BDD of the entire BNN from scratch.

Encoding BNNs as BDDs enables a wide variety of applications in security analysis and decision explanation of BNNs. In this paper, we highlight two of them within our framework, i.e., robustness analysis and interpretability. It was shown that DNNs have been suffering from poor robustness to adversarial examples [49, 50, 59]. We consider two quantitative variants of the problem: (1) how many adversarial examples does the BNN have in the input region, and (2) how many of them are misclassified to each class? We further provide an algorithm to incrementally compute the (locally) maximal Hamming distance within which the BNN satisfies the desired robustness properties.

Interpretability is an issue arisen as a result of the blackbox nature of DNNs [25, 43]. In application domains such as medical diagnosis, understanding the decisions made by DNNs is a must. We consider two problems: (1) why some inputs are (mis)classified into a class by the BNN and (2) are there any essential features in the input region that are common for all samples classified into a class?

Experimental Results. We implement our framework as a prototype tool BDD4BNN using the CUDD package [58], which scales to BNNs with up to 4 internal blocks, 200 hidden neurons, and 784 input size. To the best of our knowledge, it is the first work to precisely and quantitatively analyze such large BNNs that go significantly beyond the state-of-the-art. The experimental results show that BDD4BNN is significantly more efficient and scalable than the learning-based technique [54]. Furthermore, we demonstrate how BDD4BNN can be used in quantitative robustness analysis and decision explanation of BNNs. For quantitative robustness analysis, our experimental results show that BDD4BNN is considerably (\(5\times \) to \(1,340\times \)) faster and more accurate than the state-of-the-art approximate \(\sharp \)SAT-based approach [6]. It can also compute precisely the distribution of predicated classes of the images in the input region as well as the locally maximal Hamming distances on several BNNs. For decision explanation, we show the effectiveness of BDD4BNN in computing prime-implicant explanations and essential features of the given input region for some target classes. Note that this work focuses on quantitative verification and interpretability of BNNs and may under-perform SAT/SMT-based methods [12, 33, 45, 46] for qualitative verification of BNNs.

In general, our main contributions can be summarized as follows.

Fig. 1.
figure 1

Architecture of a BNN with \(d+1\) blocks

Full size image

  • We introduce a novel algorithmic approach for encoding BNNs into BDDs that exactly preserves the semantics of BNNs and supports incremental encoding.

  • We propose a framework for quantitative verification of BNNs and in particular, we demonstrate the robustness analysis and interpretability of BNNs.

  • We implement the framework as an end-to-end tool BDD4BNN and conduct thorough experiments on various BNNs, demonstrating the efficiency and effectiveness of BDD4BNN.

2 Preliminaries

In this section, we briefly introduce binarized neural networks (BNNs) and (reduced, ordered) binary decision diagrams (BDDs).

We denote by \(\mathbb {R}\), \(\mathbb {N}\), \({\mathbb {B}}\), and \({\mathbb {B}}_{\pm 1}\) the set of real numbers, the set of natural numbers, the standard Boolean domain \(\{0,1\}\) and the integer set \(\{+1,-1\}\). For \(n\in \mathbb {N}\), we denote by [n] the set \(\{1,\cdots ,n\}\). We will use \({\boldsymbol{W}}\), \({\boldsymbol{W}}', \dots \) to denote (2-dimensional) matrices, \({\boldsymbol{x}}, {\boldsymbol{v}},\cdots \) to denote (row) vectors, and \(x,v,\dots \) to denote scalars. We denote by \({\boldsymbol{W}}_{i,:}\) and \({\boldsymbol{W}}_{:,j}\) the i-th row and j-th column of the matrix \({\boldsymbol{W}}\). Similarly, we denote by \({\boldsymbol{x}}_j\) and \({\boldsymbol{W}}_{i,j}\) the j-th entry of \({\boldsymbol{x}}\) and \({\boldsymbol{W}}_{i,:}\) respectively. In this work, Boolean values 1/0 will be used as integers 1/0 in arithmetic computations without typecasting.

2.1 Binarized Neural Networks

A binarized neural network (BNN) [27] is a neural network where weights and activations are predominantly binarized over the domain \({\mathbb {B}}_{\pm 1}\). In this work, we consider feed-forward BNNs. As shown in Fig. 1, a BNN can be seen as a sequential composition of several internal blocks and one output block. Each internal block comprises 3 layers: a linear layer (LIN), a batch normalization layer (BN), and a binarization layer (BIN). The output block comprises a linear layer and an ARGMAX layer. Note that the input/output of internal blocks and the input of the output block are all vectors over \({\mathbb {B}}_{\pm 1}\).

Definition 1

A BNN \(\mathcal {N}:{\mathbb {B}}_{\pm 1}^{n_1}\rightarrow {\mathbb {B}}^{s}\) with s classes is given by a tuple of blocks \((t_1,\cdots ,t_d,t_{d+1})\) such that \(\mathcal {N}=t_{d+1}\circ t_{d}\circ \cdots \circ t_1,\)

  • for every \(i\in [d]\), \(t_i:{\mathbb {B}}_{\pm 1}^{n_i}\rightarrow {\mathbb {B}}_{\pm 1}^{n_{i+1}}\) is an internal block comprising a LIN layer \(t_i^{lin}\), a BN layer \(t_i^{bn}\) and a BIN \(t_i^{bin}\) with \(t_i=t_i^{bin} \circ t_i^{bn} \circ t_i^{lin}\),

  • \(t_{d+1}:{\mathbb {B}}_{\pm 1}^{n_{d+1}}\rightarrow {\mathbb {B}}^{s}\) is the output block comprising a LIN layer \(t_{d+1}^{lin}\) and an ARGMAX layer \(t_{d+1}^{am}\) with \(t_{d+1}=t_{d+1}^{am} \circ t_{d+1}^{lin}\),

where \(t_i^{bin}\), \(t_i^{bn}\), \(t_i^{lin}\) for \(i\in [d]\), \(t_{d+1}^{lin}\) and \(t_{d+1}^{am}\) are given in Table 1.

Intuitively, a LIN layer is a linear transformation. A BN layer following a LIN layer is used to standardize and normalize the output of the LIN layer. A BIN layer is used to binarize the real-numbered output vector of the BN layer. In this work, we consider the sign function which is widely used in BNNs to binarize real-numbered vectors. An ARGMAX layer follows a LIN layer and outputs the index of the largest entry as the predicted class which is represented by a one-hot vector. (In case there is more than one such entry, the first one is returned.) Formally, given a BNN \(\mathcal {N}=(t_1,\cdots ,t_d,t_{d+1})\) and an input \({\boldsymbol{x}}\in {\mathbb {B}}_{\pm 1}^{n_1}\), \(\mathcal {N}({\boldsymbol{x}})\in {\mathbb {B}}^{s}\) is a one-hot vector in which the index of the non-zero entry is the predicated class.

Table 1. Definitions of layers in BNNs, where \(n_{d+2}=s\) and \(\arg \max (\cdot )\) returns the index of the largest entry which occurs first.
Full size table

2.2 Binary Decision Diagrams

A BDD [9] is a rooted acyclic directed graph where non-terminal nodes v are labeled by Boolean variables \(\textsf {var}(v)\) and terminal nodes (leaves) v are labeled with values \(\textsf {val}(v)\in {\mathbb {B}}\), referred to as the 1-leaf and the 0-leaf respectively. Each non-terminal node v has two outgoing edges: \(\textsf {hi}(v)\) meaning \(\textsf {var}(v)=1\) and \(\textsf {lo}(v)\) meaning \(\textsf {var}(v)=0\). We will also refer to \(\textsf {hi}(v)\) and \(\textsf {lo}(v)\) as the \(\textsf {hi}\) and \(\textsf {lo}\) children of v respectively. Moreover, assuming that \(x_1,\cdots ,x_m\) is the variable ordering, for each node v with \(\textsf {var}(v)=x_i\) and each \(v'\in \{\textsf {hi}(v),\textsf {lo}(v)\}\) with \(\textsf {var}(v')=x_j\), we have \(i<j\). In the graphical representation of BDDs, \(\textsf {hi}(v)\) and \(\textsf {lo}(v)\) are depicted by solid and dashed lines respectively. Multi-Terminal Binary Decision Diagrams (MTBDDs) are a variant of BDDs in which the terminal nodes are not restricted to be 0 or 1. A BDD is reduced if it (1) has only one 1-leaf and one 0-leaf, (2) does not contain a node v such that \(\textsf {hi}(v)=\textsf {lo}(v)\), and (3) does not contain two distinct non-terminal nodes v and \(v'\) such that \(\textsf {var}(v)=\textsf {var}(v')\), \(\textsf {hi}(v)=\textsf {hi}(v')\) and \(\textsf {lo}(v)=\textsf {lo}(v')\). For example, Fig. 2 shows the reduced BDD for the Boolean function \(f(x_1,y_1,x_2,y_2)= (x_1 \Leftrightarrow y_1) \wedge (x_2 \Leftrightarrow y_2)\). Hereafter, we assume that BDDs are reduced.

Bryant [9] showed that BDDs can serve as a canonical form of Boolean functions. Given a BDD over variables \(x_1,\cdots ,x_m\), each non-terminal node v with \(\textsf {var}(v)=x_i\) represents a Boolean function \(f_v=(x_i\wedge f_{\textsf {hi}(v)})\vee (\lnot x_i\wedge f_{\textsf {lo}(v)})\). Operations on Boolean functions can usually be efficiently implemented via manipulating their BDD representations. A good variable ordering is crucial for the performance of BDD manipulations while the problem of finding an optimal ordering for a function is NP-hard. To store and manipulate BDDs efficiently, the nodes are stored in a hash table and the recent computed results are stored in a cache to avoid duplicated computations. In this work, we will use some basic BDD operations such as \(\textsc {ITE}\) for If-Then-Else, \(\textsc {Xor}\) for exclusive-OR, \(\textsc {Xnor}\) for exclusive-NOR (i.e., \(a~ \textsc {Xnor}~ b=\lnot (a~\textsc {Xor}~b)\)) and \(\textsc {SatAll}(f_v)\) for the set of all solutions of the Boolean formula \(f_v\). We denote by \(\mathcal {L}(v)\) the set \(\textsc {SatAll}(f_v)\). For easy reference, more operations are given in Table 2. By \(op(v,v')\) we denote the operation \(\textsc {Apply}(v,v',op)\).

Fig. 2.
figure 2

The reduced BDD for \(f(x_1,y_1,x_2,y_2)=(x_1\Leftrightarrow y_1) \wedge (x_2 \Leftrightarrow y_2)\)

Full size image
Table 2. Some basic BDD operations, where \(op\in \{\textsc {And},\textsc {Or},\textsc {Xor},\textsc {Xnor}\}\)
Full size table

3 BDD4BNN Design

3.1 BDD4BNN Overview

An overview of BDD4BNN is depicted in Fig. 3. BDD4BNN comprises four main components: Region2BDD, BNN2CC, BDD Model Builder, and Query Engine. For a fixed BNN \(\mathcal {N}=(t_1,\cdots ,t_d,t_{d+1})\) and a region R of the input space of \(\mathcal {N}\), BDD4BNN constructs the BDDs \(({G}_i^{out})_{i\in [s]}\) to encode the input-output relation of \(\mathcal {N}\) in the region R, where the BDD \({G}_i^{out}\) corresponds to the class \(i\in [s]\). Technically, the region R is partitioned into s parts represented by \(({G}_i^{out})_{i\in [s]}\). For each property query, BDD4BNN analyzes \(({G}_i^{out})_{i\in [s]}\) and outputs the query result.

Fig. 3.
figure 3

Overview of BDD4BNN

Full size image

The general workflow of our approach is as follows. First, Region2BDD builds up a BDD \({G}_{R}^{in}\) from the region R which represents the desired input space of \(\mathcal {N}\) for analysis. Second, BNN2CC transforms each block of the BNN \(\mathcal {N}\) into a set of cardinality constraints (CCs) similar to [6, 46]. Third, BDD Model Builder builds the BDDs \(({G}_i^{out})_{i\in [s]}\) from all the cardinality constraints and the BDD \({G}_{R}^{in}\). Finally, Query Engine answers queries by analyzing the BDDs \(({G}_i^{out})_{i\in [s]}\). Our Query Engine currently supports two types of application queries: robustness analysis and interpretability.

In the rest of this section, we first introduce the key sub-component CC2BDD, which provides an encoding of cardinality constraints into BDDs. We then provide details of the components Region2BDD, BNN2CC, and BDD Model Builder. The Query Engine will be described in Sect. 4.

3.2 CC2BDD: Cardinality Constraints to BDDs

A cardinality constraint is a constraint of the form \(\sum _{j=1}^n \ell _j\ge k\) over a vector \(\boldsymbol{x}\) of Boolean variables with length n, where the literal \(\ell _j\) is either \({\boldsymbol{x}}_j\) or \(\lnot {\boldsymbol{x}}_j\) for each \(j\in [n]\). Note that constraints of the form \(\sum _{j=1}^n \ell _j>k\), \(\sum _{j=1}^n \ell _j\le k\) and \(\sum _{j=1}^n \ell _j<k\) are equivalent to \(\sum _{j=1}^n \ell _j\ge k+1\), \(\sum _{j=1}^n \lnot \ell _j\ge n-k\) and \(\sum _{j=1}^n \lnot \ell _j\ge n-k+1\), respectively. We assume that 1 (resp. 0) is a special cardinality constraint that always holds (resp. never holds).

Fig. 4.
figure 4

Graphic representation of BDDs using Algorithm 1

Full size image

To encode \(\sum _{j=1}^n \ell _j\ge k\) as a BDD, we observe that all the possible solutions of \(\sum _{j=1}^n \ell _j\ge k\) can be compactly represented by a BDD-like graph shown in Fig. 4(a), where each node is labeled by a literal, and a solid (resp. dashed) edge from a node labeled by \(\ell _j\) means that the value of the literal \(\ell _j\) is 1 (resp. 0). Thus, each path from the \(\ell _1\)-node to the 1-leaf through the \(\ell _j\)-node (where \(1\le j\le n\)) captures a set of valuations where \(\ell _j\) followed by a (horizontal) dashed line is set to be 0 while \(\ell _j\) followed by a (vertical) solid line is set to be 1, and all the other literals which are not along the path can take arbitrary values. Clearly, for each of these valuations, there are at least k positive literals, hence the constraint \(\sum _{j=1}^n \ell _j\ge k\) holds.

figure a

Based on the above observation, we build the BDD for \(\sum _{j=1}^n \ell _j\ge k\) using Algorithm 1. It builds a BDD for each node in Fig. 4(a), row-by-row (the index i in Algorithm 1) and from right to left (the index j in Algorithm 1). For each node at the i-th row and j-th column, the label of the node must be the literal \(\ell _{i+j-1}\). We build the BDD \({G}_{i,j}= \textsc {ITE}({\boldsymbol{x}}_{i+j-1},{G}_{i+1,j},{G}_{i,j+1})\) if \(\ell _{i+j-1}\) is of the form \({\boldsymbol{x}}_{i+j-1}\) (Line 6), otherwise we build the BDD \({G}_{i,j}= \textsc {ITE}({\boldsymbol{x}}_{i+j-1},{G}_{i,j+1},{G}_{i+1,j})\) (Line 7). Finally, we obtain the BDD \({G}_{1,1}\) that encodes the solutions of \(\sum _{j=1}^n \ell _j\ge k\). Consider \(x_1+\lnot x_2 + x_3 + \lnot x_4 + x_5 + \lnot x_6 \ge 3\), Fig. 4(b) shows its BDD by Algorithm 1.

Lemma 1

For each cardinality constraint \(\sum _{j=1}^n \ell _j\ge k\), a BDD \({G}\) with \(O((n-k)\cdot k)\) nodes can be computed in \(O((n-k)\cdot k)\) time such that \(\mathcal {L}({G})\) is the set of all the solutions of \(\sum _{j=1}^n \ell _j\ge k\).

Compared with prior works [8, 42] which transform general arithmetic constraints into BDDs, we devise a dedicated BDD encoding algorithm for the cardinality constraints without applying reduction, hence it is more efficient.

3.3 Region2BDD: Input Regions to BDDs

In this paper, we consider the following two types of input regions.

  • Input region based on Hamming distance. For an input \({\boldsymbol{u}}\in {\mathbb {B}}_{\pm 1}^{n_1}\) and an integer \(r\ge 0\), \(R({\boldsymbol{u}},r)\) denotes the set \(\{{\boldsymbol{x}}\in {\mathbb {B}}_{\pm 1}^{n_1}\mid \textsf {HD}({\boldsymbol{x}},{\boldsymbol{u}})\le r\}\), where \(\textsf {HD}({\boldsymbol{x}},{\boldsymbol{u}})\) denotes the Hamming distance between \({\boldsymbol{x}}\) and \({\boldsymbol{u}}\). Intuitively, \(R({\boldsymbol{u}},r)\) includes the input vectors which differ from \(\boldsymbol{u}\) by at most r positions.

  • Input region with fixed indices. For an input \({\boldsymbol{u}}\in {\mathbb {B}}_{\pm 1}^{n_1}\) and a set of indices \(I\subseteq [n_1]\), \(R({\boldsymbol{u}},I)\) denotes the set \(\{{\boldsymbol{x}}\in {\mathbb {B}}_{\pm 1}^{n_1}\mid \forall i\in [n_1]\setminus I.\ {\boldsymbol{u}}_i={\boldsymbol{x}}_i\}\). Intuitively, \(R({\boldsymbol{u}},I)\) includes the input vectors which differ from \(\boldsymbol{u}\) only at the indices from I.

Note that both \(R({\boldsymbol{u}},n_1)\) and \(R({\boldsymbol{u}},[n_1])\) denote the entire input space \({\mathbb {B}}_{\pm 1}^{n_1}\).

Recall that each input sample is an element from \({\mathbb {B}}_{\pm 1}^{n_1}\). To represent the region R by a BDD, we transform each value \(\pm 1\) into a Boolean value 1/0. To this end, for each input \({\boldsymbol{u}}\in {\mathbb {B}}_{\pm 1}^{n_1}\), we create a new sample \({\boldsymbol{u}}^{(b)}\in {\mathbb {B}}^{n_1}\) such that for every \(i\in [n_1]\), \({\boldsymbol{u}}_i=2{\boldsymbol{u}}^{(b)}_i-1\). Therefore, \(R({\boldsymbol{u}},r)\) and \(R({\boldsymbol{u}},I)\) will be represented by \(R({\boldsymbol{u}}^{(b)},r)\) and \(R({\boldsymbol{u}}^{(b)},I)\), respectively. The transformation functions \(t_i^{lin}\), \(t_i^{bn}\), \(t_i^{bin}\) and \(t_{d+1}^{am}\) of the LIN, BN, BIN, and ARGMAX layers (cf. Table 1) will be handled accordingly. Note that for convenience, vectors over the Boolean domain \({\mathbb {B}}\) may be directly given by \({\boldsymbol{u}}\) or \({\boldsymbol{x}}\) when it is clear from the context.

Region Encoding Under Hamming Distance. Given an input \({\boldsymbol{u}}\in {\mathbb {B}}^{n_1}\) and an integer r, the region \(R({\boldsymbol{u}},r)\) can be expressed by a cardinality constraint \(\sum _{j=1}^{n_1} \ell _j\le r\) (which is equivalent to \(\sum _{j=1}^{n_1} \lnot \ell _j\ge n_1-r\)), where for every \(j\in [n_1]\), \(\ell _j={\boldsymbol{x}}_j\) if \({\boldsymbol{u}}_j=0\), otherwise \(\ell _j=\lnot {\boldsymbol{x}}_j\). For instance, consider \({\boldsymbol{u}}=(1,1,1,0,0)\) and \(r=2\), we have:

$$\textsf {HD}({\boldsymbol{u}},{\boldsymbol{x}})=1\oplus {\boldsymbol{x}}_1+1\oplus {\boldsymbol{x}}_2+1\oplus {\boldsymbol{x}}_3+ 0\oplus {\boldsymbol{x}}_4+0\oplus {\boldsymbol{x}}_5= \lnot {\boldsymbol{x}}_1+\lnot {\boldsymbol{x}}_2+\lnot {\boldsymbol{x}}_3+ {\boldsymbol{x}}_4+ {\boldsymbol{x}}_5.$$

Thus, R((1, 1, 1, 0, 0), 2) can be expressed by the cardinality constraint \(\lnot {\boldsymbol{x}}_1+\lnot {\boldsymbol{x}}_2+\lnot {\boldsymbol{x}}_3+ {\boldsymbol{x}}_4+ {\boldsymbol{x}}_5\le 2\), or equivalently \( {\boldsymbol{x}}_1+ {\boldsymbol{x}}_2+ {\boldsymbol{x}}_3+ \lnot {\boldsymbol{x}}_4+ \lnot {\boldsymbol{x}}_5\ge 3\).

By Algorithm 1, the cardinality constraint of \(R({\boldsymbol{u}},r)\) can be encoded by the BDD \(G^{in}_{{\boldsymbol{u}},r}\), such that \(\mathcal {L}(G^{in}_{{\boldsymbol{u}},r})=R({\boldsymbol{u}},r)\). Following Lemma 1, we get that:

Lemma 2

For an input region R given by an input \({\boldsymbol{u}}\in {\mathbb {B}}^{n_1}\) and an integer r, a BDD \(G^{in}_{{\boldsymbol{u}},r}\) with \(O(r\cdot (n_1-r))\) nodes can be computed in \(O(r\cdot (n_1-r))\) time such that \(\mathcal {L}(G^{in}_{{\boldsymbol{u}},r})=R({\boldsymbol{u}},r)\).

Region Encoding Under Fixed Indices. Given an input \({\boldsymbol{u}}\in {\mathbb {B}}^{n_1}\) and a set of indices \(I\subseteq [n_1]\), the region \(R({\boldsymbol{u}},I)=\{{\boldsymbol{x}}\in {\mathbb {B}}^{n_1}\mid \forall i\in [n_1]\setminus I.\ {\boldsymbol{u}}_i={\boldsymbol{x}}_i\}\) can be represented by the BDD \(G^{in}_{{\boldsymbol{u}},I}\triangleq \textsc {And}_{i\in [n_1]\setminus I} \Big ( ({\boldsymbol{u}}_i==1)? \textsc {Var}({\boldsymbol{x}}_i): \textsc {Not}(\textsc {Var}({\boldsymbol{x}}_i))\Big )\). Intuitively, \(G^{in}_{{\boldsymbol{u}},I}\) states that the value at the index \(i\in [n_1]\setminus I\) should be the same as the one in \({\boldsymbol{u}}\) while the value at the index \(i\in I\) is unrestricted. For instance, consider \({\boldsymbol{u}}=(1,0,0,0)\) and \(I=\{3,4\}\), we have:

$$\begin{aligned} R((1,0,0,0),\{3,4\})=\{(1,0,0,0),(1,0,0,1),(1,0,1,0),(1,0,1,1)\} = {\boldsymbol{x}}_1 \wedge \lnot {\boldsymbol{x}}_2. \end{aligned}$$

Lemma 3

For an input region R given by an input \({\boldsymbol{u}}\in {\mathbb {B}}^{n_1}\) and indices \(I\subseteq [n_1]\), a BDD \(G^{in}_{{\boldsymbol{u}},I}\) with \(O(n_1-|I|)\) nodes can be computed in \(O(n_1)\) time such that \(\mathcal {L}(G^{in}_{{\boldsymbol{u}},I})=R({\boldsymbol{u}},I)\).

3.4 BNN2CC: BNNs to Cardinality Constraints

As mentioned before, to encode the BNN \(\mathcal {N}=(t_1,\cdots ,t_d,t_{d+1})\) as BDDs, we transform the BNN \(\mathcal {N}\) into cardinality constraints from which the desired BDDs \(({G}_i^{out})_{i\in [s]}\) are constructed. To this end, we first transform each internal block \(t_i:{\mathbb {B}}_{\pm 1}^{n_i}\rightarrow {\mathbb {B}}_{\pm 1}^{n_{i+1}}\) into \(n_{i+1}\) cardinality constraints, each of which corresponds to one of the outputs of \(t_i\). Then we transform the output block \(t_{d+1}:{\mathbb {B}}_{\pm 1}^{n_{d+1}}\rightarrow {\mathbb {B}}^{s}\) into \(s(s-1)\) cardinality constraints, where one output class yields \((s-1)\) cardinality constraints.

For each vector-valued function t, we denote by \(t_{\downarrow j}\) the (scalar-valued) function returning the j-th entry of the output of t.

Transformation for Internal Blocks. Consider the internal block \(t_i:{\mathbb {B}}_{\pm 1}^{n_i}\rightarrow {\mathbb {B}}_{\pm 1}^{n_{i+1}}\) for \(i\in [d]\). Recall that for every \(j\in [n_{i+1}]\) and \({\boldsymbol{x}} \in {\mathbb {B}}_{\pm 1}^{n_i}\), \(t_{i\downarrow j}({\boldsymbol{x}}) = t_i^{bin}( t_i^{bn}(\langle {\boldsymbol{x}}, {\boldsymbol{W}}_{:,j}\rangle +{\boldsymbol{b}}_j))\), and each value \(\pm 1\) of an input \({\boldsymbol{u}}\in {\mathbb {B}}_{\pm 1}^{n_1}\) is replaced by 1/0 (cf. Sect. 3.3). To be consistent, the function \(t_{i\downarrow j}:{\mathbb {B}}_{\pm 1}^{n_i}\rightarrow {\mathbb {B}}_{\pm 1}\) is reformulated as the function \(t_{i\downarrow j}^{(b)}:{\mathbb {B}}^{n_i}\rightarrow {\mathbb {B}}\) such that for every \({\boldsymbol{x}}\in {\mathbb {B}}^{n_i}\), \(t_{i\downarrow j}^{(b)}({\boldsymbol{x}}) = 0.5\times (t_i^{bin}( t_i^{bn}(\langle 2{\boldsymbol{x}}-{\boldsymbol{1}}, {\boldsymbol{W}}_{:,j}\rangle +{\boldsymbol{b}}_j))+1),\) where \({\boldsymbol{1}}\) denotes the vector of 1’s with the width \(n_i\).

Let \(C_{i,j}\) be the following cardinality constraint:

$$C_{i,j}\triangleq \left\{ \begin{array}{lr} \sum _{k=1}^{n_i}\ell _k \ge \lceil \frac{1}{2}\cdot (n_i+\mu _j-{\boldsymbol{b}}_j-\frac{\gamma _j\cdot \sigma _j}{\alpha _j}) \rceil , &{} \text {if} \ \alpha _j>0;\\ 1, &{} \text {if} \ \alpha _j=0\wedge \gamma _j\ge 0;\\ 0, &{} \text {if} \ \alpha _j=0\wedge \gamma _j< 0;\\ \sum _{k=1}^{n_i}\lnot \ell _k \ge \lceil \frac{1}{2}\cdot (n_i-\mu _j+{\boldsymbol{b}}_j+\frac{\gamma _j\cdot \sigma _j}{\alpha _j}) \rceil , &{} \text {if} \ \alpha _j<0; \end{array}\right. $$

where for every \(k\in [n_i]\), \(\ell _k\) is \({\boldsymbol{x}}_k\) if \({\boldsymbol{W}}_{k,j}=+1\), and \(\ell _k\) is \(\lnot {\boldsymbol{x}}_k\) if \({\boldsymbol{W}}_{k,j}=-1\).

Proposition 1

\(t_{i\downarrow j}^{(b)}\Leftrightarrow C_{i,j}\).

Proof refers to [71].

Transformation for the Output Block. For the output block \(t_{d+1}:{\mathbb {B}}_{\pm 1}^{n_{d+1}}\rightarrow {\mathbb {B}}^{s}\), since \(t_{d+1}=t_{d+1}^{am} \circ t_{d+1}^{lin}\), then for every \(j\in [s]\), we can reformulate \(t_{d+1\downarrow j}:{\mathbb {B}}_{\pm 1}^{n_{d+1}}\rightarrow {\mathbb {B}}\) as the function \(t_{d+1\downarrow j}^{(b)}:{\mathbb {B}}^{n_{d+1}}\rightarrow {\mathbb {B}}\) such that for every \({\boldsymbol{x}}\in {\mathbb {B}}^{n_{d+1}}\), \(t_{d+1\downarrow j}^{(b)}({\boldsymbol{x}})=t_{d+1\downarrow j}(2{\boldsymbol{x}}-{\boldsymbol{1}})\).

For every \(j'\in [s]\setminus \{j\}\), we define the cardinality constraint \(C_{d+1,j'}\) as follows:

figure b

where \(\sharp \mathtt{Neg}=|\{k\in [n_{d+1}] \mid {\boldsymbol{W}}_{k,j}-{\boldsymbol{W}}_{k,j'}=-2\}|\), \(\ell _{d+1,k}\) is \({\boldsymbol{x}}_{d+1,k}\) if \({\boldsymbol{W}}_{k,j}-{\boldsymbol{W}}_{k,j'}=+2\), \(\ell _{d+1,k}\) is \(\lnot {\boldsymbol{x}}_{d+1,k}\) if \({\boldsymbol{W}}_{k,j}-{\boldsymbol{W}}_{k,j'}=-2\), and \(\ell _{d+1,k}\) is 0 if \({\boldsymbol{W}}_{k,j}-{\boldsymbol{W}}_{k,j'}=0\).

Proposition 2

\(t_{d+1\downarrow j}^{(b)} \Leftrightarrow \bigwedge _{j'\in [s], j'\ne j} C_{d+1,j'}\).

Proof refers to [71].

For each internal block \(t_i:{\mathbb {B}}_{\pm 1}^{n_i}\rightarrow {\mathbb {B}}_{\pm 1}^{n_{i+1}}\), we denote by BNN2CC\((t_i)\) the cardinality constraints \(\{C_{i,1},\cdots , C_{i,n_{i+1}}\}\). For each output class \(j\in [s]\), we denote by BNN2CC\(^j(t_{d+1})\) the cardinality constraints \(\{C_{d+1,1},\cdots C_{d+1,j-1},C_{d+1,j+1},\cdots , C_{d+1,s}\}\). By applying the above transformation to all the blocks of the BNN \(\mathcal {N}=(t_1,\cdots ,t_d,t_{d+1})\), we obtain its cardinality constraint form \(\mathcal {N}^{(b)}=(t_1^{(b)},\cdots ,t_d^{(b)},t_{d+1}^{(b)})\) such that for each \(i\in [d]\), \(t_i^{(b)}=\text {BNN2CC}(t_i)\), and \(t_{d+1}^{(b)}=(\text {BNN2CC}^1(t_{d+1}),\cdots , \text {BNN2CC}^s(t_{d+1}))\). Given an input \({\boldsymbol{u}}\in {\mathbb {B}}^{n_1}\), we denote by \(\mathcal {N}^{(b)}({\boldsymbol{u}})\) the index \(j\in [s]\) such that all the cardinality constraints in \(\text {BNN2CC}^j(t_{d+1})\) hold under the valuation \({\boldsymbol{u}}\). It is straightforward to verify:

Theorem 1

\({\boldsymbol{u}}\in {\mathbb {B}}_{\pm 1}^{n_1}\) is classified into the class j by the BNN \(\mathcal {N}\) iff \(\mathcal {N}^{(b)}({\boldsymbol{u}}^{(b)})=j\).

Example 1

Consider the BNN \(\mathcal {N}=(t_1,t_2)\) with one internal block \(t_1\) and one output block \(t_2\) as shown in Fig. 5 (left-bottom), where the elements of the Weight matrix \({\boldsymbol{W}}\) are associated to the edges, and the other parameters are given in the left-up table. The transformation functions of blocks \(t_1\) and \(t_2\) are given in the right-up table, and their cardinality constraints are given in the right-bottom table.

For instance, for each input \({\boldsymbol{x}}\in {\mathbb {B}}_{\pm 1}^{3}\), \(y_1 = \text {sign}(-x_1+x_2+x_3+2.7)\), i.e., \(y_1 =+1 \Leftrightarrow -x_1+x_2+x_3+2.7\ge 0\). By replacing \(x_i\) with \(2\times x_i^{(b)}-1\) and \(x_1^{(b)}\) with \(1-\lnot x_1^{(b)}\), we have: \(y_1 =+1 \Leftrightarrow (-x_1^{(b)}+x_2^{(b)}+x_3^{(b)} +0.85\ge 0) \Leftrightarrow (\lnot x_1^{(b)} + x_2^{(b)} + x_3^{(b)} \ge 0.15)\). Thus we get \(y_1^{(b)} \Leftrightarrow \lnot x_1^{(b)} + x_2^{(b)} + x_3^{(b)} \ge 1\) (note that \(y_1^{(b)}=0 \Leftrightarrow \lnot x_1^{(b)} + x_2^{(b)} + x_3^{(b)} < 1\)). Similarly, we can deduce that \(o_1 \Leftrightarrow y_1 - y_2\ge 0.7\), and thus \(o_1 \Leftrightarrow y_1^{(b)} - y_2^{(b)} \ge 0.35 \Leftrightarrow y_1^{(b)} +\lnot y_2^{(b)} \ge 2\).

3.5 BDD Model Builder

The construction of the BDDs \(({G}_i^{out})_{i\in [s]}\) from the BNN \(\mathcal {N}^{(b)}\) and the input region R is done iteratively throughout the blocks. Initially, the BDD for the first block is built, which can be seen as the input-output relation for the first internal block. In the i-th iteration, as the input-output relation of the first \((i-1)\) internal blocks has been encoded into the BDD, we compose this BDD with the BDD for the block \(t_i\) which is built from its cardinality constraints \(t_i^{(b)}\), resulting in the BDD for the first i internal blocks. Finally, we obtain the BDDs \(({G}_i^{out})_{i\in [s]}\) of the BNN \(\mathcal {N}\), with respect to the input region R.

Fig. 5.
figure 5

An illustrating example

Full size image

Design Choice. There are several design choices for efficiency consideration which we discuss as follows. First of all, to encode the input-output relation of an internal block \(t_i\) into BDD from its cardinality constraints \(t_i^{(b)}=\{C_{i,1},\cdots , C_{i,n_{i+1}}\}\), we need to compute \(\textsc {And}_{j\in [n_{i+1}]}\text {CC2BDD}(C_{i,j})\). A simple and straightforward approach is to initially compute a BDD \(G=\text {CC2BDD}(C_{i,1})\) and then iteratively compute the conjunction \(G=\textsc {And}(G, \text {CC2BDD}(C_{i,j}))\) of G and \(\text {CC2BDD}(C_{i,j})\) for \(2\le j\le n_{i+1}\).

Alternatively, we use a divide-and-conquer strategy to recursively compute the BDDs for the first half and the second half of the cardinality constraints respectively, and then apply the AND-operation. Our preliminary experimental results show that the latter approach often performs better (about 2 times faster) than the former one, although they generate the same BDD.

Second, constructing the BDD directly from the cardinality constraints \(t_i^{(b)}=\{C_{i,1},\cdots , C_{i,n_{i+1}}\}\) becomes prohibitively costly when \(n_i\) and \(n_{i+1}\) are large, as the BDDs \(\text {CC2BDD}(C_{i,j})\) for \(j\in [n_{i+1}]\) need to consider all the inputs in \({\mathbb {B}}^{n_i}\). To improve efficiency, we apply feasible input propagation. Namely, when we construct the BDD for the block \(t_{i+1}\), we only consider its possible inputs with respect to the output of the block \(t_i\). Our preliminary experimental results show that the optimization could significantly improve the efficiency of the BDD construction.

Third, instead of encoding the input-output relation of the BNN \(\mathcal {N}\) as a sole BDD or MTBDD, we opt to use a family of s BDDs \(({G}_i^{out})_{i\in [s]}\), each of which corresponds to one output class of \(\mathcal {N}\). Recall that each output class \(i\in [s]\) is represented by \((s-1)\) cardinality constraints. Then, we can build a BDD \({G}_i\) for the output class i, similar to the BDD construction for internal blocks. By composing \({G}_i\) with the BDD of the entire internal blocks, we obtain the BDD \({G}_i^{out}\). Building a single BDD or MTBDD for the BNN is possible from \(({G}_i^{out})_{i\in [s]}\), but our approach gives the flexibility especially when a specific target class is interested, which is common for robustness analysis.

figure c

Overall Algorithm. The overall BDD construction procedure is shown in Algorithm 2. Given a BNN \(\mathcal {N}=(t_1,\cdots ,t_d,t_{d+1})\) with s output classes and an input region \(R({\boldsymbol{u}},\tau )\), the algorithm outputs the BDDs \(({G}_i^{out})_{i\in [s]}\), encoding the input-output relation of the BNN \(\mathcal {N}\) with respect to the input region \(R({\boldsymbol{u}},\tau )\).

The procedure BNN2BDD first builds the BDD representation \({G}^{in}_{{\boldsymbol{u}},\tau }\) of the input region \(R({\boldsymbol{u}},\tau )\) and the cardinality constraints from BNN \(\mathcal {N}^{(b)}\) (Line 1). The first for-loop builds a BDD encoding the input-output relation of the entire internal blocks w.r.t. \({G}^{in}_{{\boldsymbol{u}},\tau }\). The second for-loop builds the BDDs \(({G}_i^{out})_{i\in [s]}\), each of which encodes the input-output relation of the entire BNN for a class \(i\in [s]\) w.r.t. \({G}^{in}_{{\boldsymbol{u}},\tau }\). The procedure Block2BDD receives the cardinality constraints \(\{C_m,\cdots ,C_n\}\), a BDD \({G}^{in}\) representing the feasible inputs of the block and the block index i as inputs, and returns a BDD \({G}\). If \(i=d+1\), namely, the cardinality constraints \(\{C_m,\cdots ,C_n\}\) are from the output block, the resulting BDD \({G}\) encodes the subset of \({G}^{in}_{{\boldsymbol{u}},\tau }\) that satisfy all the cardinality constraints \(\{C_m,\cdots ,C_n\}\). If \(i\ne d+1\), then the BDD \({G}\) encodes the input-output relation of the Boolean function \(f_{m,n}\) such that for every \({\boldsymbol{x}}^{i}\in \mathcal {L}(G^{in})\), \(f_{m,n}({\boldsymbol{x}}^{i})\) is the truth vector of the cardinality constraints \(\{C_m,\cdots ,C_n\}\) under the valuation \({\boldsymbol{x}}^{i}\). When \(m=1\) and \(n=n_{i+1}\), \(f_{m,n}\) is the same as \(t_i^{(b)}\), hence \(\mathcal {L}({G})=\{{\boldsymbol{x}}^i\times {\boldsymbol{x}}^{i+1}\in {G}^{in}\times {\mathbb {B}}^{n_{i+1}}\mid t_i^{(b)}({\boldsymbol{x}}^i)={\boldsymbol{x}}^{i+1}\}\). Detailed explanation refers to [71].

Theorem 2

Given a BNN \(\mathcal {N}\) with s output classes and an input region \(R({\boldsymbol{u}},\tau )\), we can compute s BDDs \(({G}_i^{out})_{i\in [s]}\) such that the BNN \(\mathcal {N}\) classifies an input \({\boldsymbol{x}}\in R({\boldsymbol{u}},\tau )\) into the class \(i\in [s]\) iff \({\boldsymbol{x}}^{(b)}\in \mathcal {L}(G_i^{out})\).

Algorithm 2 explicitly involves \(O(d+s)\) \(\textsc {RelProd}\)-operations, \(O(s^2+\sum _{i\in [d]} n_i)\) \(\textsc {And}\)-operations and O(d) \(\textsc {Exists}\)-operations.

4 Applications: Robustness Analysis and Interpretability

In this section, we present two applications within BDD4BNN, i.e., robustness analysis and interpretability of BNNs.

4.1 Robustness Analysis

Definition 2

Given a BNN \(\mathcal {N}\) and an input region \(R({\boldsymbol{u}},\tau )\), the BNN is (locally) robust w.r.t. the region \(R({\boldsymbol{u}},\tau )\) if each sample \({\boldsymbol{x}}\in R({\boldsymbol{u}},\tau )\) is classified into the same class as the ground-truth class of \({\boldsymbol{u}}\).

An adversarial example in the region \(R({\boldsymbol{u}},\tau )\) is a sample \({\boldsymbol{x}}\in R({\boldsymbol{u}},\tau )\) such that \({\boldsymbol{x}}\) is classified into a class, that differs from the ground-truth class of \({\boldsymbol{u}}\).

As mentioned in Sect. 1, qualitative verification which checks whether a BNN is robust or not is insufficient in many practical applications. In this paper, we are interested in quantitative verification of robustness which asks how many adversarial examples are there in the input region of the BNN for each class. To answer this question, given a BNN \(\mathcal {N}\) and an input region \(R({\boldsymbol{u}},\tau )\), we first obtain the BDDs \(({G}_i^{out})_{i\in [s]}\) by applying Algorithm 2 and then count the number of adversarial examples for each class in the input region \(R({\boldsymbol{u}},\tau )\). Note that counting adversarial examples amounts to computing \(|R({\boldsymbol{u}},\tau )|- |\mathcal {L}({G}_g^{out})|\), where g denotes the ground-truth class of \({\boldsymbol{u}}\), and \(|\mathcal {L}({G}_g^{out})|\) can be computed in time \(O(|{G}_{g}^{out}|)\).

In some applications, more refined analysis is needed. For instance, it may be acceptable to misclassify a dog as a cat, but unacceptable to misclassify a tree as a car. This suggests that the robustness of BNNs may depend on the classes to which samples are misclassified. To capture this, we consider the notion of targeted robustness.

Definition 3

Given a BNN \(\mathcal {N}\), an input region \(R({\boldsymbol{u}},\tau )\), and the class t, the BNN is t-target-robust w.r.t. the region \(R({\boldsymbol{u}},\tau )\) if every sample \({\boldsymbol{x}}\in R({\boldsymbol{u}},\tau )\) is never classified into the class t. (Note that we assume that the ground-truth class of \({\boldsymbol{u}}\) differs from the class t.)

The quantitative verification problem of t-target-robustness of a BNN asks how many adversarial examples in the input region \(R({\boldsymbol{u}},\tau )\) are misclassified to the class t by the BNN \(\mathcal {N}\). To answer this question, we first obtain the BDD \({G}_t^{out}\) by applying Algorithm 2 and then count the number of adversarial examples by computing \(|\mathcal {L}({G}_t^{out})|\).

Note that, if one wants to compute the (locally) maximal safe Hamming distance that satisfies a robustness property for an input sample (e.g., the proportion of adversarial examples is below a threshold), our framework can incrementally compute such a distance without constructing the BDD models of the entire BNN from scratch.

Definition 4

Given a BNN \(\mathcal {N}\), input region \(R({\boldsymbol{u}},r)\) and threshold \(\epsilon \ge 0\), \(r_1\) is the (locally) maximal safe Hamming distance of \(R({\boldsymbol{u}},\tau )\), if one of the follows holds:

  • if \(Pr(R({\boldsymbol{u}},r))>\epsilon \), then \(Pr(R({\boldsymbol{u}},r_1))\le \epsilon \) and \(Pr(R({\boldsymbol{u}},r'))>\epsilon \) for \(r':r_1< r'<r\);

  • if \(Pr(R({\boldsymbol{u}},r))\le \epsilon \), then \(Pr(R({\boldsymbol{u}},r_1+1))>\epsilon \) and \(Pr(R({\boldsymbol{u}},r'))\le \epsilon \) for \(r':r< r'\le r_1\);

where \(Pr(R({\boldsymbol{u}},r))\) is the probability \(\frac{\sum _{i\in [s].i\ne g}|\mathcal {L}({G}^{out}_i)|}{|R({\boldsymbol{u}},r)|}\) for g being the ground-truth class of \({\boldsymbol{u}}\), assuming a uniform distribution of adversarial samples.

Algorithm 3 shows the procedure to incrementally compute the maximal safe Hamming distance for a given threshold \(\epsilon \ge 0\), input region \(R({\boldsymbol{u}},r)\), and ground-truth class g of \({\boldsymbol{u}}\). Remark that \(Pr(R({\boldsymbol{u}},r))\) may not be monotonic w.r.t. the Hamming distance r.

figure d

4.2 Interpretability

In general, interpretability addresses the question of why some inputs in the input region are (mis)classified by the BNN into a specific class? We consider the interpretability of BNNs using two complementary explanations, i.e., prime implicant explanations and essential features.

Definition 5

Given a BNN \(\mathcal {N}\), an input region \(R({\boldsymbol{u}},\tau )\) and a class g, a prime implicant explanation (PI-explanation) of decisions made by the BNN \(\mathcal {N}\) on the inputs \(\mathcal {L}({G}_g^{out})\) is a minimal set of literals \(\{\ell _1,\cdots ,\ell _k\}\) such that for every \({\boldsymbol{x}}\in R({\boldsymbol{u}},\tau )\), if \({\boldsymbol{x}}\) satisfies \(\ell _1\wedge \cdots \wedge \ell _k\), then \({\boldsymbol{x}}\) is classified into the class g by the BNN \(\mathcal {N}\).

Intuitively, a PI-explanation \(\{\ell _1,\cdots ,\ell _k\}\) indicates that \(\{\textsf {var}(\ell _1),\cdots ,\textsf {var}(\ell _k)\}\) are key features, namely, if fixed, the predication is guaranteed no matter how the remaining features change. Remark that there may be more than one PI-explanation for a set of inputs \(\mathcal {L}({G}_g^{out})\). When g is set to be the class of the benign input \({\boldsymbol{u}}\), a PI-explanation on \({G}_g^{out}\) suggests why these samples are classified into g by the BNN \(\mathcal {N}\).

Definition 6

Given a BNN \(\mathcal {N}\), an input region \(R({\boldsymbol{u}},\tau )\) and a class g, the essential features for the inputs \(\mathcal {L}({G}_g^{out})\) are literals \(\{\ell _1,\cdots ,\ell _k\}\) such that every \({\boldsymbol{x}}\in R({\boldsymbol{u}},\tau )\), if \({\boldsymbol{x}}\) is classified into the class g by the BNN \(\mathcal {N}\), then \({\boldsymbol{x}}\) satisfies \(\ell _1\wedge \cdots \wedge \ell _k\).

Intuitively, the essential features \(\{\ell _1,\cdots ,\ell _k\}\) denote the key features such that all samples \({\boldsymbol{x}}\in R({\boldsymbol{u}},\tau )\) that are classified into the class g by the BNN \(\mathcal {N}\) must agree on these features. Essential features differ from PI-explanations, where the former can be seen as a necessary condition, while the latter can be seen as a sufficient condition.

BDD libraries (e.g., CUDD [58]) usually provide APIs to identify prime implicants (e.g., Cudd_bddPrintCover and Cudd_FirstPrime) and essential variables (e.g., Cudd_FindEssential). Therefore, prime implicants and essential features can be computed via queries on the BDDs \(({G}_i^{out})_{i\in [s]}\).

5 Evaluation

We have implemented our framework as a prototype tool BDD4BNN based on the CUDD package [58]. BDD4BNN is implemented with Python as the front-end to pre-process BNNs and C++ as the back-end to perform the BDD encoding and analysis. In this section, we report the experimental results, including BDD encoding, robustness analysis based on hamming distance, and interpretability.

Experimental Setup. The experiments were conducted on a machine with Intel Xeon Gold 5118 2.3GHz CPU, 64-bit Ubuntu 20.04 LTS operating systems, 128 G RAM. Each BDD encoding executed on one core limited by 8-h.

Benchmarks. We use the PyTorch (v1.0.1.post2) deep learning platform provided by NPAQ [6] to train and test BNNs. We trained 12 BNN models (P1-P12) with varying sizes using the MNIST dataset [35]. The MNIST dateset contains 70,000 gray-scale 28 \(\times \) 28 images (60,000 for training and 10,000 for testing) of handwritten digits with 10 classes. In our experiments, we downscale the images (\(28\times 28\)) to some selected input size \(n_1\) (i.e., the corresponding image is of the size \(\sqrt{n_1}\times \sqrt{n_1}\)) and then binarize the normalized pixels of the images.

Details of the BNN models are listed in Table 3, each of which has 10 classes (i.e., \(s=10\)). Column 1 shows the name of the BNN model. Column 2 shows the architecture of the BNN model, where \(n_1:\cdots :n_{d+1}:s\) denotes that the BNN model has \(d+1\) blocks, \(n_1\) inputs and s outputs; the i-th block for \(i\in [d+1]\) has \(n_i\) inputs and \(n_{i+1}\) outputs with \(n_{d+2}=s\). Recall that each internal block has 3 layers while the output block has 2 layers. Therefore, the number of layers ranges from 5 to 14, the dimension of inputs ranges from 9 to 784, and the number of hidden neurons per linear layer ranges from 10 to 100. Column 3 shows the accuracy of the BNN model on the test set of the MNIST dataset. (We can observe that the accuracy increases with the size of inputs, the number of layers, and the number of hidden neurons per layer.) We randomly choose 10 images from the training set of the MNIST dataset (one image per class) to evaluate our approach.

5.1 Performance of BDD Encoding

We evaluate BDD4BNN on the BNNs listed in Table 3 using different input regions.

BDD Encoding Using Full Input Space. We evaluate BDD4BNN on the BNNs (P1–P5), where \({\mathbb {B}}_{\pm 1}^{n_1}\) is used as the input region. The results are shown in Table 4, where \(|{G}|\) denotes the number of BDD nodes in the BDD manager. We can observe that both the execution time and the number of BDD nodes increase with the size of BNNs.

BDD Encoding Under Hamming Distance. We evaluate BDD4BNN on the BNNs (P5–P12). In this case, an input region is given by one of the 10 images and a Hamming distance r ranging from 2 to 6. The average results are shown in Table 5, where [i] (resp. (i)) indicates the number of cases that BDD4BNN runs out of memory (resp. time). Overall, the execution time and the number of BDD nodes increase with r. BDD4BNN succeeded on all the cases when \(r\le 4\), 75 cases out of 80 when \(r=5\), and 48 cases out of 80 when \(r=6\). We observe that the execution time and number of BDD nodes increase with the number of hidden neurons (P6 vs. P7, P8 vs. P9, and P11 vs. P12), while the effect of the number of layers is diverse (P6 vs. P8 vs. P10, and P7 vs. P9). From P9 and P10, we observe that the number of hidden neurons per layer is likely the key impact factor of the efficiency of BDD4BNN. Interestingly, our tool BDD4BNN works well on BNNs with large input sizes (i.e., on P11 and P12).

Table 3. BNN benchmarks
Full size table
Table 4. BDD encoding using full input space
Full size table

These results demonstrate the efficiency and scalability of BDD4BNN on BDD encoding of BNNs. We remark that, compared with the learning-based approach [54], our approach is considerably more efficient and scalable. For instance, the learning-based approach takes 403 s to encode a BNN with 64 input size, 5 hidden neurons, and 2 output size when \(r=6\), while ours takes about 3 s even for a larger network P5.

5.2 Robustness Analysis

We evaluate BDD4BNN on the robustness of BNNs, including robustness analysis under different input regions and maximal safe Hamming distance computing.

Robustness Verification with Hamming Distance. We evaluate BDD4BNN on BNNs (P7, P8, P9, and P11) using the 10 images. The input regions are given by the Hamming distance r ranging from 2 to 4, resulting in 120 instances. To the best of our knowledge, NPAQ  [6] is the only work that supports quantitative robustness verification of BNNs to which we compare BDD4BNN. Recall that NPAQ only provides PAC-style guarantees. Namely, it sets a tolerable error \(\varepsilon \) and a confidence parameter \(\delta \). The final estimated results of NPAQ have the bounded error \(\varepsilon \) with confidence of at least \(1-\delta \), i.e.,

$$\begin{aligned} Pr[(1+\varepsilon )^{-1} \mathtt{RealNum}\le \mathtt{EstimatedNum}\le (1+\varepsilon )\mathtt{RealNum}]\ge 1-\delta \end{aligned}$$
(1)

In our experiments, we set \(\varepsilon =0.8\) and \(\delta =0.2\), as done in [6].

Table 5. BDD encoding under Hamming distance
Full size table
Table 6. Robustness verification under Hamming distance
Full size table
Fig. 6.
figure 6

Details of robustness verification with Hamming distance

Full size image

The results on the average of the images are shown in Table 6. NPAQ ran out of time on 5 instances (which occur in P9 with \(r=4\) and P11 with \(r=3\) and \(r=4\)), while BDD4BNN successfully verified all the 120 instances. Table 6 only shows the results of 115 instances that can be solved by NPAQ. Columns 3, 4, and 5 (resp. 6, 7, and 8) show the number of adversarial examples, the execution time, and the proportion of adversarial examples in the input region. Column 9 shows the error rate \(\frac{\mathtt{RealNum}-\mathtt{EstimatedNum}}{\mathtt{EstimatedNum}}\), where \(\mathtt{RealNum}\) is from our result, and \(\mathtt{EstimatedNum}\) is from NPAQ. Column 10 shows the speedup of BDD4BNN compared with NPAQ. Remark that the numbers of adversarial examples are 0 for P11 on input regions with \(r=3\) and \(r=4\) that can be solved by NPAQ. There do exist input regions for P11 that cannot be solved by NPAQ but have adversarial examples (see below). On BNNs that were solved by both NPAQ and BDD4BNN, BDD4BNN is significantly (\(5\times \) to \(1,340\times \)) faster and more accurate than NPAQ. From Table 5 and Table 6, we also found that most of the verification time is spent on BDD encoding while the rest is usually less than 10 s.

Details of Robustness and Targeted Robustness. Figure 6(a) (resp. Fig. 6(b) and Fig. 6(c)) depicts the distributions of classes on P8 with Hamming distance \(r=2\) (resp. P8 with \(r=3\) and P11 with \(r=2\)), where on the x-axis \(i=0,\cdots ,9\) denotes the input region that is within the respective Hamming distance to the image of digit i (called i-region). We can observe that P8 is robust for the 0-region when \(r=2\) and robust for the 6-region when \(r=2\) and \(r=3\), but is not robust for the other regions. (Note P8 is not robust for 0-region when \(r=3\), which is hard to be visualized in Fig. 6(b) due to the small number of adversarial examples.) Most of the adversarial examples in the 1-region and 5-region are misclassified into the digit 3 by P8. P11 is not robust for the 1-region or the 5-region, but is robust for all the other regions. Though P8 and P11 are not robust on some input regions, indeed they are t-target-robust for many target classes t, e.g., P11 is t-target-robust for the 1-region when \(t\ne 2\), and the 5-region when \(t\ne 3\). (The raw data are given in [71].)

Quality Validation of NPAQ. Figure 6(d) shows the distribution of error rates of NPAQ, where the x-axis is the range of the error rate and the y-axis is the corresponding number of instances. There are 19 instances where the estimated number of adversarial examples exceeds \((1+\epsilon )\) of the real number of the adversarial examples and 7 instances where the estimated number of adversarial examples is less than \((1+\epsilon )^{-1}\) of the real number of the adversarial examples. This means that out of 115 instances, only in 89 instances the estimated number is within the allowed range, which is less than \(1-\delta =0.8\).

Maximal Safe Hamming Distance. As a representative of such an analysis, we evaluate BDD4BNN on 4 BNNs (P7, P8, P9, and P11) with 10 images for 2 robustness thresholds (\(\epsilon =0\) and \(\epsilon =0.03\)). The initial Hamming distance r is 3. Intuitively, \(\epsilon =0\) (resp. \(\epsilon =0.03\)) means that up to 0% (resp. 3%) samples in the input region can be adversarial.

Table 7 shows the results, where columns SD and Time give the maximal safe Hamming distance and the execution time, respectively. BDD4BNN solved 74 out of 80 instances. (For the remaining 6 instances, BDD4BNN ran out of time or memory, but it was still able to compute a larger safe Hamming distance.) We can observe that the maximal safe Hamming distance increases with the threshold \(\epsilon \) on several BNNs and input regions. We can also observe that P11 is more robust than others, which is consistent with their accuracies (cf. Table 3). Remark that \(\text {SD}= -1\) indicates that the input image itself is misclassified.

Table 7. Maximal safe Hamming distance
Full size table
Fig. 7.
figure 7

Graphic representation of essential features and PI-explanations

Full size image

5.3 Interpretability

To demonstrate the ability of BDD4BNN on interpretability, we consider the analysis of the BNN P12 and the image \({\boldsymbol{u}}\) of digit 1.

Essential Features. For the input region given by the Hamming distance \(r=4\), we compute two sets of essential features for the inputs \(\mathcal {L}(G_2^{out})\) and \(\mathcal {L}(G_5^{out})\), i.e., the adversarial examples in the region \(R({\boldsymbol{u}},4)\) that are misclassified into the classes 2 and 5 respectively. The essential features are depicted in Figs. 7(a) and 7(b), where black (resp. blue) color means that the value of the corresponding pixel is 1 (resp. 0), and yellow color means that the value of the corresponding pixel can take arbitrary values. Figure 7(a) (resp. Fig. 7(b)) indicates that the inputs \(\mathcal {L}(G_2^{out})\) (resp. \(\mathcal {L}(G_5^{out})\)) must agree on these black- and blue-colored pixels.

PI-Explanations. For demonstration, we assume that the input region is given by the fixed set of indices \(I=\{1,2,\cdots ,28\}\) which denotes the first row of pixels of \(28\times 28\) images. We compute two PI-explanations of the inputs \(\mathcal {L}(G_2^{out})\) and \(\mathcal {L}(G_5^{out})\). The PI-explanations are depicted in Figs. 7(c) and 7(d). Figure 7(c) (resp. Fig. 7(d)) suggests that, by the definition of the PI-explanation, all the images in the region \(R({\boldsymbol{u}},I)\) obtained by assigning arbitrary values to the yellow-colored pixels are always misclassified into the class 2 (resp. class 5), while changing one black-colored or blue-colored pixel would change the predication result since a PI-explanation is a minimal set of literals.

6 Related Work

In this section, we discuss the related work on qualitative/quantitative analysis and interpretability of DNNs. As there is a vast amount of literature regarding these topics, we will only discuss the most related ones to BDD4BNN.

Qualitative Analysis of DNNs. For real-numbered DNNs, various formal verification approaches have been proposed. Typical examples include constraint solving based approaches [17, 26, 30, 31, 51], optimization based approaches [10, 13, 15, 16, 40, 61, 67, 68], and program analysis based approaches [2, 3, 18, 20, 37,38,39, 55,56,57, 62,63,64, 69].

Existing techniques for quantized DNNs are mostly based on constraint solving, in particular, SAT/SMT solving [12, 33, 45, 46]. Following this line, verification of BNNs with ternary weights [28, 48] and quantized DNNs with multiple bits [7, 22, 24] were also studied. Recently, the SMT-based framework Marabou for real-numbered DNNs [31] has also been extended to support BNNs [1].

Quantitative Analysis of DNNs. Comparing to qualitative analysis, quantitative analysis of neural networks is currently very limited. Two sampling-based approaches were proposed to certify the robustness for both DNNs and BNNs [5, 65]. Yang et al. [69] proposed a spurious region-guided refinement approach for real-numbered DNN verification, claiming to be the first work of the quantitative robustness verification of DNNs with soundness guarantees.

Following the SAT-based qualitative analysis of BNNs [45, 46], SAT-based quantitative analysis approaches were also proposed [6, 21, 47]. In particular, approximate SAT model-counting solvers are utilized. Shih et al. [54] also proposed a BDD-based approach to tackle BNNs, similar to our work in spirit. However, our approach is able to handle BNNs of considerably larger sizes than their learning-based method.

Interpretability of DNNs. Though interpretability of DNNs is crucial for explaining predictions, it is very challenging to tackle due to the blackbox nature of DNNs. There is a large body of work on the interpretability of DNNs (cf. [25, 43] for a survey). Almost all the existing approaches are heuristic-based and restricted to finding explanations that are local in an input region. Some of them tackle the interpretability of DNNs by learning an interpretable model, such as binary decision trees [19, 70] or finite-state automata [66]. In contrast to ours, they target at DNNs and only approximate the original model in the input region. The BDD-based approach [54] mentioned above has been used to compute the PI-explanation, but essential features were not considered therein.

7 Conclusion

In this paper, we have proposed a novel BDD-based framework for quantitative verification of BNNs. We implemented the framework as a prototype tool BDD4BNN and conducted extensive experiments on 12 BNN models with varying sizes and input regions. Experimental results demonstrated that BDD4BNN is more scalable than the existing BDD-learning based approach, and significantly more efficient and accurate than the existing SAT-based approach NPAQ. This work represents the first, but a key, step of the long-term program to develop an efficient and scalable BDD-based quantitative analysis framework for BNNs.