Interpretability

Abstract

This chapter presents a method for interpreting neural networks which imposes minimal restrictions on the neural network design. The chapter demonstrates techniques for interpreting a feedforward network, including how to rank the importance of the features. An example demonstrating how to apply interpretability analysis to deep learning models for factor modeling is also presented.

Appendix

Other Interpretability Methods

Partial Dependence Plots (PDPs) evaluate the expected output w.r.t. the marginal density function of each input variable, and allow the importance of the predictors to be ranked. More precisely, partitioning the data X into an interest set, X _s, and its complement, $\boldsymbol {X}_c = \mathcal {X} \setminus \boldsymbol {X}_s$, then the “partial dependence” of the response on X _s is defined as

$$\displaystyle \begin{aligned} f_s\left(\boldsymbol{X}_s\right) = E_{\boldsymbol{X}_c}\left[\widehat{f}\left(\boldsymbol{X}_s, \boldsymbol{X}_c\right)\right] = \int \widehat{f}\left(\boldsymbol{X}_s, \boldsymbol{X}_c\right)p_{c}\left(\boldsymbol{X}_c\right)d\boldsymbol{X}_c, \end{aligned} $$

(5.20)

where $p_{c}\left (\boldsymbol {X}_c\right )$ is the marginal probability density of X _c: $p_{c}\left (\boldsymbol {X}_c\right ) = \int p\left (\boldsymbol {x}\right )d\boldsymbol {x}_s$. Equation (5.20) can be estimated from a set of training data by

$$\displaystyle \begin{aligned} \bar{f}_s\left(\boldsymbol{X}_s\right) = \frac{1}{n}\sum_{i = 1}^n\widehat{f}\left(\boldsymbol{X}_s,\boldsymbol{X}_{i, c}\right), \end{aligned} $$

(5.21)

where X _i,c $\left (i = 1, 2, \dots , n\right )$ are the observations of X _c in the training set; that is, the effects of all the other predictors in the model are averaged out. There are a number of challenges with using PDPs for model interpretability. First, the interaction effects are ignored by the simplest version of this approach. While Greenwell et al. (2018) propose a methodology extension to potentially address the modeling of interactive effects, PDPs do not provide a 1-to-1 correspondence with the coefficients in a linear regression. Instead, we would like to know, under strict control conditions, how the fitted weights and biases of the MLP correspond to the fitted coefficients of linear regression. Moreover in the context of neural networks, by treating the model as a black-box, it is difficult to gain theoretical insight in to how the choice of the network architecture affects its interpretability from a probabilistic perspective.

Garson (1991) partitions hidden-output connection weights into components associated with each input neuron using absolute values of connection weights. Garson’s algorithm uses the absolute values of the connection weights when calculating variable contributions, and therefore does not provide the direction of the relationship between the input and output variables.

Olden and Jackson (2002) determines the relative importance, r _ij = [R]_ij, of the ith output to the jth predictor variable of the model as a function of the weights, according to the expression

$$\displaystyle \begin{aligned} r_{ij}=W^{(2)}_{jk}W^{(1)}_{ki}. \end{aligned} $$

(5.22)

The approach does not account for non-linearity introduced into the activation, which is the most critical aspects of the model. Furthermore, the approach presented was limited to a single hidden layer.

Proof of Variance Bound on Jacobian

Proof

The Jacobian can be written in matrix element form as

$$\displaystyle \begin{aligned} J_{ij}=[\partial_{X} \hat{Y}]_{ij}=\sum_{k=1}^nw^{(2)}_{ik}w^{(1)}_{kj}H(I_k^{(1)})=\sum_{k=1}^nc_kH_k(I) \end{aligned} $$

(5.23)

where $c_k:=c_{ijk}:=w^{(2)}_{ik}w^{(1)}_{kj}$ and $H_k(I):=H(I_k^{(1)})$ is the Heaviside function. As a linear combination of indicator functions, we have

(5.24)

Alternatively, the Jacobian can be expressed in terms of a weighted sum of independent Bernoulli trials involving X:

(5.25)

Without loss of generality, consider the case when p = 1, the dimension of the input space is one. Then Eq. 5.25 simplifies to:

(5.26)

where $x_k:=-\frac {b^{(1)}_k}{W_k^{(1)}}$. The expectation of the Jacobian is given by

$$\displaystyle \begin{aligned} \mu_{ij}:=\mathbb{E}[J_{ij}]=\sum_{k=1}^{n}a_kp_k,\end{aligned} $$

(5.27)

where $p_k:=\Pr (x_k<X\leq x_{k+1})~\forall k=1,\dots , n-1, ~p_n:=\Pr (x_n<X).$ For finite weights, the expectation is bounded above by $\sum _{k=1}^{n}a_k$. We can write the variance of the Jacobian as:

(5.28)

Under the assumption that the mean of the Jacobian is invariant to the number of hidden units, or if the weights are constrained so that the mean is constant, then the weights are $a_k=\frac {\mu _{ij}}{np_k}$. Then the variance is bounded by the mean:

$$\displaystyle \begin{aligned} \mathbb{V}[J_{ij}]=\mu_{ij}\frac{n-1}{n}< \mu_{ij}. \end{aligned} $$

(5.29)

If we relax the assumption that μ _ij is independent of n then, under the original weights $a_k:=\sum _{i=1}^k c_i$:

$$\displaystyle \begin{aligned} \begin{array}{rcl} \mathbb{V}[J_{ij}]&\displaystyle =&\displaystyle \sum_{k=1}^{n}a_kp_k(1-p_k)\\ &\displaystyle \leq&\displaystyle \sum_{k=1}^{n}a_kp_k\\ &\displaystyle =&\displaystyle \mu_{ij}\\ &\displaystyle \leq&\displaystyle \sum_{k=1}^{n}a_k. \end{array} \end{aligned} $$

□

Russell 3000 Factor Model Description

Python Notebooks

The notebooks provided in the accompanying source code repository are designed to gain familiarity with how to implement interpretable deep networks. The examples include toy simulated data and a simple factor model. Further details of the notebooks are included in the README.md file.