Preventing deception with explanation methods using focused sampling

Abstract

Machine learning models are used in many sensitive areas where, besides predictive accuracy, their comprehensibility is also essential. Interpretability of prediction models is necessary to determine their biases and causes of errors and is a prerequisite for users’ confidence. For complex state-of-the-art black-box models, post-hoc model-independent explanation techniques are an established solution. Popular and effective techniques, such as IME, LIME, and SHAP, use perturbation of instance features to explain individual predictions. Recently, (Slack et al. in Fooling LIME and SHAP: Adversarial attacks on post-hoc explanation methods, 2020) put their robustness into question by showing that their outcomes can be manipulated due to inadequate perturbation sampling employed. This weakness would allow owners of sensitive models to deceive inspection and hide potentially unethical or illegal biases existing in their predictive models. Such possibility could undermine public trust in machine learning models and give rise to legal restrictions on their use. We show that better sampling in these explanation methods prevents malicious manipulations. The proposed sampling uses data generators that learn the training set distribution and generate new perturbation instances much more similar to the training set. We show that the improved sampling increases the LIME and SHAP’s robustness, while the previously untested method IME is the most robust. Further ablation studies show how the enhanced sampling changes the quality of explanations, reveal differences between data generators, and analyze the effect of different level of conservatism in the employment of biased classifiers.

Appendices

Derivation of IME convergence rate

The following derivation was presented by Štrumbelj and Kononenko (2010). Let $\sigma ^2_i$ be the variance of i-th feature’s sampling population for instance w, i.e. the population of $V_{\pi , w} = (f(w_{[w_i = x_i, i\in Pre^i(\pi )\cup \{i\}]}) - f(w_{[w_i = x_i, i\in Pre^i(\pi )]}))$. The value of $\sigma ^2_i$ is determined on $m_{min}$ samples drawn for each feature at the beginning of the explanation process. The desired accuracy of the IME method is given by the pair $(1-\alpha , e)$, which limits the size of the error e using the following expression (recall Eq. (2.4)):

$$\begin{aligned} P(|\hat{\phi _i} - \phi _i| < e) \ge 1 - \alpha . \end{aligned}$$

(A.1)

Let $m_i(1-\alpha , e)$ be the minimal number of samples for the i-th feature that satisfy Eq. (A.1). The value of $m_i(1-\alpha , e)$ can be calculated using

$$\begin{aligned} m_i(1-\alpha , e) = \frac{z_{1-\alpha }^2 \cdot \sigma _i^2}{e^2}, \end{aligned}$$

(A.2)

where $z_{1-\alpha }$ is implicitly defined with $P(|X| > z_{1-\alpha }) < 1-\alpha$ for ${\mathscr {N}}(0, 1)$ distributed random variable X. Let $m(1-\alpha , e)$ be the minimal number of samples that each feature satisfies (A.1). The value of $m(1-\alpha , e)$ can be calculated using

$$\begin{aligned} m(1-\alpha , e) = n\cdot \frac{z_{1-\alpha }^2 \cdot \overline{\sigma ^2}}{e^2}, \end{aligned}$$

(A.3)

where n denotes the number of features and $\overline{\sigma ^2}$ denotes the sum $\frac{1}{n}\sum _{i=1}^{n}\sigma _i^2$.

Training discriminator function of the attacker

Details of training attacker’s decision models d (see Fig. 2) is described in Algorithms 1, 2, and 3. We used a slightly different algorithm for each of the three explanation methods, LIME, SHAP, and IME, as each method uses a different sampling. Algorithms first create out-of-distribution instances by method-specific sampling. The training sets for decision models are created by labelling the created instances with 0; the instances from sample S (to which the attacker has access) from distribution $X_{dist}$ are labelled with 1. Finally, the machine learning model dModel is trained on this training set and returned as d. In our experiments, we used the random forest classifier as dModel and the training part of each evaluation dataset as S.

Heatmaps as tables

We present the information contained in Figs. 4 and 5 in a more detailed tabular form in Tables 8 and 9 respectively.

Table 8 The robustness results for gLIME (top table), gSHAP (middle table), and gIME (bottom table)

Table 9 The robustness results for gLIME (top), gSHAP (middle), and gIME (bottom) on the MvNormData dataset

Different prediction thresholds

The full results of experiments described in Sect. 5.6 are shown in Tables 10, 11, and 12.

Table 10 Percentages of instances where the sensitive feature (race) was recognized as the most important feature with gLIME for adversarial attacks with one (top table) or two unrelated features (bottom table)

Table 11 Percentage of instances where the sensitive attribute (race) was recognized as the most important feature with gSHAP for adversarial attacks with one (top table) or two unrelated features (bottom table)

Table 12 Percentages of instances where the sensitive attribute (race) was recognized as the most important feature with gIME for adversarial attacks with one (top table) or two unrelated features (bottom table)

MvNormData dataset

To test the robustness of the improved explanation methods on highly correlated data in Sect. 5.2, we created the MvNormData synthetic dataset based on the multivariate normal distribution. The dataset consists of 2000 instances described with 10 features. As we wanted our features to be zero centred, we sampled instances from ${\mathscr {N}}(0, \varSigma )$. We randomly generated the symmetric positive definite matrix $\varSigma$ as $10\times 10$ matrix by first generating its Cholesky decomposition consisting of a lower triangular $10\times 10$ matrix V. We generated each element on and below the diagonal of V independently by setting them to random integers between $-5$ and 5. To make sure that the diagonal elements are positive, we transformed them with:

$$\begin{aligned} v_{ii} \leftarrow |v_{ii}| + 1, \end{aligned}$$

where $v_{ii}$ represents the i-th diagonal element of V. We obtained the following matrix V:

$$\begin{aligned} V = \begin{bmatrix} 5 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 5 &{} 3 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 5 &{} 3 &{} 3 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ -1 &{} 1 &{} 1 &{} 4 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ -3 &{} 0 &{} 4 &{} -1 &{} 4 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ -1 &{} 5 &{} 3 &{} 0 &{} 5 &{} 4 &{} 0 &{} 0 &{} 0 &{} 0 \\ -1 &{} 4 &{} 4 &{} -3 &{} -5 &{} 4 &{} 1 &{} 0 &{} 0 &{} 0 \\ 2 &{} 1 &{} 5 &{} 0 &{} -4 &{} 3 &{} 4 &{} 4 &{} 0 &{} 0 \\ -2 &{} 2 &{} 2 &{} -2 &{} -2 &{} 5 &{} 2 &{} 5 &{} 1 &{} 0 \\ -1 &{} 5 &{} -3 &{} 2 &{} -3 &{} -4 &{} -2 &{} 2 &{} -5 &{} 5 \end{bmatrix}. \end{aligned}$$

By setting $\varSigma = V^T V$, we obtained the following $\varSigma$:

$$\begin{aligned} \varSigma = \begin{bmatrix} 25 &{} 25 &{} 25 &{} -5 &{} -15 &{} -5 &{} -5 &{} 10 &{} -10 &{} -5\\ 25 &{} 34 &{} 34 &{} -2 &{} -15 &{} 10 &{} 7 &{} 13 &{} -4 &{} 10\\ 25 &{} 34 &{} 43 &{} 1 &{} -3 &{} 19 &{} 19 &{} 28 &{} 2 &{} 1 \\ -5 &{} -2 &{} 1 &{} 19 &{} 3 &{} 9 &{} -3 &{} 4 &{} -2 &{} 11\\ -15 &{} -15&{} -3 &{} 3 &{} 42 &{} 35 &{} 2 &{} -2 &{} 8 &{} -23\\ -5 &{} 10 &{} 19 &{} 9 &{} 35 &{} 76 &{} 24 &{} 10 &{} 28 &{} -14\\ -5 &{} 7 &{} 19 &{} -3 &{} 2 &{} 24 &{} 84 &{} 58 &{} 56 &{} 0 \\ 10 &{} 13 &{} 28 &{} 4 &{} -2 &{} 10 &{} 58 &{} 87 &{} 59 &{} -12\\ -10 &{} -4 &{} 2 &{} -2 &{} 8 &{} 28 &{} 56 &{} 59 &{} 75 &{} -11\\ -5 &{} 10 &{} 1 &{} 11 &{} -23 &{} -14 &{} 0 &{} -12 &{} -11 &{} 122 \end{bmatrix}. \end{aligned}$$

With this $\varSigma$, we randomly sampled 2000 instances from ${\mathscr {N}}(0, \varSigma )$ distribution.

As we wanted our unbiased model $\psi$ to be simple and perfectly fit the output variable y, we decided y to be linearly dependent on instances x. Therefore, we sampled y randomly from $Bernoulli(X\cdot \beta )$ distribution. The coefficient vector $\beta$ was randomly generated as integers between $-5$ and 5, while $\beta _1$ was set to 0, as we chose $X_1$ as the sensitive feature and did not want it to have a direct impact on unbiased classifier $\psi$. We obtained the following $\beta$:

$$\begin{aligned} \beta ^T = [ 0, 0 , 4, -1, 5, 0, -5, -1, -3, 1] \end{aligned}$$