Compare-xAI: Toward Unifying Functional Testing Methods for Post-hoc XAI Algorithms into a Multi-dimensional Benchmark

Abstract

In recent years, Explainable AI (xAI) attracted a lot of attention as various countries turned explanations into a legal right. xAI algorithms enable humans to understand the underlying models and explain their behavior, leading to insights through which the models can be analyzed and improved beyond the accuracy metric by, e.g., debugging the learned pattern and reducing unwanted biases. However, the widespread use of xAI and the rapidly growing body of published research in xAI have brought new challenges. A large number of xAI algorithms can be overwhelming and make it difficult for practitioners to choose the correct xAI algorithm for their specific use case. This problem is further exacerbated by the different approaches used to assess novel xAI algorithms, making it difficult to compare them to existing methods. To address this problem, we introduce Compare-xAI, a benchmark that allows for a direct comparison of popular xAI algorithms with a variety of different use cases. We propose a scoring protocol employing a range of functional tests from the literature, each targeting a specific end-user requirement in explaining a model. To make the benchmark results easily accessible, we group the tests into four categories (fidelity, fragility, stability, and stress tests). We present results for 13 xAI algorithms based on 11 functional tests. After analyzing the findings, we derive potential solutions for data science practitioners as workarounds to the found practical limitations. Finally, Compare-xAI is a tentative to unify systematic evaluation and comparison methods for xAI algorithms with a focus on the end-user’s requirements. The code is made available at:

https://karim-53.github.io/cxai/.

Appendices

A Tests

For the proof-of-concept, the following list of tests is considered. Note that some tests count twice as they test both feature importance and feature attribution.

cough_and_fever:

answers the following question: Can the xAI algorithm detect symmetric binary input features?. The trained model’s equation is [Cough AND Fever]*80. The test utilize XGBRegressor model trained on a synthetic uniform distribution dataset (total size: 20000). The test procedure is as follows: train a model such that its response to the two features is exactly the same. The xAI algorithm should detect symmetric features (equal values) and allocate them equal importance. The score is calculated as follows: 1 if the xAI detect the two features are symmetric. 0 if the difference in importance is above one unit. The test is classified in the fidelity category because it is a simple tree model that demonstrate inconsistencies in explanation [9].

cough_and_fever_10_90:

answers the following question: Can the xAI algorithm detect that ’Cough’ feature is more important than ’Fever’?. The trained model’s equation is [Cough AND Fever]*80 + [Cough]*10. Cough should be more important than Fever globally. Locally for the case (Fever = yes, Cough = yes) the feature attribution of Cough should be more important. The test utilize XGBRegressor model trained on a synthetic uniform distribution dataset (total size: 20000). The test procedure is as follows: train a model with two features with unequal impact on the model. The feature with a higher influence on the output should be detected more important. The score is calculated as follows: Return 1 if Cough is more important otherwise 0. The test is classified in the fidelity category because it is a simple tree model that demonstrate inconsistencies in explanation due to the tree structure [9].

x0_plus_x1_distrib_non_uniform_stat_indep:

answers the following question: Is the xAI able to explain the model correctly despite a non-uniform distribution of the data?. The test demonstrate the effect of data distribution/causal inference. The test utilize XGBRegressor model trained on a non-uniform and statistically independent dataset (total size: 10000). The test procedure is as follows: Check if the explanation change when the distribution change. Check if non-uniform distributions affect the explanation. The score is calculated as follows: returns 1 if the two binary features obtain the same importance. The test is classified in the stability category because it assesses the impact of slightly changing the inputs [48].

x0_plus_x1_distrib_uniform_stat_dep:

answers the following question: Is the xAI able to explain the model correctly despite a statistically-dependent distribution of the data?. The test demonstrate the effect of data distribution/causal inference. The example was given in both [49] and [48]. The test utilize XGBRegressor model trained on a uniform and statistically dependent dataset (total size: 10000). The test procedure is as follows: Check if the explanation change when the distribution change. Check if statistically dependent distributions affect the explanation. The score is calculated as follows: returns 1 if the two binary features obtain the same importance. The test is classified in the stability category because To assess the impact of changing the inputs of f... This way, we are able to talk about a hypothetical scenario where the inputs are changed compared to the true features [48].

mnist:

answers the following question: Is the xAI able to detect all dummy (constant and useless) pixels?. The xAI algorithm should detect that important pixels are only in the center of the image. The test utilize an MLP model trained on the MNIST dataset (total size: 70000). The test procedure is as follows: simply train and explain the MLP model globally for every pixel. The score is calculated as follows: Return the ratio of constant pixels detected as dummy divided by the true number of constant pixels. The test is classified in the stress category because of the high number of input features. The test is adapted from [19].

fooling_perturbation_alg:

answers the following question: Is the xAI affected by an adversarial attack against perturbation-based algorithms?. Model-agnostic xAI algorithms that use feature perturbation methods might be vulnerable to this attack. The adversarial attack exploits a vulnerability to lower the feature importance of a specific feature. Setup: Let’s begin by examining the COMPAS data set. This data set consists of defendant information from Broward County, Florida. Let’s suppose that some adversary wants to mask biased or racist behavior on this data set. The test utilizes a custom function model trained on the COMPAS dataset (total size: 4629). The test procedure is as follows: The xAI algorithms need to explain the following corrupted model (custom function): if the input is from the dataset then the output is from a biased model. if not then the output is from a fair model. The score is calculated as follows: Return 1 if Race is the most important feature despite the adversarial attack. The score decreases while its rank decrease. The test is classified in the fragility category because fragility includes all adversarial attacks [47].

counterexample_dummy_axiom:

answers the following question: Is the xAI able to detect unused input features?. This is a counter-example used in literature to verify that SHAP CES do not satisfy the dummy axiom while BSHAP succeeds in this test. The test utilizes a custom function model trained on a synthetic dataset (total size: 20000). The test procedure is as follows: Train a model with one extra feature B that is dummy. The score is calculated as follows: returns 1 if the dummy feature B obtains a null importance. The test is classified in the fidelity category because assigning an importance of zero to a dummy feature reflects the model behavior (Fidelity) but also helps the data scientist to quickly understand the model.

a_and_b_or_c:

answers the following question: Can the xAI algorithm detect that input feature ’A’ is more important than ’B’ or ’C’?. This is a baseline test that the xAI should succeed in all cases. Model: A and (B or C). Goal: make sure that A is more important than B, C. Noise effect: even if the model output is not exactly equal to 1 still we expect the xai to give a correct answer. The test utilize XGBRegressor model trained on a synthetic dataset (total size: 20000). The test procedure is as follows: The model learns the following equation: A and (B or C). The explanation should prove that A is more important. The score is calculated as follows: If A is the most important feature then return 1. If A is the 2nd most important feature then return 0.5 i.e. 1- (1/nb of feature more important than A). If A is the last one: return 0 (completely wrong). The test is classified in the fidelity category because of the same reason as cough and fever 10–90: A’s effect on the output is higher than B or C.

correlated_features:

answers the following question: Can the xAI algorithm detect, that two of three input features are perfectly correlated?. This is a fragility test, which attacks the xAI algorithm by introducing a third feature ’C’ which is perfectly correlated with feature ’B’. The test utilizes a XGBRegressor model trained on a synthetic dataset with three input features (total size: 20000). The test procedure is as follows: The model is trained on a synthetic dataset with the three input features ’A’, ’B’ and ’C’, of which ’B’ and ’C’ are perfectly correlated. The xAI algorithm should detect this correlation and only assign non-zero importance to one of the two correlated features. The score is calculated as follows: Since we do not care about which correlated feature is assigned the non zero importance value, we emply following scoring metric: 1-(min(B, C)/max(B, C)). This function assigns a value of zero if the assigned importance of the correlated features is equal, and 1 if one of the features is assigned an importance of zero. The test is classified in the fragility category because it tests the xAI algorithms ability to adjust to perfect correlation in the input features.

B xAI Algorithms

archipelago:

 [20] separate the input features into sets. all features inside a set interact and there is no interaction outside a set. ArchAttribute is an interaction attribution method. ArchDetect is the corresponding interaction detector. The xAI algorithm is model agnostic i.e. it can explain any AI model. The xAI algorithm can output the following explanations: Feature interaction (local explanation).

baseline_random:

 [33] Output a random explanation. It is not a real explainer. It helps measure the baseline score and processing time. The xAI algorithm is model agnostic i.e. it can explain any AI model. The xAI algorithm can output the following explanations: Feature attribution (local explanation), Feature importance (global explanation), Feature interaction (local explanation).

exact_shapley_values:

 [5] is a permutation-based xAI algorithm following a game theory approach: Iteratively Order the features randomly, then add them to the input one at a time following this order, and calculate their expected marginal contribution [4]. The output is unique given a set of constrains defined in the original paper. The xAI algorithm is model agnostic i.e. it can explain any AI model. The xAI algorithm can output the following explanations: Feature importance (global explanation). The following information are required by the xAI algorithm:, A reference dataset (input only), The model’s predict function

kernel_shap:

 [7] it approximates the Shapley values with a constant noise [48]. The xAI algorithm is model agnostic i.e. it can explain any AI model. The xAI algorithm can output the following explanations: Feature attribution (local explanation), Feature importance (global explanation). The following information are required by the xAI algorithm:, A reference dataset (input only), The model’s predict function

lime:

 [1] it explains the model locally by generating an interpretable model approximating the original one. The xAI algorithm is model agnostic i.e. it can explain any AI model. The xAI algorithm can output the following explanations: Feature attribution (local explanation), Feature importance (global explanation). The following information are required by the xAI algorithm:, A reference dataset (input only), The model’s predict probability function, Nature of the ML task (regression/classification), The model’s predict function

maple:

 [44] is a supervised neighborhood approach that combines ideas from local linear models and ensembles of decision trees [44]. The xAI algorithm is model agnostic i.e. it can explain any AI model. The xAI algorithm can output the following explanations: Feature attribution (local explanation), Feature importance (global explanation). The following information are required by the xAI algorithm:, AI model’s structure, A reference dataset (input only), The train set, The model’s predict function

partition:

 [7] Partition SHAP approximates the Shapley values using a hierarchy of feature coalitions. The xAI algorithm is model agnostic i.e. it can explain any AI model. The xAI algorithm can output the following explanations: Feature attribution (local explanation), Feature importance (global explanation). The following information are required by the xAI algorithm:, A reference dataset (input only), The model’s predict function

permutation:

is a shuffle-based feature importance. It permutes the input data and compares it to the normal prediction The xAI algorithm is model agnostic i.e. it can explain any AI model. The xAI algorithm can output the following explanations: Feature attribution (local explanation), Feature importance (global explanation). The following information are required by the xAI algorithm:, input features, A reference dataset (input only), The model’s predict function

permutation_partition:

is a combination of permutation and partition algorithm from shap. The xAI algorithm is model agnostic i.e. it can explain any AI model. The xAI algorithm can output the following explanations: Feature attribution (local explanation), Feature importance (global explanation). The following information are required by the xAI algorithm:, input features, A reference dataset (input only), The model’s predict function

saabas:

explain tree based models by decomposing each prediction into bias and feature contribution components The xAI algorithm can explain tree-based models. The xAI algorithm can output the following explanations: Feature attribution (local explanation), Feature importance (global explanation). The following information are required by the xAI algorithm:, AI model’s structure

sage:

 [19] Compute feature importance based on Shapley value but faster. The features that are most critical for the model to make good predictions will have large importance and only features that make the model’s performance worse will have negative values.

Disadvantage: The convergence of the algorithm depends on 2 parameters: ‘thres‘ and ‘gap‘. The algorithm can be trapped in a potential infinite loop if we do not fine tune them. The xAI algorithm is model agnostic i.e. it can explain any AI model. The xAI algorithm can output the following explanations: Feature importance (global explanation). The following information are required by the xAI algorithm:, True output of the data points to explain, A reference dataset (input only), The model’s predict function

shap_interaction:

 [45] SI: Shapley Interaction Index. The xAI algorithm is model agnostic i.e. it can explain any AI model. The xAI algorithm can output the following explanations: Feature interaction (local explanation).

shapley_taylor_interaction:

 [46] STI: Shapley Taylor Interaction Index. The xAI algorithm is model agnostic i.e. it can explain any AI model. The xAI algorithm can output the following explanations: Feature interaction (local explanation).

tree_shap:

 [9] accurately compute the shap values using the structure of the tree model. The xAI algorithm can explain tree-based models. The xAI algorithm can output the following explanations: Feature attribution (local explanation), Feature importance (global explanation). The following information are required by the xAI algorithm:, AI model’s structure, A reference dataset (input only)

tree_shap_approximation:

is a faster implementation of shap reserved for tree based models. The xAI algorithm can explain tree-based models. The xAI algorithm can output the following explanations: Feature attribution (local explanation), Feature importance (global explanation). The following information are required by the xAI algorithm:, AI model’s structure, A reference dataset (input only)

joint_shapley:

is an extension of the axioms and intuitions of Shapley values proposed by [56]. This xAI algorithm creates a powerset up to a user specified power k of all the input features and computes the average Shapley values for all different subsets. This leads to a more accurate attribution of importance for each input feature, but significantly increases the run-time.

C Test Results

Table 2 contains test results without using any filter. Tests is the number of completed tests. Time is the average execution time per test. It informs the user about the relative difference in execution time between algorithms.

Table 2. Results for all Tests