From Black Boxes to Conversations: Incorporating XAI in a Conversational Agent

Abstract

The goal of Explainable AI (XAI) is to design methods to provide insights into the reasoning process of black-box models, such as deep neural networks, in order to explain them to humans. Social science research states that such explanations should be conversational, similar to human-to-human explanations. In this work, we show how to incorporate XAI in a conversational agent, using a standard design for the agent comprising natural language understanding and generation components. We build upon an XAI question bank, which we extend by quality-controlled paraphrases, to understand the user’s information needs. We further systematically survey the literature for suitable explanation methods that provide the information to answer those questions, and present a comprehensive list of suggestions. Our work is the first step towards truly natural conversations about machine learning models with an explanation agent. The comprehensive list of XAI questions and the corresponding explanation methods may support other researchers in providing the necessary information to address users’ demands. To facilitate future work, we release our source code and data https://github.com/bach1292/XAGENT/.

Appendices

Appendix

A GPT-3 Paraphrase Prompting

We finetune the GPT-3 model with two instances for each reference question in the initial XAI question bank (2-shot). Each instance consists of the reference question and two paraphrases of this question. Subsequently, we prompt the model to generate paraphrases with a new question (see Fig. 3 for an example). We repeat the prompt multiple times for each reference question.

Fig. 3.

Example GPT-3 finetuning, prompt and output to generate XAI paraphrase candidates

B Phrase Annotation Details

The distribution of annotation scores varies among each question category (see Fig. 4). In general, most of the score medians are above 4, indicating the good quality of GPT-3 in generating paraphrases. However, varying interquartile ranges suggest that GPT-3 generates better paraphrases in specific categories such as How to be that or Why not, and mixed paraphrases in others, such as What if or Other.

Figure 5 depicts the average annotator score per phrase pair. Phrase pairs are ranked by their score, separately for the 310 paraphrase pairs and 59 negative pairs. Most of the paraphrase pairs that were generated by GPT-3 have a score $\ge 4$, and thus are perceived as being similar, indicating that GPT-3 generates high-quality paraphrases in general. Conversely, most negative pairs, which were sampled from different questions, have an average score $<4$, supporting the quality of the human annotations. However, there are a few outliers of negative pairs which are annotated with a high similarity score. This is likely caused by our choice of negative phrases, sampled at random from a different question. These pairs may not be truly negative, as one question may be more general than the other or they can be interpreted in different ways (see Table 3 for examples). Furthermore, annotators disagree on ambiguous pairs and agree on unambiguous pairs (Table 4), further supporting the good quality of the dataset.

Fig. 4.

Annotation score distribution for each question category.

Fig. 5.

Average human annotation score for all phrase pairs ranked by score. Negative pairs are paraphrases from different questions.

C Representation Methods

We test two different feature representation methods: classical TF-IDF weighting, and sentence embeddings. For TF-IDF weighting, we follow a standard preprocessing pipeline: We select tokens of 2 or more alphanumeric characters (punctuation is ignored and always treated as a token separator) and stem the text using the Porter Stemmer [29] to obtain our token dictionary. Maximum and minimum DF thresholds are subject to hyperparameter optimization (see full list of hyperparameters in Table 5). We embed sentences (i.e., question instances) using SimCSE [12] to obtain an alternative feature representation to TF-IDF. We employ the pretrained RoBERTa-large model [27] as base model in SimCSE.

Table 3. Example negative pairs with average score > 4

Table 4. Phrase pairs with highest agreement/disagreement between annotators (bold indicates the reference questions in the question bank)

Table 5. Hyperparameters for Grid Search, bold indicates the chosen hyperparameters. For the other hyperparameters, we use default value in scikit-learn [38].

Fig. 6.

Confusion matrix for SimCSE + NN (Color figure online)

D Details on NLU Evaluation

Figure 6 shows the confusion matrix for SimCSE + NN’s. The blue lines separate the questions in each category (see Table 1), and the diagonal contains number of the True Positive rate for each question. This prominent diagonal reflects the high accuracy of the approaches. The squares around the diagonal are sub-confusion matrices between questions in the same group. The high number of gray color in these squares indicates that questions in the same category are harder to distinguish than questions in different category (note that the numbers on x and y axes indicate the merged labels, not IDs).

Table 6. XAI methods and selection criteria (Abbreviation: Cls = Classification, Reg = Regression, RL = Reinforcement Learning)

E XAI Method Overview

Table 6 shows the criteria, which are mentioned in Sect. 5.2 in the paper, to choose the proper XAI method for each XAI question.

F Conversation Scenarios

1.1 F.1 Random Forest Classifier on Adult Data

In this section, we show an example conversation between a prototype implementation of our proposed framework and a user on tabular data (Adult dataset (See footnote 1)) with a Random Forest (RF) classifier.

The task on this data set is to predict whether the income exceeds $50.000/year (abbreviated 50K) based on census data. We train the classifier using the sklearn library and its standard parameter settings⁶. The mean accuracy of the classifier using 3-fold cross-validation is 0.85. For explanations, we retrain the RF classifier with the same parameter settings on the full data set. The data set and the classifier are loaded at the beginning of the conversation.

Figure 1 in the main body of the paper shows a conversation with the prototype agent (X-Agent). At the beginning of the conversation, the user provides information about her features by answering retrieval questions from the agent. These questions can be generated based on DataSheets [14] of the data set. We omit this part of the conversation in Fig. 1 and show how the X-Agent reacts to several questions about the model.

The first question is the request: Give me the reason for this prediction! The natural language understanding (NLU) component matches this question to the reference question Why is this instance given this prediction? in the question bank (question 47 in Table 1). The Question-XAI method mapping (QX) selects SHAP [28] as the XAI method to provide the information for the answer. The natural language generation (NLG) component combines SHAP’s feature importance information with the predefined text “The above graph ...” to respond to the user question.

For the next question, Why is this profile predicted $\le$50K instead of >50K, the labels $\le$50K and >50K are replaced by the token <class> before matching to reference question 53 in Table 2 (main body of the paper) Why is this instance predicted P instead of Q?. The QX component identifies DICE [34] as the explanation method for this reference question, and the information is translated into natural language. In detail, DICE returns a counterfactual instance with the desired target label (>50K), yielding two features (Age and Workclass) that need to change in order to obtain the desired prediction. The NLG component extracts the relations between feature values of the original instance (Age: 39, Workclass: State-gov) and counterfactual instance (Age: 66.3, Workclass: Self-emp-inc). In comparison to the counterfactual, Age of the original instance is lower and Workclass differs. These relations are converted and rendered as text in the final answer by the NLG component.

For the final question That’s hard, how could I change only Occupation to get >50K prediction?, the words “Occupation” and “>50K” are substituted by tokens <feature> and <class> respectively. Then, the question is matched to reference question 13 (see Table 1) How should this feature change to get a different prediction?. DICE is again determined as the XAI method for providing the required information to answer this question. However, this question asks for a specific feature, i.e., constrains the search space of DICE for counterfactuals. Finally, the provided information is again translated to natural language.

Fig. 7.

Conversation example to explain a Convolutional Neural Network on MNIST (Color figure online)

1.2 F.2 Convolutional Neural Network on MNIST

We use the MNIST data set and a pre-trained convolutional neural network [46] to showcase a conversation on an image data set (see Fig. 7). First, the NLU component matches the first question Why did you predict that? to reference question 47 Why is this instance given this prediction? (see Table 1). Then, QX maps this question to SHAP [28] as the explanation technique. SHAP highlights the important parts on the image that lead to prediction 7. The NLG component adds an explanation in form of natural language text to the information provided by SHAP (the image). For the second question How should this image change to get number 9 predicted?, number 9 is replaced by token <class>. NLU maps this processed question to reference question 12 (see Table 1). QX identifies CFProto [46] as the method to answer this question. CFProto outputs the modified image that is closer to number 9. Finally, NLG generates the explanation text along with the output of CFProto.