Interpreting Pretrained Language Models via Concept Bottlenecks

Abstract

Pretrained language models (PLMs) have made significant strides in various natural language processing tasks. However, the lack of interpretability due to their “black-box” nature poses challenges for responsible implementation. Although previous studies have attempted to improve interpretability by using, e.g., attention weights in self-attention layers, these weights often lack clarity, readability, and intuitiveness. In this research, we propose a novel approach to interpreting PLMs by employing high-level, meaningful concepts that are easily understandable for humans. For example, we learn the concept of “Food” and investigate how it influences the prediction of a model’s sentiment towards a restaurant review. We introduce C$^3$M, which combines human-annotated and machine-generated concepts to extract hidden neurons designed to encapsulate semantically meaningful and task-specific concepts. Through empirical evaluations on real-world datasets, we show that our approach offers valuable insights to interpret PLM behavior, helps diagnose model failures, and enhances model robustness amidst noisy concept labels.

Appendices

A Definitions of Training Strategies

Given a text input $x \in \mathbb {R}^d$, concepts $c\in \mathbb {R}^k$ and its label y, the strategies for fine-tuning the text encoder $f_\theta$, the projector $p_\psi$ and the label predictor $g_\phi$ are defined as follows:

i) Vanilla fine-tuning a PLM: The concept labels are ignored, and then the text encoder $f_\theta$ and the label predictor $g_\phi$ are fine-tuned either as follows:

$$\begin{aligned} \theta , \phi = \textrm{argmin}_{\theta , \phi } L_{CE} (g_\phi (f_\theta (x), y), \end{aligned}$$

or as follows (frozen text encoder $f_\theta$):

$$\begin{aligned} \phi = \textrm{argmin}_{\phi } L_{CE} (g_\phi (f_\theta (x), y), \end{aligned}$$

where $L_{CE}$ indicates the cross-entropy loss. In this work we only consider the former option for its significant better performance.

ii) Independently training PLM with the concept and task labels: The text encoder $f_\theta$, the projector $p_\psi$ and the label predictor $g_\phi$ are trained separately with ground truth concepts labels and task labels as follows:

$$\begin{aligned} \begin{aligned} \theta , \psi &= \textrm{argmin}_{\theta , \psi } L_{CE} (p_\psi (f_\theta (x)),c), \\ \phi &= \textrm{argmin}_{\phi } L_{CE} (g_{\phi }(c),y). \end{aligned} \end{aligned}$$

During inference, the label predictor will use the output from the projector rather than the ground-truth concepts.

iii) Sequentially training PLM with the concept and task labels: We first learn the concept encoder as the independent training strategy above, and then use its output to train the label predictor:

$$\begin{aligned} \begin{aligned} \phi = \textrm{argmin}_{\phi } L_{CE} (g_{\phi }(p_\psi (f_\theta (x),y). \end{aligned} \end{aligned}$$

iv) Jointly training PLM with the concept and task labels: Learn the concept encoder and label predictor via a weighted sum $L_{joint}$ of the two objectives described above:

$$\begin{aligned} \begin{aligned} \theta , \psi , \phi &= \textrm{argmin}_{\theta , \psi , \phi } L_{joint}(x, c, y) \\ {} &= \textrm{argmin}_{\theta , \psi , \phi } [L_{CE} (g_{\phi }(p_\psi (f_\theta (x),y) \\ {} &+ \gamma L_{CE} (p_\psi (f_\theta (x)),c)]. \end{aligned} \end{aligned}$$

It’s worth noting that the CBE-PLMs trained jointly are sensitive to the loss weight $\gamma$. We report the most effective results here, tested value for $\gamma$ are given in Table 2 in Appendix D.

B Details of the Manual Concept Annotation for the IMDB Dataset

Our annotation policy is following a previous work [5] for NLP datasets annotating. For the IMDB-C dataset, we annotate the four concepts (Acting, Stroyline, Emotional Arousal, Cinematography) manually. Even though the concepts are naturally understandable by humans, two Master students familiar with sentiment analysis are selected as annotators for independent annotation with the annotation tool introduced by [30]. The strict quadruple matching F1 score between two annotators is $85.74\%$, which indicates a consistent agreement between the two annotators [13]. In case of disagreement, a third expert will be asked to make the final decision.

C Implementation Detail

In this section, we provide more details on the implementation settings of our experiments. Specifically, we implement our framework with PyTorch [24] and HuggingFace [29] and train our framework on a single 80 GB Nvidia A100 GPU. We follow a prior work [1] for backbone implementation. All backbone models have a maximum token number of 512 and a batch size of 8. We use the Adam optimizer to update the backbone, projector, and label predictor according to Sect. 3.1. The values of other hyperparameters (Table 2 in Appendix D) for each specific PLM type are determined through grid search. We run all the experiments on an Nvidia A100 GPU with 80 GB RAM.

D Parameters and Notations

In this section, we provide used notations in this paper along with their descriptions for comprehensive understanding. We also list their experimented values and optimal ones, as shown in Table 2.

Table 2. Key parameters in this paper with their annotations and evaluated values. Note that bold values indicate the optimal ones.

E Statistics of Data Splits

The Statistics and split policies of the experimented datasets, including the source concept dataset $\mathcal {D}_s$, the unlabeled concept dataset $\mathcal {D}_u$, and their augmented versions. The specific details are presented in Table 3.

Table 3. Statistics of experimented datasets. k denotes the number of concepts.

F Statistics of Concepts in Transformed Datasets

The Statistics and split policies of the transformed datasets of experimented datasets are presented in Table 4.

Table 4. Statistics of concepts in transformed datasets ($\tilde{\mathcal {D}}$). Human-specified concepts are underlined. Concepts shown in gray are not used in experiments as the portion of the “Unknown” label is too large.

G More Results on Explainable Predictions

Case studies on explainable predictions for both CEBaB and IMDB-C datasets are given in Fig. 5 and Fig. 6 respectively.

Fig. 5.

Illustration of the explanable prediction for an example from the CEBaB dataset.

Fig. 6.

Illustration of the explanable prediction for an example from the IMDB-C dataset.

H A Case Study on Test-Time Intervention

We present a case study of Test-time Intervention using an example from the transformed unlabeled concept data $\tilde{\mathcal {D}}_u$ of the CEBaB dataset, as shown in Fig. 7. The first row displays the target concept labels generated by ChatGPT. The second row shows the predictions from the trained CBE-PLM model, which mispredicts two concepts ("Waiting time" and "Waiting area"). The third row demonstrates test-time intervention using ChatGPT as the oracle, which corrects the predicted task labels. Finally, the fourth row implements test-time intervention with a human oracle, rectifying the concept that ChatGPT originally mislabeled.

Fig. 7.

Illustration of the explanable prediction for an example from the transformed unlabeled concept data $\tilde{\mathcal {D}}_{u}$ of the CEBaB dataset. The brown box with dash lines indicates the test-time intervention on corresponding concepts. (Color figure online)

I Examples of Querying ChatGPT

In this paper, we query ChatGPT for 1) augmenting the concept set, and 2) annotate missing concept labels. Note that in practice, we query ChatGPT (GPT4) via OpenAI API. Here we demonstrate examples from the ChatGPT (GPT4) GUI for better illustration. The illustrations are given in Fig. 8 and Fig. 9.

Fig. 8.

The illustration of querying ChatGPT for additional concepts for the IMDB-C dataset.

Fig. 9.

The illustration of querying ChatGPT for annotating a missing concept label for the IMDB-C dataset.