Active neural learners for text with dual supervision

Abstract

Dual supervision for text classification and information retrieval, which involves training the machine with class labels augmented with text annotations that are indicative of the class, has been shown to provide significant improvements, both in and beyond active learning (AL) settings. Annotations in the simplest form are highlighted portions of the text that are indicative of the class. In this work, we aim to identify and realize the full potential of unsupervised pretrained word embeddings for text-related tasks in AL settings by training neural nets—specifically, convolutional and recurrent neural nets—through dual supervision. We propose an architecture-independent algorithm for training neural networks with human rationales for class assignments and show how unsupervised embeddings can be better leveraged in active learning settings using the said algorithm. The proposed solution involves the use of gradient-based feature attributions for constraining the machine to follow the user annotations; further, we discuss methods for overcoming the architecture-specific challenges in the optimization. Our results on the sentiment classification task show that one annotated and labeled document can be worth up to seven labeled documents, giving accuracies of up to 70% for as few as ten labeled and annotated documents, and shows promise in significantly reducing user effort for total-recall information retrieval task in systematic literature reviews.

Appendices

Appendix 1: CNN and RNN Architectures

1.1 CNN for text classification

We will use the CNN architecture proposed in Kim [25] and investigated in detail by Zhang and Wallace [68]. A simplified architecture is shown in Fig. 8 (adapted from Zhang and Wallace [68] and Kim [25]).

Fig. 8

CNN architecture: An input sentence is converted into a sentence matrix, M, of dimension $N \times d$ where N is the number of words and d is the dimension of word embedding. The activated feature maps are computed by applying 1-D convolution over M using k filters of sizes $r_i \times d$, where $r_i$, the region size, indicates the number of adjacent words jointly considered. The feature vector, $v_s$, is constructed by applying 1-max pooling on each of the activated feature maps; the dimensions of $v_s$ are therefore equal to k. $v_s$ is used as feature set for softmax classifier

We used 50 filters each of sizes 3,4 and 5 as suggested in Zhang et al. [70] and Zhang and Wallace [68]; the weights of the filters and their biases and the weights and biases of the softmax are trainable parameters. However, we trained the machine with static embeddings as suggested by Zhang and Wallace [68] (unlike Zhang et al. [70]). We applied dropout regularization to the penultimate layer ($v_s$) with a dropout probability of 0.3.

1.2 RNN for text classification

We use an adaptation of the hierarchical attention network Yang et al. [61] for text classification; specifically, we use single GRU (gated recurrent units) instead of bidirectional GRU and we ignore the hierarchy and consider the document as one long sentence. The architecture is shown in Fig. 9.

Fig. 9

RNN architecture: The input sequence $w_i$ of words represented by their embeddings and the hidden states $h_i$ are computed according to standard GRU formulation. The attentions $\alpha _i$ are derived using the context vector u (Eq. 11), and feature vector $v_s$ is computed by taking the weighted mean of the hidden activations (Eq. 12). $v_s$ is used as feature set for softmax classifier

The hidden states $h_i$ are derived using standard GRU formulation. The attentions $\alpha _i$ are derived using context vector u (Bahdanau et al. [4]) as :

$$\begin{aligned} \alpha _i = \frac{exp(u_i^{T}u)}{\sum _j exp(u_j^{T}u)} \end{aligned}$$

(11)

where $u_i = tanh(W_A h_i + b_A)$. The feature vector $v_s$ is later obtained as:

$$\begin{aligned} v_s = \sum _i \alpha _i h_i \end{aligned}$$

(12)

The trainable parameters include the various weights and biases used for computing $h_i$ along with the attention parameters ($W_A$ and $b_A$) and the context vector u. We used hidden units of size 100 and context vector of size 50. We found that the RNNs needed more regularization than CNNs and used all of the following: variational dropout between recurrent states Gal and Ghahramani [15], input embedding dropout, activity regularization of recurrent states Merity et al. [35] and L2 regularization.

Appendix 2: Architecture-specific adaptations of misattribution error

1.1 CNNs

In the CNN architecture considered, the misattribution error is hard to optimize because the gradients through the 1-max pooling will contain zeros and will not be informative enough. Essentially, the machine should first suppress values from nonuseful features and then wait around till the right values are chosen by the machine. This increases train time. Further, another challenge is that if the machine does end up selecting most of the user-suggested features, the zero gradients to the nonuser-suggested features do not really guarantee that the activation values of the nonuser-suggested features are sufficiently suppressed. This makes the optimization hard.

Let us call the given CNN network defined in Fig. 8 as N. We propose computing the attributions for the misattribution error by considering a proxy network P instantiated with the exact same weights as N but with 1-max pooling replaced with sum; i.e., the sentence features are computed by summing across all of the feature maps instead of computing max. This enables the machine to adjust attributions of all words, in parallel. Let $\tilde{A}^{P}$ be the attributions computed using the network P.

$$\begin{aligned} Misattribution_{Error} = \mu _{c}(\mathbf {A^{U}},{\tilde{\varvec{A}}}^{P}) \end{aligned}$$

(13)

However, there is one problem with this approach: The machine gets the same reward irrespective of whether it chooses to minimize the objective function by using several filters or just a few filters. In order to better demonstrate, consider the hypothetical feature maps shown in Tables 4 and 5; the shaded columns correspond to the activation values from the user-suggested features. Observe that both of them satisfy the objective function in Eq. 13, although the misattribution error for the true network N would be truly minimized if the feature maps looked like Table 2. Another side effect if the machine chose to behave like the one shown in Table 1 is that the machine ends up underutilizing the number of filters and reduces the capacity. Note that this is just one of the many possible ways to not result in desired attribution distribution.

Table 4 Hypothetical feature maps after training with Eq. 1 when three filters A, B and C of the same region size are used

Table 5 Hypothetical feature maps after training with Eq. 1 when three filters A, B and C are used

The fix to this problem would be to introduce an additional term in the objective function encouraging the machine to optimize Eq. 13 by using features across different filters instead of features across the same filter. Note that we are primarily concerned in maximizing the number of filters which select user-suggested features when 1-max pooling is applied. Let us first consider the nonzero attributions to the user-suggested features from each of the filters such that each filter is allowed to transfer information from only one set of input words; these can be extracted greedily by applying 1-max pooling only to activations corresponding to user-suggested features (Ex: along A3, B2 and C2 for feature maps in Table 1). It is easy to see that pushing these attribution scores higher would encourage the proxy network P to use features across many filters. Let $A_i^{B,c}$ be the attribution assigned to $i^{th}$ user-suggested feature along the best-possible path leading to logit of class c. We choose to maximize the sum of these attributions $\sigma _c = \sum _{i}A^{B,c}_i$ as we do not know the exact distribution of attribution among user-defined features; however, as we do not know the required magnitude of these attributions, we need to maximize it in comparison with another number. For simplicity, we choose to maximize the attributions with respect to the expected class E in comparison with the other classes. Let us quantify the total attribution from user-specified features leading up to expected class, E, with respect to other classes using softmax as $\gamma _E$:

$$\begin{aligned} \gamma _E = \frac{\exp {\sigma _E}}{\sum _{c}\exp {\sigma _c}} \end{aligned}$$

(14)

Finally, we can define the misattribution error using cross-entropy loss as:

$$\begin{aligned} Misattribution_{Error} = \mu _{c}(A^{U},\tilde{A}^{P}) + log(\gamma _E) \end{aligned}$$

(15)

Interestingly, if the activation used is ReLU or Leaky-ReLU, the second term is equivalent to adding a new example having only user-labeled features. Thus, masking features for factoring in user suggestions, like in traditional methods, is applicable in certain cases such as this and is a special case of attribution; however, unlike in traditional models, this is similar to adding a new example rather than modifying an existing data point.

In the text classification task: If $\mu _{c}(A^{U},\tilde{A}^{P})$ is used alone, we observe that the accuracies with training pool smaller than 50 examples remain unchanged, but saturate way earlier as the machine starts using fewer filters and has a reduced capacity as illustrated. Further, if the machine is trained with just $log(\gamma _E)$, the accuracies with larger number ($\ge 100$) of training examples remain relatively unchanged but those with fewer examples have a lesser accuracy; this is because the machine does not get automatically tuned to ignore nonannotated words when there are fewer examples. In the information retrieval task, however, we did not notice any advantage (even for higher number of training examples) of using the second term as the number of user-suggested features was very few.

1.2 RNNs

For the same reasons as in max pooling, optimizing through the attention is hard because the gradients are too small for the nonattended features. As we chose to compute attributions by setting F to class logits in CNNs, we choose to optimize using feature vector $v_s$ (Eq. 12) in RNNs. Following the same idea as in CNNs, we propose to compute attributions using hidden states without attention (i.e., $h_i$ instead of $\alpha _ih_i$) and independently optimize the $\alpha _i$s to select user-suggested features. In order to optimize attentions, we can easily write the misattribution error by equating the normalized attributions $\tilde{A}_i$ with the attentions $\alpha _i$:

$$\begin{aligned} Misattribution_{Error} = \mu _f\left( A^{U},\alpha _i \right) \end{aligned}$$

(16)

Fig. 10

RNN (without attentions and softmax) with an example sentence. The annotated words (shaded) can be interpreted as those words worth accumulating in the hidden state. The thickened arrows indicate those terms which the machine might remember

We need to now factor user suggestions into the computation of hidden features: In order to do that, we can very easily compute the attributions by setting F to be the sequence of hidden states; in other words, the gradients from all hidden states at time steps $\ge t$ would be considered to compute attribution of word $w_t$ as shown in Eq. 17 [the derivatives are computed at $\mathbf {V_0=w_0,V_1=w_1 ... V_n=w_n}$]

$$\begin{aligned} A_i = \mathbf {w_i \cdot } \left( \frac{\partial \mathbf {h_i}}{\partial \mathbf {V_i}} + \frac{\partial \mathbf {h_{i+1}}}{\partial \mathbf {V_{i}}} + \dots + \frac{\partial \mathbf {h_{n}}}{\partial \mathbf {V_{i}}} \right) \end{aligned}$$

(17)

Using this, we can write out the complete misattribution error as:

$$\begin{aligned} Misattribution_{Error} = \mu _f\left( A^{U},\alpha \right) + \mu _f\left( A^{U},\tilde{A}\right) \end{aligned}$$

(18)

However, this is computationally expensive and we propose approximating it by just considering the gradients from hidden state at time $t=i$. It turns out that this approximation works quite well not only for those cases where all the hidden states are used but also where the last hidden state is used. Mathematically, if $A_i$ is large, $\mathbf {w_i \cdot }\frac{\partial \mathbf {h_i}}{\partial \mathbf {V_i}}$ should be large as well; this is because, if the gradient of hidden state at $i^{th}$ time step with respect to $w_i$ is small, the gradients of hidden states at later time steps would be smaller as they are a function of $h_i$.

Gated recurrent neural networks can be interpreted as machines which read through a sequence of inputs and accumulate useful information in the hidden states from the inputs that it finds important (Fig. 10). From the figure, it is easy to see how the approximation satisfies attribution with respect to both feature vectors: mean or weighted mean of all hidden states and last hidden state. Note that this is just an approximation to speed up the training for single recurrent layer.

Appendix 3: Additional text classification results

In this section, we illustrate the advantages of combining annotations with unsupervised word embeddings by benchmarking with Sharma et al. [44]. We simulated the human labeler by extracting features as described in the paper: We choose only the positive (negative) features that had the highest $\chi ^2$ (chi-squared) statistic in at least 5% of the positive (negative) documents. Further, the artificial labeler returns any one word, rather than the top word or all the words, as rationale. Here is the description of the datasets:

• WvsH This classification task consists of classifying between comp.os.ms-windows.misc and comp.sys.ibm.pc.hardware in the 20-newsgroup classification task.

• Nova This is a binary classification task derived from the 20-newsgroup classification task. More details can be found at Guyon et al. [18].

• SRAA This is a binary classification task consisting of 48K documents that discuss either auto or aviation.

The improvement in the AUCs is illustrated in Table 6. We derived the AUCs by taking a mean of the AUCs obtained by RNNs and CNNs when trained on ten randomly chosen documents—five from each class. The AUCs for Sharma et al. [44] are extracted from the paper.

Table 6 Text classification

Additionally, we also empirically evaluated the gains we would get when we perform one-shot text classification with annotations using Yahoo! Answer Classification dataset. In this ten-way classification task, we randomly chose one document from each class and manually annotated for the purposes of the experiments. In the traditional machine learning method, we obtained an accuracy of about 14% with annotations, while we obtained 28% with RNNs and 31% with CNNs.

1.1 RNN attribution distribution

The attribution distribution obtained for RNNs is shown in Fig. 11.

Fig. 11

Attribution distribution for RNN: The graphs show the distribution of shares allocated to class-indicating words in predicting the class as the training progresses. For RNNs trained with as few as 40 labeled and annotated documents, 50% of the unlabeled documents are assigned a share of at least 0.5 to the class-indicating words

Appendix 4: Results of information retrieval

The details of the experiments are presented in Table 7. The experimental results of other three datasets are presented in Fig. 12.

Table 7 Although the query is a subset of the keywords in all of the datasets and the keywords significantly overlap with the search strings used to collect candidate studies, we report improvements in recall up to 5%

Fig. 12

Recall curves for Kitchenham, Wahono and Radjenovic: The graphs show the percentage of relevant documents retrieved over five review rounds. The first ten documents are retrieved using BM25 scoring based on the query terms (Table 3). In subsequent rounds, the top ten most relevant documents in the unlabeled pool, as predicted by the machine, are shown to the user. The curves corresponding to CNN-A and RNN-A are obtained by training the machine with the corresponding keywords (see Table 3) as corpus-level annotations. Interestingly, the machines with annotations retrieve about 5% more relevant documents although all the documents in the corpus have at least one of these terms