Neural Query-Biased Abstractive Summarization Using Copying Mechanism

A query-biased summarizer takes a query in addition to a source document as an input, and outputs a summary with respect to the query, as in Table 1. The generated summaries are intended to be used, for example, for snippets as the results of search engines. Query-biased summarization has been studied for decades [3, 4, 16, 18]. Conventional approaches are mostly extractive, and often use the overlapping words as cues to calculate the salience score of a sentence [14, 16, 18].

A copying mechanism is a network to primarily include the words in the source document in the summary [5, 6, 17]. To achieve this, the copying mechanism increases the probability of including the words in the source document. A copying mechanism can be seen as an extension of the pointer-network [19], which only copies words in the input and does not output words other than in the input. Gu et al. [5], Gulcehre et al. [6], and Miao et al. [12] extended the pointer-network to copying mechanisms by using a function to balance copying and generation. See et al. [17] and Chen and Lapata [2] applied the copying mechanism to single-document summarization tasks without a query. We came up with an idea of using copying mechanisms to include the overlapping words in the summary. However, the copying mechanisms were originally designed for the settings without the query information, and it is not necessarily clear how we can integrate the mechanisms into a query-biased summarizer.

Encoder-decoders for the query-biased setting have been proposed. Hasselqvist et al. [7] proposed an architecture being able to copy the words in the source document, while our copying mechanisms copy the overlapping words and their surroundings explicitly. Nema et al. [13] presented a dataset extracted from Debatepedia. They proposed a method to gain the diversity of the summary, while we focus on copying mechanisms.

We propose three copying mechanisms designed for query-biased summarizers: copying from the source, copying the overlapping words, and copying the overlapping words and their surroundings. We empirically show that the models copying the overlapping words perform better. These results support the fact that the strategy to include the overlapping words, which was shown useful for conventional query-biased summarizers, also works well for neural network-based query-biased summarizers.

We first explain a base query-biased neural abstractive summarizer proposed by Nema et al. [13], into which we integrate our copying mechanisms in the next section.

Open image in new window

Encoders: The base model has two bi-directional Long Short-term Memory (LSTM) [8]-based encoders; one is for the query \(\mathbf{q} =\{q_{1}, ..., q_{|\mathbf{q} |}\}\) and another is for the source document \(\mathbf{d} =\{w_{1}, ..., w_{|\mathbf{d} |}\}\). In each encoder, the outputs of the forward and the backward LSTM are concatenated into a vector. We refer to the generated vector for the i-th word in the query as \(h^{q}_{i}\), and the j-th word in the source document as \(h^{d}_{j}\).

Decoder with Query- and Source Document- Attentions: The decoder outputs the summary. The final state \(h^{d}_{|\mathbf {d}|}\) is used to initialize the first state of the LSTM in the decoder. In each time step t, the decoder calculates the attention weights \(a^{q}_{t,i}= v_{q} \cdot \texttt {tanh}(W_{q}s_{t} + U_{q}h^{q}_{i})\) for every word in the query. \(W_{q}, U_{q} \in \mathbb {R}^{l \times l}\) are weight matrices, and \(v_{q} \in \mathbb {R}^{l}\) is a \(l-\)dimensional weight vector, where each element is automatically learned from the training data. \(s_{t}=LSTM_{d}(s_{t-1}, [y_{t-1}; d_{t-1}])\) is the output of the LSTM in the decoder. Here, \(y_{t-1}\) is the embedding vector of the previously generated word and \(d_{t}\) is a document representation explained later. The weights are converted into probabilities \(\alpha ^{q}_{t,i}=\frac{\texttt {exp}(a^{q}_{t,i})}{\sum _{i=1}^{|\mathbf{q} |}{} \texttt {exp}(a^{q}_{t,i})}\). We now obtain a query vector: \(q_{t} = \sum _{i=1}^{|\mathbf{q} |}\alpha ^{q}_{t,i}h^{q}_{i}.\)

The source document attention mechanism further calculates the attention weights \(a^{d}_{t,j}\) for every word in the source document and converts them into probabilities \(\alpha ^{d}_{t,j}\):

Open image in new window

(1)

\(W_{d}, U_{d}, Z \in \mathbb {R}^{l \times l}\) and \(v_{d} \in \mathbb {R}^{l}\) are learnable parameters. Note that Eq. (1) contains \(q_{t}\), which means that the weights are calculated by considering the query. We then take the weighted average to obtain a document vector: \(d_{t} = \sum _{j=1}^{|\mathbf{w} |} \alpha ^{d}_{t,j}h^{d}_{j}\).

Objective Function: All learnable matrices are tuned to minimize the negative likelihood for the reference summaries y in the training data D: \(-\frac{1}{|D|}\sum _{D}{log\ p(x|y)}\).

We discuss the copying mechanisms for query-biased summarizers. Figure 1 shows the overview of a query-biased summarizer with a copying mechanism. In the following subsections, we present three mechanisms; SOURCE, OVERLAPand OVERLAP-WIND.

In the equations above, Q refers to the set of content words¹ in the query. Thus, \(Q \cap M\) represents the overlapping words. \(\lambda _{d} \in \mathbb {R}> 0\) is a hyperparameter that controls the importance of the overlapping words. By using \(\lambda _{d}\), this model can assign a relatively high probability for the overlapping words.Thus, the overlapping words are more likely to be included in the summary. \(\lambda _{d}\) is tuned on validation data.

We used the publicly available dataset² provided by Nema et al. [13]. The data contains the tuples of a source document, a query and a summary, extracted from Debatepedia³. We used 80% of the data for training, and the remaining was equally split for parameter tuning and testing.

We used Adam [9] for the optimizer with \(\beta _{1}=0.9\) and \(\beta _{2}=0.999\). The initial learning rate was set to 0.0004. The word embeddings for both the query and the source document were initialized by GloVe [15] and further tuned during training the models. We selected the best-performing value from 200, 300 and 400 for the dimension size for LSTM by using the data for tuning. We used 32 for the batch size. We used all the vocabulary in the training data as the pre-defined dictionary.

We prepared three baselines without any copying mechanisms. The first, ENC-DEC, was a simple encoder-decoder based summarizer without the query encoder. This model uses Eq. (1) without \(q_{t}\). The second, ENC-DEC QUERY, was the query-aware encoder-decoder explained in Sect. 2. The third, DIVERSE, was the state-of-the-art model proposed by Nema et al. [13]. We adopted full-length ROUGE [11] for the automatic evaluation metric. In addition, we conducted manual evaluation by human judges. 55 randomly selected document/query pairs and their summaries generated by DIVERSE, SOURCE, and OVERLAP-WINDwere shown to crowdworkers on Amazon Mechanical Turk. We assigned 10 workers for each set of document/query/summaries and asked them to rank the summaries. We adopted readability and responsiveness as the manual evaluation criteria, following the evaluation metric in DUC2007⁴. The workers were allowed to give the same rank to multiple summaries.

ROUGE Scores: ENC-DEC, without query information, achieved very low performance. Adding the query encoder (ENC-DEC QUERY) improved the score. Furthermore, copying the words in the source document (SOURCE) achieved better scores than those of the best-performing model without a copying mechanism (DIVERSE). Thus, integrating the copying mechanism improved the performance even in the query-biased setting. Among our models, OVERLAP  and OVERLAP-WIND  (\(L_{d}=1\))  achieved the better performances than SOURCE. The dagger (\({\dagger }\)) indicates that the differences between the scores of SOURCE, OVERLAP  and those of our best-performing model (OVERLAP-WIND(\(L_{d}=1\))) are statistically significant with the paired bootstrap resampling test used in Koehn et al. [10] (\(p < 0.05\)). This supports our assumption that the strategy to copy the overlapping words is shown effective even for RNN-based summarizers.

Rankings by Human Judges: Our best model OVERLAP-WIND  (\(L_{d}=1\)) is ranked higher than the state-of-the-art DIVERSE and SOURCE. The differences between OVERLAP-WIND  (\(L_{d}=1\)) and SOURCE are statistically significant (\(p < 0.05\)) with the paired bootstrap resampling test [10]. The results of manual evaluation also support our assumption.

We proposed the copying mechanisms designed for query-biased summarizers to primarily include the words overlapping between the source document and the query. Our experimental results showed that the mechanisms to primarily include the overlapping words between the source document and the query achieved the better performances in terms of both ROUGE and rankings by human judges. The results suggested that the strategy to include the overlapping words, which has been shown useful for conventional non-neural summarizers, also works well for RNN-based summarizers.