Skip to main content
View expanded cover

European Conference on Information Retrieval

ECIR 2021: Advances in Information Retrieval pp 384–392Cite as

Windowing Models for Abstractive Summarization of Long Texts

  • 1060 Accesses

Part of the Lecture Notes in Computer Science book series (LNISA,volume 12657)

Abstract

Neural summarization models have a fixed-size input limitation: if text length surpasses the model’s maximal input length, some document content (possibly summary-relevant) gets truncated. Independently summarizing windows of maximal input size disallows for information flow between windows and leads to incoherent summaries. We propose windowing models for neural abstractive summarization of (arbitrarily) long texts. We extend the sequence-to-sequence model augmented with pointer generator network by (1) allowing the encoder to slide over different windows of the input document and (2) sharing the decoder and retaining its state across different input windows. We explore two windowing variants: Static Windowing precomputes the number of tokens for the decoder to generate from each window (based on training corpus statistics); in Dynamic Windowing the decoder learns to emit a token signaling the shift to the next input window. Empirical results render our models effective in intended use-case: summarizing long texts with relevant content not bound to document beginning.

Keywords

  • Abstractive summarization
  • Dynamic long text summarization

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-72240-1_39
  • Chapter length: 9 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   149.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-72240-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   199.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.

Notes

  1. 1.

    We experimented also with Transformer [17] encoder/decoder, but obtained worse results.

  2. 2.

    We pad the last window(s), if shorter than \(T_w\) tokens.

  3. 3.

    For example, with \(d=1.2\) and \(k=0.8\), the early windows will receive larger weights than the later windows.

  4. 4.

    This is a rudimentary method for computing semantic sentence similarity. We will experiment with cutting-edge sentence embedding models [4, 5, 9, 19, inter alia] in subsequent work.

  5. 5.

    Depending on \(T_w\) and ss, a sentence can appear in more than one window. In such cases, we map the sentence to its last containing window.

  6. 6.

    https://tinyurl.com/y3y69h3z.

References

  1. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: Proceedings of ICLR (2015). http://arxiv.org/abs/1409.0473

  2. Celikyilmaz, A., Bosselut, A., He, X., Choi, Y.: Deep communicating agents for abstractive summarization. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 1 (Long Papers), pp. 1662–1675 (2018)

    Google Scholar 

  3. Cohan, A., Dernoncourt, F., Kim, D.S., Bui, T., Kim, S., Chang, W., Goharian, N.: A discourse-aware attention model for abstractive summarization of long documents. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 2 (Short Papers), pp. 615–621 (2018)

    Google Scholar 

  4. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., Bordes, A.: Supervised learning of universal sentence representations from natural language inference data. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 670–680 (2017)

    Google Scholar 

  5. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186 (2019)

    Google Scholar 

  6. Hermann, K.M., et al.: Teaching machines to read and comprehend. In: Advances in Neural Information Processing Systems, pp. 1693–1701 (2015)

    Google Scholar 

  7. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)

    Google Scholar 

  8. Koupaee, M., Wang, W.Y.: Wikihow: a large scale text summarization dataset. CoRR abs/1810.09305 (2018). http://arxiv.org/abs/1810.09305

  9. Kusner, M., Sun, Y., Kolkin, N., Weinberger, K.: From word embeddings to document distances. In: International Conference on Machine Learning, pp. 957–966 (2015)

    Google Scholar 

  10. Luong, M., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: Proceedings of EMNLP, pp. 1412–1421 (2015). http://arxiv.org/abs/1508.04025

  11. Makino, T., Iwakura, T., Takamura, H., Okumura, M.: Global optimization under length constraint for neural text summarization. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1039–1048 (2019). https://www.aclweb.org/anthology/P19-1099

  12. Nallapati, R., Xiang, B., Zhou, B.: Sequence-to-sequence RNNs for text summarization. In: Proceedings of ICLR: Workshop Track (2016). http://arxiv.org/abs/1602.06023

  13. Nenkova, A., McKeown, K.R.: Automatic summarization. Found. Trends Inf. Retr. 5(2–3), 103–233 (2011)

    CrossRef  Google Scholar 

  14. Paulus, R., Xiong, C., Socher, R.: A deep reinforced model for abstractive summarization. In: Proceedings of ICLR (2018). http://arxiv.org/abs/1705.04304

  15. See, A., Liu, P.J., Manning, C.D.: Get to the point: summarization with pointer-generator networks. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073–1083 (2017)

    Google Scholar 

  16. Tan, J., Wan, X., Xiao, J.: Abstractive document summarization with a graph-based attentional neural model. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1171–1181. Association for Computational Linguistics (2017). https://doi.org/10.18653/v1/P17-1108. http://aclweb.org/anthology/P17-1108

  17. Vaswani, A., et al.: Attention is all you need. In: Proceedings of NeurIPS (2017)

    Google Scholar 

  18. You, Y., Jia, W., Liu, T., Yang, W.: Improving abstractive document summarization with salient information modeling. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2132–2141 (2019). https://www.aclweb.org/anthology/P19-1205

  19. Zhelezniak, V., Savkov, A., Shen, A., Moramarco, F., Flann, J., Hammerla, N.Y.: Don’t settle for average, go for the max: fuzzy sets and max-pooled word vectors. In: Proceedings of ICLR (2019)

    Google Scholar 

Download references

Acknowledgment

The work of Goran Glavaš is supported by the Baden Württemberg Stiftung (Eliteprogramm, AGREE grant).

Author information

Authors and Affiliations

  1. Data and Web Science Group, University of Mannheim, Mannheim, Germany

    Leon Schüller & Goran Glavaš

  2. inovex GmbH, Pforzheim, Germany

    Florian Wilhelm & Nico Kreiling

Authors
  1. Leon Schüller
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Florian Wilhelm
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nico Kreiling
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Goran Glavaš
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Goran Glavaš .

Editor information

Editors and Affiliations

  1. Radboud University Nijmegen, Nijmegen, The Netherlands

    Djoerd Hiemstra

  2. Department of Computer Science, Katholieke Universiteit Leuven, Heverlee, Belgium

    Prof. Marie-Francine Moens

  3. Toulouse, Toulouse Institute of Computer Science Research, Toulouse, France

    Josiane Mothe

  4. Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche, Pisa, Italy

    Raffaele Perego

  5. Leipzig University, Leipzig, Germany

    Martin Potthast

  6. Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche, Pisa, Italy

    Fabrizio Sebastiani

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Schüller, L., Wilhelm, F., Kreiling, N., Glavaš, G. (2021). Windowing Models for Abstractive Summarization of Long Texts. In: Hiemstra, D., Moens, MF., Mothe, J., Perego, R., Potthast, M., Sebastiani, F. (eds) Advances in Information Retrieval. ECIR 2021. Lecture Notes in Computer Science(), vol 12657. Springer, Cham. https://doi.org/10.1007/978-3-030-72240-1_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-72240-1_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-72239-5

  • Online ISBN: 978-3-030-72240-1

  • eBook Packages: Computer ScienceComputer Science (R0)