Recognizing Textual Entailment with Attentive Reading and Writing Operations

  • Liang Liu
  • Huan Huo
  • Xiufeng Liu
  • Vasile Palade
  • Dunlu Peng
  • Qingkui Chen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10827)


Inferencing the entailment relations between natural language sentence pairs is fundamental to artificial intelligence. Recently, there is a rising interest in modeling the task with neural attentive models. However, those existing models have a major limitation to keep track of the attention history because usually only one single vector is utilized to memorize the past attention information. We argue its importance based on our observation that the potential alignment clues are not always centralized. Instead, they may diverge substantially, which could cause the problem of long-range dependency. In this paper, we propose to facilitate the conventional attentive reading operations with two sophisticated writing operations - forget and update. Instead of utilizing a single vector that accommodates the attention history, we write the past attention information directly into the sentence representations. Therefore, higher memory capacity of attention history could be achieved. Experiments on Stanford Natural Language Inference corpus (SNLI) demonstrate the superior efficacy of our proposed architecture.


  1. 1.
    Dagan, I., Glickman, O., Magnini, B.: The PASCAL recognising textual entailment challenge. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 177–190. Springer, Heidelberg (2006). Scholar
  2. 2.
    Lakoff, G.: Linguistics and natural logic. Synthese 22(1), 151–271 (1970)CrossRefGoogle Scholar
  3. 3.
    MacCartney, B.: Natural Language Inference. Stanford University, Stanford (2009)Google Scholar
  4. 4.
    Pavlick, E.: Compositional lexical semantics in natural language inference. Ph.D. dissertation, University of Pennsylvania (2017)Google Scholar
  5. 5.
    Bowman, S.R., Potts, C., Manning, C.D.: Recursive neural networks for learning logical semantics. CoRR, abs/1406.1827 (2014).
  6. 6.
    Bowman, S.R., Potts, C., Manning, C.D.: Learning distributed word representations for natural logic reasoning. In: Proceedings of the Association for the Advancement of Artificial Intelligence Spring Symposium, AAAI, pp. 10–13 (2015)Google Scholar
  7. 7.
    Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. CoRR, abs/1508.05326 (2015).
  8. 8.
    Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473 (2014).
  9. 9.
    Daniluk, M., Rocktäschel, T., Welbl, J., Riedel, S.: Frustratingly short attention spans in neural language modeling. CoRR, abs/1702.04521 (2017).
  10. 10.
    Graves, A., Wayne, G., Danihelka, I.: Neural turing machines. CoRR, abs/1410.5401 (2014).
  11. 11.
    Parikh, A.P., Täckström, O., Das, D., Uszkoreit, J.: A decomposable attention model for natural language inference. CoRR, abs/1606.01933 (2016).
  12. 12.
    Rocktäschel, T., Grefenstette, E., Hermann, K.M., Kociský, T., Blunsom, P.: Reasoning about entailment with neural attention. CoRR, abs/1509.06664 (2015).
  13. 13.
    Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 1532–1543 (2014)Google Scholar
  14. 14.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)Google Scholar
  15. 15.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). Scholar
  16. 16.
    McDermott, K.B., Roediger, H.L.: Memory (Encoding, Storage, Retrieval). Noba Textbook Series: Psychology. DEF Publishers, Champaign (2016). Scholar
  17. 17.
    Meng, F., Lu, Z., Li, H., Liu, Q.: Interactive attention for neural machine translation. CoRR, abs/1610.05011 (2016).
  18. 18.
    Wang, M., Lu, Z., Li, H., Liu, Q.: Memory-enhanced decoder for neural machine translation. CoRR, abs/1606.02003 (2016).
  19. 19.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR, abs/1412.6980 (2014).
  20. 20.
    Wang, S., Jiang, J.: Learning natural language inference with LSTM. CoRR, abs/1512.08849 (2015).
  21. 21.
    Liu, P., Qiu, X., Chen, J., Huang, X.: Deep fusion LSTMs for text semantic matching. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany, 7–12 August 2016, vol. 1, Long Papers. The Association for Computer Linguistics (2016).
  22. 22.
    Liu, P., Qiu, X., Huang, X.: Modelling interaction of sentence pair with coupled-LSTMs. CoRR, abs/1605.05573 (2016).
  23. 23.
    Cheng, J., Dong, L., Lapata, M.: Long short-term memory-networks for machine reading. CoRR, abs/1601.06733 (2016).
  24. 24.
    Sha, L., Chang, B., Sui, Z., Li, S.: Reading and thinking: re-read LSTM unit for textual entailment recognition. In: Calzolari, N., Matsumoto, Y., Prasad, R. (eds.) 26th International Conference on Computational Linguistics, COLING 2016. Proceedings of the Conference, Technical Papers, Osaka, Japan, 11–16 December 2016, pp. 2870–2879. ACL (2016).
  25. 25.
    Weston, J., Chopra, S., Bordes, A.: Memory networks. CoRR, abs/1410.3916 (2014).
  26. 26.
    Sukhbaatar, S., Szlam, A., Weston, J., Fergus, R.: End-to-end memory networks. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Annual Conference on Neural Information Processing Systems. Advances in Neural Information Processing Systems, Montreal, Quebec, Canada, 7–12 December 2015, vol. 28, pp. 2440–2448 (2015).
  27. 27.
    Meng, F., Lu, Z., Tu, Z., Li, H., Liu, Q.: Neural transformation machine: a new architecture for sequence-to-sequence learning. CoRR, abs/1506.06442 (2015).
  28. 28.
    Feng, Y., Zhang, S., Zhang, A., Wang, D., Abel, A.: Memory-augmented neural machine translation. CoRR, abs/1708.02005 (2017).

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Liang Liu
    • 1
  • Huan Huo
    • 1
  • Xiufeng Liu
    • 2
  • Vasile Palade
    • 3
  • Dunlu Peng
    • 1
  • Qingkui Chen
    • 1
  1. 1.Department of Computer ScienceUniversity of Shanghai for Science and TechnologyShanghaiChina
  2. 2.Technical University of DenmarkKongens LyngbyDenmark
  3. 3.Faculty of Engineering, Environment and ComputingCoventry UniversityCoventryUK

Personalised recommendations