Recognizing Textual Entailment with Attentive Reading and Writing Operations

  • Liang Liu
  • Huan Huo
  • Xiufeng Liu
  • Vasile Palade
  • Dunlu Peng
  • Qingkui Chen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10827)

Abstract

Inferencing the entailment relations between natural language sentence pairs is fundamental to artificial intelligence. Recently, there is a rising interest in modeling the task with neural attentive models. However, those existing models have a major limitation to keep track of the attention history because usually only one single vector is utilized to memorize the past attention information. We argue its importance based on our observation that the potential alignment clues are not always centralized. Instead, they may diverge substantially, which could cause the problem of long-range dependency. In this paper, we propose to facilitate the conventional attentive reading operations with two sophisticated writing operations - forget and update. Instead of utilizing a single vector that accommodates the attention history, we write the past attention information directly into the sentence representations. Therefore, higher memory capacity of attention history could be achieved. Experiments on Stanford Natural Language Inference corpus (SNLI) demonstrate the superior efficacy of our proposed architecture.

References

  1. 1.
    Dagan, I., Glickman, O., Magnini, B.: The PASCAL recognising textual entailment challenge. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 177–190. Springer, Heidelberg (2006).  https://doi.org/10.1007/11736790_9CrossRefGoogle Scholar
  2. 2.
    Lakoff, G.: Linguistics and natural logic. Synthese 22(1), 151–271 (1970)CrossRefGoogle Scholar
  3. 3.
    MacCartney, B.: Natural Language Inference. Stanford University, Stanford (2009)Google Scholar
  4. 4.
    Pavlick, E.: Compositional lexical semantics in natural language inference. Ph.D. dissertation, University of Pennsylvania (2017)Google Scholar
  5. 5.
    Bowman, S.R., Potts, C., Manning, C.D.: Recursive neural networks for learning logical semantics. CoRR, abs/1406.1827 (2014). http://arxiv.org/abs/1406.1827
  6. 6.
    Bowman, S.R., Potts, C., Manning, C.D.: Learning distributed word representations for natural logic reasoning. In: Proceedings of the Association for the Advancement of Artificial Intelligence Spring Symposium, AAAI, pp. 10–13 (2015)Google Scholar
  7. 7.
    Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. CoRR, abs/1508.05326 (2015). http://arxiv.org/abs/1508.05326
  8. 8.
    Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473 (2014). http://arxiv.org/abs/1409.0473
  9. 9.
    Daniluk, M., Rocktäschel, T., Welbl, J., Riedel, S.: Frustratingly short attention spans in neural language modeling. CoRR, abs/1702.04521 (2017). http://arxiv.org/abs/1702.04521
  10. 10.
    Graves, A., Wayne, G., Danihelka, I.: Neural turing machines. CoRR, abs/1410.5401 (2014). http://arxiv.org/abs/1410.5401
  11. 11.
    Parikh, A.P., Täckström, O., Das, D., Uszkoreit, J.: A decomposable attention model for natural language inference. CoRR, abs/1606.01933 (2016). http://arxiv.org/abs/1606.01933
  12. 12.
    Rocktäschel, T., Grefenstette, E., Hermann, K.M., Kociský, T., Blunsom, P.: Reasoning about entailment with neural attention. CoRR, abs/1509.06664 (2015). http://arxiv.org/abs/1509.06664
  13. 13.
    Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 1532–1543 (2014)Google Scholar
  14. 14.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)Google Scholar
  15. 15.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997).  https://doi.org/10.1162/neco.1997.9.8.1735CrossRefGoogle Scholar
  16. 16.
    McDermott, K.B., Roediger, H.L.: Memory (Encoding, Storage, Retrieval). Noba Textbook Series: Psychology. DEF Publishers, Champaign (2016). https://doi.org/nobaproject.comGoogle Scholar
  17. 17.
    Meng, F., Lu, Z., Li, H., Liu, Q.: Interactive attention for neural machine translation. CoRR, abs/1610.05011 (2016). http://arxiv.org/abs/1610.05011
  18. 18.
    Wang, M., Lu, Z., Li, H., Liu, Q.: Memory-enhanced decoder for neural machine translation. CoRR, abs/1606.02003 (2016). http://arxiv.org/abs/1606.02003
  19. 19.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR, abs/1412.6980 (2014). http://arxiv.org/abs/1412.6980
  20. 20.
    Wang, S., Jiang, J.: Learning natural language inference with LSTM. CoRR, abs/1512.08849 (2015). http://arxiv.org/abs/1512.08849
  21. 21.
    Liu, P., Qiu, X., Chen, J., Huang, X.: Deep fusion LSTMs for text semantic matching. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany, 7–12 August 2016, vol. 1, Long Papers. The Association for Computer Linguistics (2016). http://aclweb.org/anthology/P/P16/P16-1098.pdf
  22. 22.
    Liu, P., Qiu, X., Huang, X.: Modelling interaction of sentence pair with coupled-LSTMs. CoRR, abs/1605.05573 (2016). http://arxiv.org/abs/1605.05573
  23. 23.
    Cheng, J., Dong, L., Lapata, M.: Long short-term memory-networks for machine reading. CoRR, abs/1601.06733 (2016). http://arxiv.org/abs/1601.06733
  24. 24.
    Sha, L., Chang, B., Sui, Z., Li, S.: Reading and thinking: re-read LSTM unit for textual entailment recognition. In: Calzolari, N., Matsumoto, Y., Prasad, R. (eds.) 26th International Conference on Computational Linguistics, COLING 2016. Proceedings of the Conference, Technical Papers, Osaka, Japan, 11–16 December 2016, pp. 2870–2879. ACL (2016). http://aclweb.org/anthology/C/C16/C16-1270.pdf
  25. 25.
    Weston, J., Chopra, S., Bordes, A.: Memory networks. CoRR, abs/1410.3916 (2014). http://arxiv.org/abs/1410.3916
  26. 26.
    Sukhbaatar, S., Szlam, A., Weston, J., Fergus, R.: End-to-end memory networks. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Annual Conference on Neural Information Processing Systems. Advances in Neural Information Processing Systems, Montreal, Quebec, Canada, 7–12 December 2015, vol. 28, pp. 2440–2448 (2015). http://papers.nips.cc/paper/5846-end-to-end-memory-networks
  27. 27.
    Meng, F., Lu, Z., Tu, Z., Li, H., Liu, Q.: Neural transformation machine: a new architecture for sequence-to-sequence learning. CoRR, abs/1506.06442 (2015). http://arxiv.org/abs/1506.06442
  28. 28.
    Feng, Y., Zhang, S., Zhang, A., Wang, D., Abel, A.: Memory-augmented neural machine translation. CoRR, abs/1708.02005 (2017). http://arxiv.org/abs/1708.02005

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Liang Liu
    • 1
  • Huan Huo
    • 1
  • Xiufeng Liu
    • 2
  • Vasile Palade
    • 3
  • Dunlu Peng
    • 1
  • Qingkui Chen
    • 1
  1. 1.Department of Computer ScienceUniversity of Shanghai for Science and TechnologyShanghaiChina
  2. 2.Technical University of DenmarkKongens LyngbyDenmark
  3. 3.Faculty of Engineering, Environment and ComputingCoventry UniversityCoventryUK

Personalised recommendations