Advertisement

Text Feature Extraction and Selection Based on Attention Mechanism

  • Longxuan MaEmail author
  • Lei Zhang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11440)

Abstract

Selecting features that represent a particular corpus is important to the success of many machine learning and text mining applications. However, the previous attention-based work only focused on feature augmentation in the lexical level, lacking the exploration of feature enhancement in the sentence level. In this paper, we exploit a novel feature extraction and selection model for information retrieval, denoted by Dynamic Feature Generation Network (DFGN). In sentence dimension, features are firstly extracted by a variety of different attention mechanisms, then dynamically filtered by thresholds automatically learned. Different kinds of characteristics are distilled according to specific tasks, enhancing the practicability and robustness of the model. DFGN relies solely on the text itself, requires no external feature engineering. Our approach outperforms previous work on multiple well-known answer selection datasets. Through the analysis of the experiments, we prove that DFGN provides excellent retrieval and interpretative abilities.

Keywords

Feature extraction and selection Machine learning Question answering 

References

  1. 1.
    Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473 (2014)Google Scholar
  2. 2.
    Bian, W., Li, S., Yang, Z., Chen, G., Lin, Z.: A compare-aggregate model with dynamic-clip attention for answer selection. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM 2017, Singapore, 06–10 November 2017, pp. 1987–1990. ACM (2017)Google Scholar
  3. 3.
    Chen, Q., Zhu, X., Ling, Z., Wei, S., Jiang, H., Inkpen, D.: Enhanced LSTM for natural language inference. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Long Papers, Vancouver, Canada, 30 July–4 August, vol. 1, pp. 1657–1668. Association for Computational Linguistics (2017)Google Scholar
  4. 4.
    Feng, M., Xiang, B., Glass, M.R., Wang, L., Zhou, B.: Applying deep learning to answer selection: a study and an open task. In: 2015 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2015, Scottsdale, AZ, USA, 13–17 December 2015, pp. 813–820. IEEE (2015)Google Scholar
  5. 5.
    Gong, Y., Luo, H., Zhang, J.: Natural language inference over interaction space. CoRR abs/1709.04348 (2017)Google Scholar
  6. 6.
    He, H., Gimpel, K., Lin, J.J.: Multi-perspective sentence similarity modeling with convolutional neural networks. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, 17–21 September 2015, pp. 1576–1586. The Association for Computational Linguistics (2015)Google Scholar
  7. 7.
    He, H., Lin, J.J.: Pairwise word interaction modeling with deep neural networks for semantic similarity measurement. In: NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, 12–17 June 2016, pp. 937–948. The Association for Computational Linguistics (2016)Google Scholar
  8. 8.
    Kim, S., Hong, J., Kang, I., Kwak, N.: Semantic sentence matching with densely-connected recurrent and co-attentive information. CoRR abs/1805.11360 (2018)Google Scholar
  9. 9.
    Lin, Z., et al.: A structured self-attentive sentence embedding. CoRR abs/1703.03130 (2017)Google Scholar
  10. 10.
    Liu, Y., Sun, C., Lin, L., Wang, X.: Learning natural language inference using bidirectional LSTM model and inner-attention. CoRR abs/1605.09090 (2016)Google Scholar
  11. 11.
    Parikh, A.P., Täckström, O., Das, D., Uszkoreit, J.: A decomposable attention model for natural language inference. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, 1–4 November 2016, pp. 2249–2255. The Association for Computational Linguistics (2016)Google Scholar
  12. 12.
    Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, A Meeting of SIGDAT, a Special Interest Group of the ACL, Doha, Qatar, 25–29 October 2014, pp. 1532–1543. ACL (2014)Google Scholar
  13. 13.
    Rao, J., He, H., Lin, J.J.: Noise-contrastive estimation for answer selection with deep neural networks. In: Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, 24–28 October 2016, pp. 1913–1916. ACM (2016)Google Scholar
  14. 14.
    Rocktäschel, T., Grefenstette, E., Hermann, K.M., Kociský, T., Blunsom, P.: Reasoning about entailment with neural attention. CoRR abs/1509.06664 (2015)Google Scholar
  15. 15.
    dos Santos, C.N., Tan, M., Xiang, B., Zhou, B.: Attentive pooling networks. CoRR abs/1602.03609 (2016)Google Scholar
  16. 16.
    Shen, G., Yang, Y., Deng, Z.: Inter-weighted alignment network for sentence pair modeling. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, 9–11 September 2017, pp. 1179–1189. Association for Computational Linguistics (2017)Google Scholar
  17. 17.
    Tay, Y., Tuan, L.A., Hui, S.C.: A compare-propagate architecture with alignment factorization for natural language inference. CoRR abs/1801.00102 (2018)Google Scholar
  18. 18.
    Tay, Y., Tuan, L.A., Hui, S.C.: Multi-cast attention networks for retrieval-based question answering and response prediction. CoRR abs/1806.00778 (2018)Google Scholar
  19. 19.
    Tran, N.K., Niederée, C.: Multihop attention networks for question answer matching. In: The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, 08–12 July 2018, pp. 325–334. ACM (2018)Google Scholar
  20. 20.
    Wang, B., Liu, K., Zhao, J.: Inner attention based recurrent neural networks for answer selection. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Long Papers, Berlin, Germany, 7–12 August 2016, vol. 1. The Association for Computer Linguistics (2016)Google Scholar
  21. 21.
    Wang, M., Smith, N.A., Mitamura, T.: What is the jeopardy model? A quasi-synchronous grammar for QA. In: EMNLP-CoNLL 2007, Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Prague, Czech Republic, 28–30 June 2007, pp. 22–32. ACL (2007)Google Scholar
  22. 22.
    Wang, S., Jiang, J.: A compare-aggregate model for matching text sequences. CoRR abs/1611.01747 (2016)Google Scholar
  23. 23.
    Wang, S., Jiang, J.: Learning natural language inference with LSTM. In: NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, 12–17 June 2016, pp. 1442–1451. The Association for Computational Linguistics (2016)Google Scholar
  24. 24.
    Wang, Z., Hamza, W., Florian, R.: Bilateral multi-perspective matching for natural language sentences. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, 19–25 August 2017, pp. 4144–4150. ijcai.org (2017)Google Scholar
  25. 25.
    Yang, Y., Yih, W., Meek, C.: WikiQA: a challenge dataset for open-domain question answering. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, 17–21 September 2015, pp. 2013–2018. The Association for Computational Linguistics (2015)Google Scholar
  26. 26.
    Yin, W., Schütze, H., Xiang, B., Zhou, B.: ABCNN: attention-based convolutional neural network for modeling sentence pairs. TACL 4, 259–272 (2016)Google Scholar
  27. 27.
    Zhang, X., Li, S., Sha, L., Wang, H.: Attentive interactive neural networks for answer selection in community question answering. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, California, USA, 4–9 February 2017, pp. 3525–3531. AAAI Press (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Beijing University of Posts and TelecommunicationsBeijingChina

Personalised recommendations