Skip to main content

Lightweight Multiple Perspective Fusion with Information Enriching for BERT-Based Answer Selection

Part of the Lecture Notes in Computer Science book series (LNAI,volume 12430)

Abstract

Answer selection (AS), as one of the hottest topics in the field of natural language processing, has developed rapidly with outstanding performances reported, especially with the emergency of pretrained model (e.g., BERT). However, the current BERT based AS methods applied BERT only by fine-tuning or stacking other modules such as CNN and RNN, but ignored to exploit the discrimination embedded inside the BERT. In this paper, we proposed a novel method LMPF-IE, i.e., Lightweight Multiple Perspective Fusion with Information Enriching. The method can mine and fuse the multi-layer discrimination inside different layers of BERT and can use Question Category and Name Entity Recognition to enrich the information which can help BERT better understand the relationship between questions and answers. We test the proposed BERT layer-wised attention model in 5 benchmark datasets of answer selection task. The experimental results clearly verify better performances than the baseline models can be achieved by our method.

Keywords

  • Answer selection
  • BERT
  • Question answer

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-60450-9_43
  • Chapter length: 12 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   119.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-60450-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   159.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.

References

  1. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of NAACL: Human Language Technologies. Association for Computational Linguistics, June 2019

    Google Scholar 

  2. Li, D., Yu, Y., Chen, Q., Li, X.: BERTSel: answer selection with pre-trained models. CoRR abs/1905.07588 (2019)

    Google Scholar 

  3. Loshchilov, I., Hutter, F.: Fixing weight decay regularization in adam. CoRR abs/1711.05101 (2017)

    Google Scholar 

  4. Madabushi, H.T., Lee, M., Barnden, J.: Integrating question classification and deep learning for improved answer selection. In: Bender, E.M., Derczynski, L., Isabelle, P. (eds.) Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, 20–26 August 2018, pp. 3283–3294. Association for Computational Linguistics (2018)

    Google Scholar 

  5. Mozafari, J., Fatemi, A., Nematbakhsh, M.A.: BAS: an answer selection method using BERT language model. CoRR abs/1911.01528 (2019)

    Google Scholar 

  6. Nakov, P., et al.: Semeval-2017 task 3: community question answering. In: Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017 (2017)

    Google Scholar 

  7. Nakov, P., et al.: Semeval-2016 task 3: community question answering. In: Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016 (2016)

    Google Scholar 

  8. Peters, M., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of NAACL: Human Language Technologies. Association for Computational Linguistics, June 2018

    Google Scholar 

  9. Peters, M.E., Ruder, S., Smith, N.A.: To tune or not to tune? Adapting pretrained representations to diverse tasks (2019)

    Google Scholar 

  10. Qi, P., Zhang, Y., Zhang, Y., Bolton, J., Manning, C.D.: Stanza: a Python natural language processing toolkit for many human languages. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (2020)

    Google Scholar 

  11. Qiao, Y., Xiong, C., Liu, Z., Liu, Z.: Understanding the behaviors of BERT in ranking (2019)

    Google Scholar 

  12. Qin, S., Rong, W., Shi, L., Yang, J., Yang, H., Xiong, Z.: Syntax tree aware adversarial question rewriting for answer selection. In: 2019 IJCNN, July 2019

    Google Scholar 

  13. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019)

    Google Scholar 

  14. Rossetto, F., Gravina, A., Severini, S., Attardi, G.: A comparative study of models for answer sentence selection. In: Proceedings of the 6th CLiC-it (2019)

    Google Scholar 

  15. dos Santos, C., Tan, M., Xiang, B., Zhou, B.: Attentive pooling networks (2016)

    Google Scholar 

  16. Severyn, A., Moschitti, A.: Learning to rank short text pairs with convolutional deep neural networks. In: Proceedings of the 38th ACM SIGIR Conference on Research and Development in Information Retrieval (2015)

    Google Scholar 

  17. Sha, L., Zhang, X., Qian, F., Chang, B., Sui, Z.: A multi-view fusion neural network for answer selection (2018)

    Google Scholar 

  18. Tang, D., Rong, W., Qin, S., Yang, J., Xiong, Z.: A n-gated recurrent unit with review for answer selection. Neurocomputing 371, 158–165 (2020)

    CrossRef  Google Scholar 

  19. Tay, Y., Phan, M.C., Luu, A.T., Hui, S.C.: Learning to rank question answer pairs with holographic dual LSTM architecture. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (2017)

    Google Scholar 

  20. Tay, Y., Tuan, L.A., Hui, S.C.: Hyperbolic representation learning for fast and efficient neural question answering. In: Proceedings of the 11th WSDM, WSDM 2018, Association for Computing Machinery (2018)

    Google Scholar 

  21. Tran, N.K., Niederée, C.: Multihop attention networks for question answer matching. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018 (2018)

    Google Scholar 

  22. Wang, B., Liu, K., Zhao, J.: Inner attention based recurrent neural networks for answer selection. In: Proceedings of the 54th ACL. Association for Computational Linguistics, August 2016

    Google Scholar 

  23. Wang, C., Jiang, F., Yang, H.: A hybrid framework for text modeling with convolutional RNN. In: the 23rd ACM SIGKDD Conference (2017)

    Google Scholar 

  24. Wang, D., Nyberg, E.: A long short-term memory model for answer sentence selection in question answering. In: Proceedings of the 53rd ACL and the 7th IJCNLP. Association for Computational Linguistics, July 2015

    Google Scholar 

  25. Wang, M., Manning, C.: Probabilistic tree-edit models with structured latent variables for textual entailment and question answering. In: Proceedings of the 23rd ICCL (Coling 2010). Coling 2010 Organizing Committee, August 2010

    Google Scholar 

  26. Wang, M., Smith, N.A., Mitamura, T.: What is the Jeopardy model? A quasi-synchronous grammar for QA. In: Proceedings of the 2007 Joint Conference on EMNLP and CoNLL. Association for Computational Linguistics, June 2007

    Google Scholar 

  27. Wang, Z., Hamza, W., Florian, R.: Bilateral multi-perspective matching for natural language sentences. In: 26th IJCAI (2017)

    Google Scholar 

  28. Yang, Y., Yih, W.T., Meek, C.: WikiQA: A challenge dataset for open-domain question answering. In: Proceedings of the 2015 Conference on EMNLP. Association for Computational Linguistics, September 2015

    Google Scholar 

  29. Yih, W.T., Chang, M.W., Meek, C., Pastusiak, A.: Question answering using enhanced lexical semantic models. In: Proceedings of the 51st ACL. Association for Computational Linguistics, August 2013

    Google Scholar 

  30. Yu, L., Hermann, K.M., Blunsom, P., Pulman, S.: Deep learning for answer sentence selection. CoRR abs/1412.1632 (2014)

    Google Scholar 

Download references

Acknowledgement

This work is partially supported by National Natural Science Foundation of China (Grants no. 61772568), Guangdong Basic and Applied Basic Research Foundation (Grant no. 2019A1515012029), and Guangdong Special Support Program.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Meng Yang .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Gu, Y., Yang, M., Lin, P. (2020). Lightweight Multiple Perspective Fusion with Information Enriching for BERT-Based Answer Selection. In: Zhu, X., Zhang, M., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2020. Lecture Notes in Computer Science(), vol 12430. Springer, Cham. https://doi.org/10.1007/978-3-030-60450-9_43

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-60450-9_43

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-60449-3

  • Online ISBN: 978-3-030-60450-9

  • eBook Packages: Computer ScienceComputer Science (R0)