Skip to main content
SpringerLink
Log in

Error Investigation of Pre-trained BERTology Models on Vietnamese Natural Language Inference

  • Conference paper
  • First Online:
Recent Challenges in Intelligent Information and Database Systems (ACIIDS 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1716))

Included in the following conference series:

Abstract

Natural Language Inference tasks have emerged in recent years and attracted significant attention from the natural language processing research community. There has been much success in this task with many quality datasets in English and Chinese for research and demonstrating the impressive performance of machine learning models. Pre-trained models play a crucial role, which is reflected in their superior performance compared to other models. However, they are still far from perfect and have many obstacles to the characteristics of the data. Especially in Vietnamese, we have just seen the emergence of the ViNLI benchmark dataset to serve the research community. In this paper, we experiment and analyze how the characteristics in the ViNLI benchmark dataset affect the performance of the pre-trained BETology-based models. In addition, the data parameters of ViNLI are also measured and analyzed on the accuracy of these models to see if it has any impact on the accuracy of the model.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amirkhani, H., et al.: FarsTail: a Persian natural language inference dataset. arXiv preprint arXiv:2009.08820 (2020)

  2. Bowman, S., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632–642 (2015)

    Google Scholar 

  3. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., Specia, L.:. SemEval-2017 task 1: semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055 (2017)

  4. Chen, J., Choi, E., Durrett, G.: Can NLI models verify QA systems’ predictions? In: Findings of the Association for Computational Linguistics, EMNLP 2021, pp. 3841–3854 (2021)

    Google Scholar 

  5. Chen, Z., Zhang, H., Zhang, X., Zhao, L.: Quora question pairs (2018). https://www.kaggle.com/c/quora-question-pairs

  6. Conneau, A., et al.: Unsupervised cross-lingual representation learning at scale (2019). arXiv preprint arXiv:1911.02116

  7. Conneau, A., et al.: XNLI: evaluating cross-lingual sentence representations. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475–2485 (2018)

    Google Scholar 

  8. Cooper, R., et al.: Using the framework (1996)

    Google Scholar 

  9. Dagan, I., Glickman, O., Magnini, B.: The PASCAL recognising textual entailment challenge. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 177–190. Springer, Heidelberg (2006). https://doi.org/10.1007/11736790_9

    Chapter  Google Scholar 

  10. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  11. Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990)

    Google Scholar 

  12. Ghaeini, R.: Dependent reading bidirectional LSTM for natural language inference. arXiv preprint arXiv:1802.05577 (2018)

  13. Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S.R., Smith, N.A.: Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324 (2018)

  14. Ham, J., Choe, Y.J., Park, K., Choi, I., Soh, H.: KorNLI and korSTS: new benchmark datasets for Korean natural language understanding. arXiv preprint arXiv:2004.03289 (2020)

  15. Hu, H., Richardson, K., Xu, L., Li, L., Kübler, S., Moss, L.S.: OCNLI: original Chinese natural language inference. In: Findings of the Association for Computational Linguistics, EMNLP 2020, pp. 3512–3526 (2020)

    Google Scholar 

  16. Van Huynh, T., Van Nguyen, K., Nguyen, N.L.-T.: ViNLI: a Vietnamese corpus for studies on open-domain natural language inference. In: Proceedings of the 29th International Conference on Computational Linguistics (Accepted) (2022)

    Google Scholar 

  17. Mahendra, R., Aji, A.F., Louvan, S., Rahman, F., Vania, C.: IndoNLI: a natural language inference dataset for Indonesian. arXiv preprint arXiv:2110.14566 (2021)

  18. Mishra, A., Patel, D., Vijayakumar, A., Li, X., Kapanipathi, P., Talamadupula, K.: Reading comprehension as natural language inference: a semantic analysis. In: Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pp. 12–19 (2020)

    Google Scholar 

  19. Nguyen, D.Q., Nguyen, A.T.: PhoBERT: pre-trained language models for Vietnamese. In: Findings of the Association for Computational Linguistics: EMNLP 2020, 1037–1042 (2020)

    Google Scholar 

  20. Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., Kiela, D.: Adversarial NLI: a new benchmark for natural language understanding. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885–4901 (2020)

    Google Scholar 

  21. Poliak, A., Naradowsky, J., Haldar, A., Rudinger, R., Van Durme, B.: Hypothesis only baselines in natural language inference. arXiv preprint arXiv:1805.01042 (2018)

  22. Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392 (2016)

    Google Scholar 

  23. Reimers, N., Gurevych, I.: Sentence-BERT: sentence embeddings using Siamese BERT-networks. arXiv preprint arXiv:1908.10084 (2019)

  24. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  25. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.: GLUE: a multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353–355 (2018)

    Google Scholar 

  26. Williams, A., Nangia, N., Bowman, S.R.: A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426 (2017)

  27. Xue, L., et al.: mT5: a massively multilingual pre-trained text-to-text transformer. In: NAACL-HLT (2021)

    Google Scholar 

  28. Zellers, R., Bisk, Y., Schwartz, R., Choi, Y.: A large-scale adversarial dataset for grounded commonsense inference. In: EMNLP, Swag (2018)

    Google Scholar 

Download references

Acknowledgement

This research is funded by Vietnam National University HoChiMinh City (VNU-HCM) under grant number DS2022-26-01. Tin Van Huynh was funded by Vingroup JSC and supported by the Master, PhD Scholarship Programme of Vingroup Innovation Foundation (VINIF), Institute of Big Data, code VINIF.2021.ThS.49.

Author information

Authors and Affiliations

  1. Faculty of Information Science and Engineering, University of Information Technology, Ho Chi Minh, Vietnam

    Tin Van Huynh, Huy Quoc To, Kiet Van Nguyen & Ngan Luu-Thuy Nguyen

  2. Vietnam National University, Ho Chi Minh City, Vietnam

    Tin Van Huynh, Huy Quoc To, Kiet Van Nguyen & Ngan Luu-Thuy Nguyen

Authors
  1. Tin Van Huynh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huy Quoc To
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kiet Van Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ngan Luu-Thuy Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tin Van Huynh .

Editor information

Editors and Affiliations

  1. University of Newcastle Australia, Newcastle, NSW, Australia

    Edward Szczerbicki

  2. Wrocław University of Science and Technology, Wrocław, Poland

    Krystian Wojtkiewicz

  3. International University - VNU-HCM, Ho Chi Minh City, Vietnam

    Sinh Van Nguyen

  4. Wrocław University of Science and Technology, Wrocław, Poland

    Marcin Pietranik

  5. Wrocław University of Science and Technology, Wrocław, Poland

    Marek Krótkiewicz

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Van Huynh, T., To, H.Q., Van Nguyen, K., Nguyen, N.LT. (2022). Error Investigation of Pre-trained BERTology Models on Vietnamese Natural Language Inference. In: Szczerbicki, E., Wojtkiewicz, K., Nguyen, S.V., Pietranik, M., Krótkiewicz, M. (eds) Recent Challenges in Intelligent Information and Database Systems. ACIIDS 2022. Communications in Computer and Information Science, vol 1716. Springer, Singapore. https://doi.org/10.1007/978-981-19-8234-7_14

Download citation

  • DOI: https://doi.org/10.1007/978-981-19-8234-7_14

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-19-8233-0

  • Online ISBN: 978-981-19-8234-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation