Skip to main content
SpringerLink
Log in

Analyzing Vietnamese Legal Questions Using Deep Neural Networks with Biaffine Classifiers

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2021)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 13109))

Included in the following conference series:

Abstract

In this paper, we propose using deep neural networks to extract important information from Vietnamese legal questions, a fundamental task towards building a question answering system in the legal domain. Given a legal question in natural language, the goal is to extract all the segments that contain the needed information to answer the question. We introduce a deep model that solves the task in three stages. First, our model leverages recent advanced autoencoding language models to produce contextual word embeddings, which are then combined with character-level and POS-tag information to form word representations. Next, bidirectional long short-term memory networks are employed to capture the relations among words and generate sentence-level representations. At the third stage, borrowing ideas from graph-based dependency parsing methods which provide a global view on the input sentence, we use biaffine classifiers to estimate the probability of each pair of start-end words to be an important segment. Experimental results on a public Vietnamese legal dataset show that our model outperforms the previous work by a large margin, achieving 94.79% in the F\(_1\) score. The results also prove the effectiveness of using contextual features extracted from pre-trained language models combined with other types of features such as character-level and POS-tag features when training on a limited dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://pytorch.org/.

  2. 2.

    https://huggingface.co/transformers/.

References

  1. Bach, N.X., Cham, L.T.N., Thien, T.H.N., Phuong, T.M.: Question analysis for vietnamese legal question answering. In: Proceedings of KSE, pp. 154–159 (2017)

    Google Scholar 

  2. Xuan Bach, N., Khuong Duy, T., Minh Phuong, T.: A POS tagging model for Vietnamese social media text using BiLSTM-CRF with rich features. In: Nayak, A.C., Sharma, A. (eds.) PRICAI 2019. LNCS (LNAI), vol. 11672, pp. 206–219. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-29894-4_16

    Chapter  Google Scholar 

  3. Bach, N.X., Thanh, P.D., Oanh, T.T.: Question analysis towards a Vietnamese question answering system in the education domain. Cybern. Inf. Technol. 20(1), 112–128 (2020)

    Google Scholar 

  4. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. TACL 5, 135–146 (2017)

    Article  Google Scholar 

  5. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp. 4171–4186 (2019)

    Google Scholar 

  6. Dozat, T., Manning, C.D.: Deep biaffine attention for neural dependency parsing. In: Proceedings of ICLR (2017)

    Google Scholar 

  7. Duong, H.-T., Ho, B.-Q.: A Vietnamese question answering system in Vietnam’s legal documents. In: Saeed, K., Snášel, V. (eds.) CISIM 2014. LNCS, vol. 8838, pp. 186–197. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-45237-0_19

    Chapter  Google Scholar 

  8. Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18(5–6), 602–610 (2005)

    Article  Google Scholar 

  9. He, Z., Wang, X., Wei, W., Feng, S., Mao, X., Jiang, S.: A survey on recent advances in sequence labeling from deep learning models. arXiv preprint arXiv:2011.06727v1 (2020)

  10. He, P., Liu, X., Gao, J., Chen, W.: DeBERTa: decoding-enhanced BERT with disentangled attention. In: Proceedings of ICLR (2021)

    Google Scholar 

  11. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  12. Kien, P.M., Nguyen, H.T., Bach, N.X., Tran, V., Nguyen, M.L., Phuong, T.M.: Answering legal questions by learning neural attentive text representation. In: Proceedings of COLING, pp. 988–998 (2020)

    Google Scholar 

  13. Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of EMNLP, pp. 1746–1751 (2014)

    Google Scholar 

  14. Lafferty, J., McCallum, A., Pereira, F.: Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of ICML, pp. 282–289 (2001)

    Google Scholar 

  15. Le-Hong, P., Bui, D.T.: A factoid question answering system for Vietnamese. In: Proceedings of Web Conference Companion, Workshop Track, pp. 1049–1055 (2018)

    Google Scholar 

  16. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(110), 2278–2324 (1998)

    Article  Google Scholar 

  17. Li, Y., Li, Z., Zhang, M., Wang, R., Li, S., Si, L.: Self-attentive biaffine dependency parsing. In: Proceedings of IJCAI, pp. 5067–5073 (2019)

    Google Scholar 

  18. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692v1 (2019)

  19. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: Proceedings of ICLR (2019)

    Google Scholar 

  20. Mikolov, T., Chen, K., Corrado, G.S., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of ICLR (2013)

    Google Scholar 

  21. Mrini, K., Dernoncourt, F., Tran, Q.H., Bui, T., Chang, W., Nakashole, N.: Rethinking self-attention: towards interpretability in neural parsing. In: Proceedings of EMNLP Findings, pp. 731–742 (2020)

    Google Scholar 

  22. Nguyen, D.Q., Nguyen, D.Q., Pham., S.Q.: A Vietnamese question answering system. In: Proceedings of KSE, pp. 26–32 (2009)

    Google Scholar 

  23. Nguyen, D.Q., Verspoor, K.: End-to-end neural relation extraction using deep biaffine attention. In: Azzopardi, L., Stein, B., Fuhr, N., Mayr, P., Hauff, C., Hiemstra, D. (eds.) ECIR 2019. LNCS, vol. 11437, pp. 729–738. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-15712-8_47

    Chapter  Google Scholar 

  24. Nguyen, D.Q., Nguyen, A.T.: PhoBERT: pre-trained language models for Vietnamese. In: Proceedings of EMNLP, pp. 1037–1042 (2020)

    Google Scholar 

  25. Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of EMNLP, pp. 1532–1543 (2014)

    Google Scholar 

  26. Song, X., Petrak, J., Roberts, A.: A deep neural network sentence level classification method with context information. In: Proceedings of EMNLP, pp. 900–904 (2018)

    Google Scholar 

  27. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Proceedings of NIPS (2014)

    Google Scholar 

  28. Tran, V.M., Nguyen, V.D., Tran, O.T., Pham, U.T.T., Ha, T.Q.: An experimental study of Vietnamese question answering system. In: Proceedings of IALP, pp. 152–155 (2009)

    Google Scholar 

  29. Tran, V.M., Le, D.T., Tran, X.T., Nguyen, T.T.: A model of Vietnamese person named entity question answering system. In: Proceedings of PACLIC, pp. 325–332 (2012)

    Google Scholar 

  30. Tran, O.T., Ngo, B.X., Le Nguyen, M., Shimazu, A.: Answering legal questions by mining reference information. In: Nakano, Y., Satoh, K., Bekki, D. (eds.) JSAI-isAI 2013. LNCS (LNAI), vol. 8417, pp. 214–229. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10061-6_15

    Chapter  Google Scholar 

  31. Vaswani, A., et al.: Attention is all you need. In: Proceedings of NIPS, pp. 6000–6010 (2017)

    Google Scholar 

  32. Vu, T., Nguyen, D.Q., Nguyen, D.Q., Dras, M., Johnson, M.: VnCoreNLP: a Vietnamese natural language processing toolkit. In: Proceedings NAACL Demonstrations, pp. 56–60 (2018)

    Google Scholar 

  33. Yadav, V., Bethard, S.: A survey on recent advances in named entity recognition from deep learning models. In: Proceedings of COLING, pp. 2145–2158 (2018)

    Google Scholar 

  34. Yang, K., Deng, J.: Strongly incremental constituency parsing with graph neural networks. In: Proceedings of NeurIPS (2020)

    Google Scholar 

  35. Yang, S., Wang, Y., Chu, X.: A survey of deep learning techniques for neural machine translation. arXiv preprint arXiv:2002.07526v1 (2020)

  36. Yu, J., Bohnet, B., Poesio, M.: Named entity recognition as dependency parsing. In: Proceedings of ACL, pp. 6470–6476 (2020)

    Google Scholar 

Download references

Acknowledgements

We would like to thank FPT Technology Research Institute, FPT University for financial support which made this work possible.

Author information

Authors and Affiliations

  1. Department of Computer Science, Posts and Telecommunications Institute of Technology, Hanoi, Vietnam

    Nguyen Anh Tu, Hoang Thi Thu Uyen, Tu Minh Phuong & Ngo Xuan Bach

  2. FPT Technology Research Institute, FPT University, Hanoi, Vietnam

    Nguyen Anh Tu

Authors
  1. Nguyen Anh Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hoang Thi Thu Uyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tu Minh Phuong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ngo Xuan Bach
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ngo Xuan Bach .

Editor information

Editors and Affiliations

  1. Sampoerna University, Jakarta, Indonesia

    Teddy Mantoro

  2. Kyungpook National University, Daegu, Korea (Republic of)

    Minho Lee

  3. Sampoerna University, Jakarta, Indonesia

    Media Anugerah Ayu

  4. Murdoch University, Murdoch, WA, Australia

    Kok Wai Wong

  5. Universitas Indonesia, Depok, Indonesia

    Achmad Nizar Hidayanto

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Anh Tu, N., Thi Thu Uyen, H., Minh Phuong, T., Xuan Bach, N. (2021). Analyzing Vietnamese Legal Questions Using Deep Neural Networks with Biaffine Classifiers. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Lecture Notes in Computer Science(), vol 13109. Springer, Cham. https://doi.org/10.1007/978-3-030-92270-2_44

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-92270-2_44

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-92269-6

  • Online ISBN: 978-3-030-92270-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation