Abstract
We present a summary of the 7th Competition on Legal Information Extraction and Entailment. The competition consists of four tasks on case law and statute law. The case law component includes an information retrieval task (Task 1), and the confirmation of an entailment relation between an existing case and an unseen case (Task 2). The statute law component includes an information retrieval task (Task 3) and an entailment/question answering task (Task 4). Participation was open to any group based on any approach. Ten different teams participated in the case law competition tasks, most of them in more than one task. We received results from 9 teams for Task 1 (22 runs) and 8 teams for Task 2 (22 runs). On the statute law task, there were 14 different teams participating, most in more than one task. Eleven teams submitted a total of 28 runs for Task 3, and 13 teams submitted a total of 30 runs for Task 4. We summarize the approaches, our official evaluation, and analysis on our data and submission results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
There is one question (R1-23-1: relevant articles are 554 and 1002) that have a relevant article excluded by this competition (1002). We also calculated the results by excluding this question, but there is no significant difference with official evaluation results. So we use the official evaluation results for this paper.
- 3.
“In cases where a mortgage is created with respect to a building on leased land, the mortgage may not be exercised against the right of lease.”.
- 4.
“(1) If the owner of a first thing attaches a second thing that the owner owns to the first thing to serve the ordinary use of the first thing, the thing that the owner attaches is an appurtenance. (2) An appurtenance is disposed of together with the principal thing if the principal thing is disposed of.”.
References
Alberts, H., Ipek, A., Lucas, R., Wozny, P.: COLIEE 2020: Legal information retrieval and entailment with legal embeddings and boosting. In: COLIEE (2020)
Aydemir, A., de Castro Souza, P., Gelfman, A.: Using BERT and TF-IDF to predict entailment in law-based queries. In: COLIEE (2020)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018)
Hayashi, R., Kiyota, N., Fujita, M., Kano, Y.: Legal bar exam solver integrating legal logic language proleg and argument structure analysis with legal linguistic dictionary. In: COLIEE (2020)
Hudzina, J., et al.: Information extraction and entailment of common law and civil code. In: COLIEE (2020)
Leburu-Dingalo, T., Thuma, E., Motlogelwa, N., Mudongo, M.: Ub\(\_\)Botswana at COLIEE 2020 case law retrieval. In: COLIEE (2020)
MacAvaney, S., Yates, A., Cohan, A., Goharian, N.: Cedr: contextualized embeddings for document ranking. In: SIGIR (2019)
Mandal, A., Ghosh, K., Pal, A., Ghosh, S.: Automatic catchphrase identification from legal court case documents. In: Proceedings of the Conference on Information and Knowledge Management, pp. 2187–2190. ACM, New York, NY, USA (2017)
Mandal, A., Ghosh, S., Ghosh, K., Mandal, S.: Significance of textual representation in legal case retrieval and entailment. In: COLIEE (2020)
Nguyen, H.T., et al.: JNLP team: deep learning for legal processing in COLIEE 2020. In: COLIEE (2020)
Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of NAACL (2018)
Rabelo, J., Kim, M.Y., Goebel, R.: Application of text entailment techniques in COLIEE 2020. In: COLIEE (2020)
Shao, H.L., Chen, Y.C., Huang, S.: BERT-based ensemble model for the statute law retrieval and legal information entailment. In: COLIEE (2020)
Shao, Y., Liu, B., Mao, J., Liu, Y., Zhang, M., Ma, S.: THUIR@COLIEE-2020: leveraging semantic understanding and exact matching for legal case retrieval and entailment. In: COLIEE (2020)
Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: a language model-based search engine for complex queries. In: Proceedings of the International Conference on Intelligent Analysis, pp. 2–6 (2005)
Suematsu, Y., Matsuyoshi, S., Utsumi, A.: Recognizing textual entailment for Japanese legal text using lexical simplication and tuple-based matching and similarity features. In: COLIEE (2020)
Wehnert, S., et al.: Legal information retrieval and entailment detection: hybrid approaches of traditional machine learning and deep learning. In: COLIEE (2020)
Westermann, H., Šavelka, J., Benyekhlef, K.: Paragraph similarity scoring and fine-tuned BERT for legal information retrieval and entailment. In: COLIEE (2020)
Yoshioka, M., Suzuki, Y.: HUKB at COLIEE 2020 information retrieval task. In: COLIEE (2020)
Yurochkin, M., Claici, S., Chien, E., Mirzazadeh, F., Solomon, J.: Hierarchical optimal transport for document representation (2019)
Acknowledgements
This research was supported by JSPS KAKENHI Grant Numbers, JP17H06103 and JP19H05470, the National Institute of Informatics, Shizuoka University, Hokkaido University, and the University of Alberta’s Alberta Machine Intelligence Institute (Amii). Special thanks to Colin Lachance from vLex for his unwavering support in the development of the case law data set, and to continued support from Ross Intelligence and Intellicon.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Rabelo, J., Kim, MY., Goebel, R., Yoshioka, M., Kano, Y., Satoh, K. (2021). COLIEE 2020: Methods for Legal Document Retrieval and Entailment. In: Okazaki, N., Yada, K., Satoh, K., Mineshima, K. (eds) New Frontiers in Artificial Intelligence. JSAI-isAI 2020. Lecture Notes in Computer Science(), vol 12758. Springer, Cham. https://doi.org/10.1007/978-3-030-79942-7_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-79942-7_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-79941-0
Online ISBN: 978-3-030-79942-7
eBook Packages: Computer ScienceComputer Science (R0)