Abstract
We present a summary of the 9th Competition on Legal Information Extraction and Entailment (COLIEE 2022). The competition consists of four tasks on case law and statute law. The case law component includes an information retrieval task (Task 1), and the confirmation of an entailment relation between an existing case and an unseen case (Task 2). The statute law component includes an information retrieval task (Task 3) and an entailment/question answering task (Task 4). Participation was open to any group, using any approach. Ten different teams participated in the case law competition tasks, most of them in more than one task. We received competition submissions from 9 teams for Task 1 (26 runs) and 5 teams for Task 2 (14 runs). On the statute law task, there were 11 different teams participating, most in more than one task. Five teams submitted a total of 15 runs for Task 3, and 6 teams submitted a total of 18 runs for Task 4. We summarize the technical details of all approaches, describe our official evaluation, and provide an overall analysis on our data and submission results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
“Notice” is a legal technical term that denotes a legal case description that is considered to be relevant to a query case.
- 2.
for a description of the previous Task 1 formulation, please see the COLIEE 2020 summary [10].
- 3.
Results from the LLNTU and UOttawa teams were not considered because they had an f1-score of 0, which may indicate some problem in their submission.
- 4.
For example, when recall is zero it means all FN errors where captured, but there might be FP errors which would not be captured.
- 5.
Since Task 3 is a pre-processing step for the legal textual entailment (task 4), it is important to have all relevant articles in the retrieved results. So we emphasize recall in this evaluation.
References
Abolghasemi, A., Althammer, S., Hanbury, A., Verberne, S.: Dossier@coliee2022: dense retrieval and neural re-ranking for legal case retrieval. In: Sixteenth International Workshop on Juris-informatics (JURISIN) (2022)
Askari, A., Peikos, G., Pasi, G., Verberne, S.: Leibi@coliee 2022: aggregating tuned lexical models with a cluster-driven bert-based model for case law retrieval. In: Sixteenth International Workshop on Juris-informatics (JURISIN) (2022)
Bui, M.Q., Nguyen, C., Do, D.T., Le, N.K., Nguyen, D.H., Nguyen, T.T.T.: Using deep learning approaches for tackling legal’s challenges (COLIEE 2022). In: Sixteenth International Workshop on Juris-informatics (JURISIN) (2022)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018)
Fink, T., Recski, G., Kusa, W., Hanbury, A.: Statute-enhanced lexical retrieval of court cases for COLIEE 2022. In: Sixteenth International Workshop on Juris-informatics (JURISIN) (2022)
Fujita, M., Onaga, T., Ueyama, A., Kano, Y.: Legal textual entailment using ensemble of rule-based and bert-based method with data augmentations including generation without excess or deficiency. In: Sixteenth International Workshop on Juris-informatics (JURISIN) (2022)
Lin, M., Huang, S.C., Shao, H.L.: Rethinking attention: an attempting on revaluing attention weight with disjunctive union of longest uncommon subsequence for legal queries answering. In: Sixteenth International Workshop on Juris-informatics (JURISIN) (2022)
Nigam, S.K., Goel, N.: Legal case retrieval and entailment using cascading of lexical and semantic-based models. In: Sixteenth International Workshop on Juris-informatics (JURISIN) (2022)
Rabelo, J., Kim, M.Y., Goebel, R.: Semantic-based classification of relevant case law. In: Sixteenth International Workshop on Juris-informatics (JURISIN) (2022)
Rabelo, J., Kim, M.Y., Goebel, R., Yoshioka, M., Kano, Y., Satoh, K.: COLIEE 2020: methods for legal document retrieval and entailment, pp. 196–210, June 2021. https://doi.org/10.1007/978-3-030-79942-7_13
Rabelo, J., Kim, M.Y., Goebel, R., Yoshioka, M., Kano, Y., Satoh, K.: Overview and discussion of the competition on legal information extraction/entailment (COLIEE) 2021. Rev. Socionetwork Strateg. 16, 111–133 (2022.) https://doi.org/10.1007/s12626-022-00105-z
Robertson, S.E., Walker, S.: Okapi/Keenbow at TREC-8. In: Proceedings of the TREC-8, pp. 151–162 (2000)
Rosa, G.M., Bonifacio, L.H., Jeronymo, V., de Alencar Lotufo, R., Nogueira, R.: 3b parameters are worth more than in-domain training data: a case study in the legal case entailment task. In: Sixteenth International Workshop on Juris-informatics (JURISIN) (2022)
Wehnert, S., Kutty, L., Luca, E.W.D.: Using textbook knowledge for statute retrieval and entailment classification. In: Sixteenth International Workshop on Juris-informatics (JURISIN) (2022)
Wen, J., Zhong, Z., Bai, Y., Zhao, X., Yang, M.: Siat@coliee-2022: legal case retrieval with longformer-based contrastive learning. In: Sixteenth International Workshop on Juris-informatics (JURISIN) (2022)
Yoshioka, M., Suzuki, Y., Aoki, Y.: HUKB at the COLIEE 2022 statute law task. In: Sixteenth International Workshop on Juris-informatics (JURISIN) (2022)
Acknowledgements
This competition would not be possible without the significant support of Colin Lachance from vLex, Compass Law and Jurisage, and the guidance of Jimoh Ovbiagele of Ross Intelligence and Young-Yik Rhim of Intellicon. Our work to create and run the COLIEE competition is also supported by our institutions: the National Institute of Informatics (NII), Shizuoka University and Hokkaido University in Japan, and the University of Alberta and the Alberta Machine Intelligence Institute in Canada. We also acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), [including DGECR-2022-00369, RGPIN-2022-0346]. This work was also supported by JSPS KAKENHI Grant Numbers, JP17H06103 and JP19H05470 and JST, AIP Trilateral AI Research, Grant Number JPMJCR20G4.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kim, MY., Rabelo, J., Goebel, R., Yoshioka, M., Kano, Y., Satoh, K. (2023). COLIEE 2022 Summary: Methods for Legal Document Retrieval and Entailment. In: Takama, Y., Yada, K., Satoh, K., Arai, S. (eds) New Frontiers in Artificial Intelligence. JSAI-isAI 2022. Lecture Notes in Computer Science(), vol 13859. Springer, Cham. https://doi.org/10.1007/978-3-031-29168-5_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-29168-5_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-29167-8
Online ISBN: 978-3-031-29168-5
eBook Packages: Computer ScienceComputer Science (R0)