Skip to main content
SpringerLink
Log in

A Summary of the COLIEE 2019 Competition

  • Conference paper
  • First Online:
New Frontiers in Artificial Intelligence (JSAI-isAI 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12331))

Included in the following conference series:

Abstract

We summarize the evaluation of the 6th Competition on Legal Information Extraction/Entailment (COLIEE 2019). The competition consists of four tasks: two on case law and two on statute law. The case law component includes an information retrieval task (Task 1), and the confirmation of an entailment relation between an existing case and an unseen case (Task 2). The statute law component also includes an information retrieval task (Task 3) and an entailment/question answering task (Task 4), which attempts to confirm whether a particular statute applies to a yes/no question. Participation was open to any group in the world, based on any approach. Eleven different teams participated in the case law competition tasks, some of them in more than one task. We received results from 7 teams for Task 1 (15 runs) and 7 teams for Task 2 (18 runs). For the statute law tasks, 8 different teams participated, some in more than one task. Seven teams submitted a total of 13 runs for Task 3, and 7 teams submitted a total of 15 runs for Task 4. Here we summarize each team’s approaches, our official evaluation, and some analysis of the variety of methods that produced the evaluation results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    This is an interesting approach worth further investigation, however the paper describing the method lacked important information and thus was not accepted for publication on the COLIEE proceedings.

  2. 2.

    https://www.elastic.co/.

  3. 3.

    https://radimrehurek.com/gensim/.

  4. 4.

    https://scikit-learn.org.

References

  1. Dang, T.B., Nguyen, T., Nguyen, L.M.: An approach to statute law retrieval task in COLIEE-2019. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE 2019 (2019)

    Google Scholar 

  2. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018). http://arxiv.org/abs/1810.04805

  3. El Hamdani, R., Troussel, A., Houvenagel, C.: COLIEE case law competition task 1: the legal case retrieval task. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE’2019 (2019)

    Google Scholar 

  4. Gain, B., Bandyopadhyay, D., Saikh, T., Ekbal, A.: IITP@COLIEE 2019: legal information retrieval using BM25 and BERT. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE 2019 (2019)

    Google Scholar 

  5. Hayashi, R., Kano, Y.: Searching relevant articles for legal bar exam by Doc2Vec and TF-IDF. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE 2019 (2019)

    Google Scholar 

  6. Hoshino, R., Kiyota, N., Kano, Y.: Question answering system for legal bar examination using predicate argument structures focusing on exceptions. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE 2019 (2019)

    Google Scholar 

  7. Hudzina, J., Vacek, T., Madan, K., Tonya, C., Schilder, F.: Statutory entailment using similarity features and decomposable attention models. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE 2019 (2019)

    Google Scholar 

  8. Kano, Y., Kim, M.Y., Goebel, R., Satoh, K.: Overview of COLIEE 2017. In: Satoh, K., Kim, M.Y., Kano, Y., Goebel, R., Oliveira, T. (eds.) COLIEE 2017. 4th Competition on Legal Information Extraction and Entailment. EPiC Series in Computing, vol. 47, pp. 1–8. EasyChair (2017). https://doi.org/10.29007/fm8f. https://easychair.org/publications/paper/Fglr

  9. Kim, M.Y., Goebel, R., Kano, Y., Satoh, K.: COLIEE-2016: evaluation of the competition on legal information extraction and entailment. In: International Workshop on Juris-Informatics (JURISIN 2016) (2016)

    Google Scholar 

  10. Kim, M.Y., Goebel, R., Satoh, K.: COLIEE-2015: evaluation of legal question answering. In: Ninth International Workshop on Juris-Informatics (JURISIN 2015) (2015)

    Google Scholar 

  11. Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: Proceedings of the 31st International Conference on International Conference on Machine Learning, ICML 2014, vol. 32, pp. II-1188–II-1196 (2014). JMLR.org. http://dl.acm.org/citation.cfm?id=3044805.3045025

  12. Macdonald, C., McCreadie, R., Santos, R.L., Ounis, I.: From puppy to maturity: experiences in developing terrier. In: Proceedings of the SIGIR 2012 Workshop in Open Source Information Retrieval, pp. 60–63 (2012)

    Google Scholar 

  13. Madan, K., Hudzina, J., Vacek, T., Schilder, F., Custis, T.: Textual entailment using word embeddings and linguistic similarity. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE 2019 (2019)

    Google Scholar 

  14. Mi-Young, K., Rabelo, J., Goebel, R.: Statute law information retrieval and entailment. In: Proceedings of the 17th International Conference on Artificial Intelligence and Law. ICAIL 2019 (2019)

    Google Scholar 

  15. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)

    Google Scholar 

  16. Nguyen, H.T., Tran, V., Nguyen, L.M.: A deep learning approach for statute law entailment task in COLIEE-2019. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE 2019 (2019)

    Google Scholar 

  17. Paulino-Passos, G., Toni, F.: Retrieving legal cases with vector representations of text. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE 2019 (2019)

    Google Scholar 

  18. Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of NAACL (2018)

    Google Scholar 

  19. Rabelo, J., Mi-Young, K., Goebel, R.: Combining similarity and transformer methods for case law entailment. In: Proceedings of the 17th International Conference on Artificial Intelligence and Law. ICAIL 2019 (2019)

    Google Scholar 

  20. Raiyani, K., Quaresma, P.: Keyword & machine learning based japanese statute law retrieval and entailment task at COLIEE-2019. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE 2019 (2019)

    Google Scholar 

  21. Rossi, J., Kanoulas, E.: Legal information retrieval with generalized language models. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE 2019 (2019)

    Google Scholar 

  22. Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: a language-model based search engine for complex queries. In: Proceedings of the International Conference on Intelligent Analysis. Technical report (2005)

    Google Scholar 

  23. Tran, V., Nguyen, M.L., Satoh, K.: Building legal case retrieval systems with lexical matching and summarization using a pre-trained phrase scoring model. In: Proceedings of the 17th International Conference on Artificial Intelligence and Law. ICAIL 2019 (2019)

    Google Scholar 

  24. Wehnert, S., Hoque, S.A., Fenske, W., Saake, G.: Threshold-based retrieval and textual entailment detection on legal bar exam questions. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE 2019 (2019)

    Google Scholar 

  25. Yoshioka, M., Kano, Y., Kiyota, N., Satoh, K.: Overview of Japanese statute law retrieval and entailment task at COLIEE-2018. In: The Proceedings of the 12th International Workshop on Juris-Informatics (JURISIN2018), pp. 117–128. The Japanese Society of Artificial Intelligence (2018)

    Google Scholar 

  26. Yoshioka, M., Song, Z.: HUKB at COLIEE 2019 information retrieval task - utilization of metadata for relevant case retrieval. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE 2019 (2019)

    Google Scholar 

Download references

Acknowledgements

This research was supported by the National Institute of Informatics, Shizuoka University, Hokkaido University, and the University of Alberta’s Alberta Machine Intelligence Institute (Amii). Special thanks to Colin Lachance from vLex for his unwavering support in the development of the case law data set, and to continued support from Ross Intelligence and Intellicon.

Author information

Authors and Affiliations

  1. Alberta Machine Intelligence Institute, Edmonton, AB, Canada

    Juliano Rabelo, Mi-Young Kim & Randy Goebel

  2. University of Alberta, Edmonton, AB, Canada

    Juliano Rabelo & Randy Goebel

  3. Department of Science, Augustana Faculty, Camrose, AB, Canada

    Mi-Young Kim

  4. Graduate School of Information Science and Technology, Kita-ku, Sapporo-shi, Hokkaido, Japan

    Masaharu Yoshioka

  5. Global Station for Big Date and Cybersecurity, Global Institution for Collaborative Research and Education, Kita-ku, Sapporo-shi, Hokkaido, Japan

    Masaharu Yoshioka

  6. Hokkaido University, Kita-ku, Sapporo-shi, Hokkaido, Japan

    Masaharu Yoshioka

  7. Faculty of Informatics, Shizuoka University, Naka-ku, Hamamatsu-shi, Shizuoka, Japan

    Yoshinobu Kano

  8. National Institute of Informatics, Hitotsubashi, Chiyoda-ku, Tokyo, Japan

    Ken Satoh

Authors
  1. Juliano Rabelo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mi-Young Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Randy Goebel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Masaharu Yoshioka
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yoshinobu Kano
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ken Satoh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juliano Rabelo .

Editor information

Editors and Affiliations

  1. The University of Electro-Communications, Tokyo, Japan

    Maki Sakamoto

  2. Tokyo Institute of Technology, Tokyo, Japan

    Naoaki Okazaki

  3. Keio University, Tokyo, Japan

    Koji Mineshima

  4. National Institute of Informatics, Tokyo, Japan

    Ken Satoh

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rabelo, J., Kim, MY., Goebel, R., Yoshioka, M., Kano, Y., Satoh, K. (2020). A Summary of the COLIEE 2019 Competition. In: Sakamoto, M., Okazaki, N., Mineshima, K., Satoh, K. (eds) New Frontiers in Artificial Intelligence. JSAI-isAI 2019. Lecture Notes in Computer Science(), vol 12331. Springer, Cham. https://doi.org/10.1007/978-3-030-58790-1_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58790-1_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58789-5

  • Online ISBN: 978-3-030-58790-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation