Skip to main content
SpringerLink
Log in

Towards the Automated Evaluation of Legal Casenote Essays

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13355))

Included in the following conference series:

Abstract

A legal casenote essay is a commonly assigned writing task to first-year law students aiming to promote their understanding of legal reasoning and help them acquire writing skills in a legal domain. To ensure law students master the legal casenote writing, it is critical that instructors monitor and evaluate students’ progress, and provide a timely and specific feedback. This is, however, a challenging task to many instructors as they often need to dedicate excessive time and effort to evaluate writing of and provide formative feedback to each individual student. We posit a computational tool that can afford at-scale evaluation of legal casenote writing may help remedy this challenge. Although quite some automatic writing evaluation (AWE) tools have been applied in the domain of education, the AWE tool that can analyse rhetoric of a legal casenote essay (i.e., specific rhetorical elements required by this task) is yet to be developed. We made the first step towards developing such a tool. We manually annotated each sentence in a corpus of 1,020 authentic casenote essays written over 6 offerings of the first-year legal writing course and developed one traditional machine learning classifier (Random Forest) and two deep learning classifiers (based on vanilla BERT and Legal BERT pre-trained language models). We found that the deep learning classifier based on Legal BERT could correctly identify more than 86% of rhetorical moves in a casenote. Our findings may be of a particular interest for educational researchers and practitioners who seek to use the methods of artificial intelligence to support legal writing education.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://scikit-learn.org/.

  2. 2.

    https://scikit-learn.org/stable/modules/grid_search.html.

  3. 3.

    https://liwc.wpengine.com/.

  4. 4.

    https://huggingface.co/nlpaueb/legal-bert-base-uncased.

  5. 5.

    https://www.nltk.org/.

  6. 6.

    Source files of the casenote classifier developed in this study are publicly available at https://bit.ly/3roDWTC.

References

  1. Ädel, A.: Selecting quantitative data for qualitative analysis: a case study connecting a lexicogrammatical pattern to rhetorical moves. J. Engl. Acad. Purp. 16, 68–80 (2014)

    Article  Google Scholar 

  2. Allen, L.K., Jacovina, M.E., McNamara, D.S.: Computer-based writing instruction(2016)

    Google Scholar 

  3. Almatrafi, O., Johri, A., Rangwala, H.: Needle in a haystack: identifying learner posts that require urgent response in MOOC discussion forums. Comput. Educ. 118, 1–9 (2018)

    Article  Google Scholar 

  4. Anthony, L., Lashkia, G.V.: Mover: a machine learning tool to assist in the reading and writing of technical papers. IEEE TPC 46(3), 185–193 (2003)

    Google Scholar 

  5. Artstein, R., Poesio, M.: Inter-coder agreement for computational linguistics. Comput. Linguist. 34, 555–596 (2008)

    Article  Google Scholar 

  6. Barbosa, G., et al.: Towards automatic cross-language classification of cognitive presence in online discussions. In: LAK, pp. 605–614 (2020)

    Google Scholar 

  7. Bhatia, V.K.: Simplification v. easification-the case of legal texts1. Appl. Linguis. 4(1), 42–54 (1983)

    Article  Google Scholar 

  8. Bhatia, V.K.: Analysing genre: Language use in professional settings. Routledge (2014)

    Google Scholar 

  9. Bird, S., Loper, E.: Nltk: the natural language toolkit. ACM (2004)

    Google Scholar 

  10. Bransford, J.D., Barclay, J.R., Franks, J.J.: Sentence memory: a constructive versus interpretive approach. Cogn. Psychol. 3(2), 193–209 (1972)

    Article  Google Scholar 

  11. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)

    Article  Google Scholar 

  12. Caines, A., Pastrana, S., Hutchings, A., Buttery, P.J.: Automatically identifying the function and intent of posts in underground forums. Crime Sci. 7(1), 1–14 (2018). https://doi.org/10.1186/s40163-018-0094-4

    Article  Google Scholar 

  13. Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., Androutsopoulos, I.: Legal-bert: The muppets straight out of law school (2020). arXiv preprint arXiv:2010.02559

  14. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: synthetic minority over-sampling technique. J. Artifi. Intell. Res. 16, 321–357 (2002)

    Article  Google Scholar 

  15. Cotos, E., Huffman, S., Link, S.: Understanding graduate writers’ interaction with and impact of the research writing tutor during revision. J. Writing Res. 12(1), 187–232 (2020)

    Article  Google Scholar 

  16. Crossley, S.: A chronotopic approach to genre analysis: an exploratory study. Engl. Specif. Purp. 26(1), 4–24 (2007)

    Article  Google Scholar 

  17. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding (2018). arXiv preprint arXiv:1810.04805

  18. Driessen, E., Van Der Vleuten, C.: Matching student assessment to problem-based learning: lessons from experience in a law faculty. Stud. Contin. Educ. 22(2), 235–248 (2000)

    Article  Google Scholar 

  19. Dunlosky, J., Hartwig, M.K., Rawson, K.A., Lipko, A.R.: Improving college students’ evaluation of text learning using idea-unit standards. Quart. J. Exp. Psychol. 64(3), 467–484 (2011)

    Article  Google Scholar 

  20. Ferreira, M., Rolim, V., Mello, R.F., Lins, R.D., Chen, G., Gašević, D.: Towards automatic content analysis of social presence in transcripts of online discussions. In: LAK, pp. 141–150 (2020)

    Google Scholar 

  21. Hao, Y., Dong, L., Wei, F., Xu, K.: Visualizing and understanding the effectiveness of bert (2019). arXiv preprint arXiv:1908.05620

  22. Haykin, S., Lippmann, R.: Neural networks, a comprehensive foundation. Int. J. Neural Syst. 5(4), 363–364 (1994)

    Article  Google Scholar 

  23. Hendry, G.D., Bromberger, N., Armstrong, S.: Constructive guidance and feedback for learning: the usefulness of exemplars, marking sheets and different types of feedback in a first year law subject. Ass. Evalu. High. Educ. 36(1), 1–11 (2011)

    Article  Google Scholar 

  24. Knight, S., Abel, S., Shibani, A., Goh, Y.K., Conijn, R., Gibson, A., Vajjala, S., Cotos, E., Sándor, Á., Shum, S.B.: Are you being rhetorical? a description of rhetorical move annotation tools and open corpus of sample machine-annotated rhetorical moves. J. Learn. Analy. 7(3), 138–154 (2020)

    Article  Google Scholar 

  25. Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., Wight, R., Lucas, C., Sandor, A., Kitto, K., et al.: Acawriter: a learning analytics tool for formative feedback on academic writing. J. Writing Res. 12(1), 141–186 (2020)

    Article  Google Scholar 

  26. Knight, S., Shum, S.B., Ryan, P., Sándor, Á., Wang, X.: Designing academic writing analytics for civil law student self-assessment. Int. J. AIED 28(1), 1–28 (2018)

    Google Scholar 

  27. Kovanović, V., et al.: Towards automated content analysis of discussion transcripts: A cognitive presence case. In: LAK, pp. 15–24 (2016)

    Google Scholar 

  28. Levine, J.M.: Legal writing as a discipline: Past, present, and future. ABA Legal Writing Sourcebook, 3rd ed., Duquesne University School of Law Research Paper (2020)

    Google Scholar 

  29. Lucas, C., Shum, S.B., Liu, M., Bebawy, M.: Implementing acawriter as a novel strategy to support pharmacy students’ reflective practice in scientific research. In: JPE (2021)

    Google Scholar 

  30. Neto, V., et al.: Automated analysis of cognitive presence in online discussions written in portuguese. In: Pammer-Schindler, V., Pérez-Sanagustín, M., Drachsler, H., Elferink, R., Scheffel, M. (eds.) EC-TEL 2018. LNCS, vol. 11082, pp. 245–261. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98572-5_19

    Chapter  Google Scholar 

  31. Neumann Jr., R.K., Margolis, E., Stanchi, K.M.: Legal reasoning and legal writing. Aspen Publishers (2021)

    Google Scholar 

  32. Nguyen, T.-S., Nguyen, L.-M., Tojo, S., Satoh, K., Shimazu, A.: Recurrent neural network-based models for recognizing requisite and effectuation parts in legal texts. Arti. Intell. Law 26(2), 169–199 (2018). https://doi.org/10.1007/s10506-018-9225-1

    Article  Google Scholar 

  33. Pardo, A., Jovanovic, J., Dawson, S., Gašević, D., Mirriahi, N.: Using learning analytics to scale the provision of personalised feedback. Br. J. Edu. Technol. 50(1), 128–138 (2019)

    Article  Google Scholar 

  34. Pennebaker, J.W., Francis, M.E., Booth, R.J.: Linguistic inquiry and word count: Liwc 2001, vol. 71. Lawrence Erlbaum Associates, Mahway (2001)

    Google Scholar 

  35. Ren, H., Li, Y.: A comparison study on the rhetorical moves of abstracts in published research articles and master’s foreign-language theses. Engl. Lang. Teach. 4(1), 162–166 (2011)

    Article  MathSciNet  Google Scholar 

  36. Robson, R.: Law students as legal scholars: an essay/review of scholarly writing for law students and academic legal writing. NY City L. Rev. 7, 195 (2004)

    Google Scholar 

  37. Roscoe, R.D., McNamara, D.S.: Writing pal: Feasibility of an intelligent writing strategy tutor in the high school classroom. J. Educ. Psychol. 105(4), 1010 (2013)

    Google Scholar 

  38. Sándor, Á.: Modeling metadiscourse conveying the author’s rhetorical strategy in biomedical research abstracts. Revue française de linguistique appliquée 12(2), 97–108 (2007)

    Article  Google Scholar 

  39. Sha, L., et al.: Which hammer should i use? a systematic evaluation of approaches for classifying educational forum posts. In: EDM (2021)

    Google Scholar 

  40. Shibani, A., Knight, S., Shum, S.B.: Contextualizable learning analytics design: a generic model and writing analytics evaluations. In: LAK, pp. 210–219 (2019)

    Google Scholar 

  41. Silveira, R., Fernandes, C., Neto, J.A.M., Furtado, V., Pimentel Filho, J.E.: Topic modelling of legal documents via legal-bert. In: Proceedings CEUR 1613 (2021)

    Google Scholar 

  42. Speiser, J.L., Miller, M.E., Tooze, J., Ip, E.: A comparison of random forest variable selection methods for classification prediction modeling. Expert Syst. Appl. 134, 93–101 (2019)

    Article  Google Scholar 

  43. Strobl, C., Ailhaud, E., Benetos, K., Devitt, A., Kruse, O., Proske, A., Rapp, C.: Digital support for academic writing: a review of technologies and pedagogies. Comput. Educ. 131, 33–48 (2019)

    Article  Google Scholar 

  44. Swales, J.M.: Aspects of article introductions. No. 1. University of Michigan Press (2011)

    Google Scholar 

  45. Tessuto, G.: Investigating English legal genres in academic and professional contexts. Cambridge Scholars Publishing (2013)

    Google Scholar 

  46. Tessuto, G.: Generic structure and rhetorical moves in English-language empirical law research articles: sites of interdisciplinary and interdiscursive cross-over. Engl. Specif. Purp. 37, 13–26 (2015)

    Article  Google Scholar 

  47. Vihikan, W.O., Mistica, M., Levy, I., Christie, A., Baldwin, T.: Automatic resolution of domain name disputes. In: NLP Workshop 2021, pp. 228–238 (2021)

    Google Scholar 

  48. Wenestam, A.: Labelling factual information in legal cases using fine-tuned bert models (2021)

    Google Scholar 

  49. Xing, W., Tang, H., Pei, B.: Beyond positive and negative emotions: looking into the role of achievement emotions in discussion forums of MOOCS. Internet High. Educ. 43, 100690 (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre for Learning Analytics, Faculty of Information Technology, Monash University, Clayton, Victoria, Australia

    Mladen Raković, Lele Sha, Dragan Gašević & Guanliang Chen

  2. Faculty of Law, Monash University, Clayton, Victoria, Australia

    Gerry Nagtzaam, Nick Young & Patrick Stratmann

  3. School of Informatics, University of Edinburgh, Edinburgh, UK

    Dragan Gašević

  4. Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia

    Dragan Gašević

Authors
  1. Mladen Raković
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lele Sha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gerry Nagtzaam
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nick Young
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Patrick Stratmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dragan Gašević
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Guanliang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Mladen Raković or Guanliang Chen .

Editor information

Editors and Affiliations

  1. Ateneo De Manila University, Quezon, Philippines

    Maria Mercedes Rodrigo

  2. Department of Computer Science, North Carolina State University, Raleigh, NC, USA

    Noburu Matsuda

  3. Durham University, Durham, UK

    Alexandra I. Cristea

  4. University of Leeds, Leeds, UK

    Vania Dimitrova

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Raković, M. et al. (2022). Towards the Automated Evaluation of Legal Casenote Essays. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2022. Lecture Notes in Computer Science, vol 13355. Springer, Cham. https://doi.org/10.1007/978-3-031-11644-5_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-11644-5_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-11643-8

  • Online ISBN: 978-3-031-11644-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation