Boosting court judgment prediction and explanation using legal entities

Abstract

The automatic prediction of court case judgments using Deep Learning and Natural Language Processing is challenged by the variety of norms and regulations, the inherent complexity of the forensic language, and the length of legal judgments. Although state-of-the-art transformer-based architectures and Large Language Models (LLMs) are pre-trained on large-scale datasets, the underlying model reasoning is not transparent to the legal expert. This paper jointly addresses court judgment prediction and explanation by not only predicting the judgment but also providing legal experts with sentence-based explanations. To boost the performance of both tasks we leverage a legal named entity recognition step, which automatically annotates documents with meaningful domain-specific entity tags and masks the corresponding fine-grained descriptions. In such a way, transformer-based architectures and Large Language Models can attend to in-domain entity-related information in the inference process while neglecting irrelevant details. Furthermore, the explainer can boost the relevance of entity-enriched sentences while limiting the diffusion of potentially sensitive information. We also explore the use of in-context learning and lightweight fine-tuning to tailor LLMs to the legal language style and the downstream prediction and explanation tasks. The results obtained on a benchmark dataset from the Indian judicial system show the superior performance of entity-aware approaches to both judgment prediction and explanation.

Appendix

We report in appendix

• A detailed description of the hardware and configuration settings used in our experiments (see Sect. A.1);

• Additional results for the CJP task (see Sect. A.2);

• A deeper analysis of the impact of the boosting parameter on explanations on plausibility (see Sect. A.3);

• additional results on the JUSTICE dataset (see Sect. A.4).

1.1 Experimental settings

Training parameters. In the L-NER task, we trained the models based on LUKE architecture using a batch size of 256 and a learning rate of $10^{-4}$. In contrast, we trained BERT-based models using a batch size of 1 and a learning rate of $10^{-5}$. Both models undergo a maximum of 10 training epochs, with an early stopping criterion applied. Both models also have a weight decay of 0.01 and a warmup ratio of 0.06.

For the CJP task, we trained sentence encoders for a maximum of 15 epochs, utilizing a learning rate of $5\cdot 10^{-5}$, a warmup ratio of 0.06, a weight decay of 0.01, and a batch size of 64. The hierarchical transformer-based architecture has a maximum length of 256 tokens and undergoes a maximum of 100 epochs of training, with a learning rate of $5\cdot 10^{-5}$ and a batch size of 256.

Tables 12 and 13 summarize the main per-model parameter settings, whereas Table 14 reports the training and evaluation times for L-NER and CJP models in terms of the number of samples processed per second.

Hardware. The experiments were run on a machine equipped with Intel$^{\circledR }$ Core$^{\textrm{TM}}$ i9-10980XE CPU, 2 $\times$ Nvidia$^{\circledR }$ RTX A6000 GPU, 128 GB of RAM running Ubuntu 22.04 LTS. We provide detailed information about the models used for the evaluation and the fine-tuning procedure in the official project repository.³

Table 12 Summary of the used models and parameters for the NER task

Table 13 Summary of the used models and parameters for the CJP task

Table 14 Performance Analysis: training and evaluation samples per second for L-NER and CJP models. CJP models have four MLP layers; however, variations in the number of MLP layers do not affect the training and validation time

1.2 Additional results for court judgment prediction

Here we report the values of additional performance metrics on CJP, i.e., Precision and Recall. Table 15 reports the result obtained including the NER-masking step in the CJP pipeline whereas Table 16 shows the results obtained by exploiting original document sentences to make predictions.

Some relevant facts emerge from the analysis of these results:

• Relationship between Recall and F1 Score: The best recall values are consistently associated with the best F1 scores. This implies that when the approach performs well in terms of recall, it tends to achieve a higher F1 score, which is a balanced measure considering both precision and recall. This indicates that a higher recall, or the ability to correctly identify positive instances, generally leads to better overall performance as reflected by the F1 score.

• Precision and Recall with NER-Masking: The results show that when NER-masking is included in the pipeline, the precision and recall values tend to be higher compared to the approach without NER-masking. This suggests that incorporating NER-masking has a positive impact on the model’s ability to accurately identify positive instances and capture relevant information. In most cases, the precision and recall values achieved with NER-masking are consistently higher, indicating better performance in terms of correctly identifying entities of interest. However, there is one exception since the precision obtained without NER-masking on the development set is an instance where the precision value is higher compared to the precision achieved with NER-masking.

Table 15 CJP results obtained applying a hierarchical approach, masking the text with NER tags

Table 16 CJP results obtained applying a hierarchical approach, without masking the text with NER tags

1.3 Impact of boosting parameter on plausibility

Fig. 4

Plausibility results varying the degree of boosting $\beta$. We reported with a star the results when no boosting is applied ($\beta$=0)

We study the impact of the boosting parameter $\beta$ on plausibility. We compute the explanations for the explain set for which ground truth explanations are available We report the results in Fig. 4. Generally, boosting is highly beneficial for Gradient (G) and GradientXInput (GxI) with no masking. These explainers are indeed the ones with the increasing percentage of documents whose explanations are affected by the boosting. The impact is low or none for LOO without masking and G with masking, for which it is measured a low percentage of modified explanations also when increasing $\beta$.

1.4 Court judgment predictions on JUSTICE

We replicated the court judgment prediction experiments, with entity masking (H-NER-masked) and without (H-unmasked), on the JUSTICE dataset as well. The results are reported in Table 17. Notice that the entities recognized by the NER step cannot be validated because the JUSTICE dataset lacks on entity-level annotations.

Entity masking slightly improves prediction performance. The reason behind the more limited performance improvement compared to to Malik et al. (2021), are: (1) The lack of domain-specific NER annotations (we can just reuse the NER module trained on Indian court judgments), and (2) The more limited redundancy level of JUSTICE’s judgments, which contain only the facts and no extra information about the verdit.

Table 17 Results on JUSTICE dataset