Appellate Court Modifications Extraction for Portuguese


Appellate Court Modifications Extraction consists of, given an Appellate Court decision, identifying the proposed modifications by the upper Court of the lower Court judge’s decision. In this work, we propose a system to extract Appellate Court Modifications for Portuguese. Information extraction for legal texts has been previously addressed using different techniques and for several languages. Our proposal differs from previous work in two ways: (1)  our corpus is composed of Brazilian Appellate Court decisions, in which we look for a set of modifications provided by the Court; and (2) to automatically extract the modifications, we use a traditional Machine Learning approach and a Deep Learning approach, both as alternative solutions and as a combined solution. We tackle the Appellate Court Modifications Extraction task, experimenting with a wide variety of methods. In order to train and evaluate the system, we have built the KauaneJunior corpus, using public data disclosed by the Appellate State Court of Rio de Janeiro jurisprudence database. Our best method, which is a Bidirectional Long Short-Term Memory network combined with Conditional Random Fields, obtained an \(F_{\beta = 1}\) score of 94.79%.

This is a preview of subscription content, access via your institution.

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9


  1. 1.

    There are recent works discussing Brazil’s Supreme Court’s (Supremo Tribunal Federal) prominent role in describing the contemporary Brazilian judiciary system. One of these papers offers a brief description of judicial review in Brazilian law, which the reader may find useful to verify some of the claims made here. See da Silva (2013).

  2. 2.

    On the importance of previous judicial decisions as a constraint on legal decision-making, see Marmor (2012).

  3. 3.

    We used an in-house BILSTM-based tagger trained for Portuguese.

  4. 4.

  5. 5.

  6. 6.

  7. 7.

  8. 8.

  9. 9.

    We used a BILSTM based tagger trained for Portuguese.


  1. Angelidis I, Chalkidis I, Koubarakis M (2018) Named entity recognition, linking and generation for greek legislation. In: JURIX, pp 1–10

  2. Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw 5(2):157–166

    Article  Google Scholar 

  3. Bevan R, Torrisi A, Atkinson K, Bollegala D, Coenen F (2018) Efficient and effective case reject-accept filtering: a study using machine learning. In: JURIX, pp 171–175

  4. Boella G, Di Caro L, Ruggeri A, Robaldo L (2014) Learning from syntax generalizations for automatic semantic annotation. J Intell Inf Syst 43(2):231–246

    Article  Google Scholar 

  5. Brasil (2008) Código civil brasileiro e legislação correlata, 2nd edn. Senado Federal, Subsecretaria de Edições Técnicas, [Online]. Accessed 28 Sept 2018

  6. Carletta J (1996) Assessing agreement on classification tasks: the kappa statistic. Comput. Linguist. 22(2):249–254

    Google Scholar 

  7. Chalkidis I, Androutsopoulos I (2017) A deep learning approach to contract element extraction. In: JURIX, pp 155–164

  8. Chalkidis I, Androutsopoulos I, Michos A (2017) Extracting contract elements. In: Proceedings of the 16th edition of the international conference on artificial intelligence and law. ACM, pp 19–28

  9. Chiarcos C (2012) Powla: modeling linguistic corpora in OWL/DL. In: Extended semantic web conference. Springer, pp 225–239

  10. Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:14061078

  11. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:14123555

  12. Cohn T, Blunsom P (2005) Semantic role labelling with tree conditional random fields. In: Proceedings of the ninth conference on computational natural language learning. Association for Computational Linguistics, pp 169–172

  13. Curran JR, Clark S, Bos J (2007) Linguistically motivated large-scale NLP with c&c and boxer. In: Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions. Association for Computational Linguistics, pp 33–36

  14. da Silva VA (2013) Deciding without deliberating. Int J Const Law 11(3):557–584

    Google Scholar 

  15. de Araujo DA, Rigo SJ, Barbosa JLV (2017) Ontology-based information extraction for juridical events with case studies in Brazilian legal realm. Artif Intell Law 25(4):379–396

    Article  Google Scholar 

  16. de Justiça CN (2017) Justiça em números 2017: ano-base 2016., [Online]. Accessed 14 Apr 2018

  17. Dozat T, Manning CD (2017) Deep biaffine attention for neural dependency parsing. In: Proceedings of the 5th international conference on learning representations

  18. Dragoni M, Villata S, Rizzi W, Governatori G (2016) Combining NLP approaches for rule extraction from legal documents. In: 1st workshop on mining and reasoning with legal texts (MIREL 2016)

  19. Dyer C, Ballesteros M, Ling W, Matthews A, Smith NA (2015) Transition-based dependency parsing with stack long short-term memory. In: Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), no. 1, pp 334–343

  20. Fillmore CJ (1977) Scenes-and-frames semantics. Linguist Struct Process 59:55–88

    Google Scholar 

  21. García-Constantino M, Atkinson K, Bollegala D, Chapman K, Coenen F, Roberts C, Robson K (2017) CLIEL: context-based information extraction from commercial law documents. In: Proceedings of the 16th edition of the international conference on artificial intelligence and law. ACM, pp 79–87

  22. Gianfelice D, Lesmo L, Palmirani M, Perlo D, Radicioni DP (2013) Modificatory provisions detection: a hybrid NLP approach. In: Proceedings of the fourteenth international conference on artificial intelligence and law. ACM, pp 43–52

  23. Gimpel K, Schneider N, O’Connor B, Das D, Mills D, Eisenstein J, Heilman M, Yogatama D, Flanigan J, Smith NA (2011) Part-of-speech tagging for twitter: annotation, features, and experiments. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies: short papers, vol 2. Association for Computational Linguistics, pp 42–47

  24. Gonçalves CR (2018) Direito civil brasileiro v. 6–Responsabilidade civil. Editora Saraiva

  25. Graves A, Schmidhuber J (2005) Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Netw. 18(5–6):602–610

    Article  Google Scholar 

  26. Graves A, Mohamed Ar, Hinton G (2013) Speech recognition with deep recurrent neural networks. In: 2013 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 6645–6649

  27. Hermann KM, Kočiský T, Grefenstette E, Espeholt L, Kay W, Suleyman M, Blunsom P (2015) Teaching machines to read and comprehend. In: Advances in neural information processing systems, pp 1693–1701  

  28. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  29. Huang Z, Xu W, Yu K (2015) Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:150801991

  30. Klein D, Manning CD (2003) Accurate unlexicalized parsing. In: Proceedings of the 41st annual meeting of the association for computational linguistics

  31. Lafferty J, McCallum A, Pereira FC (2001) Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the eighteenth international conference on machine learning, pp 282–289

  32. Ma X, Hovy E (2016) End-to-end sequence labeling via bi-directional LSTM–CNNS–CRF. In: Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), no. 1, pp 1064–1074

  33. Marmor A (2012) The Routledge companion to philosophy of law. Routledge, London

    Google Scholar 

  34. McCallum A, Li W (2003) Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In: Proceedings of the seventh conference on natural language learning at HLT-NAACL 2003, vol 4. Association for Computational Linguistics, pp 188–191

  35. Mikolov T, Chen K, Corrado G, Dean J (2013a) Efficient estimation of word representations in vector space. arXiv preprint arXiv:13013781

  36. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013b) Distributed representations of words and phrases and their compositionality. In: Burges CJC, Bottou L, Welling M, Ghahramani Z, Weinberger KQ (eds) Advances in neural information processing systems, vol 26, pp 3111–3119

  37. Nanda R, John AK, Caro LD, Boella G, Robaldo L (2017a) Legal information retrieval using topic clustering and neural networks. In: COLIEE@ ICAIL, pp 68–78

  38. Nanda R, Siragusa G, Di Caro L, Theobald M, Boella G, Robaldo L, Costamagna F (2017b) Concept recognition in European and national law. In: JURIX, pp 193–198

  39. Nguyen TS, Nguyen LM, Tojo S, Satoh K, Shimazu A (2018) Recurrent neural network-based models for recognizing requisite and effectuation parts in legal texts. Artif Intell Law 26(2):169–199

    Article  Google Scholar 

  40. Peters ME, Ammar W, Bhagavatula C, Power R (2017) Semi-supervised sequence tagging with bidirectional language models. In: Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), no. 1, pp 1756–1765

  41. Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer LS (2018b) Deep contextualized word representations. In: NAACL HLT 2018: 16th annual conference of the North American chapter of the association for computational linguistics: human language technologies, vol 1, pp 2227–2237

  42. Peters ME, Neumann M, Zettlemoyer L, Yih WT (2018a) Dissecting contextual word embeddings: architecture and representation. arXiv preprint arXiv:1808.08949

  43. Ramshaw L, Marcus M (1995) Text chunking using transformation-based learning. In: Proceedings of the 3rd ACL workshop on very large Corpora, pp 82–94

  44. Sha F, Pereira F (2003) Shallow parsing with conditional random fields. In: Proceedings of the 2003 conference of the North American chapter of the association for computational linguistics on human language technology, vol 1. Association for Computational Linguistics, pp 134–141

  45. Strubell E, Verga P, Belanger D, McCallum A (2017) Fast and accurate entity recognition with iterated dilated convolutions. In: Proceedings of the 2017 conference on empirical methods in natural language processing, pp 2670–2680

  46. Trompper M, Winkels R (2016) Automatic assignment of section structure to texts of Dutch court judgments. In: JURIX, pp 167–172

  47. Wen TH, Gasic M, Mrkšić N, Su PH, Vandyke D, Young S (2015) Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In: Proceedings of the 2015 conference on empirical methods in natural language processing. Association for Computational Linguistics, pp 1711–1721. 10.18653/v1/D15-1199,

  48. Yang Z, Yang D, Dyer C, He X, Smola A, Hovy E (2016) Hierarchical attention networks for document classification. In: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pp 1480–1489

Download references


Funding was provided by “Conselho Nacional de Desenvolvimento Científico e Tecnológico” and “Coordenação de Aperfeiçoamento de Pessoal de Nível Superior”.

Author information



Corresponding author

Correspondence to William Paulo Ducca Fernandes.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Fernandes, W.P.D., Silva, L.J.S., Frajhof, I.Z. et al. Appellate Court Modifications Extraction for Portuguese. Artif Intell Law 28, 327–360 (2020).

Download citation


  • Natural language processing
  • Deep Learning
  • Recurrent Neural Networks
  • Long Short-Term Memory
  • Gated Recurrent Units
  • Machine Learning
  • Conditional Random Fields
  • Information extraction
  • Law
  • Modificatory provisions