The banking industry has lately been under pressure, notably from regulators and NGOs, to report various Environmental, Societal and Governance (ESG) metrics (e.g., the carbon footprint of loans). For years at Crédit Agricole, a specialized division examined ESG and Corporate Social Responsibility (CSR) reports to ensure, e.g., the bank’s commitment to de-fund coal activities, and companies with social or environmental issues. With both an intensification of the aforementioned exterior pressure, and of the number of companies making such reports publicly available, the tedious process of going through each report has become unsustainable.
In this work, we present two adaptations of previously published models for joint entity and relation extraction. We train them on a private dataset consisting in ESG and CSR reports annotated internally at Crédit Agricole. We show that we are able to effectively detect entities such as coal activities and environmental or social issues, as well as relations between these entities, thus enabling the financial industry to quickly grasp the creditworthiness of clients and prospects w.r.t. ESG criteria. The resulting model is provided at https://github.com/adimajo/renard_joint.
- Named Entity Recognition
- Relation extraction
Supported by Groupe Crédit Agricole; analyses and opinions of the authors expressed in this work are their own. The authors wish to thank the ESG team at CACIB for the document annotations and their valuable comments.
This is a preview of subscription content, access via your institution.
Tax calculation will be finalised at checkout
Purchases are for personal use onlyLearn about institutional subscriptions
Some of these reports are becoming mandatory, e.g. in France as part of the “document d’enregistrement universel” required by the regulating authority, and audited.
The incorporation of ESG criteria alongside traditional financial metrics; see e.g. https://www.unepfi.org/banking/bankingprinciples/, https://www.ca-cib.com/our-solutions/sustainable-banking.
Available at https://corporate.arcelormittal.com/corporate-library.
Baldini Soares, L., FitzGerald, N., Ling, J., Kwiatkowski, T.: Matching the blanks: Distributional similarity for relation learning. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2895–2905. Association for Computational Linguistics, Florence, Italy, July 2019. https://doi.org/10.18653/v1/P19-1279, https://www.aclweb.org/anthology/P19-1279
Bekoulis, G., Deleu, J., Demeester, T., Develder, C.: Joint entity recognition and relation extraction as a multi-head selection problem. Expert Syst. Appl. 114, 34–45 (2018)
Devalle, A., Fiandrino, S., Cantino, V.: The linkage between ESG performance and credit ratings: a firm-level perspective analysis. Int. J. Bus. Manage. 12, 53 (2017). https://doi.org/10.5539/ijbm.v12n9p53
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Eberts, M., Ulges, A.: Span-based joint entity and relation extraction with transformer pre-training. In: 24th European Conference on Artificial Intelligence (2020)
de Guindos, L.: Shining a light on climate risks: the ECB’s economy-wide climate stress test (2021). https://www.ecb.europa.eu/press/blog/date/2021/html/ecb.blog210318~3bbc68ffc5.en.html
Han, X., Wang, L.: A novel document-level relation extraction method based on BERT and entity information. IEEE Access 8, 96912–96919 (2020)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Li, X., et al.: Entity-relation extraction as multi-turn question answering. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1340–1350. Association for Computational Linguistics, Florence, Italy, July 2019. https://doi.org/10.18653/v1/P19-1129, https://www.aclweb.org/anthology/P19-1129
Luan, Y., He, L., Ostendorf, M., Hajishirzi, H.: Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In: Proceedings of Conference on Empirical Methods Natural Language Processing (EMNLP) (2018)
Luan, Y., Wadden, D., He, L., Shah, A., Ostendorf, M., Hajishirzi, H.: A general framework for information extraction using dynamic span graphs. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3036–3046. Association for Computational Linguistics, Minneapolis, Minnesota, June 2019. https://doi.org/10.18653/v1/N19-1308, https://www.aclweb.org/anthology/N19-1308
Martins, P.H., Marinho, Z., Martins, A.F.T.: Joint learning of named entity recognition and entity linking. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pp. 190–196. Association for Computational Linguistics, Florence, Italy, July 2019. https://doi.org/10.18653/v1/P19-2026, https://www.aclweb.org/anthology/P19-2026
Nayak, T., Ng, H.T.: Effective modeling of encoder-decoder architecture for joint entity and relation extraction. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 8528–8535 (2020)
Poidatz, A.: Banques : des engagements climat à prendre au 4ème degré (2020). https://www.oxfamfrance.org/rapports/banques-des-engagements-climat-a-prendre-au-4eme-degre/
Straková, J., Straka, M., Hajic, J.: Neural architectures for nested NER through linearization. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5326–5331. Association for Computational Linguistics, Florence, Italy, July 2019. https://doi.org/10.18653/v1/P19-1527, https://www.aclweb.org/anthology/P19-1527
Takanobu, R., Zhang, T., Liu, J., Huang, M.: A hierarchical framework for relation extraction with reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 7072–7079 (2019)
Taylor, W.L.: “Cloze procedure”: a new tool for measuring readability. Journalism Q. 30(4), 415–433 (1953)
Vaswani, A., et al.: Attention is all you need. In: Guyon, I. et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
Verga, P., Strubell, E., McCallum, A.: Simultaneously self-attending to all mentions for full-abstract biological relation extraction. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 872–884. Association for Computational Linguistics, New Orleans, Louisiana, June 2018. https://doi.org/10.18653/v1/N18-1080, https://www.aclweb.org/anthology/N18-1080
Wadden, D., Wennberg, U., Luan, Y., Hajishirzi, H.: Entity, relation, and event extraction with contextualized span representations. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5784–5789. Association for Computational Linguistics, Hong Kong, China, November 2019. https://doi.org/10.18653/v1/D19-1585, https://www.aclweb.org/anthology/D19-1585
Wang, H., et al.: Extracting multiple-relations in one-pass with pre-trained transformers. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1371–1377. Association for Computational Linguistics, Florence, Italy, July 2019. https://doi.org/10.18653/v1/P19-1132, https://www.aclweb.org/anthology/P19-1132
Wang, J., Lu, W.: Two are better than one: joint entity and relation extraction with table-sequence encoders. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1706–1721. Association for Computational Linguistics, Online, November 2020. https://doi.org/10.18653/v1/2020.emnlp-main.133, https://www.aclweb.org/anthology/2020.emnlp-main.133
Editors and Affiliations
A Evolution of Test F1 for the IBM Model
B Named Entity Recognition Representation
A popular output representation of NER is BIO (Begin, In, Out) embedding, where each word is marked as the beginning, inside, or outside of an entity (see e.g. [19, 22]); however, this representation does not allow overlapping entities. On the other hand, span-based methods , which classify spans of words, can extract the spans of these overlapping entities. Figure 8 gives examples of BIO and span-based entity representations.
In the ClimLL dataset, even though the entities are presented in the span-based format in the dataset, there is no overlapping entity. Thus, it is also possible to convert to BIO format. Multiple relations can exist in the same sentence but relations cannot span across sentences. This facilitates splitting the paragraphs by sentence.
C Single versus Multiple Relation Extraction
Relation extraction algorithms are divided into two categories: Single Relation Extraction  (SRE) algorithms which expect only one relation per input sentence and multiple relation extraction [7, 21] (MRE) where multiple relations may exist in a single input sentence (Fig. 9).
In this work, multiple relations are considered.
1.1 D.1 Address a Shortcoming in Evaluation
While re-implementing the model, we noticed that, in the evaluation process, SpERT considers an incorrectly predicted entity span or relation as two negative observations. An example is presented in Fig. 10, where the model returns a set of predicted entities with “SpaceX” incorrectly classified as a person. In this case, the original evaluation process would iterate through the union of the true entity and the predicted entity sets. If an entity (including its span and type) is only presented in one of the sets, then it is considered to be classified as non-entity in the other. With this approach, “SpaceX” is considered to be incorrectly classified twice.
Thus, instead of iterating through the union of the true entity and predicted entity sets that include both entity spans and types, we only consider the union of the true entity spans with the predicted entity spans. Similarly for relations, we only take the union of the true and predicted spans of the source and target entity pair. As a result, we obtain a more accurate evaluation step.
1.2 D.2 Proposed Improvements
Furthermore, we also proposed two improvements to the prediction stages. Because SpERT classifies spans into entities, when dealing with datasets in BIO representation, it has to discard overlapping entities. In the original implementation, predicted entity spans are looped through in no specific order and any span that overlaps with previous spans is discarded. We suggest, instead, prioritizing discarding spans with low classification confidence.
Secondly, we noticed that the true pairs of entity types are not considered in the relation prediction stage of the original SpERT: For example, the model can only predict that an entity pair has a “live in” relation if the source entity is a person and the target entity is a location, irrespective of the probability given by the relation prediction stage. Thus, we modified the model so that it only predicts a relation if this relation fits the types of the source and target entities.
E Evolution of Loss Functions
The entity and relation losses as well as the F1 score on the validation set throughout the training process (30 epochs) of SpERT on ClimLL are displayed on Fig. 11. Both entity and relation losses reached their minimum after only a few epochs while the validation F1 score kept improving.
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Ehrhardt, A., Nguyen, M.T. (2021). Automated ESG Report Analysis by Joint Entity and Relation Extraction. In: Kamp, M., et al. Machine Learning and Principles and Practice of Knowledge Discovery in Databases. ECML PKDD 2021. Communications in Computer and Information Science, vol 1525. Springer, Cham. https://doi.org/10.1007/978-3-030-93733-1_23
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-93732-4
Online ISBN: 978-3-030-93733-1