Skip to main content
SpringerLink
Log in

UniCausal: Unified Benchmark and Repository for Causal Text Mining

  • Conference paper
  • First Online:
Big Data Analytics and Knowledge Discovery (DaWaK 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14148))

Included in the following conference series:

  • 492 Accesses

Abstract

Current causal text mining datasets vary in objectives, data coverage, and annotation schemes. These inconsistent efforts prevent modeling capabilities and fair comparisons of model performance. Furthermore, few datasets include cause-effect span annotations, which are needed for end-to-end causal relation extraction. To address these issues, we propose UniCausal, a unified benchmark for causal text mining across three tasks: (I) Causal Sequence Classification, (II) Cause-Effect Span Detection and (III) Causal Pair Classification. We consolidated and aligned annotations of six high quality, mainly human-annotated, corpora, resulting in a total of 58,720, 12,144 and 69,165 examples for each task respectively. Since the definition of causality can be subjective, our framework was designed to allow researchers to work on some or all datasets and tasks. To create an initial benchmark, we fine-tuned BERT pre-trained language models to each task, achieving 70.10% Binary F1, 52.42% Macro F1, and 84.68% Binary F1 scores respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Our repository is at https://github.com/tanfiona/UniCausal. Links to our trained baseline models are available in the repository.

  2. 2.

    https://github.com/chridey/altlex.

  3. 3.

    https://github.com/duncanka/BECAUSE/tree/2.0.

  4. 4.

    https://github.com/paramitamirza/Causal-TimeBank.

  5. 5.

    https://github.com/tommasoc80/EventStoryLine.

  6. 6.

    https://catalog.ldc.upenn.edu/LDC2019T05.

  7. 7.

    E.g.: Bootstrapped versions of AltLex [11] and SCITE [14]; Causal KBs: CauseNet [9], CausalNet [16] and CausalBank [15]; Semantic KBs that include causal relations: FrameNet [26] and ConceptNet [27].

  8. 8.

    (\(\texttt {<ARG0>}\), \(\texttt {</ARG0>}\)) marks the boundaries of a Cause span, while (\(\texttt {<ARG1>}\), \(\texttt {</ARG1>}\)) marks the boundaries of a corresponding Effect span.

References

  1. Asghar, N.: Automatic extraction of causal relations from natural language texts: a comprehensive survey. CoRR abs/1605.07895 (2016). http://arxiv.org/abs/1605.07895

  2. Ayyanar, R., Koomullil, G., Ramasangu, H.: Causal relation classification using convolutional neural networks and grammar tags. In: 2019 IEEE 16th India Council International Conference (INDICON), pp. 1–3 (2019). https://doi.org/10.1109/INDICON47234.2019.9028985

  3. Cao, P., et al.: Knowledge-enriched event causality identification via latent structure induction networks. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4862–4872. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.acl-long.376. https://aclanthology.org/2021.acl-long.376

  4. Caselli, T., Vossen, P.: The event StoryLine corpus: a new benchmark for causal and temporal relation extraction. In: Proceedings of the Events and Stories in the News Workshop, Vancouver, Canada, pp. 77–86. Association for Computational Linguistics (2017). https://doi.org/10.18653/v1/W17-2711. https://aclanthology.org/W17-2711

  5. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171–4186. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/N19-1423. https://aclanthology.org/N19-1423

  6. Dunietz, J., Burnham, G., Bharadwaj, A., Rambow, O., Chu-Carroll, J., Ferrucci, D.: To test machine comprehension, start by defining comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7839–7859. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.701. https://aclanthology.org/2020.acl-main.701

  7. Dunietz, J., Levin, L., Carbonell, J.: The BECauSE corpus 2.0: annotating causality and overlapping relations. In: Proceedings of the 11th Linguistic Annotation Workshop, Valencia, Spain, pp. 95–104. Association for Computational Linguistics (2017). https://doi.org/10.18653/v1/W17-0812. https://aclanthology.org/W17-0812

  8. Girju, R.: Automatic detection of causal relations for question answering. In: Proceedings of the ACL 2003 Workshop on Multilingual Summarization and Question Answering, Sapporo, Japan, pp. 76–83. Association for Computational Linguistics (2003). https://doi.org/10.3115/1119312.1119322. https://aclanthology.org/W03-1210

  9. Heindorf, S., Scholten, Y., Wachsmuth, H., Ngomo, A.N., Potthast, M.: Causenet: towards a causality graph extracted from the web. In: d’Aquin, M., Dietze, S., Hauff, C., Curry, E., Cudré-Mauroux, P. (eds.) CIKM 2020: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, 19–23 October 2020, pp. 3023–3030. ACM (2020). https://doi.org/10.1145/3340531.3412763

  10. Hendrickx, I., et al.: SemEval-2010 task 8: multi-way classification of semantic relations between pairs of nominals. In: Proceedings of the 5th International Workshop on Semantic Evaluation, Uppsala, Sweden, pp. 33–38. Association for Computational Linguistics (2010). https://aclanthology.org/S10-1006

  11. Hidey, C., McKeown, K.: Identifying causal relations using parallel Wikipedia articles. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, pp. 1424–1433. Association for Computational Linguistics (2016). https://doi.org/10.18653/v1/P16-1135. https://aclanthology.org/P16-1135

  12. Ittoo, A., Bouma, G.: Minimally-supervised learning of domain-specific causal relations using an open-domain corpus as knowledge base. Data Knowl. Eng. 88, 142–163 (2013). https://doi.org/10.1016/j.datak.2013.08.004

  13. Kyriakakis, M., Androutsopoulos, I., Saudabayev, A., Ginés i Ametllé, J.: Transfer learning for causal sentence detection. In: Proceedings of the 18th BioNLP Workshop and Shared Task, Florence, Italy, pp. 292–297. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/W19-5031. https://aclanthology.org/W19-5031

  14. Li, Z., Li, Q., Zou, X., Ren, J.: Causality extraction based on self-attentive BiLSTM-CRF with transferred embeddings. Neurocomputing 423, 207–219 (2021). https://doi.org/10.1016/j.neucom.2020.08.078

  15. Li, Z., Ding, X., Liu, T., Hu, J.E., Durme, B.V.: Guided generation of cause and effect. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3629–3636. ijcai.org (2020). https://doi.org/10.24963/ijcai.2020/502

  16. Luo, Z., Sha, Y., Zhu, K.Q., Hwang, S., Wang, Z.: Commonsense causal reasoning between short texts. In: Baral, C., Delgrande, J.P., Wolter, F. (eds.) Principles of Knowledge Representation and Reasoning: Proceedings of the Fifteenth International Conference, KR 2016, Cape Town, South Africa, 25–29 April 2016, pp. 421–431. AAAI Press (2016). http://www.aaai.org/ocs/index.php/KR/KR16/paper/view/12818

  17. Mariko, D., Abi-Akl, H., Labidurie, E., Durfort, S., De Mazancourt, H., El-Haj, M.: The financial document causality detection shared task (FinCausal 2020). In: Proceedings of the 1st Joint Workshop on Financial Narrative Processing and MultiLing Financial Summarisation, Barcelona, Spain, pp. 23–32. COLING (Online) (2020). https://aclanthology.org/2020.fnp-1.3

  18. Mariko, D., Akl, H.A., Labidurie, E., Durfort, S., de Mazancourt, H., El-Haj, M.: The financial document causality detection shared task (FinCausal 2021). In: Proceedings of the 3rd Financial Narrative Processing Workshop, Lancaster, United Kingdom, pp. 58–60. Association for Computational Linguistics (2021). https://aclanthology.org/2021.fnp-1.10

  19. Mirza, P., Sprugnoli, R., Tonelli, S., Speranza, M.: Annotating causality in the TempEval-3 corpus. In: Proceedings of the EACL 2014 Workshop on Computational Approaches to Causality in Language (CAtoCL), Gothenburg, Sweden, pp. 10–19. Association for Computational Linguistics (2014). https://doi.org/10.3115/v1/W14-0702. https://aclanthology.org/W14-0702

  20. Mirza, P., Tonelli, S.: An analysis of causality between events and its relation to temporal information. In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Ireland, pp. 2097–2106. Dublin City University and Association for Computational Linguistics (2014). https://aclanthology.org/C14-1198

  21. Nakayama, H.: seqeval: a python framework for sequence labeling evaluation (2018). https://github.com/chakki-works/seqeval

  22. Niki, Y., Sakaji, H., Izumi, K., Matsushima, H.: Causality existence classification from multilingual texts using end-to-end LSTM models. In: Papapetrou, P., Cheng, X., He, Q. (eds.) 2019 International Conference on Data Mining Workshops, ICDM Workshops 2019, Beijing, China, 8–11 November 2019, pp. 17–23. IEEE (2019). https://doi.org/10.1109/ICDMW.2019.00011

  23. Ponti, E.M., Korhonen, A.: Event-related features in feedforward neural networks contribute to identifying causal relations in discourse. In: Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-Level Semantics, Valencia, Spain, pp. 25–30. Association for Computational Linguistics (2017). https://doi.org/10.18653/v1/W17-0903. https://aclanthology.org/W17-0903

  24. Radinsky, K., Horvitz, E.: Mining the web to predict future events. In: Leonardi, S., Panconesi, A., Ferragina, P., Gionis, A. (eds.) Sixth ACM International Conference on Web Search and Data Mining, WSDM 2013, Rome, Italy, 4–8 February 2013, pp. 255–264. ACM (2013). https://doi.org/10.1145/2433396.2433431

  25. Ramshaw, L., Marcus, M.: Text chunking using transformation-based learning. In: Third Workshop on Very Large Corpora (1995). https://aclanthology.org/W95-0107

  26. Ruppenhofer, J., Ellsworth, M., Schwarzer-Petruck, M., Johnson, C.R., Scheffczyk, J.: Framenet II: extended theory and practice. Technical report, International Computer Science Institute (2016)

    Google Scholar 

  27. Speer, R., Chin, J., Havasi, C.: Conceptnet 5.5: an open multilingual graph of general knowledge. In: Singh, S.P., Markovitch, S. (eds.) Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 4–9 February 2017, San Francisco, California, USA, pp. 4444–4451. AAAI Press (2017). http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14972

  28. Stasaski, K., Rathod, M., Tu, T., Xiao, Y., Hearst, M.A.: Automatically generating cause-and-effect questions from passages. In: Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 158–170. Association for Computational Linguistics, Online (2021). https://aclanthology.org/2021.bea-1.17

  29. Tan, F.A., et al.: Event causality identification with causal news corpus - shared task 3, CASE 2022. In: Proceedings of the 5th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE), Abu Dhabi, United Arab Emirates (Hybrid), pp. 195–208. Association for Computational Linguistics (2022). https://aclanthology.org/2022.case-1.28

  30. Tan, F.A., et al.: The causal news corpus: annotating causal relations in event sentences from news. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference, Marseille, France, pp. 2298–2310. European Language Resources Association (2022). https://aclanthology.org/2022.lrec-1.246

  31. Tan, F.A., et al.: The causal news corpus: annotating causal relations in event sentences from news. In: Proceedings of the Language Resources and Evaluation Conference, Marseille, France, pp. 2298–2310. European Language Resources Association (2022). https://aclanthology.org/2022.lrec-1.246

  32. Webber, B., Prasad, R., Lee, A., Joshi, A.: The penn discourse treebank 3.0 annotation manual. University of Pennsylvania, Philadelphia (2019)

    Google Scholar 

  33. Yang, J., Han, S.C., Poon, J.: A survey on extraction of causal relations from natural language text. Knowl. Inf. Syst. (2022). https://doi.org/10.1007/s10115-022-01665-w

  34. Zuo, X., et al.: Improving event causality identification via self-supervised representation learning on external causal statement. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2162–2172. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.findings-acl.190. https://aclanthology.org/2021.findings-acl.190

  35. Zuo, X., et al.: Improving event causality identification via self-supervised representation learning on external causal statement. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2162–2172. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.findings-acl.190. https://aclanthology.org/2021.findings-acl.190

  36. Zuo, X., Chen, Y., Liu, K., Zhao, J.: KnowDis: knowledge enhanced data augmentation for event causality detection via distant supervision. In: Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, pp. 1544–1550. International Committee on Computational Linguistics (Online) (2020). https://doi.org/10.18653/v1/2020.coling-main.135. https://aclanthology.org/2020.coling-main.135

Download references

Acknowledgements

This research/project is supported by the National Research Foundation, Singapore under its Industry Alignment Fund - Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. Additionally, we thank Jiatong Han for helping with the creation of tutorials.

Author information

Authors and Affiliations

  1. Institute of Data Science, National University of Singapore, Singapore, Singapore

    Fiona Anting Tan & See-Kiong Ng

  2. Tencent Technology, Beijing, China

    Xinyu Zuo

Authors
  1. Fiona Anting Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xinyu Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. See-Kiong Ng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fiona Anting Tan .

Editor information

Editors and Affiliations

  1. Poznań University of Technology, Poznan, Poland

    Robert Wrembel

  2. Free University of Bozen-Bolzano, Bozen-Bolzano, Italy

    Johann Gamper

  3. Johannes Kepler University Linz, Linz, Austria

    Gabriele Kotsis

  4. Vienna University of Technology, Vienna, Austria

    A Min Tjoa

  5. Johannes Kepler University Linz, Linz, Austria

    Ismail Khalil

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tan, F.A., Zuo, X., Ng, SK. (2023). UniCausal: Unified Benchmark and Repository for Causal Text Mining. In: Wrembel, R., Gamper, J., Kotsis, G., Tjoa, A.M., Khalil, I. (eds) Big Data Analytics and Knowledge Discovery. DaWaK 2023. Lecture Notes in Computer Science, vol 14148. Springer, Cham. https://doi.org/10.1007/978-3-031-39831-5_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-39831-5_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-39830-8

  • Online ISBN: 978-3-031-39831-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation