Abstract
Annotating a corpus with argument structures is a complex task, and it is even more challenging when addressing text genres where argumentative discourse markers do not abound. We explore a corpus of opinion articles annotated by multiple annotators, providing diverse perspectives of the argumentative content therein. New annotation aggregation methods are explored, diverging from the traditional ones that try to minimize presumed errors from annotator disagreement. The impact of our methods is assessed for the task of argument density prediction, seen as an initial step in the argument mining pipeline. We evaluate and compare models trained for this regression task in different generated datasets, considering their prediction error and also from a ranking perspective. Results confirm the expectation that addressing argument density from a ranking perspective is more promising than looking at the problem as a mere regression task. We also show that probabilistic aggregation, which weighs tokens by considering all annotators, is a more interesting approach, achieving encouraging results as it accommodates different annotator perspectives. The code and models are publicly available at https://github.com/DARGMINTS/argument-density.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
This does not hold for the z-test “All” vs. \(\langle A, C, D \rangle \) (probabilistic) where the p-value is 0.059, although it stays very close to a statistical significance.
- 3.
References
Al-Khatib, K., Wachsmuth, H., Kiesel, J., Hagen, M., Stein, B.: A news editorial corpus for mining argumentation strategies. In: Proceedings of COLING 2016, Osaka, Japan, pp. 3433–3443 (2016)
Bal, B.K., Saint Dizier, P.: Towards building annotated resources for analyzing opinions and argumentation in news editorials. In: Proceedings 7th Language Resources and Evaluation Conference, ELRA, Valletta, Malta (2010)
Basile, V.: It’s the end of the gold standard as we know it. In: Baldoni, M., Bandini, S. (eds.) AIxIA 2020. LNCS (LNAI), vol. 12414, pp. 441–453. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77091-4_26
Basile, V., et al.: We need to consider disagreement in evaluation. In: Proceedings 1st Workshop on Benchmarking: Past, Present and Future, pp. 15–21. ACL, Online (2021)
Cunningham, H., et al.: Developing Language Processing Components with GATE Version 8 (a User Guide), University of Sheffield Department of Computer Science (2014)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings 2019 NAACL, pp. 4171–4186. ACL, Minneapolis, Minnesota (2019)
Dumitrache, A., Inel, O., Aroyo, L., Timmermans, B., Welty, C.: Crowdtruth 2.0: Quality metrics for crowdsourcing with disagreement. arXiv preprint arXiv:1808.06080 (2018)
Eger, S., Daxenberger, J., Gurevych, I.: Neural end-to-end learning for computational argumentation mining. In: Proceedings 55th Annual Meeting of the ACL, pp. 11–22. ACL, Vancouver, Canada (2017)
Goudas, T., Louizos, C., Petasis, G., Karkaletsis, V.: Argument extraction from news, blogs, and social media. In: Likas, A., Blekas, K., Kalles, D. (eds.) Artificial Intelligence: Methods and Applications, pp. 287–299. Springer International Publishing, Cham (2014)
Krahmer, E., Theune, M. (eds.): EACL/ENLG -2009. LNCS (LNAI), vol. 5790. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15573-4
Krippendorff, K.: Measuring the reliability of qualitative text analysis data. Qual. Quant. 38, 787–800 (2004)
Lawrence, J., Reed, C.: Argument mining: a survey. Comput. Linguist. 45(4), 765–818 (2019)
Levy, R., Bilu, Y., Hershcovich, D., Aharoni, E., Slonim, N.: Context dependent claim detection. In: Proceedings COLING 2014, pp. 1489–1500. ACL, Dublin, Ireland (2014)
Lippi, M., Torroni, P.: Argumentation mining: state of the art and emerging trends. ACM Trans. Int. Technol. 16(2), 1–25 (2016)
Meyer, C.M., Mieskes, M., Stab, C., Gurevych, I.: DKPro agreement: An open-source Java library for measuring inter-rater agreement. In: Proceedings COLING 2014, pp. 105–109. ACL, Dublin, Ireland (2014)
Moens, M.F.: Argumentation mining: how can a machine acquire common sense and world knowledge? Argument Comput. 9(1), 1–14 (2017)
Ovesdotter Alm, C.: Subjective natural language problems: motivations, applications, characterizations, and implications. In: Proceedings 49th Annual Meeting of the ACL, pp. 107–112. ACL, Portland, Oregon, USA (2011)
Palau, R.M., Moens, M.F.: Argumentation mining: the detection, classification and structure of arguments in text. In: Proceedings 12th International Conference on Artificial Intelligence and Law, p. 98–107. ACM, New York, NY, USA (2009)
Park, J., Cardie, C.: Identifying appropriate support for propositions in online user comments. In: Proceedings 1st Workshop on Argumentation Mining, pp. 29–38. ACL (2014)
Pavlick, E., Kwiatkowski, T.: Inherent disagreements in human textual inferences. Trans. Assoc. Comput. Linguist. 7, 677–694 (2019)
Peldszus, A., Stede, M.: Joint prediction in MST-style discourse parsing for argumentation mining. In: Proceedings EMNLP, pp. 938–948. ACL, Lisbon, Portugal (2015)
Plank, B., Hovy, D., Søgaard, A.: Linguistically debatable or just plain wrong? In: Proceedings 52nd Annual Meeting of the ACL. pp. 507–511. ACL, Baltimore, Maryland (2014)
Rinott, R., Dankin, L., Alzate Perez, C., Khapra, M.M., Aharoni, E., Slonim, N.: Show me your evidence - an automatic method for context dependent evidence detection. In: Proceedings 2015 EMNLP, pp. 440–450. ACL, Lisbon, Portugal (2015)
Rocha, G., et al.: Annotating arguments in a corpus of opinion articles. In: Proceedings 13th Language Resources and Evaluation Conference, ELRA (2022)
Saint-Dizier, P.: Challenges of argument mining: Generating an argument synthesis based on the qualia structure. In: Proceedings 9th International Natural Language Generation Conference, pp. 79–83. ACL, Edinburgh, UK (2016)
Schaefer, R., Stede, M.: Argument mining on twitter: a survey. IT-Inf. Technol. 63(1), 45–58 (2021)
Simpson, E.D., Gurevych, I.: A Bayesian approach for sequence tagging with crowds. In: Proceedings 2019 EMNLP and 9th IJCNLP, pp. 1093–1104. ACL, Hong Kong, China (2019)
Stab, C., Gurevych, I.: Identifying argumentative discourse structures in persuasive essays. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 46–56. Association for Computational Linguistics, Doha, Qatar (2014)
Stab, C., Miller, T., Schiller, B., Rai, P., Gurevych, I.: Cross-topic argument mining from heterogeneous sources. In: Proceedings 2018 EMNLP, pp. 3664–3674. ACL (2018)
Teufel, S.: Argumentative Zoning: Information Extraction from Scientific Text. Ph.D. thesis, University of Edinburgh (1999)
Teufel, S., Siddharthan, A., Batchelor, C.: Towards discipline-independent argumentative zoning: evidence from chemistry and computational linguistics. In: Proceedings 2009 EMNLP, pp. 1493–1502. ACL, USA (2009)
Trautmann, D., Daxenberger, J., Stab, C., Schütze, H., Gurevych, I.: Fine-grained argument unit recognition and classification. In: Proceedings of the AAAI Conference on AI, vol. 34, no. 05, pp. 9048–9056 (2020)
Visser, J., Konat, B., Duthie, R., Koszowy, M., Budzynska, K., Reed, C.: Argumentation in the 2016 US presidential elections: annotated corpora of television debates and social media reaction. Lang. Res. Eval. 54(1), 123–154 (2019). https://doi.org/10.1007/s10579-019-09446-8
Acknowledgements
This research is supported by project DARGMINTS (POCI/01/0145/FEDER/031460), LIACC (FCT/UID/ CEC/0027/2020), INESC-ID (UIDB/50021/2020) and CLUP (UIDB/00022/2020), funded by Fundação para a Ciência e a Tecnologia (FCT). Gil Rocha is supported by a PhD studentship (SFRH/BD/140125/2018) from FCT. Bernardo Leite is supported by a PhD studentship (2021.05432.BD) from FCT.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Rocha, G. et al. (2022). Predicting Argument Density from Multiple Annotations. In: Rosso, P., Basile, V., Martínez, R., Métais, E., Meziane, F. (eds) Natural Language Processing and Information Systems. NLDB 2022. Lecture Notes in Computer Science, vol 13286. Springer, Cham. https://doi.org/10.1007/978-3-031-08473-7_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-08473-7_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-08472-0
Online ISBN: 978-3-031-08473-7
eBook Packages: Computer ScienceComputer Science (R0)