Abstract
Constructive feedback is important for improving critical thinking skills. However, little work has been done to automatically generate such feedback for an argument. In this work, we experiment with an annotation protocol for collecting user-generated counter-arguments via crowdsourcing. We conduct two parallel crowdsourcing experiments, where workers are instructed to produce (i) a counter-argument, and (ii) a counter-argument after identifying a fallacy. Our analysis indicates that we can collect counter-arguments that are useful as constructive feedback, especially when workers are first asked to identify a fallacy type.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
We calculate the Cohen’s kappa after filtering out unsure instances.
References
El Khoiri, N., Widiati, U.: Logical fallacies in Indonesian EFL learners’ argumentative writing: students’ perspectives. Dinamika Ilmu 17(1), 71–81 (2017)
Ghosh, D., Khanam, A., Han, Y., Muresan, S.: Coarse-grained argumentation features for scoring persuasive essays. In: Proceedings of the 54th Annual Meeting of ACL (Volume 2: Short Papers), pp. 549–554 (2016)
Habernal, I., Pauli, P., Gurevych, I.: Adapting serious game for fallacious argumentation to German: pitfalls, insights, and best practices. In: Proceedings of the Eleventh International Conference on LREC, pp. 3329–3335 (2018)
Habernal, I., Wachsmuth, H., Gurevych, I., Stein, B.: The argument reasoning comprehension task: identification and reconstruction of implicit warrants. In: Proceedings of the 2018 Conference of NAACL: HLT, Volume 1 (Long Papers), pp. 1930–1940. Association for Computational Linguistics (2018)
Hua, X., Wang, L.: Neural argument generation augmented with externally retrieved evidence. In: Proceedings of the 56th Annual Meeting of ACL (Volume 1: Long Papers), pp. 219–230 (2018)
Indah, R.N., Kusuma, A.W.: Fallacies in English department students’ claims: a rhetorical analysis of critical thinking. Jurnal Pendidikan Humaniora 3(4), 295–304 (2015)
Lucas, C., Gibson, A., Buckingham Shum, S.: Utilization of a novel online reflective learning tool for immediate formative feedback to assist pharmacy students’ reflective writing skills. Am. J. Pharm. Educ. (2018)
Nguyen, H.V., Litman, D.J.: Argument mining for improving the automated scoring of persuasive essays. In: The Thirty-Second AAAI Conference on Artificial Intelligence, pp. 5892–5899 (2018)
Oktavia, W., Yasin, A., et al.: An analysis of students’ argumentative elements and fallacies in students’ discussion essays. Engl. Lang. Teach. 2(3) (2014)
Persing, I., Davis, A., Ng, V.: Modeling organization in student essays. In: Proceedings of the 2010 Conference on EMNLP, pp. 229–239. Association for Computational Linguistics (2010)
Persing, I., Ng, V.: Modeling thesis clarity in student essays. In: Proceedings of the 51st Annual Meeting of ACL (Volume 1: Long Papers), vol. 1, pp. 260–269 (2013)
Persing, I., Ng, V.: Modeling stance in student essays. In: Proceedings of the 54th Annual Meeting of ACL (Volume 1: Long Papers), vol. 1, pp. 2174–2184 (2016)
Wachsmuth, H., Al-Khatib, K., Stein, B.: Using argument mining to assess the argumentation quality of essays. In: Proceedings of the 26th International Conference on COLING, pp. 1680–1692 (2016)
Wachsmuth, H., Syed, S., Stein, B.: Retrieval of the best counterargument without prior topic knowledge. In: Proceedings of the 56th Annual Meeting of ACL (Volume 1: Long Papers), vol. 1, pp. 241–251 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Reisert, P., Vallejo, G., Inoue, N., Gurevych, I., Inui, K. (2019). An Annotation Protocol for Collecting User-Generated Counter-Arguments Using Crowdsourcing. In: Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., Luckin, R. (eds) Artificial Intelligence in Education. AIED 2019. Lecture Notes in Computer Science(), vol 11626. Springer, Cham. https://doi.org/10.1007/978-3-030-23207-8_43
Download citation
DOI: https://doi.org/10.1007/978-3-030-23207-8_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-23206-1
Online ISBN: 978-3-030-23207-8
eBook Packages: Computer ScienceComputer Science (R0)