An Annotation Protocol for Collecting User-Generated Counter-Arguments Using Crowdsourcing

  • Paul ReisertEmail author
  • Gisela Vallejo
  • Naoya Inoue
  • Iryna Gurevych
  • Kentaro Inui
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11626)


Constructive feedback is important for improving critical thinking skills. However, little work has been done to automatically generate such feedback for an argument. In this work, we experiment with an annotation protocol for collecting user-generated counter-arguments via crowdsourcing. We conduct two parallel crowdsourcing experiments, where workers are instructed to produce (i) a counter-argument, and (ii) a counter-argument after identifying a fallacy. Our analysis indicates that we can collect counter-arguments that are useful as constructive feedback, especially when workers are first asked to identify a fallacy type.


Critical thinking Counter-argument Fallacy Crowdsourcing Annotation study Constructive feedback 


  1. 1.
    El Khoiri, N., Widiati, U.: Logical fallacies in Indonesian EFL learners’ argumentative writing: students’ perspectives. Dinamika Ilmu 17(1), 71–81 (2017)CrossRefGoogle Scholar
  2. 2.
    Ghosh, D., Khanam, A., Han, Y., Muresan, S.: Coarse-grained argumentation features for scoring persuasive essays. In: Proceedings of the 54th Annual Meeting of ACL (Volume 2: Short Papers), pp. 549–554 (2016)Google Scholar
  3. 3.
    Habernal, I., Pauli, P., Gurevych, I.: Adapting serious game for fallacious argumentation to German: pitfalls, insights, and best practices. In: Proceedings of the Eleventh International Conference on LREC, pp. 3329–3335 (2018)Google Scholar
  4. 4.
    Habernal, I., Wachsmuth, H., Gurevych, I., Stein, B.: The argument reasoning comprehension task: identification and reconstruction of implicit warrants. In: Proceedings of the 2018 Conference of NAACL: HLT, Volume 1 (Long Papers), pp. 1930–1940. Association for Computational Linguistics (2018)Google Scholar
  5. 5.
    Hua, X., Wang, L.: Neural argument generation augmented with externally retrieved evidence. In: Proceedings of the 56th Annual Meeting of ACL (Volume 1: Long Papers), pp. 219–230 (2018)Google Scholar
  6. 6.
    Indah, R.N., Kusuma, A.W.: Fallacies in English department students’ claims: a rhetorical analysis of critical thinking. Jurnal Pendidikan Humaniora 3(4), 295–304 (2015)Google Scholar
  7. 7.
    Lucas, C., Gibson, A., Buckingham Shum, S.: Utilization of a novel online reflective learning tool for immediate formative feedback to assist pharmacy students’ reflective writing skills. Am. J. Pharm. Educ. (2018)Google Scholar
  8. 8.
    Nguyen, H.V., Litman, D.J.: Argument mining for improving the automated scoring of persuasive essays. In: The Thirty-Second AAAI Conference on Artificial Intelligence, pp. 5892–5899 (2018)Google Scholar
  9. 9.
    Oktavia, W., Yasin, A., et al.: An analysis of students’ argumentative elements and fallacies in students’ discussion essays. Engl. Lang. Teach. 2(3) (2014)Google Scholar
  10. 10.
    Persing, I., Davis, A., Ng, V.: Modeling organization in student essays. In: Proceedings of the 2010 Conference on EMNLP, pp. 229–239. Association for Computational Linguistics (2010)Google Scholar
  11. 11.
    Persing, I., Ng, V.: Modeling thesis clarity in student essays. In: Proceedings of the 51st Annual Meeting of ACL (Volume 1: Long Papers), vol. 1, pp. 260–269 (2013)Google Scholar
  12. 12.
    Persing, I., Ng, V.: Modeling stance in student essays. In: Proceedings of the 54th Annual Meeting of ACL (Volume 1: Long Papers), vol. 1, pp. 2174–2184 (2016)Google Scholar
  13. 13.
    Wachsmuth, H., Al-Khatib, K., Stein, B.: Using argument mining to assess the argumentation quality of essays. In: Proceedings of the 26th International Conference on COLING, pp. 1680–1692 (2016)Google Scholar
  14. 14.
    Wachsmuth, H., Syed, S., Stein, B.: Retrieval of the best counterargument without prior topic knowledge. In: Proceedings of the 56th Annual Meeting of ACL (Volume 1: Long Papers), vol. 1, pp. 241–251 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Paul Reisert
    • 1
    • 3
    Email author
  • Gisela Vallejo
    • 2
  • Naoya Inoue
    • 1
    • 3
  • Iryna Gurevych
    • 2
  • Kentaro Inui
    • 1
    • 3
  1. 1.RIKEN Center for Advanced Intelligence Project (AIP)TokyoJapan
  2. 2.Ubiquitous Knowledge Processing Lab (UKP), Department of Computer ScienceTechnische Universität DarmstadtDarmstadtGermany
  3. 3.Tohoku UniversitySendaiJapan

Personalised recommendations