Skip to main content
SpringerLink
Log in

Assessing Good, Bad and Ugly Arguments Generated by ChatGPT: a New Dataset, its Methodology and Associated Tasks

  • Conference paper
  • First Online:
Progress in Artificial Intelligence (EPIA 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14115))

Included in the following conference series:

  • 511 Accesses

Abstract

The recent success of Large Language Models (LLMs) has sparked concerns about their potential to spread misinformation. As a result, there is a pressing need for tools to identify “fake arguments” generated by such models. To create these tools, examples of texts generated by LLMs are needed. This paper introduces a methodology to obtain good, bad and ugly arguments from argumentative essays produced by ChatGPT, OpenAI’s LLM. We then describe a novel dataset containing a set of diverse arguments, ArGPT. We assess the effectiveness of our dataset and establish baselines for several argumentation-related tasks. Finally, we show that the artificially generated data relates well to human argumentation and thus is useful as a tool to train and test systems for the defined tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://openai.com/blog/chatgpt.

  2. 2.

    All code, data, and experiments for this paper are available at: https://github.com/C4AI/ArGPT.

  3. 3.

    https://research.ibm.com/interactive/project-debater/.

  4. 4.

    https://www.ets.org/toefl/test-takers/ibt/about/content/writing.html.

  5. 5.

    https://www.kaggle.com/c/asap-aes.

  6. 6.

    For the sake of space, we discuss only the differences of our methodology in respect to theirs.

  7. 7.

    ChatGPT has emerged as a valuable annotation tool, often outperforming manual annotations. (e.g., [6, 19]). Nonetheless, despite our best efforts, we could not teach ChatGPT to generate annotations adhering to our methodology. This limitation is reasonable, considering that even human annotators require training to perform such tasks effectively.

  8. 8.

    We excluded “Adherence to the theme” from this account.

References

  1. Accuosto, P., Saggion, H.: Mining arguments in scientific abstracts with discourse-level embeddings. Data Knowl. Eng. 129, 101840 (2020)

    Article  Google Scholar 

  2. Anand, Y., Nussbaum, Z., Duderstadt, B., Schmidt, B., Mulyar, A.: Gpt4all: training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo. github.com/nomic-ai/gpt4all (2023)

    Google Scholar 

  3. Blanchard, D., Tetreault, J., Higgins, D., Cahill, A., Chodorow, M.: Toefl11: A corpus of non-native english. ETS Research Report Series 2013, i–15 (2013)

    Article  Google Scholar 

  4. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y.T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M.T., Zhang, Y.: Sparks of Artificial General Intelligence: Early Experiments with gpt-4 (2023)

    Google Scholar 

  5. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: NAACL HLT 2019–2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies—Proceedings of the Conference, vol. 1, pp. 4171–4186. Association for Computational Linguistics (ACL) (2019)

    Google Scholar 

  6. Gilardi, F., Alizadeh, M., Kubli, M.: Chatgpt outperforms crowd-workers for text-annotation tasks (2023)

    Google Scholar 

  7. Hidayaturrahman, Dave, E., Suhartono, D., Arymurthy, A.M.: Enhancing argumentation component classification using contextual language model. J. Big Data 8(1), 103 (2021)

    Google Scholar 

  8. Kashefi, O., Afrin, T., Dale, M., Olshefski, C., Godley, A., Litman, D., Hwa, R.: ArgRewrite vol. 2: an annotated argumentative revisions corpus. Lang. Res. Eval. 56(3), 881–915 (2022)

    Google Scholar 

  9. Lagakis, P., Demetriadis, S.: Automated essay scoring: a review of the field. In: 2021 International Conference on Computer, Information and Telecommunication Systems (CITS), pp. 1–6 (2021)

    Google Scholar 

  10. Lawrence, J., Reed, C.: Argument mining: a survey. Comput. Linguist. 45(4), 765–818 (2020)

    Google Scholar 

  11. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A Robustly Optimized BERT Pretraining Approach (2019). arxiv:abs/1907.11692

  12. Mathias, S., Bhattacharyya, P.: ASAP++: enriching the ASAP automated essay grading dataset with essay attribute scores. In: International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan (2018)

    Google Scholar 

  13. Mayer, T., Cabrio, E., Villata, S.: Transformer-based argument mining for healthcare applications. In: Frontiers in Artificial Intelligence and Applications. vol. 325, pp. 2108–2115. IOS Press BV (2020)

    Google Scholar 

  14. Morio, G., Ozaki, H., Morishita, T., Yanai, K.: End-to-end Argument Mining with Cross-corpora Multi-task Learning. Trans. Assoc. Comput. Linguist. 10, 639–658 (2022). https://doi.org/10.1162/tacl_a_00481

  15. Park, J., Cardie, C.: A corpus of eRulemaking user comments for measuring evaluability of arguments. In: International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan, May 2018

    Google Scholar 

  16. Peldszus, A., Stede, M.: An annotated corpus of argumentative microtexts. European Conference on Argumentation (ECA’16), pp. 801–816 (2016)

    Google Scholar 

  17. Stab, C., Gurevych, I.: Parsing argumentation structures in persuasive essays. Comput. Linguist. 43(3), 619–659 (2017)

    Google Scholar 

  18. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., Hashimoto, T.B.: Stanford Alpaca: An Instruction-Following LLaMA Model (2023)

    Google Scholar 

  19. Törnberg, P.: Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning (2023)

    Google Scholar 

  20. Veselovsky, V., Ribeiro, M.H., West, R.: Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks (2023)

    Google Scholar 

  21. Yannakoudakis, H., Cummins, R.: Evaluating the performance of automated text scoring systems. In: Workshop on Innovative Use of NLP for Building Educational Applications, pp. 213–223. Association for Computational Linguistics, Denver, Colorado, Jun 2015

    Google Scholar 

Download references

Acknowledgements

This work was carried out at the Center for Artificial Intelligence (C4AI-USP), with support by FAPESP grant 2019/07665-4 and by the IBM Corporation. Victor Hugo is partially supported by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) grant 88887.616392/2021-00. Paulo is supported by the FAPESP grant 2019/26762-0. Denis is partially supported by grants FAPESP #2022/02937-9 and CNPq #305136/2022-4. Fabio is partially supported by CNPq #305753/2022-3. Igor is partially supported by CAPES grant 88887.635492/2021-00. We acknowledge support by CAPES - Finance Code 001.

Author information

Authors and Affiliations

  1. Escola Politécnica, São Paulo, Brazil

    Victor Hugo Nascimento Rocha & Fabio Gagliardi Cozman

  2. Instituto de Matemática e Estatística, São Paulo, Brazil

    Igor Cataneo Silveira & Denis Deratani Mauá

  3. Instituto de Estudos Avançados Universidade de São Paulo, C4AI (c4ai.inova.usp.br) Av. Prof. Lúcio Martins Rodrigues, 370, São Paulo, SP, 05508-020, Brazil

    Paulo Pirozelli

Authors
  1. Victor Hugo Nascimento Rocha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Igor Cataneo Silveira
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paulo Pirozelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Denis Deratani Mauá
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fabio Gagliardi Cozman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Victor Hugo Nascimento Rocha .

Editor information

Editors and Affiliations

  1. Lucy Family Institute for Data & Society, Notre Dame, IN, USA

    Nuno Moniz

  2. GECAD, Polytechnic of Porto, Porto, Portugal

    Zita Vale

  3. GRIA - LIACC, University of Azores, Ponta-Delgada, Portugal

    José Cascalho

  4. CISUC, University of Coimbra, Coimbra, Portugal

    Catarina Silva

  5. IEETA, University of Aveiro, Aveiro, Portugal

    Raquel Sebastião

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rocha, V.H.N., Silveira, I.C., Pirozelli, P., Mauá, D.D., Cozman, F.G. (2023). Assessing Good, Bad and Ugly Arguments Generated by ChatGPT: a New Dataset, its Methodology and Associated Tasks. In: Moniz, N., Vale, Z., Cascalho, J., Silva, C., Sebastião, R. (eds) Progress in Artificial Intelligence. EPIA 2023. Lecture Notes in Computer Science(), vol 14115. Springer, Cham. https://doi.org/10.1007/978-3-031-49008-8_34

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-49008-8_34

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-49007-1

  • Online ISBN: 978-3-031-49008-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation