Skip to main content
SpringerLink
Log in

Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPT

  • Conference paper
  • First Online:
Artificial Intelligence in Education Technologies: New Development and Innovative Practices (AIET 2023)

Part of the book series: Lecture Notes on Data Engineering and Communications Technologies ((LNDECT,volume 190))

  • 802 Accesses

Abstract

Recently, generative AIs like ChatGPT have become available to the wide public. These tools can for instance be used by students to generate essays or whole theses. But how does a teacher know whether a text is written by a student or an AI? In our work, we explore traditional and new features to (1) detect text generated by AI from scratch and (2) text rephrased by AI. Since we found that classification is more difficult when the AI has been instructed to create the text in a way that a human would not recognize that it was generated by an AI, we also investigate this more advanced case. For our experiments, we produced a new text corpus covering 10 school topics. Our best systems to classify basic and advanced human-generated/AI-generated texts have F1-scores of over 96%. Our best systems for classifying basic and advanced human-generated/AI-rephrased texts have F1-scores of more than 78%. The systems use a combination of perplexity, semantic, list lookup, error-based, readability, AI feedback, and text vector features. Our results show that the new features substantially help to improve the performance of many classifiers. Our best basic text rephrasing detection system even outperforms GPTZero by 183.8% relative in F1-score.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/LorenzM97/human-AI-generatedTextCorpus.

  2. 2.

    https://gptzero.me.

  3. 3.

    https://writer.com/ai-content-detector.

  4. 4.

    https://openai-openai-detector.hf.space.

  5. 5.

    https://github.com/openai/gpt-2.

  6. 6.

    https://textblob.readthedocs.io/en/dev/quickstart.html#sentiment-analysis.

  7. 7.

    https://github.com/jxmorris12/language_tool_python.

References

  1. Pelau, C., Dabija, D.-C., Ene, I.: What makes an AI device human-like? The role of interaction quality, empathy and perceived psychological anthropomorphic characteristics in the acceptance of artificial intelligence in the service industry. Comput. Hum. Behav. 122, 106855 (2021)

    Article  Google Scholar 

  2. Adiwardana, D., et al.: Towards a human-like open-domain Chatbot. arXiv preprint arXiv:2001.09977 (2020)

  3. Dibitonto, M., Leszczynska, K., Tazzi, F., Medaglia, C.M.: Chatbot in a campus environment: design of LiSA, a virtual assistant to help students in their university life. In: Kurosu, M. (ed.) HCI 2018. LNCS, vol. 10903, pp. 103–116. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91250-9_9

    Chapter  Google Scholar 

  4. Arteaga, D., Arenas, J., Paz, F., Tupia, M., Bruzza, M.: Design of information system architecture for the recommendation of tourist sites in the city of Manta, Ecuador through a Chatbot. In: 2019 14th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–6. IEEE (2019)

    Google Scholar 

  5. Falala-Séchet, C., Antoine, L., Thiriez, I., Bungener, C.: OWLIE: a Chatbot that provides emotional support for coping with psychological difficulties. In: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, pp. 236–237 (2019)

    Google Scholar 

  6. Taecharungroj, V.: “What Can ChatGPT Do?’’ analyzing early reactions to the innovative AI Chatbot on Twitter. Big Data Cogn. Comput. 7(1), 35 (2023)

    Article  Google Scholar 

  7. Baidoo-Anu, D., Owusu Ansah, L.: Education in the era of generative artificial intelligence (AI): understanding the potential benefits of ChatGPT in promoting teaching and learning. Available at SSRN 4337484 (2023)

    Google Scholar 

  8. Jeblick, K., et al.: ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports. arXiv e-prints (2022)

    Google Scholar 

  9. Jiao, W., Wang, W., Huang, J.-T., Wang, X., Tu, Z.: Is ChatGPT a good translator? A preliminary study. arXiv preprint arXiv:2301.08745 (2023)

  10. Brown, T.B., et al.: Language models are few-shot learners. CoRR abs/2005.14165 (2020)

    Google Scholar 

  11. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACLHLT, pp. 4171–4186 (2019)

    Google Scholar 

  12. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019). arXiv arXiv:1907.11692

  13. Roberts, A., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. Technical report, Google (2019)

    Google Scholar 

  14. Mitrović, S., Andreoletti, D., Ayoub, O.: ChatGPT or human? Detect and explain. Explaining decisions of machine learning model for detecting short ChatGPT-generated text. arXiv preprint arXiv:2301.13852 (2023)

  15. Soni, M., Wade, V.: Comparing abstractive summaries generated by Chat-GPT to real summaries through blinded reviewers and text classification algorithms (2023)

    Google Scholar 

  16. OpenAI: New AI classifier for indicating AI-written text (2023). https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text. Accessed 21 Apr 2023

  17. Shijaku, R., Canhasi, E.: ChatGPT generated text detection (2023). https://doi.org/10.13140/RG.2.2.21317.52960

  18. Zaitsu, W., Jin, M.: Distinguishing ChatGPT(\(-\)3.5, \(-\)4)-generated and human-written papers through Japanese Stylometric Analysis (2023)

    Google Scholar 

  19. Guo, B., et al.: How close is ChatGPT to human experts? Comparison corpus, evaluation, and detection (2023)

    Google Scholar 

  20. Vu, N.T., Schlippe, T., Kraus, F., Schultz, T.: Rapid bootstrapping of five Eastern European languages using the rapid language adaptation toolkit. In: INTERSPEECH (2010)

    Google Scholar 

  21. Gehrmann, S., Strobelt, H., Rush, A.: GLTR: statistical detection and visualization of generated text. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Florence, Italy, pp. 111–116. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/P19-3019

  22. Bird, S., Loper, E.: NLTK: the natural language toolkit. In: Proceedings of the ACL Interactive Poster and Demonstration Sessions, Barcelona, Spain, pp. 214–217. Association for Computational Linguistics (2004). https://aclanthology.org/P04-3031

  23. Wankhade, M., Rao, A., Kulkarni, C.: A survey on sentiment analysis methods, applications, and challenges. Artif. Intell. Rev. 55, 5731–5780 (2022). https://doi.org/10.1007/s10462-022-10144-1

    Article  Google Scholar 

  24. Rakhmanov, O., Schlippe, T.: Sentiment analysis for Hausa: classifying students’ comments. In: SIGUL 2022, Marseille, France (2022)

    Google Scholar 

  25. Mabokela, K.R., Schlippe, T.: AI for social good: sentiment analysis to detect social challenges in South Africa. In: SACAIR (2022)

    Google Scholar 

  26. Natalie: What is ChatGPT? (2023). https://help.openai.com/en/articles/6783457-what-is-chatgpt. Accessed 21 Apr 2023

  27. Nadeau, D., Sekine, S.: A survey of named entity recognition and classification. Lingvisticae Investigationes 30(1), 3–26 (2007). https://doi.org/10.1075/li.30.1.03nad

    Article  Google Scholar 

  28. Kumarage, T., Garland, J., Bhattacharjee, A., Trapeznikov, K., Ruston, S., Liu, H.: Stylometric detection of AI-generated text in Twitter timelines (2023)

    Google Scholar 

  29. Flesch, R.F.: A new readability yardstick. J. Appl. Psychol. 32(3), 221–233 (1948)

    Article  Google Scholar 

  30. Kincaid, J.P., Fishburne, R.P., Rogers, R.L., Chissom, B.S.: Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel (1975)

    Google Scholar 

  31. Solaiman, I., et al.: Release strategies and the social impacts of language models (2019)

    Google Scholar 

  32. Reimers, N., Gurevych, I.: Sentence-BERT: sentence embeddings using siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3982–3992. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/D19-1410. https://aclanthology.org/D19-1410

Download references

Author information

Authors and Affiliations

  1. IU International University of Applied Sciences, Erfurt, Germany

    Lorenz Mindner, Tim Schlippe & Kristina Schaaff

Authors
  1. Lorenz Mindner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tim Schlippe
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kristina Schaaff
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tim Schlippe .

Editor information

Editors and Affiliations

  1. IU International University of Applied Sciences, Erfurt, Germany

    Tim Schlippe

  2. The Education University of Hong Kong, Hong Kong S.A.R., China

    Eric C. K. Cheng

  3. Swinburne University of Technology, Melbourne, VIC, Australia

    Tianchong Wang

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mindner, L., Schlippe, T., Schaaff, K. (2023). Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPT. In: Schlippe, T., Cheng, E.C.K., Wang, T. (eds) Artificial Intelligence in Education Technologies: New Development and Innovative Practices. AIET 2023. Lecture Notes on Data Engineering and Communications Technologies, vol 190. Springer, Singapore. https://doi.org/10.1007/978-981-99-7947-9_12

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-7947-9_12

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-7946-2

  • Online ISBN: 978-981-99-7947-9

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation