Abstract
Recently, generative AIs like ChatGPT have become available to the wide public. These tools can for instance be used by students to generate essays or whole theses. But how does a teacher know whether a text is written by a student or an AI? In our work, we explore traditional and new features to (1) detect text generated by AI from scratch and (2) text rephrased by AI. Since we found that classification is more difficult when the AI has been instructed to create the text in a way that a human would not recognize that it was generated by an AI, we also investigate this more advanced case. For our experiments, we produced a new text corpus covering 10 school topics. Our best systems to classify basic and advanced human-generated/AI-generated texts have F1-scores of over 96%. Our best systems for classifying basic and advanced human-generated/AI-rephrased texts have F1-scores of more than 78%. The systems use a combination of perplexity, semantic, list lookup, error-based, readability, AI feedback, and text vector features. Our results show that the new features substantially help to improve the performance of many classifiers. Our best basic text rephrasing detection system even outperforms GPTZero by 183.8% relative in F1-score.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
References
Pelau, C., Dabija, D.-C., Ene, I.: What makes an AI device human-like? The role of interaction quality, empathy and perceived psychological anthropomorphic characteristics in the acceptance of artificial intelligence in the service industry. Comput. Hum. Behav. 122, 106855 (2021)
Adiwardana, D., et al.: Towards a human-like open-domain Chatbot. arXiv preprint arXiv:2001.09977 (2020)
Dibitonto, M., Leszczynska, K., Tazzi, F., Medaglia, C.M.: Chatbot in a campus environment: design of LiSA, a virtual assistant to help students in their university life. In: Kurosu, M. (ed.) HCI 2018. LNCS, vol. 10903, pp. 103–116. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91250-9_9
Arteaga, D., Arenas, J., Paz, F., Tupia, M., Bruzza, M.: Design of information system architecture for the recommendation of tourist sites in the city of Manta, Ecuador through a Chatbot. In: 2019 14th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–6. IEEE (2019)
Falala-Séchet, C., Antoine, L., Thiriez, I., Bungener, C.: OWLIE: a Chatbot that provides emotional support for coping with psychological difficulties. In: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, pp. 236–237 (2019)
Taecharungroj, V.: “What Can ChatGPT Do?’’ analyzing early reactions to the innovative AI Chatbot on Twitter. Big Data Cogn. Comput. 7(1), 35 (2023)
Baidoo-Anu, D., Owusu Ansah, L.: Education in the era of generative artificial intelligence (AI): understanding the potential benefits of ChatGPT in promoting teaching and learning. Available at SSRN 4337484 (2023)
Jeblick, K., et al.: ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports. arXiv e-prints (2022)
Jiao, W., Wang, W., Huang, J.-T., Wang, X., Tu, Z.: Is ChatGPT a good translator? A preliminary study. arXiv preprint arXiv:2301.08745 (2023)
Brown, T.B., et al.: Language models are few-shot learners. CoRR abs/2005.14165 (2020)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACLHLT, pp. 4171–4186 (2019)
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019). arXiv arXiv:1907.11692
Roberts, A., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. Technical report, Google (2019)
Mitrović, S., Andreoletti, D., Ayoub, O.: ChatGPT or human? Detect and explain. Explaining decisions of machine learning model for detecting short ChatGPT-generated text. arXiv preprint arXiv:2301.13852 (2023)
Soni, M., Wade, V.: Comparing abstractive summaries generated by Chat-GPT to real summaries through blinded reviewers and text classification algorithms (2023)
OpenAI: New AI classifier for indicating AI-written text (2023). https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text. Accessed 21 Apr 2023
Shijaku, R., Canhasi, E.: ChatGPT generated text detection (2023). https://doi.org/10.13140/RG.2.2.21317.52960
Zaitsu, W., Jin, M.: Distinguishing ChatGPT(\(-\)3.5, \(-\)4)-generated and human-written papers through Japanese Stylometric Analysis (2023)
Guo, B., et al.: How close is ChatGPT to human experts? Comparison corpus, evaluation, and detection (2023)
Vu, N.T., Schlippe, T., Kraus, F., Schultz, T.: Rapid bootstrapping of five Eastern European languages using the rapid language adaptation toolkit. In: INTERSPEECH (2010)
Gehrmann, S., Strobelt, H., Rush, A.: GLTR: statistical detection and visualization of generated text. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Florence, Italy, pp. 111–116. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/P19-3019
Bird, S., Loper, E.: NLTK: the natural language toolkit. In: Proceedings of the ACL Interactive Poster and Demonstration Sessions, Barcelona, Spain, pp. 214–217. Association for Computational Linguistics (2004). https://aclanthology.org/P04-3031
Wankhade, M., Rao, A., Kulkarni, C.: A survey on sentiment analysis methods, applications, and challenges. Artif. Intell. Rev. 55, 5731–5780 (2022). https://doi.org/10.1007/s10462-022-10144-1
Rakhmanov, O., Schlippe, T.: Sentiment analysis for Hausa: classifying students’ comments. In: SIGUL 2022, Marseille, France (2022)
Mabokela, K.R., Schlippe, T.: AI for social good: sentiment analysis to detect social challenges in South Africa. In: SACAIR (2022)
Natalie: What is ChatGPT? (2023). https://help.openai.com/en/articles/6783457-what-is-chatgpt. Accessed 21 Apr 2023
Nadeau, D., Sekine, S.: A survey of named entity recognition and classification. Lingvisticae Investigationes 30(1), 3–26 (2007). https://doi.org/10.1075/li.30.1.03nad
Kumarage, T., Garland, J., Bhattacharjee, A., Trapeznikov, K., Ruston, S., Liu, H.: Stylometric detection of AI-generated text in Twitter timelines (2023)
Flesch, R.F.: A new readability yardstick. J. Appl. Psychol. 32(3), 221–233 (1948)
Kincaid, J.P., Fishburne, R.P., Rogers, R.L., Chissom, B.S.: Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel (1975)
Solaiman, I., et al.: Release strategies and the social impacts of language models (2019)
Reimers, N., Gurevych, I.: Sentence-BERT: sentence embeddings using siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3982–3992. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/D19-1410. https://aclanthology.org/D19-1410
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Mindner, L., Schlippe, T., Schaaff, K. (2023). Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPT. In: Schlippe, T., Cheng, E.C.K., Wang, T. (eds) Artificial Intelligence in Education Technologies: New Development and Innovative Practices. AIET 2023. Lecture Notes on Data Engineering and Communications Technologies, vol 190. Springer, Singapore. https://doi.org/10.1007/978-981-99-7947-9_12
Download citation
DOI: https://doi.org/10.1007/978-981-99-7947-9_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-7946-2
Online ISBN: 978-981-99-7947-9
eBook Packages: EducationEducation (R0)