Can Users Distinguish Narrative Texts Written by an Artificial Intelligence Writing Tool from Purely Human Text?

Abstract

The present user study investigates narrative texts generated by an AI-tool (the Generative Pretrained Transformer 2 Model, GPT-2). In particular, we examined whether readers can distinguish texts written mainly by a GPT-2-based interactive interface for creative writing from texts written without this tool. Nine participants with a literature-specific professional background were provided with the first few lines of 18 unfamiliar poems and short stories written by classic authors (two texts per participant). They created two continuations for each of these texts, one without and one with the help of the AI-tool (only minor human edits allowed). In an evaluation phase, they were presented with all the continuations not written by themselves (16 triples of original continuation, human continuation, AI-based continuation) without knowing each continuation’s category. Their task was to assign each continuation to the correct category. Results showed that participants misclassified 18% of the AI-based continuations (14% as written by other participants without AI-tool and 3% as being the original continuation). Additionally, participants misclassified 35% of the purely human continuations written by other participants (13% as being AI-based continuations and 22% as being the original). These findings indicate that even professionals are no longer able to perfectly distinguish between narrative texts mainly written by an AI-writing-tool and purely human text.