Skip to main content

Can Users Distinguish Narrative Texts Written by an Artificial Intelligence Writing Tool from Purely Human Text?

  • Conference paper
  • First Online:
HCI International 2021 - Posters (HCII 2021)

Abstract

The present user study investigates narrative texts generated by an AI-tool (the Generative Pretrained Transformer 2 Model, GPT-2). In particular, we examined whether readers can distinguish texts written mainly by a GPT-2-based interactive interface for creative writing from texts written without this tool. Nine participants with a literature-specific professional background were provided with the first few lines of 18 unfamiliar poems and short stories written by classic authors (two texts per participant). They created two continuations for each of these texts, one without and one with the help of the AI-tool (only minor human edits allowed). In an evaluation phase, they were presented with all the continuations not written by themselves (16 triples of original continuation, human continuation, AI-based continuation) without knowing each continuation’s category. Their task was to assign each continuation to the correct category. Results showed that participants misclassified 18% of the AI-based continuations (14% as written by other participants without AI-tool and 3% as being the original continuation). Additionally, participants misclassified 35% of the purely human continuations written by other participants (13% as being AI-based continuations and 22% as being the original). These findings indicate that even professionals are no longer able to perfectly distinguish between narrative texts mainly written by an AI-writing-tool and purely human text.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Makridakis, S.: The forthcoming Artificial Intelligence (AI) revolution: its impact on society and firms. Futures 90, 46–60 (2017). https://doi.org/10.1016/j.futures.2017.03.006

    Article  Google Scholar 

  2. Bernhart, T., Richter, S.: Frühe digitale Poesie. Informatik Spektrum 44, 11–18 (2021). https://doi.org/10.1007/s00287-021-01329-z

    Article  Google Scholar 

  3. Shneiderman, B.: Bridging the gap between ethics and practice: guidelines for reliable, safe, and trustworthy human-centered AI systems. ACM Trans. Interact. Intell. Syst. 10(4), 1–31 (2020). https://doi.org/10.1145/3419764

    Article  Google Scholar 

  4. Boden, M., et al.: Principles of robotics: regulating robots in the real world. Connect. Sci. 29(2), 124–129 (2017). https://doi.org/10.1080/09540091.2016.1271400

  5. Shneiderman, B.: Design lessons from AI’s two grand goals: human emulation and useful applications. IEEE Trans. Technol. Soc. 1(2), 73–82 (2020). https://doi.org/10.1109/TTS.2020.2992669

    Article  Google Scholar 

  6. Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for deep learning in NLP. arXiv:1906.02243 (2019)

  7. Mazzone, M., Elgammal, A.: Art, creativity, and the potential of artificial intelligence. Arts 8(1), 26 (2019). https://doi.org/10.3390/arts8010026

    Article  Google Scholar 

  8. Köbis, N., Mossink, L.D.: Artificial intelligence versus Maya Angelou: experimental evidence that people cannot differentiate AI-generated from human-written poetry. Comput. Hum. Behav. 114, (2021). https://doi.org/10.1016/j.chb.2020.106553

    Article  Google Scholar 

  9. Boneh, D., Grotto, A.J., McDaniel, P., Papernot, N.: How relevant is the turing test in the age of sophisbots? IEEE Secur. Priv. 17(6), 64–71 (2019). https://doi.org/10.1109/MSEC.2019.2934193

    Article  Google Scholar 

  10. Zacharakis, A., Kaliakatsos-Papakostas, M., Kalaitzidou, S., Cambouropoulos, E.: Evaluating human-computer co-creative processes in music: a case study on the CHAMELEON melodic harmonizer. Front. Psychol. 12, 322 (2021). https://doi.org/10.3389/fpsyg.2021.603752

    Article  Google Scholar 

  11. Rayner, K., Chace, K.H., Slattery, T.J., Ashby, J.: Eye movements as reflections of comprehension processes in reading. Sci. Stud. Read. 10(3), 241–255 (2006). https://doi.org/10.1207/s1532799xssr1003_3

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Vivian Emily Gunser , Sandra Richter or Peter Gerjets .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gunser, V.E., Gottschling, S., Brucker, B., Richter, S., Gerjets, P. (2021). Can Users Distinguish Narrative Texts Written by an Artificial Intelligence Writing Tool from Purely Human Text?. In: Stephanidis, C., Antona, M., Ntoa, S. (eds) HCI International 2021 - Posters. HCII 2021. Communications in Computer and Information Science, vol 1419. Springer, Cham. https://doi.org/10.1007/978-3-030-78635-9_67

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-78635-9_67

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-78634-2

  • Online ISBN: 978-3-030-78635-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics