An Eye-Tracking Study of Differences in Reading Between Automated and Human-Written News

  • Chenyan JiaEmail author
  • Jacek Gwizdka
Conference paper
Part of the Lecture Notes in Information Systems and Organisation book series (LNISO, volume 43)


An eye-tracking experiment (N = 24) was conducted to study differences in reading between automated and human-written news. This work adopted expectation-confirmation theory to examine readers’ prior expectations and actual perceptions of both human-written news and automated news. Results revealed that nine eye-tracking variables were significantly different when people read automated news vs. human-written news. Findings also showed promising classification results of 31 eye-tracking-derived features. Self-reported results showed that the readability of human-written news was perceived as significantly higher than that of automated news.


Automated journalism Human-written news Eye-tracking Expectation-confirmation theory Readability 



We thank Lan Li, who is a master student in the School of Information at the University of Texas at Austin, for contributing to the data collection.


  1. 1.
    Napoli, P.M.: Automated media: an institutional theory perspective on algorithmic media production and consumption. Commun. Theory 24, 340–360 (2014). Scholar
  2. 2.
    Graefe, A.: Guide to Automated Journalism. Columbia Journalism School, Tow Center for Digital Journalism, New York (2016)Google Scholar
  3. 3.
    Celeste LeCompte: Automation in the newsroom: How algorithms are helping reporters expand coverage, engage audiences, and respond to breaking news (2015).
  4. 4.
    Dubay, W.H.: The Principles of Readability. Impact Information, Costa Mesa (2004)Google Scholar
  5. 5.
    Dalecki, L., Lasorsa, D.L., Lewis, S.C.: The news readability problem. Journal. Pract. 3, 7–8 (2009)Google Scholar
  6. 6.
    Diakopoulos, N.: Algorithmic accountability. Digital Journal. 3(3), 398–415 (2015). Scholar
  7. 7.
    Clerwall, C.: Enter the robot journalist. Journal. Practice 5, 519–531 (2014). Scholar
  8. 8.
    Haim, M., Graefe, A.: Automated news: better than expected? Digital Journal. 5, 1044–1059 (2017)CrossRefGoogle Scholar
  9. 9.
    Jia, C.: Chinese automated journalism: a comparison between expectations and perceived quality. Int. J. Commun. 14, 2611–2632 (2020)Google Scholar
  10. 10.
    Carlson, M.: The robotic reporter. Digital Journal. 3(3), 416–431 (2015). Scholar
  11. 11.
    van der Kaa, H., Krahmer, E.: Journalist versus news consumer: the perceived credibility of machine written news. BMJ 2(5147), 305 (2014)Google Scholar
  12. 12.
    Shyam Sundar, S.: Exploring receivers’ criteria for perception of print and online news. Journal. Mass Commun. Q. 76(2), 373–386 (1999). Scholar
  13. 13.
    Graefe, A., Haim, M., Haarmann, B., Brosius, H.-B.: Readers’ perception of computer-generated news: credibility, expertise, and readability. Journalism 19(5), 595–610 (2018). Scholar
  14. 14.
    Seib, C.: Papers need to work on handling the English language, Austin American-Statesman, p. B25 (1976)Google Scholar
  15. 15.
    Fishkin, S.F.: From Fact to Fiction: Journalism & Imaginative Writing in America. Oxford University Press, Oxford (1985)Google Scholar
  16. 16.
    Flesch, R.F.: A new readability yardstick. J. Appl. Psychol. 32, 221–233 (1948)CrossRefGoogle Scholar
  17. 17.
    Flesch, R.F.: The Art of Readable Writing. Harper, New York (1949)Google Scholar
  18. 18.
    Gedeon, T., Caldwell, S.: Effects of text difficulty and readers on predicting reading comprehension from eye movements. In: 2015 6th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), Gyor, pp. 407–412 (2015).
  19. 19.
    Rayner, K., Chace, K.H., Slattery, T.J., Ashby, J.: Eye movements as reflections of comprehension processes in reading. Sci. Stud. Read. 10, 241–255 (2006)CrossRefGoogle Scholar
  20. 20.
    Martínez-Gómez, P., Aizawa, A.: Recognition of understanding level and language skill using measurements of reading behavior. In: Proceedings of the 19th International Conference on Intelligent User Interfaces 2014, pp. 95–104 (2014).
  21. 21.
    Gwizdka, J., Hosseini, R., Cole, M., Wang, S.: Temporal dynamics of eye-tracking and EEG during reading and relevance decisions. J. Assoc. Inf. Sci. Technol. 68(10), 2299–2312 (2017). Scholar
  22. 22.
    Gwizdka, J.: Differences in reading between word search and information relevance decisions: evidence from eye-tracking. In: Davis, F.D., Riedl, R., vom Brocke, J., Léger, P.-M., Randolph, A.B. (eds.) Information Systems and Neuroscience. LNISO, vol. 16, pp. 141–147. Springer, Cham (2017). Scholar
  23. 23.
    Cole, M.J., Gwizdka, J., Liu, C., Bierig, R., Belkin, N.J., Zhang, X.: Task and user effects on reading patterns in information search. Interact. Comput. 23(4), 346–362 (2011). Scholar
  24. 24.
    Shojaeizadeh, M., Djamasbi, S.: Eye movements and reading behavior of younger and older users: an exploratory eye-tacking study. In: Zhou, J., Salvendy, G. (eds.) ITAP 2018. LNCS, vol. 10926, pp. 377–391. Springer, Cham (2018). Scholar
  25. 25.
    Sundar, R.P., Becker, M.W., Bello, N.M., Bix, L.: Quantifying age-related differences in information processing behaviors when viewing prescription drug labels. PLoS ONE 7(6) (2012)
  26. 26.
    Angela, S., Tanya, B.: The influence of science reading comprehension on South African township learners’ learning of science. S. Afr. J. Sci. 115(1), 72–80 (2019)Google Scholar
  27. 27.
    Bhattacherjee, A.: Understanding information systems continuous: an expectation-confirmation model. MIS Q. 25(3), 351–370 (2001). Scholar
  28. 28.
    Lin, C.S., Sheng, W., Tsai, R.J.: Integrating perceived playfulness into expectation-confirmation model for web portal context. Inf. Manag. 42(5), 683–693 (2005)CrossRefGoogle Scholar
  29. 29.
    Oliver, R.L.: A cognitive model of the antecedents and consequences of satisfaction decisions. J. Mark. Res. 17, 460–469 (1980). Scholar
  30. 30.
    Oliver, R.L.: Cognitive, affective, and attribute bases of the satisfaction response. J. Consum. Res. 20, 418–430 (1993).
  31. 31.
    Oliver, R.L.: Satisfaction. A Behavioral Perspective on the Consumer, 2nd edn. Routledge, London (2015)Google Scholar
  32. 32.
    Salvucci, D.D., Goldberg, J.H.: Identifying fixations and saccades in eye-tracking protocols. In: Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (ETRA 2000), 71–78 (2000).
  33. 33.
    Spreng, R.A., Olshavsky, R.W.: A desires congruency model of consumer satisfaction. J. Acad. Mark. Sci. 21(3), 169–177 (1993)CrossRefGoogle Scholar
  34. 34.
    Witten, I.H., Frank, E., Hall, M.A.: Data Mining, Fourth Edition: Practical Machine Learning Tools and Techniques. Morgan Kaufmann Publishers Inc., San Francisco (2016)Google Scholar
  35. 35.
    Chawla, N.V., Bowyer, K.W., Hall, L.O., Philip Kegelmeyer,W.: SMOTE: synthetic minority over-sampling technique. arXiv:1106.1813 Cs (2011)
  36. 36.
    Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2011)CrossRefGoogle Scholar
  37. 37.
    Rayner, K., Pollatsek, A.: The Psychology of Reading. Lawrence Erlbaum Associates, Mahwah (1989)Google Scholar

Copyright information

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of JournalismUniversity of Texas at AustinAustinUSA
  2. 2.School of InformationUniversity of Texas at AustinAustinUSA

Personalised recommendations