User Modeling and User-Adapted Interaction

, Volume 29, Issue 5, pp 977–1011 | Cite as

Gaze analysis of user characteristics in magazine style narrative visualizations

  • Dereck TokerEmail author
  • Cristina Conati
  • Giuseppe Carenini


Previous research has shown that various user characteristics (e.g., cognitive abilities, personality traits, and learning abilities) can influence user experience during information visualization tasks. These findings have prompted researchers to investigate user-adaptive information visualizations that can help users by providing personalized support based on their specific needs. Whereas existing work has been mostly limited to tasks involving just visualizations, the aim of our research is to broaden this work to include scenarios where users process textual documents with embedded visualizations, i.e., Magazine Style Narrative Visualizations, or MSNVs for short. In this paper, we analyze eye tracking data collected from a user study with MSNVs to uncover processing behaviors that are negatively impacting user experience (i.e., time on task) for users with low abilities in these user characteristics. Our analysis leverages Linear Mixed-Effects Models to evaluate the relationships among user characteristics, gaze processing behaviors, and task performance. Our results identify several MSNV processing behaviors within the visualization that contribute to poor task performance for users with low reading proficiency. For instance, we identify that users with low reading proficiency transition significantly more often compared to their counterparts between relevant and non-relevant bars, and transition more often from bars to the labels. We present our findings as a step toward designing user-adaptive support mechanisms to alleviate these difficulties with MSNVs, and provide suggestions on how our results can be leveraged for creating a set of meaningful interventions for future evaluation (e.g., dynamically highlighting relevant bars and labels in the visualization to help users with low reading proficiency locate them more effectively).


Narrative visualization User modeling Eye tracking Mixed models Adaptive visualizations Individual differences Users characteristics 



  1. Akaike, H.: A new look at the statistical model identification. IEEE Trans. Autom. Control 19(6), 716–723 (1974)MathSciNetzbMATHGoogle Scholar
  2. Allen, B.: Individual differences and the conundrums of user-centered design: two experiments. J. Am. Soc. Inf. Sci. 51(6), 508–520 (2000)Google Scholar
  3. Baddeley, A.: Oxford Psychology Series, No. 11. Working Memory. Clarendon Press/Oxford University Press, New York (1986)Google Scholar
  4. Benjamini, Y., Hochberg, Y.: Controlling the false discovery rate—a practical and powerful approach to multiple testing. J. R. Stat. Soc. 57(1), 289–300 (1995)MathSciNetzbMATHGoogle Scholar
  5. Blair, J.R., Spreen, O.: Predicting premorbid IQ: a revision of the national adult reading test. Clin. Neuropsychol. 3(2), 129–136 (1989)Google Scholar
  6. Boy, J., Rensink, R.A., Bertini, E., et al.: A principled way of assessing visualization literacy. IEEE Trans. Vis. Comput. Graph. 20(12), 1963–1972 (2014)Google Scholar
  7. Cacioppo, J.T., Petty, R.E., Kao, C.F.: The efficient assessment of need for cognition. J. Pers. Assess. 48(3), 306–307 (1984)Google Scholar
  8. Carenini, G., Conati, C., Hoque, E., et al.: User task adaptation in multimedia presentations. In: Proceedings of the 1st International Workshop on User-Adaptive Information Visualization (WUAV 2013), in Conjunction with the 21st Conference on User Modeling, Adaptation and Personalization (UMAP 2013) (2013)Google Scholar
  9. Carenini, G., Conati, C., Hoque, E., et al.: Highlighting interventions and user differences: informing adaptive information visualization support. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, pp. 1835–1844 (2014)Google Scholar
  10. Çöltekin, A., Fabrikant, S.I., Lacayo, M.: Exploring the efficiency of users’ visual analytics strategies based on sequence analysis of eye movement recordings. Int. J. Geogr. Inf. Sci. 24(10), 1559–1575 (2010)Google Scholar
  11. Conati, C., Maclaren, H.: Exploring the role of individual differences in information visualization. In: Proceedings of the Working Conference on Advanced visual Interfaces, New York, NY, USA. ACM, pp. 199–206 (2008)Google Scholar
  12. Conati, C., Carenini, G., Hoque, E., et al.: Evaluating the impact of user characteristics and different layouts on an interactive visualization for decision making. In: Computer Graphics Forum, Wiley Online Library, pp. 371–380 (2014)Google Scholar
  13. Conati, C., Lallé, S., Rahman, M.A., et al.: Further results on predicting cognitive abilities for adaptive visualizations. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Australia. AAAI Press (2017)Google Scholar
  14. D’Mello, S., Olney, A., Williams, C., et al.: Gaze tutor: a gaze-reactive intelligent tutoring system. Int. J. Hum. Comput. Stud. 70(5), 377–398 (2012)Google Scholar
  15. D’Mello, S., Mills, C., Bixler, R., et al.: Zone out no more: mitigating mind wandering during computerized reading. In: Proceedings of the 10th International Conference on Educational Data Mining, Wuhan, China, pp. 8–15 (2017)Google Scholar
  16. Dyson, M.C., Haselgrove, M.: The influence of reading speed and line length on the effectiveness of reading from screen. Int. J. Hum. Comput. Stud. 54(4), 585–612 (2001)Google Scholar
  17. Ekstrom, R.B., French, J.W., Harman, H.H., et al.: Manual for Kit of Factor Referenced Cognitive Tests. Educational Testing Service, Princeton (1976)Google Scholar
  18. Field, A.P.: How to Design and Report Experiments. Sage, London (2003)Google Scholar
  19. Folker, S., Ritter, H., Sichelschmidt, L.: Processing and integrating multimodal material—the influence of color-coding. In: Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 27(27), pp. 690–695 (2005)Google Scholar
  20. Gingerich, M.J., Conati, C.: Constructing models of user and task characteristics from eye gaze data for user-adaptive information highlighting. In: Twenty-Ninth AAAI Conference on Artificial Intelligence (2015)Google Scholar
  21. Gotz, D., Wen, Z.: Behavior-driven visualization recommendation. In: Proceedings of the 14th International Conference on Intelligent User Interfaces, IUI’09, New York, NY, USA. ACM, pp. 315–324 (2009)Google Scholar
  22. Grabe, W., Jiang, X.: Assessing reading. In: Kunnan, A.J. (ed.) The Companion to Language Assessment, pp. 185–200. Wiley, Hoboken (2013)Google Scholar
  23. Grawemeyer, B.: Evaluation of ERST: an external representation selection tutor. In: Proceedings of the 4th International Conference on Diagrammatic Representation and Inference, Diagrams’06. Springer, Berlin, Heidelberg, pp. 154–167 (2006)Google Scholar
  24. Green, T.M., Fisher, B.: Towards the personal equation of interaction: the impact of personality factors on visual analytics interface interaction. In: 2010 IEEE Symposium on Visual Analytics Science and Technology (VAST), pp. 203–210 (2010)Google Scholar
  25. Green, N.L., Carenini, G., Kerpedjiev, S., et al.: AutoBrief: an experimental system for the automatic generation of briefings in integrated text and information graphics. Int. J. Hum. Comput. Stud. 61(1), 32–70 (2004)Google Scholar
  26. Hegarty, M., Just, M.A.: Constructing mental models of machines from text and diagrams. J. Mem. Lang. 32(6), 717–742 (1993)Google Scholar
  27. Huang, D., Tory, M., Aseniero, B.A., et al.: Personal visualization and personal visual analytics. IEEE Trans. Vis. Comput. Graph. 21(3), 420–433 (2015)Google Scholar
  28. Human Factors (HF): Guidelines on the Multimodality of Icons, Symbols and Pictograms. European Telecommunications Standards Institute, Sophia Antipolis (2008)Google Scholar
  29. Just, M.A., Carpenter, P.A.: Eye fixations and cognitive processes. Cogn. Psychol. 8(4), 441–480 (1976)Google Scholar
  30. Kalyuga, S.: Managing Cognitive Load in Adaptive Multimedia Learning. Information Science Reference, Hershey (2009)Google Scholar
  31. Kalyuga, S., Chandler, P., Sweller, J.: Levels of expertise and instructional design. Hum. Factors 40(1), 1–17 (1998)Google Scholar
  32. Kalyuga, S., Law, Y.K., Lee, C.H.: Expertise reversal effect in reading Chinese texts with added causal words. Instr. Sci. 41(3), 481–497 (2013)Google Scholar
  33. Kong, N., Hearst, M.A., Agrawala, M.: Extracting references between text and charts via crowdsourcing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Toronto, Canada. ACM, pp. 31–40 (2014)Google Scholar
  34. Kules, B., Capra, R.: Influence of training and stage of search on gaze behavior in a library catalog faceted search interface. J. Am. Soc. Inform. Sci. Technol. 63(1), 114–138 (2012)Google Scholar
  35. Kuznetsova, A., Brockhoff, P.B., Christensen, R.H.B.: lmerTest package: tests in linear mixed effects models. J. Stat. Softw. 82(13) (2017). Accessed 12 October 2018
  36. Lallé, S., Toker, D., Conati, C., et al.: Prediction of users’ learning curves for adaptation while using an information visualization. In: Proceedings of the 20th International Conference on Intelligent User Interfaces. ACM, pp. 357–368 (2015)Google Scholar
  37. Lallé, S., Conati, C., Carenini, G.: Predicting confusion in information visualization from eye tracking and interaction data. In: Proceedings on the 25th International Joint Conference on Artificial Intelligence, pp. 2529–2535 (2016)Google Scholar
  38. Lallé, S., Conati, C., Carenini, G.: Impact of individual differences on user experience with a visualization interface for public engagement. In: Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization, UMAP’17, New York, NY, USA. ACM, pp. 247–252 (2017)Google Scholar
  39. Lankow, J., Ritchie, J., Crooks, R.: Infographics: The Power of Visual Storytelling. Wiley, Hoboken (2012)Google Scholar
  40. Loboda, T.D., Brusilovsky, P., Brunstein, J.: Inferring word relevance from eye-movements of readers. In: ACM Press, pp. 175 (2011)Google Scholar
  41. Logie, R.H.: Visuo-spatial Working Memory. Nachdr. Psychology Press, Hove (2009)Google Scholar
  42. Martínez-Gómez, P., Aizawa, A.: Recognition of understanding level and language skill using measurements of reading behavior. In: ACM Press, pp. 95–104 (2014)Google Scholar
  43. Mayer, R.E.: Multimedia Learning, 2nd edn. Cambridge University Press, Cambridge (2009)Google Scholar
  44. Meara, P.: EFL Vocabulary Tests, 2nd edn. Lognostics, Swansea (2010)Google Scholar
  45. Meara, P., Jones, G.: Eurocentres Vocabulary Size Test 10KA. Eurocentres Learning Service, Zurich (1990)Google Scholar
  46. Metoyer, R., Zhi, Q., Janczuk, B., et al.: Coupling story to visualization: using textual analysis as a bridge between data and interpretation. In: ACM Press, pp. 503–507 (2018)Google Scholar
  47. Munzner, T.: Visualization Analysis and Design. CRC Press, Taylor & Francis Group (2014)Google Scholar
  48. Nazemi, K., Retz, R., Bernard, J., et al.: Adaptive semantic visualization for bibliographic entries. In: Advances in Visual Computing, Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, pp. 13–24 (2013)Google Scholar
  49. Olsen, A.: The tobii i-vt fixation filter. Tobii Technology (2012). Accessed 13 September 2015
  50. Ooms, K., De Maeyer, P., Fack, V., et al.: Interpreting maps through the eyes of expert and novice users. Int. J. Geogr. Inf. Sci. 26(10), 1773–1788 (2012)Google Scholar
  51. Ooms, K., De Maeyer, P., Fack, V.: Study of the attentive behavior of novice and expert map users using eye tracking. Cartogr. Geogr. Inf. Sci. 41(1), 37–54 (2014)Google Scholar
  52. Ottley, A., Yang, H., Chang, R.: Personality as a predictor of user strategy: how locus of control affects search strategies on tree visualizations. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea. ACM, pp. 3251–3254 (2015)Google Scholar
  53. Ozcelik, E., Arslan-Ari, I., Cagiltay, K.: Why does signaling enhance multimedia learning? Evidence from eye movements. Comput. Hum. Behav. 26(1), 110–117 (2010)Google Scholar
  54. Scheiter, K., Wiebe, E., Holsanova, J.: Theoretical and Instructional Aspects of Learning with Visualizations. Instructional Design: Concepts, Methodologies, Tools and Applications. IGI Global, Hershey (2011)Google Scholar
  55. Segel, E., Heer, J.: Narrative visualization: telling stories with data. IEEE Trans. Vis. Comput. Graph. 16(6), 1139–1148 (2010)Google Scholar
  56. Steichen, B., Conati, C., Carenini, G.: Inferring visualization task properties, user performance, and user cognitive abilities from eye gaze data. ACM Trans. Interact. Intell. Syst. 4, 11 (2014)Google Scholar
  57. Strauss, E., Sherman, E.M.S., Spreen, O., et al.: A Compendium of Neuropsychological Tests: Administration, Norms, and Commentary, 3rd edn. Oxford University Press, Oxford (2006)Google Scholar
  58. Tai, R.H., Loehr, J.F., Brigham, F.J.: An exploration of the use of eye-gaze tracking to study problem-solving on standardized science assessments. Int. J. Res. Method Educ. 29(2), 185–208 (2006)Google Scholar
  59. Tang, H., Topczewski, J.J., Topczewski, A.M., et al.: Permutation test for groups of scanpaths using normalized Levenshtein distances and application in NMR questions. In: ACM Press, pp. 169 (2012)Google Scholar
  60. Toker, D., Conati, C.: Eye tracking to understand user differences in visualization processing with highlighting interventions. In: Proceedings of the 22nd International Conference on User Modeling, Adaptation, and Personalization, UMAP’14. Springer, Aalborg, Denmark (2014)Google Scholar
  61. Toker, D., Conati, C., Carenini, G., et al.: Towards adaptive information visualization: on the influence of user characteristics. In: Proceedings of the 20th International Conference on User Modeling, Adaptation, and Personalization, UMAP’12. Springer, Berlin, Heidelberg, pp. 274–285 (2012)Google Scholar
  62. Toker, D., Conati, C., Steichen, B., et al.: Individual user characteristics and information visualization: connecting the dots through eye tracking. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’13. ACM, New York, NY, USA, pp. 295–304 (2013)Google Scholar
  63. Toker, D., Lallé, S., Conati, C.: Pupillometry and head distance to the screen to predict skill acquisition during information visualization tasks. In: Proceedings of the 22nd International Conference on Intelligent User Interfaces. ACM Press, pp. 221–231 (2017)Google Scholar
  64. Toker, D., Conati, C., Carenini, G.: User-adaptive support for processing magazine style narrative visualizations: identifying user characteristics that matter. In: ACM Press, pp. 199–204 (2018)Google Scholar
  65. Tufte, E.R.: Visual Explanations: Images and Quantities, Evidence and Narrative. Graphics Press, Cheshire (1997)zbMATHGoogle Scholar
  66. Turner, M.L., Engle, R.W.: Is working memory capacity task dependent? J. Mem. Lang. 28(2), 127–154 (1989)Google Scholar
  67. van Gog, T.: The signaling (or cueing) principle in multimedia learning. In: Mayer, Richard (ed.) The Cambridge Handbook of Multimedia Learning, pp. 263–278. Cambridge University Press, Cambridge (2014)Google Scholar
  68. Velez, M.C., Silver, D., Tremaine, M.: Understanding visualization through spatial ability differences. In: Proceedings of the IEEE Conference on Visualization, Minneapolis, MN, USA. IEEE, pp. 511–518 (2005)Google Scholar
  69. Vogel, E.K., Woodman, G.F., Luck, S.J.: Storage of features, conjunctions, and objects in visual working memory. J. Exp. Psychol. Hum. Percept. Perform. 27(1), 92–114 (2001)Google Scholar
  70. Waddell, T.F., Auriemma, J.R., Sundar, S.S.: Make it simple, or force users to read? Paraphrased design improves comprehension of end user license agreements. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’16. ACM Press, pp. 5252–5256 (2016)Google Scholar
  71. Walker, D.A.: Converting Kendall’s Tau for correlational or meta-analytic analyses. J. Mod. Appl. Stat. Methods 2(2), 525–530 (2003)Google Scholar
  72. Wiley, J., Sanchez, C.A., Jaeger, A.J.: The individual differences in working memory capacity principle in multimedia learning. The Cambridge Handbook of Multimedia Learning, pp. 598–620. Cambridge University Press, Cambridge (2014)Google Scholar
  73. Wobbrock, J.O., Findlater, L., Gergle, D., et al.: The aligned rank transform for nonparametric factorial analyses using only anova procedures. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, pp. 143–146 (2011)Google Scholar
  74. Ziemkiewicz, C., Crouser, R.J., Yauilla, A.R., et al.: How locus of control influences compatibility with visualization style. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, Providence, RI, USA. IEEE, pp. 81–90 (2011)Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  • Dereck Toker
    • 1
    Email author
  • Cristina Conati
    • 1
  • Giuseppe Carenini
    • 1
  1. 1.University of British ColumbiaVancouverCanada

Personalised recommendations