Advertisement

User Modeling and User-Adapted Interaction

, Volume 26, Issue 4, pp 307–345 | Cite as

Prediction of individual learning curves across information visualizations

  • Sébastien LalléEmail author
  • Cristina Conati
  • Giuseppe Carenini
Article

Abstract

Confident usage of information visualizations is thought to be influenced by cognitive aspects as well as amount of exposure and training. To support the development of individual competency in visualization processing, it is important to ascertain if we can track users’ progress or difficulties they might have while working with a given visualization. In this paper, we extend previous work on predicting in real time a user’s learning curve—a mathematical model that can represent a user’s skill acquisition ability—when working with a visualization. First, we investigate whether results we previously obtained in predicting users’ learning curves during visualization processing generalize to a different visualization. Second, we study to what extent we can make predictions on a user’s learning curve without information on the visualization being used. Our models leverage various data sources, including a user’s gaze behavior, pupil dilation, and cognitive abilities. We show that these models outperform a baseline that leverages knowledge on user task performance so far. Our best performing model achieves good accuracies in predicting users’ learning curves even after observing users’ performance on a few tasks only. These results represent an important step toward understanding how to support users in learning a new visualization.

Keywords

Information visualization Adaptive visualization User modeling Machine learning Eye tracking Learning curve 

Notes

Acknowledgments

This publication is based upon work supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), under Grant No. STPG381322-09. We also thank Dereck Toker for his help in revising the paper.

References

  1. Akaike, H.: A new look at the statistical model identification. IEEE Trans. Autom. Control. 19(6), 716–723 (1974)MathSciNetzbMATHCrossRefGoogle Scholar
  2. Amar, R., Eagan, J., Stasko, J.: Low-level components of analytic activity in information visualization. In: Proceedings of the 2005 IEEE Symposium on Information Visualization. pp. 15–21. IEEE Computer Society, Washington, DC (2005)Google Scholar
  3. Anderson, J.R.: Cognitive Skills and Their Acquisition. Psychology Press, Hove (1981)Google Scholar
  4. Anderson, J.R.: Rules of the Mind. Psychology Press, Hove (2014)Google Scholar
  5. Aoki, H., Hansen, J.P., Itoh, K.: Learning to interact with a computer by gaze. Behav. Inf. Technol. 27(4), 339–344 (2008)CrossRefGoogle Scholar
  6. Bailey, C.D., McIntyre, E.V.: The relation between fit and prediction for alternative forms of learning curves and relearning curves. IIE Trans. 29(6), 487–495 (1997)Google Scholar
  7. Bautista, J., Carenini, G.: An empirical evaluation of interactive visualizations for preferential choice. In: Proceedings of the Working Conference on Advanced Visual Interfaces. pp. 207–214. ACM, New York, NY (2008)Google Scholar
  8. Beatty, J.: Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychol. Bull. 91(2), 276–292 (1982)CrossRefGoogle Scholar
  9. Beck, J.E., Sison, J.: Using knowledge tracing in a noisy environment to measure student reading proficiencies. Int. J. Artif. Intell. Educ. 16(2), 129–143 (2006)Google Scholar
  10. Bednarik, R., Eivazi, S., Vrzakova, H.: A computational approach for prediction of problem-solving behavior using support vector machines and eye-tracking data. In: Nakano, Y., Conati, C., Bader, T. (eds.) Eye Gaze in Intelligent User Interfaces, pp. 111–134. Springer, London (2013)CrossRefGoogle Scholar
  11. Bixler, R., Kopp, K., D’Mello, S.: Evaluation of a personalized method for proactive mind wandering reduction. In: Proceedings of the 4th Workshop on Personalization Approaches for Learning Environments. 22nd Conference on User Modeling, Adaptation, and Personalization, pp. 33–41. Springer, Aalborg (2014)Google Scholar
  12. Bondareva, D., Conati, C., Feyzi-Behnagh, R., Harley, J.M., Azevedo, R., Bouchet, F.: Inferring learning from gaze data during interaction with an environment to support self-regulated learning. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education. pp. 229–238. Springer, Memphis, TN (2013)Google Scholar
  13. Bradley, M.M., Miccoli, L., Escrig, M.A., Lang, P.J.: The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology 45(4), 602–607 (2008)CrossRefGoogle Scholar
  14. Bradshaw, J.: Pupil size as a measure of arousal during information processing. Nature 216, 515–516 (1967)CrossRefGoogle Scholar
  15. Bunt, A., Conati, C., McGrenere, J.: Supporting interface customization using a mixed-initiative approach. In: Proceedings of the 12th International Conference on Intelligent User Interfaces. pp. 92–101. ACM, Honolulu, HI (2007)Google Scholar
  16. Carenini, G., Conati, C., Hoque, E., Steichen, B., Toker, D., Enns, J.T.: Highlighting interventions and user differences: Informing adaptive information visualization support. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 1835–1844. ACM, Toronto (2014)Google Scholar
  17. Carenini, G., Loyd, J.: ValueCharts: Analyzing linear models expressing preferences and evaluations. In: Proceedings of the Working Conference on Advanced Visual Interfaces. pp. 150–157. ACM, Gallipoli (2004)Google Scholar
  18. Chamberlain, B.C., Carenini, G., Oberg, G., Poole, D., Taheri, H.: A decision support system for the design and evaluation of sustainable wastewater solutions. IEEE Trans. Comput. 63(1), 129–141 (2014)MathSciNetCrossRefGoogle Scholar
  19. Çöltekin, A., Fabrikant, S.I., Lacayo, M.: Exploring the efficiency of users’ visual analytics strategies based on sequence analysis of eye movement recordings. Int. J. Geogr. Inf. Sci. 24(10), 1559–1575 (2010)CrossRefGoogle Scholar
  20. Conati, C., Maclaren, H.: Exploring the role of individual differences in information visualization. In: Proceedings of the Working Conference on Advanced Visual Interfaces. pp. 199–206. ACM, New York, NY (2008)Google Scholar
  21. Conati, C., Carenini, G., Steichen, B.: Dereck toker: Evaluating the impact of user characteristics and different layouts on an interactive visualization for decision making. Comput. Graph. Forum 33(3), 371–380 (2014)CrossRefGoogle Scholar
  22. Conati, C., Carenini, G., Toker, D., Lallé, S.: Towards user-adaptive information visualization. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence. pp. 4100–4106. AAAI Press, Austin, TX (2015)Google Scholar
  23. D’Mello, S., Olney, A., Williams, C., Hays, P.: Gaze tutor: A gaze-reactive intelligent tutoring system. Int. J. Human-Comput. Stud. 70(5), 377–398 (2012)CrossRefGoogle Scholar
  24. Dork, M., Carpendale, S., Collins, C., Williamson, C.: VisGets: Coordinated visualizations for web-based information exploration and discovery. IEEE Trans. Vis. Comput. Graph. 14(6), 1205–1212 (2008)CrossRefGoogle Scholar
  25. Ekstrom, R.B., Harman, H.H.: Manual for Kit of Factor-Referenced Cognitive Tests. Educational Testing Service, Princeton (1976)Google Scholar
  26. Few, S.: Now you see it: Simple visualization techniques for quantitative analysis. Analytics Press, Oakland, CA (2009)Google Scholar
  27. Fisher, D., Drucker, S.M., Fernandez, R., Ruble, S.: Visualizations everywhere: a multiplatform infrastructure for linked visualizations. IEEE Trans. Vis. Comput. Graph. 16(6), 1157–1163 (2010)CrossRefGoogle Scholar
  28. Fukuda, K., Vogel, E.K.: Human variation in overriding attentional capture. J. Neurosci. 29(27), 8726–8733 (2009)CrossRefGoogle Scholar
  29. Ghazarian, A., Noorhosseini, S.M.: Automatic detection of users’ skill levels using high-frequency user interface events. User Model. User-Adapt. Interact. 20(2), 109–146 (2010)CrossRefGoogle Scholar
  30. Gingerich, M., Conati, C.: Constructing models of user and task characteristics from eye gaze data for user-adaptive information highlighting. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence. pp. 1728–1734. AAAI Press, Austin, TX (2015)Google Scholar
  31. Granholm, E., Steinhauer, S.R.: Pupillometric measures of cognitive and emotional processes. Int. J. Psychophysiol. 52(1), 1–6 (2004)CrossRefGoogle Scholar
  32. Green, T.M., Fisher, B.: Towards the personal equation of interaction: The impact of personality factors on visual analytics interface interaction. In: Proceedings of the 2010 IEEE Symposium on Visual Analytics Science and Technology. pp. 203–210. IEEE, Salt Lake City, UT (2010)Google Scholar
  33. Harrower, M., Brewer, C.A.: ColorBrewer.org: An online tool for selecting colour schemes for maps. Cartogr. J. 40(1), 27–37 (2003)CrossRefGoogle Scholar
  34. Hess, E.H., Polt, J.M.: Pupil size in relation to mental activity during simple problem-solving. Science 143, 1190–1192 (1964)CrossRefGoogle Scholar
  35. Holmes, G., Donkin, A., Witten, I.H.: Weka: A machine learning workbench. In: Second Australian and New Zealand Conference on Intelligent Information Systems. pp. 357–361 (1994)Google Scholar
  36. Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., van de Weijer, J.: Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford University Press, Oxford (2011)Google Scholar
  37. Huang, W., Eades, P., Hong, S.-H.: Measuring effectiveness of graph visualizations: A cognitive load perspective. Inf. Vis. 8(3), 139–152 (2009)CrossRefGoogle Scholar
  38. Huang, D., Tory, M., Aseniero, B.A., Bartram, L., Bateman, S., Carpendale, S., Tang, A., Woodbury, R.: Personal visualization and personal visual analytics. IEEE Trans. Vis. Comput. Graph. 21(3), 420–433 (2015)CrossRefGoogle Scholar
  39. Hullman, J., Adar, E., Shah, P.: Benefitting InfoVis with visual difficulties. IEEE Trans. Vis. Comput. Graph. 17(12), 2213–2222 (2011)CrossRefGoogle Scholar
  40. Hur, I., Kim, S.-H., Samak, A., Yi, J.S.: A comparative study of three sorting techniques in performing cognitive tasks on a tabular representation. Int. J. Human-Comput. Interact. 29(6), 379–390 (2013)CrossRefGoogle Scholar
  41. Hurst, A., Hudson, S.E., Mankoff, J.: Dynamic detection of novice vs. skilled use without a task model. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 271–280. ACM, San Jose, CA (2007)Google Scholar
  42. Iqbal, S.T., Adamczyk, P.D., Zheng, X.S., Brian P. Bailey: Towards an index of opportunity: Understanding changes in mental workload during task execution. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 311–320. ACM, Portland, OR (2005)Google Scholar
  43. Jain, M., Balakrishnan, R.: User learning and performance with bezel menus. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 2221–2230. ACM, Austin, TX (2012)Google Scholar
  44. Jaques, N., Conati, C., Harley, J.M., Azevedo, R.: Predicting Affect from gaze data during interaction with an intelligent tutoring system. In: Proceedings of the 12th International Conference on Intelligent Tutoring Systems. pp. 29–38. Springer, Honolulu, HI (2014)Google Scholar
  45. Kardan, S., Conati, C.: Comparing and combining eye gaze and interface actions for determining user learning with an interactive simulation. Proceedings on the 21th International Conference on User Modeling. Adaptation, and Personalization, pp. 215–227. Springer, Rome (2013)Google Scholar
  46. Kodagoda, N., Wong, B.L.W., Rooney, C., Khan, N.: Interactive visualization for low literacy users: From lessons learnt to design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 1159–1168. ACM, New York, NY (2012)Google Scholar
  47. Kong, N., Hearst, M.A., Agrawala, M.: Extracting references between text and charts via crowdsourcing. In: Proceedings of the SIGCHI conference on Human Factors in Computing Systems. pp. 31–40. ACM, Toronto (2014)Google Scholar
  48. Kruskal, W.: Relative importance by averaging over orderings. Am. Stat. 41(1), 6–10 (1987)zbMATHGoogle Scholar
  49. Lallé, S., Conati, C., Carenini, G.: Predicting confusion in information visualization from eye tracking and interaction data. In: Proceedings on the 25th International Joint Conference on Artificial Intelligence. pp. 2529–2535. AAAI Press, New York, NY (2016)Google Scholar
  50. Lallé, S., Mostow, J., Luengo, V., Guin, N.: Comparing student models in different formalisms by predicting their impact on help success. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education. pp. 161–170. Springer, Memphis, TN (2013)Google Scholar
  51. Lallé, S., Toker, D., Conati, C., Carenini, G.: Prediction of users’ learning curves for adaptation while using an information visualization. In: Proceedings of the 20th International Conference on Intelligent User Interfaces. pp. 357–368. ACM, Atlanta, GA (2015)Google Scholar
  52. Lee, B., Isenberg, P., Riche, N.H., Carpendale, S.: Beyond mouse and keyboard: Expanding design considerations for information visualization interactions. IEEE Trans. Vis. Comput. Graph. 18(12), 2689–2698 (2012)CrossRefGoogle Scholar
  53. Ma, J., Liao, I., Ma, K.-L., Frazier, J.: Living liquid: Design and evaluation of an exploratory visualization tool for museum visitors. IEEE Trans. Vis. Comput. Graph. 18(12), 2799–2808 (2012)CrossRefGoogle Scholar
  54. Martínez-Gómez, P., Aizawa, A.: Recognition of understanding level and language skill using measurements of reading behavior. In: Proceedings of the 19th International Conference on Intelligent User Interfaces. pp. 95–104. ACM, Haifa (2014)Google Scholar
  55. Ooms, K., De Maeyer, P., Fack, V.: Study of the attentive behavior of novice and expert map users using eye tracking. Cartogr. Geogr. Inf. Sci. 41(1), 37–54 (2014)CrossRefGoogle Scholar
  56. Paas, F., Tuovinen, J.E., Tabbers, H., Van Gerven, P.W.: Cognitive load measurement as a means to advance cognitive load theory. Educ. Psychol. 38(1), 63–71 (2003)CrossRefGoogle Scholar
  57. Partala, T., Surakka, V.: Pupil size variation as an indication of affective processing. Int. J. Human-Comput. Stud. 59(1), 185–198 (2003)CrossRefGoogle Scholar
  58. Pommeranz, A., Broekens, J., Wiggers, P., Brinkman, W.-P., Jonker, C.M.: Designing interfaces for explicit preference elicitation: A user-centered investigation of preference representation and elicitation process. User Model. User-Adapt. Interact. 22(4–5), 357–397 (2012)CrossRefGoogle Scholar
  59. Pousman, Z., Stasko, J., Mateas, M.: Casual information visualization: Depictions of data in everyday life. IEEE Trans. Vis. Comput. Graph. 13(6), 1145–1152 (2007)CrossRefGoogle Scholar
  60. Prendinger, H., Hyrskykari, A., Nakayama, M., Istance, H., Bee, N., Takahasi, Y.: Attentive interfaces for users with disabilities: Eye gaze for intention and uncertainty estimation. Univ. Access Inf. Soc. 8(4), 339–354 (2009)CrossRefGoogle Scholar
  61. Rotter, J.B.: Generalized expectancies for internal versus external control of reinforcement. Psychol. Monogr. Gen. Appl. 80(1), 1–28 (1966)CrossRefGoogle Scholar
  62. Ruchikachorn, P., Mueller, K.: Learning visualizations by analogy: Promoting visual literacy through visualization morphing. IEEE Trans. Vis. Comput. Graph. 21(9), 1028–1044 (2015)CrossRefGoogle Scholar
  63. Saraiya, P., North, C., Duca, K.: An insight-based methodology for evaluating bioinformatics visualizations. IEEE Trans. Vis. Comput. Graph. 11(4), 443–456 (2005)CrossRefGoogle Scholar
  64. Shneiderman, B., Plaisant, C.: Strategies for evaluating information visualization tools: Multi-dimensional in-depth long-term case studies. In: Proceedings of the 2006 AVI Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization. pp. 1–7. ACM, New York, NY (2006)Google Scholar
  65. Speelman, C.P., Kirsner, K.: Beyond the Learning Curve: The Construction of Mind. Oxford University Press, Oxford (2005)CrossRefGoogle Scholar
  66. Steichen, B., Conati, C., Carenini, G.: Inferring visualization task properties, user performance, and user cognitive abilities from eye gaze data. ACM Trans. Interact. Intell. Syst. 4(2), Article 11 (2014)Google Scholar
  67. Toker, D., Conati, C.: Eye tracking to understand user differences in visualization processing with highlighting interventions. In: Proceedings of the 22nd International Conference on User Modeling. Adaptation, and Personalization, pp. 219–230. Springer, Aalborg (2014)Google Scholar
  68. Toker, D., Conati, C., Carenini, G., Haraty, M.: Towards adaptive information visualization: On the influence of user characteristics. In: Proceedings of the 20th International Conference on User Modeling. Adaptation, and Personalization, pp. 274–285. Springer, Montréal (2012)Google Scholar
  69. Toker, D., Conati, C., Steichen, B., Carenini, G.: Individual user characteristics and information visualization: Connecting the dots through eye tracking. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 295–304. ACM, Paris (2013)Google Scholar
  70. Toker, D., Steichen, B., Gingerich, M., Conati, C., Carenini, G.: Towards facilitating user skill acquisition: Identifying Untrained visualization users through eye tracking. In: Proceedings of the 19th International Conference on Intelligent User Interfaces. pp. 105–114. ACM, Haifa (2014)Google Scholar
  71. Tominski, C., Forsell, C., Johansson, J.: Interaction support for visual comparison inspired by natural behavior. IEEE Trans. Vis. Comput. Graph. 18(12), 2719–2728 (2012)CrossRefGoogle Scholar
  72. Turner, M.L., Engle, R.W.: Is working memory capacity task dependent? J. Mem. Lang. 28(2), 127–154 (1989)CrossRefGoogle Scholar
  73. Yudelson, M.V., Koedinger, K.R., Gordon, G.J.: Individualized Bayesian knowledge tracing models. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education. pp. 171–180. Springer, Memphis, TN (2013)Google Scholar
  74. Zhou, J., Sun, J., Chen, F., Wang, Y., Taib, R., Khawaji, A., Li, Z.: Measurable decision making with GSR and pupillary analysis for intelligent user interface. ACM Trans. Computer-Hum. Interact. ToCHI 21(6), 33 (2015)Google Scholar
  75. Zhu, Y.: Measuring effective data visualization. In: Proceedings of the 3rd International Symposium on Advances in Visual Computing. pp. 652–661. Springer, Lake Tahoe, NV (2007)Google Scholar
  76. Ziemkiewicz, C., Kosara, R.: The shaping of information by visual metaphors. IEEE Trans. Vis. Comput. Graph. 14(6), 1269–1276 (2008)CrossRefGoogle Scholar
  77. Ziemkiewicz, C., Ottley, A., Crouser, R.J., Chauncey, K., Su, S.L., Chang, R.: Understanding visualization by understanding individual users. IEEE Comput. Graph. Appl. 32(6), 88–94 (2012)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  • Sébastien Lallé
    • 1
    Email author
  • Cristina Conati
    • 1
  • Giuseppe Carenini
    • 1
  1. 1.Department of Computer ScienceUniversity of British ColumbiaVancouverCanada

Personalised recommendations