Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Prediction of individual learning curves across information visualizations

Abstract

Confident usage of information visualizations is thought to be influenced by cognitive aspects as well as amount of exposure and training. To support the development of individual competency in visualization processing, it is important to ascertain if we can track users’ progress or difficulties they might have while working with a given visualization. In this paper, we extend previous work on predicting in real time a user’s learning curve—a mathematical model that can represent a user’s skill acquisition ability—when working with a visualization. First, we investigate whether results we previously obtained in predicting users’ learning curves during visualization processing generalize to a different visualization. Second, we study to what extent we can make predictions on a user’s learning curve without information on the visualization being used. Our models leverage various data sources, including a user’s gaze behavior, pupil dilation, and cognitive abilities. We show that these models outperform a baseline that leverages knowledge on user task performance so far. Our best performing model achieves good accuracies in predicting users’ learning curves even after observing users’ performance on a few tasks only. These results represent an important step toward understanding how to support users in learning a new visualization.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Notes

  1. 1.

    The standard tests are: for PS, the Kit of Fact or-Referenced Cognitive Tests-P3 (Ekstrom and Harman 1976); for Verbal WM, the OSPAN test (Turner and Engle 1989); for Visual WM, the Fukuda & Vogel’s test (Fukuda and Vogel 2009); for Locus of Control, the Rotter’s test (Rotter 1966).

  2. 2.

    The post-questionnaires from the two studies are not used in this paper because each specifically targeted a different visualization, whereas our goal here is to investigate whether the proposed approach can generalize across different types of bar-chart-based visualizations.

  3. 3.

    Video and demo: www.cs.ubc.ca/group/iui/VALUECHARTS.

  4. 4.

    EMDAT: https://github.com/ATUAV/EMDAT.

  5. 5.

    While other variants of regression might produce even better results, we only used backward stepwise linear regression in this work to keep the analysis simple, given that our main goal is to determine the feasibility of predicting learning curve in real time across our three datasets.

  6. 6.

    We did not perform an extensive model selection process because our objective here is to provide a proof-of concept indication of accuracy, to practically qualify the performances reported via RMSE, as opposed to finding the model with the best possible accuracy.

  7. 7.

    Throughout the paper statistical significance is reported at the .05 level after adjustments.

  8. 8.

    For simplicity, we will denote this effect as “effect of generic AOI sets”, although this is a misnomer for the NoAOI set.

References

  1. Akaike, H.: A new look at the statistical model identification. IEEE Trans. Autom. Control. 19(6), 716–723 (1974)

  2. Amar, R., Eagan, J., Stasko, J.: Low-level components of analytic activity in information visualization. In: Proceedings of the 2005 IEEE Symposium on Information Visualization. pp. 15–21. IEEE Computer Society, Washington, DC (2005)

  3. Anderson, J.R.: Cognitive Skills and Their Acquisition. Psychology Press, Hove (1981)

  4. Anderson, J.R.: Rules of the Mind. Psychology Press, Hove (2014)

  5. Aoki, H., Hansen, J.P., Itoh, K.: Learning to interact with a computer by gaze. Behav. Inf. Technol. 27(4), 339–344 (2008)

  6. Bailey, C.D., McIntyre, E.V.: The relation between fit and prediction for alternative forms of learning curves and relearning curves. IIE Trans. 29(6), 487–495 (1997)

  7. Bautista, J., Carenini, G.: An empirical evaluation of interactive visualizations for preferential choice. In: Proceedings of the Working Conference on Advanced Visual Interfaces. pp. 207–214. ACM, New York, NY (2008)

  8. Beatty, J.: Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychol. Bull. 91(2), 276–292 (1982)

  9. Beck, J.E., Sison, J.: Using knowledge tracing in a noisy environment to measure student reading proficiencies. Int. J. Artif. Intell. Educ. 16(2), 129–143 (2006)

  10. Bednarik, R., Eivazi, S., Vrzakova, H.: A computational approach for prediction of problem-solving behavior using support vector machines and eye-tracking data. In: Nakano, Y., Conati, C., Bader, T. (eds.) Eye Gaze in Intelligent User Interfaces, pp. 111–134. Springer, London (2013)

  11. Bixler, R., Kopp, K., D’Mello, S.: Evaluation of a personalized method for proactive mind wandering reduction. In: Proceedings of the 4th Workshop on Personalization Approaches for Learning Environments. 22nd Conference on User Modeling, Adaptation, and Personalization, pp. 33–41. Springer, Aalborg (2014)

  12. Bondareva, D., Conati, C., Feyzi-Behnagh, R., Harley, J.M., Azevedo, R., Bouchet, F.: Inferring learning from gaze data during interaction with an environment to support self-regulated learning. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education. pp. 229–238. Springer, Memphis, TN (2013)

  13. Bradley, M.M., Miccoli, L., Escrig, M.A., Lang, P.J.: The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology 45(4), 602–607 (2008)

  14. Bradshaw, J.: Pupil size as a measure of arousal during information processing. Nature 216, 515–516 (1967)

  15. Bunt, A., Conati, C., McGrenere, J.: Supporting interface customization using a mixed-initiative approach. In: Proceedings of the 12th International Conference on Intelligent User Interfaces. pp. 92–101. ACM, Honolulu, HI (2007)

  16. Carenini, G., Conati, C., Hoque, E., Steichen, B., Toker, D., Enns, J.T.: Highlighting interventions and user differences: Informing adaptive information visualization support. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 1835–1844. ACM, Toronto (2014)

  17. Carenini, G., Loyd, J.: ValueCharts: Analyzing linear models expressing preferences and evaluations. In: Proceedings of the Working Conference on Advanced Visual Interfaces. pp. 150–157. ACM, Gallipoli (2004)

  18. Chamberlain, B.C., Carenini, G., Oberg, G., Poole, D., Taheri, H.: A decision support system for the design and evaluation of sustainable wastewater solutions. IEEE Trans. Comput. 63(1), 129–141 (2014)

  19. Çöltekin, A., Fabrikant, S.I., Lacayo, M.: Exploring the efficiency of users’ visual analytics strategies based on sequence analysis of eye movement recordings. Int. J. Geogr. Inf. Sci. 24(10), 1559–1575 (2010)

  20. Conati, C., Maclaren, H.: Exploring the role of individual differences in information visualization. In: Proceedings of the Working Conference on Advanced Visual Interfaces. pp. 199–206. ACM, New York, NY (2008)

  21. Conati, C., Carenini, G., Steichen, B.: Dereck toker: Evaluating the impact of user characteristics and different layouts on an interactive visualization for decision making. Comput. Graph. Forum 33(3), 371–380 (2014)

  22. Conati, C., Carenini, G., Toker, D., Lallé, S.: Towards user-adaptive information visualization. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence. pp. 4100–4106. AAAI Press, Austin, TX (2015)

  23. D’Mello, S., Olney, A., Williams, C., Hays, P.: Gaze tutor: A gaze-reactive intelligent tutoring system. Int. J. Human-Comput. Stud. 70(5), 377–398 (2012)

  24. Dork, M., Carpendale, S., Collins, C., Williamson, C.: VisGets: Coordinated visualizations for web-based information exploration and discovery. IEEE Trans. Vis. Comput. Graph. 14(6), 1205–1212 (2008)

  25. Ekstrom, R.B., Harman, H.H.: Manual for Kit of Factor-Referenced Cognitive Tests. Educational Testing Service, Princeton (1976)

  26. Few, S.: Now you see it: Simple visualization techniques for quantitative analysis. Analytics Press, Oakland, CA (2009)

  27. Fisher, D., Drucker, S.M., Fernandez, R., Ruble, S.: Visualizations everywhere: a multiplatform infrastructure for linked visualizations. IEEE Trans. Vis. Comput. Graph. 16(6), 1157–1163 (2010)

  28. Fukuda, K., Vogel, E.K.: Human variation in overriding attentional capture. J. Neurosci. 29(27), 8726–8733 (2009)

  29. Ghazarian, A., Noorhosseini, S.M.: Automatic detection of users’ skill levels using high-frequency user interface events. User Model. User-Adapt. Interact. 20(2), 109–146 (2010)

  30. Gingerich, M., Conati, C.: Constructing models of user and task characteristics from eye gaze data for user-adaptive information highlighting. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence. pp. 1728–1734. AAAI Press, Austin, TX (2015)

  31. Granholm, E., Steinhauer, S.R.: Pupillometric measures of cognitive and emotional processes. Int. J. Psychophysiol. 52(1), 1–6 (2004)

  32. Green, T.M., Fisher, B.: Towards the personal equation of interaction: The impact of personality factors on visual analytics interface interaction. In: Proceedings of the 2010 IEEE Symposium on Visual Analytics Science and Technology. pp. 203–210. IEEE, Salt Lake City, UT (2010)

  33. Harrower, M., Brewer, C.A.: ColorBrewer.org: An online tool for selecting colour schemes for maps. Cartogr. J. 40(1), 27–37 (2003)

  34. Hess, E.H., Polt, J.M.: Pupil size in relation to mental activity during simple problem-solving. Science 143, 1190–1192 (1964)

  35. Holmes, G., Donkin, A., Witten, I.H.: Weka: A machine learning workbench. In: Second Australian and New Zealand Conference on Intelligent Information Systems. pp. 357–361 (1994)

  36. Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., van de Weijer, J.: Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford University Press, Oxford (2011)

  37. Huang, W., Eades, P., Hong, S.-H.: Measuring effectiveness of graph visualizations: A cognitive load perspective. Inf. Vis. 8(3), 139–152 (2009)

  38. Huang, D., Tory, M., Aseniero, B.A., Bartram, L., Bateman, S., Carpendale, S., Tang, A., Woodbury, R.: Personal visualization and personal visual analytics. IEEE Trans. Vis. Comput. Graph. 21(3), 420–433 (2015)

  39. Hullman, J., Adar, E., Shah, P.: Benefitting InfoVis with visual difficulties. IEEE Trans. Vis. Comput. Graph. 17(12), 2213–2222 (2011)

  40. Hur, I., Kim, S.-H., Samak, A., Yi, J.S.: A comparative study of three sorting techniques in performing cognitive tasks on a tabular representation. Int. J. Human-Comput. Interact. 29(6), 379–390 (2013)

  41. Hurst, A., Hudson, S.E., Mankoff, J.: Dynamic detection of novice vs. skilled use without a task model. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 271–280. ACM, San Jose, CA (2007)

  42. Iqbal, S.T., Adamczyk, P.D., Zheng, X.S., Brian P. Bailey: Towards an index of opportunity: Understanding changes in mental workload during task execution. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 311–320. ACM, Portland, OR (2005)

  43. Jain, M., Balakrishnan, R.: User learning and performance with bezel menus. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 2221–2230. ACM, Austin, TX (2012)

  44. Jaques, N., Conati, C., Harley, J.M., Azevedo, R.: Predicting Affect from gaze data during interaction with an intelligent tutoring system. In: Proceedings of the 12th International Conference on Intelligent Tutoring Systems. pp. 29–38. Springer, Honolulu, HI (2014)

  45. Kardan, S., Conati, C.: Comparing and combining eye gaze and interface actions for determining user learning with an interactive simulation. Proceedings on the 21th International Conference on User Modeling. Adaptation, and Personalization, pp. 215–227. Springer, Rome (2013)

  46. Kodagoda, N., Wong, B.L.W., Rooney, C., Khan, N.: Interactive visualization for low literacy users: From lessons learnt to design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 1159–1168. ACM, New York, NY (2012)

  47. Kong, N., Hearst, M.A., Agrawala, M.: Extracting references between text and charts via crowdsourcing. In: Proceedings of the SIGCHI conference on Human Factors in Computing Systems. pp. 31–40. ACM, Toronto (2014)

  48. Kruskal, W.: Relative importance by averaging over orderings. Am. Stat. 41(1), 6–10 (1987)

  49. Lallé, S., Conati, C., Carenini, G.: Predicting confusion in information visualization from eye tracking and interaction data. In: Proceedings on the 25th International Joint Conference on Artificial Intelligence. pp. 2529–2535. AAAI Press, New York, NY (2016)

  50. Lallé, S., Mostow, J., Luengo, V., Guin, N.: Comparing student models in different formalisms by predicting their impact on help success. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education. pp. 161–170. Springer, Memphis, TN (2013)

  51. Lallé, S., Toker, D., Conati, C., Carenini, G.: Prediction of users’ learning curves for adaptation while using an information visualization. In: Proceedings of the 20th International Conference on Intelligent User Interfaces. pp. 357–368. ACM, Atlanta, GA (2015)

  52. Lee, B., Isenberg, P., Riche, N.H., Carpendale, S.: Beyond mouse and keyboard: Expanding design considerations for information visualization interactions. IEEE Trans. Vis. Comput. Graph. 18(12), 2689–2698 (2012)

  53. Ma, J., Liao, I., Ma, K.-L., Frazier, J.: Living liquid: Design and evaluation of an exploratory visualization tool for museum visitors. IEEE Trans. Vis. Comput. Graph. 18(12), 2799–2808 (2012)

  54. Martínez-Gómez, P., Aizawa, A.: Recognition of understanding level and language skill using measurements of reading behavior. In: Proceedings of the 19th International Conference on Intelligent User Interfaces. pp. 95–104. ACM, Haifa (2014)

  55. Ooms, K., De Maeyer, P., Fack, V.: Study of the attentive behavior of novice and expert map users using eye tracking. Cartogr. Geogr. Inf. Sci. 41(1), 37–54 (2014)

  56. Paas, F., Tuovinen, J.E., Tabbers, H., Van Gerven, P.W.: Cognitive load measurement as a means to advance cognitive load theory. Educ. Psychol. 38(1), 63–71 (2003)

  57. Partala, T., Surakka, V.: Pupil size variation as an indication of affective processing. Int. J. Human-Comput. Stud. 59(1), 185–198 (2003)

  58. Pommeranz, A., Broekens, J., Wiggers, P., Brinkman, W.-P., Jonker, C.M.: Designing interfaces for explicit preference elicitation: A user-centered investigation of preference representation and elicitation process. User Model. User-Adapt. Interact. 22(4–5), 357–397 (2012)

  59. Pousman, Z., Stasko, J., Mateas, M.: Casual information visualization: Depictions of data in everyday life. IEEE Trans. Vis. Comput. Graph. 13(6), 1145–1152 (2007)

  60. Prendinger, H., Hyrskykari, A., Nakayama, M., Istance, H., Bee, N., Takahasi, Y.: Attentive interfaces for users with disabilities: Eye gaze for intention and uncertainty estimation. Univ. Access Inf. Soc. 8(4), 339–354 (2009)

  61. Rotter, J.B.: Generalized expectancies for internal versus external control of reinforcement. Psychol. Monogr. Gen. Appl. 80(1), 1–28 (1966)

  62. Ruchikachorn, P., Mueller, K.: Learning visualizations by analogy: Promoting visual literacy through visualization morphing. IEEE Trans. Vis. Comput. Graph. 21(9), 1028–1044 (2015)

  63. Saraiya, P., North, C., Duca, K.: An insight-based methodology for evaluating bioinformatics visualizations. IEEE Trans. Vis. Comput. Graph. 11(4), 443–456 (2005)

  64. Shneiderman, B., Plaisant, C.: Strategies for evaluating information visualization tools: Multi-dimensional in-depth long-term case studies. In: Proceedings of the 2006 AVI Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization. pp. 1–7. ACM, New York, NY (2006)

  65. Speelman, C.P., Kirsner, K.: Beyond the Learning Curve: The Construction of Mind. Oxford University Press, Oxford (2005)

  66. Steichen, B., Conati, C., Carenini, G.: Inferring visualization task properties, user performance, and user cognitive abilities from eye gaze data. ACM Trans. Interact. Intell. Syst. 4(2), Article 11 (2014)

  67. Toker, D., Conati, C.: Eye tracking to understand user differences in visualization processing with highlighting interventions. In: Proceedings of the 22nd International Conference on User Modeling. Adaptation, and Personalization, pp. 219–230. Springer, Aalborg (2014)

  68. Toker, D., Conati, C., Carenini, G., Haraty, M.: Towards adaptive information visualization: On the influence of user characteristics. In: Proceedings of the 20th International Conference on User Modeling. Adaptation, and Personalization, pp. 274–285. Springer, Montréal (2012)

  69. Toker, D., Conati, C., Steichen, B., Carenini, G.: Individual user characteristics and information visualization: Connecting the dots through eye tracking. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 295–304. ACM, Paris (2013)

  70. Toker, D., Steichen, B., Gingerich, M., Conati, C., Carenini, G.: Towards facilitating user skill acquisition: Identifying Untrained visualization users through eye tracking. In: Proceedings of the 19th International Conference on Intelligent User Interfaces. pp. 105–114. ACM, Haifa (2014)

  71. Tominski, C., Forsell, C., Johansson, J.: Interaction support for visual comparison inspired by natural behavior. IEEE Trans. Vis. Comput. Graph. 18(12), 2719–2728 (2012)

  72. Turner, M.L., Engle, R.W.: Is working memory capacity task dependent? J. Mem. Lang. 28(2), 127–154 (1989)

  73. Yudelson, M.V., Koedinger, K.R., Gordon, G.J.: Individualized Bayesian knowledge tracing models. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education. pp. 171–180. Springer, Memphis, TN (2013)

  74. Zhou, J., Sun, J., Chen, F., Wang, Y., Taib, R., Khawaji, A., Li, Z.: Measurable decision making with GSR and pupillary analysis for intelligent user interface. ACM Trans. Computer-Hum. Interact. ToCHI 21(6), 33 (2015)

  75. Zhu, Y.: Measuring effective data visualization. In: Proceedings of the 3rd International Symposium on Advances in Visual Computing. pp. 652–661. Springer, Lake Tahoe, NV (2007)

  76. Ziemkiewicz, C., Kosara, R.: The shaping of information by visual metaphors. IEEE Trans. Vis. Comput. Graph. 14(6), 1269–1276 (2008)

  77. Ziemkiewicz, C., Ottley, A., Crouser, R.J., Chauncey, K., Su, S.L., Chang, R.: Understanding visualization by understanding individual users. IEEE Comput. Graph. Appl. 32(6), 88–94 (2012)

Download references

Acknowledgments

This publication is based upon work supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), under Grant No. STPG381322-09. We also thank Dereck Toker for his help in revising the paper.

Author information

Correspondence to Sébastien Lallé.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lallé, S., Conati, C. & Carenini, G. Prediction of individual learning curves across information visualizations. User Model User-Adap Inter 26, 307–345 (2016). https://doi.org/10.1007/s11257-016-9179-5

Download citation

Keywords

  • Information visualization
  • Adaptive visualization
  • User modeling
  • Machine learning
  • Eye tracking
  • Learning curve