Skip to main content
SpringerLink
Log in

Prediction of individual learning curves across information visualizations

  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

Confident usage of information visualizations is thought to be influenced by cognitive aspects as well as amount of exposure and training. To support the development of individual competency in visualization processing, it is important to ascertain if we can track users’ progress or difficulties they might have while working with a given visualization. In this paper, we extend previous work on predicting in real time a user’s learning curve—a mathematical model that can represent a user’s skill acquisition ability—when working with a visualization. First, we investigate whether results we previously obtained in predicting users’ learning curves during visualization processing generalize to a different visualization. Second, we study to what extent we can make predictions on a user’s learning curve without information on the visualization being used. Our models leverage various data sources, including a user’s gaze behavior, pupil dilation, and cognitive abilities. We show that these models outperform a baseline that leverages knowledge on user task performance so far. Our best performing model achieves good accuracies in predicting users’ learning curves even after observing users’ performance on a few tasks only. These results represent an important step toward understanding how to support users in learning a new visualization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. The standard tests are: for PS, the Kit of Fact or-Referenced Cognitive Tests-P3 (Ekstrom and Harman 1976); for Verbal WM, the OSPAN test (Turner and Engle 1989); for Visual WM, the Fukuda & Vogel’s test (Fukuda and Vogel 2009); for Locus of Control, the Rotter’s test (Rotter 1966).

  2. The post-questionnaires from the two studies are not used in this paper because each specifically targeted a different visualization, whereas our goal here is to investigate whether the proposed approach can generalize across different types of bar-chart-based visualizations.

  3. Video and demo: www.cs.ubc.ca/group/iui/VALUECHARTS.

  4. EMDAT: https://github.com/ATUAV/EMDAT.

  5. While other variants of regression might produce even better results, we only used backward stepwise linear regression in this work to keep the analysis simple, given that our main goal is to determine the feasibility of predicting learning curve in real time across our three datasets.

  6. We did not perform an extensive model selection process because our objective here is to provide a proof-of concept indication of accuracy, to practically qualify the performances reported via RMSE, as opposed to finding the model with the best possible accuracy.

  7. Throughout the paper statistical significance is reported at the .05 level after adjustments.

  8. For simplicity, we will denote this effect as “effect of generic AOI sets”, although this is a misnomer for the NoAOI set.

References

  • Akaike, H.: A new look at the statistical model identification. IEEE Trans. Autom. Control. 19(6), 716–723 (1974)

    Article  MathSciNet  MATH  Google Scholar 

  • Amar, R., Eagan, J., Stasko, J.: Low-level components of analytic activity in information visualization. In: Proceedings of the 2005 IEEE Symposium on Information Visualization. pp. 15–21. IEEE Computer Society, Washington, DC (2005)

  • Anderson, J.R.: Cognitive Skills and Their Acquisition. Psychology Press, Hove (1981)

    Google Scholar 

  • Anderson, J.R.: Rules of the Mind. Psychology Press, Hove (2014)

    Google Scholar 

  • Aoki, H., Hansen, J.P., Itoh, K.: Learning to interact with a computer by gaze. Behav. Inf. Technol. 27(4), 339–344 (2008)

    Article  Google Scholar 

  • Bailey, C.D., McIntyre, E.V.: The relation between fit and prediction for alternative forms of learning curves and relearning curves. IIE Trans. 29(6), 487–495 (1997)

    Google Scholar 

  • Bautista, J., Carenini, G.: An empirical evaluation of interactive visualizations for preferential choice. In: Proceedings of the Working Conference on Advanced Visual Interfaces. pp. 207–214. ACM, New York, NY (2008)

  • Beatty, J.: Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychol. Bull. 91(2), 276–292 (1982)

    Article  Google Scholar 

  • Beck, J.E., Sison, J.: Using knowledge tracing in a noisy environment to measure student reading proficiencies. Int. J. Artif. Intell. Educ. 16(2), 129–143 (2006)

    Google Scholar 

  • Bednarik, R., Eivazi, S., Vrzakova, H.: A computational approach for prediction of problem-solving behavior using support vector machines and eye-tracking data. In: Nakano, Y., Conati, C., Bader, T. (eds.) Eye Gaze in Intelligent User Interfaces, pp. 111–134. Springer, London (2013)

    Chapter  Google Scholar 

  • Bixler, R., Kopp, K., D’Mello, S.: Evaluation of a personalized method for proactive mind wandering reduction. In: Proceedings of the 4th Workshop on Personalization Approaches for Learning Environments. 22nd Conference on User Modeling, Adaptation, and Personalization, pp. 33–41. Springer, Aalborg (2014)

  • Bondareva, D., Conati, C., Feyzi-Behnagh, R., Harley, J.M., Azevedo, R., Bouchet, F.: Inferring learning from gaze data during interaction with an environment to support self-regulated learning. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education. pp. 229–238. Springer, Memphis, TN (2013)

  • Bradley, M.M., Miccoli, L., Escrig, M.A., Lang, P.J.: The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology 45(4), 602–607 (2008)

    Article  Google Scholar 

  • Bradshaw, J.: Pupil size as a measure of arousal during information processing. Nature 216, 515–516 (1967)

    Article  Google Scholar 

  • Bunt, A., Conati, C., McGrenere, J.: Supporting interface customization using a mixed-initiative approach. In: Proceedings of the 12th International Conference on Intelligent User Interfaces. pp. 92–101. ACM, Honolulu, HI (2007)

  • Carenini, G., Conati, C., Hoque, E., Steichen, B., Toker, D., Enns, J.T.: Highlighting interventions and user differences: Informing adaptive information visualization support. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 1835–1844. ACM, Toronto (2014)

  • Carenini, G., Loyd, J.: ValueCharts: Analyzing linear models expressing preferences and evaluations. In: Proceedings of the Working Conference on Advanced Visual Interfaces. pp. 150–157. ACM, Gallipoli (2004)

  • Chamberlain, B.C., Carenini, G., Oberg, G., Poole, D., Taheri, H.: A decision support system for the design and evaluation of sustainable wastewater solutions. IEEE Trans. Comput. 63(1), 129–141 (2014)

    Article  MathSciNet  Google Scholar 

  • Çöltekin, A., Fabrikant, S.I., Lacayo, M.: Exploring the efficiency of users’ visual analytics strategies based on sequence analysis of eye movement recordings. Int. J. Geogr. Inf. Sci. 24(10), 1559–1575 (2010)

    Article  Google Scholar 

  • Conati, C., Maclaren, H.: Exploring the role of individual differences in information visualization. In: Proceedings of the Working Conference on Advanced Visual Interfaces. pp. 199–206. ACM, New York, NY (2008)

  • Conati, C., Carenini, G., Steichen, B.: Dereck toker: Evaluating the impact of user characteristics and different layouts on an interactive visualization for decision making. Comput. Graph. Forum 33(3), 371–380 (2014)

    Article  Google Scholar 

  • Conati, C., Carenini, G., Toker, D., Lallé, S.: Towards user-adaptive information visualization. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence. pp. 4100–4106. AAAI Press, Austin, TX (2015)

  • D’Mello, S., Olney, A., Williams, C., Hays, P.: Gaze tutor: A gaze-reactive intelligent tutoring system. Int. J. Human-Comput. Stud. 70(5), 377–398 (2012)

    Article  Google Scholar 

  • Dork, M., Carpendale, S., Collins, C., Williamson, C.: VisGets: Coordinated visualizations for web-based information exploration and discovery. IEEE Trans. Vis. Comput. Graph. 14(6), 1205–1212 (2008)

    Article  Google Scholar 

  • Ekstrom, R.B., Harman, H.H.: Manual for Kit of Factor-Referenced Cognitive Tests. Educational Testing Service, Princeton (1976)

    Google Scholar 

  • Few, S.: Now you see it: Simple visualization techniques for quantitative analysis. Analytics Press, Oakland, CA (2009)

  • Fisher, D., Drucker, S.M., Fernandez, R., Ruble, S.: Visualizations everywhere: a multiplatform infrastructure for linked visualizations. IEEE Trans. Vis. Comput. Graph. 16(6), 1157–1163 (2010)

    Article  Google Scholar 

  • Fukuda, K., Vogel, E.K.: Human variation in overriding attentional capture. J. Neurosci. 29(27), 8726–8733 (2009)

    Article  Google Scholar 

  • Ghazarian, A., Noorhosseini, S.M.: Automatic detection of users’ skill levels using high-frequency user interface events. User Model. User-Adapt. Interact. 20(2), 109–146 (2010)

    Article  Google Scholar 

  • Gingerich, M., Conati, C.: Constructing models of user and task characteristics from eye gaze data for user-adaptive information highlighting. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence. pp. 1728–1734. AAAI Press, Austin, TX (2015)

  • Granholm, E., Steinhauer, S.R.: Pupillometric measures of cognitive and emotional processes. Int. J. Psychophysiol. 52(1), 1–6 (2004)

    Article  Google Scholar 

  • Green, T.M., Fisher, B.: Towards the personal equation of interaction: The impact of personality factors on visual analytics interface interaction. In: Proceedings of the 2010 IEEE Symposium on Visual Analytics Science and Technology. pp. 203–210. IEEE, Salt Lake City, UT (2010)

  • Harrower, M., Brewer, C.A.: ColorBrewer.org: An online tool for selecting colour schemes for maps. Cartogr. J. 40(1), 27–37 (2003)

    Article  Google Scholar 

  • Hess, E.H., Polt, J.M.: Pupil size in relation to mental activity during simple problem-solving. Science 143, 1190–1192 (1964)

    Article  Google Scholar 

  • Holmes, G., Donkin, A., Witten, I.H.: Weka: A machine learning workbench. In: Second Australian and New Zealand Conference on Intelligent Information Systems. pp. 357–361 (1994)

  • Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., van de Weijer, J.: Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford University Press, Oxford (2011)

    Google Scholar 

  • Huang, W., Eades, P., Hong, S.-H.: Measuring effectiveness of graph visualizations: A cognitive load perspective. Inf. Vis. 8(3), 139–152 (2009)

    Article  Google Scholar 

  • Huang, D., Tory, M., Aseniero, B.A., Bartram, L., Bateman, S., Carpendale, S., Tang, A., Woodbury, R.: Personal visualization and personal visual analytics. IEEE Trans. Vis. Comput. Graph. 21(3), 420–433 (2015)

    Article  Google Scholar 

  • Hullman, J., Adar, E., Shah, P.: Benefitting InfoVis with visual difficulties. IEEE Trans. Vis. Comput. Graph. 17(12), 2213–2222 (2011)

    Article  Google Scholar 

  • Hur, I., Kim, S.-H., Samak, A., Yi, J.S.: A comparative study of three sorting techniques in performing cognitive tasks on a tabular representation. Int. J. Human-Comput. Interact. 29(6), 379–390 (2013)

    Article  Google Scholar 

  • Hurst, A., Hudson, S.E., Mankoff, J.: Dynamic detection of novice vs. skilled use without a task model. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 271–280. ACM, San Jose, CA (2007)

  • Iqbal, S.T., Adamczyk, P.D., Zheng, X.S., Brian P. Bailey: Towards an index of opportunity: Understanding changes in mental workload during task execution. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 311–320. ACM, Portland, OR (2005)

  • Jain, M., Balakrishnan, R.: User learning and performance with bezel menus. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 2221–2230. ACM, Austin, TX (2012)

  • Jaques, N., Conati, C., Harley, J.M., Azevedo, R.: Predicting Affect from gaze data during interaction with an intelligent tutoring system. In: Proceedings of the 12th International Conference on Intelligent Tutoring Systems. pp. 29–38. Springer, Honolulu, HI (2014)

  • Kardan, S., Conati, C.: Comparing and combining eye gaze and interface actions for determining user learning with an interactive simulation. Proceedings on the 21th International Conference on User Modeling. Adaptation, and Personalization, pp. 215–227. Springer, Rome (2013)

  • Kodagoda, N., Wong, B.L.W., Rooney, C., Khan, N.: Interactive visualization for low literacy users: From lessons learnt to design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 1159–1168. ACM, New York, NY (2012)

  • Kong, N., Hearst, M.A., Agrawala, M.: Extracting references between text and charts via crowdsourcing. In: Proceedings of the SIGCHI conference on Human Factors in Computing Systems. pp. 31–40. ACM, Toronto (2014)

  • Kruskal, W.: Relative importance by averaging over orderings. Am. Stat. 41(1), 6–10 (1987)

    MATH  Google Scholar 

  • Lallé, S., Conati, C., Carenini, G.: Predicting confusion in information visualization from eye tracking and interaction data. In: Proceedings on the 25th International Joint Conference on Artificial Intelligence. pp. 2529–2535. AAAI Press, New York, NY (2016)

  • Lallé, S., Mostow, J., Luengo, V., Guin, N.: Comparing student models in different formalisms by predicting their impact on help success. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education. pp. 161–170. Springer, Memphis, TN (2013)

  • Lallé, S., Toker, D., Conati, C., Carenini, G.: Prediction of users’ learning curves for adaptation while using an information visualization. In: Proceedings of the 20th International Conference on Intelligent User Interfaces. pp. 357–368. ACM, Atlanta, GA (2015)

  • Lee, B., Isenberg, P., Riche, N.H., Carpendale, S.: Beyond mouse and keyboard: Expanding design considerations for information visualization interactions. IEEE Trans. Vis. Comput. Graph. 18(12), 2689–2698 (2012)

    Article  Google Scholar 

  • Ma, J., Liao, I., Ma, K.-L., Frazier, J.: Living liquid: Design and evaluation of an exploratory visualization tool for museum visitors. IEEE Trans. Vis. Comput. Graph. 18(12), 2799–2808 (2012)

    Article  Google Scholar 

  • Martínez-Gómez, P., Aizawa, A.: Recognition of understanding level and language skill using measurements of reading behavior. In: Proceedings of the 19th International Conference on Intelligent User Interfaces. pp. 95–104. ACM, Haifa (2014)

  • Ooms, K., De Maeyer, P., Fack, V.: Study of the attentive behavior of novice and expert map users using eye tracking. Cartogr. Geogr. Inf. Sci. 41(1), 37–54 (2014)

    Article  Google Scholar 

  • Paas, F., Tuovinen, J.E., Tabbers, H., Van Gerven, P.W.: Cognitive load measurement as a means to advance cognitive load theory. Educ. Psychol. 38(1), 63–71 (2003)

    Article  Google Scholar 

  • Partala, T., Surakka, V.: Pupil size variation as an indication of affective processing. Int. J. Human-Comput. Stud. 59(1), 185–198 (2003)

    Article  Google Scholar 

  • Pommeranz, A., Broekens, J., Wiggers, P., Brinkman, W.-P., Jonker, C.M.: Designing interfaces for explicit preference elicitation: A user-centered investigation of preference representation and elicitation process. User Model. User-Adapt. Interact. 22(4–5), 357–397 (2012)

    Article  Google Scholar 

  • Pousman, Z., Stasko, J., Mateas, M.: Casual information visualization: Depictions of data in everyday life. IEEE Trans. Vis. Comput. Graph. 13(6), 1145–1152 (2007)

    Article  Google Scholar 

  • Prendinger, H., Hyrskykari, A., Nakayama, M., Istance, H., Bee, N., Takahasi, Y.: Attentive interfaces for users with disabilities: Eye gaze for intention and uncertainty estimation. Univ. Access Inf. Soc. 8(4), 339–354 (2009)

    Article  Google Scholar 

  • Rotter, J.B.: Generalized expectancies for internal versus external control of reinforcement. Psychol. Monogr. Gen. Appl. 80(1), 1–28 (1966)

    Article  Google Scholar 

  • Ruchikachorn, P., Mueller, K.: Learning visualizations by analogy: Promoting visual literacy through visualization morphing. IEEE Trans. Vis. Comput. Graph. 21(9), 1028–1044 (2015)

    Article  Google Scholar 

  • Saraiya, P., North, C., Duca, K.: An insight-based methodology for evaluating bioinformatics visualizations. IEEE Trans. Vis. Comput. Graph. 11(4), 443–456 (2005)

    Article  Google Scholar 

  • Shneiderman, B., Plaisant, C.: Strategies for evaluating information visualization tools: Multi-dimensional in-depth long-term case studies. In: Proceedings of the 2006 AVI Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization. pp. 1–7. ACM, New York, NY (2006)

  • Speelman, C.P., Kirsner, K.: Beyond the Learning Curve: The Construction of Mind. Oxford University Press, Oxford (2005)

    Book  Google Scholar 

  • Steichen, B., Conati, C., Carenini, G.: Inferring visualization task properties, user performance, and user cognitive abilities from eye gaze data. ACM Trans. Interact. Intell. Syst. 4(2), Article 11 (2014)

  • Toker, D., Conati, C.: Eye tracking to understand user differences in visualization processing with highlighting interventions. In: Proceedings of the 22nd International Conference on User Modeling. Adaptation, and Personalization, pp. 219–230. Springer, Aalborg (2014)

  • Toker, D., Conati, C., Carenini, G., Haraty, M.: Towards adaptive information visualization: On the influence of user characteristics. In: Proceedings of the 20th International Conference on User Modeling. Adaptation, and Personalization, pp. 274–285. Springer, Montréal (2012)

  • Toker, D., Conati, C., Steichen, B., Carenini, G.: Individual user characteristics and information visualization: Connecting the dots through eye tracking. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 295–304. ACM, Paris (2013)

  • Toker, D., Steichen, B., Gingerich, M., Conati, C., Carenini, G.: Towards facilitating user skill acquisition: Identifying Untrained visualization users through eye tracking. In: Proceedings of the 19th International Conference on Intelligent User Interfaces. pp. 105–114. ACM, Haifa (2014)

  • Tominski, C., Forsell, C., Johansson, J.: Interaction support for visual comparison inspired by natural behavior. IEEE Trans. Vis. Comput. Graph. 18(12), 2719–2728 (2012)

    Article  Google Scholar 

  • Turner, M.L., Engle, R.W.: Is working memory capacity task dependent? J. Mem. Lang. 28(2), 127–154 (1989)

    Article  Google Scholar 

  • Yudelson, M.V., Koedinger, K.R., Gordon, G.J.: Individualized Bayesian knowledge tracing models. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education. pp. 171–180. Springer, Memphis, TN (2013)

  • Zhou, J., Sun, J., Chen, F., Wang, Y., Taib, R., Khawaji, A., Li, Z.: Measurable decision making with GSR and pupillary analysis for intelligent user interface. ACM Trans. Computer-Hum. Interact. ToCHI 21(6), 33 (2015)

    Google Scholar 

  • Zhu, Y.: Measuring effective data visualization. In: Proceedings of the 3rd International Symposium on Advances in Visual Computing. pp. 652–661. Springer, Lake Tahoe, NV (2007)

  • Ziemkiewicz, C., Kosara, R.: The shaping of information by visual metaphors. IEEE Trans. Vis. Comput. Graph. 14(6), 1269–1276 (2008)

    Article  Google Scholar 

  • Ziemkiewicz, C., Ottley, A., Crouser, R.J., Chauncey, K., Su, S.L., Chang, R.: Understanding visualization by understanding individual users. IEEE Comput. Graph. Appl. 32(6), 88–94 (2012)

    Article  Google Scholar 

Download references

Acknowledgments

This publication is based upon work supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), under Grant No. STPG381322-09. We also thank Dereck Toker for his help in revising the paper.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of British Columbia, 2366 Main Mall, Vancouver, BC, V6T 1Z4, Canada

    Sébastien Lallé, Cristina Conati & Giuseppe Carenini

Authors
  1. Sébastien Lallé
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cristina Conati
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Giuseppe Carenini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sébastien Lallé.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lallé, S., Conati, C. & Carenini, G. Prediction of individual learning curves across information visualizations. User Model User-Adap Inter 26, 307–345 (2016). https://doi.org/10.1007/s11257-016-9179-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-016-9179-5

Keywords

Search

Navigation