User Modeling and User-Adapted Interaction

, Volume 27, Issue 1, pp 55–88

Enhancing learning outcomes through self-regulated learning support with an Open Learner Model



Open Learner Models (OLMs) have great potential to support students’ Self-Regulated Learning (SRL) in Intelligent Tutoring Systems (ITSs). Yet few classroom experiments have been conducted to empirically evaluate whether and how an OLM can enhance students’ domain level learning outcomes through the scaffolding of SRL processes in an ITS. In two classroom experiments with a total of 302 7th- and 8th-grade students, we investigated the effect of (a) an OLM that supports students’ self-assessment of their equation-solving skills and (b) shared control over problem selection, on students’ equation-solving abilities, enjoyment of learning with the tutor, self-assessment accuracy, and problem selection decisions. In the first, smaller experiment, the hypothesized main effect of the OLM on students’ learning outcomes was confirmed; we found no main effect of shared control of problem selection, nor an interaction. In the second, larger experiment, the hypothesized main effects were not confirmed, but we found an interaction such that the students who had access to the OLM learned significantly better equation-solving skills than their counterparts when shared control over problem selection was offered in the system. Thus, the two experiments support the notion that an OLM can enhance students’ domain-level learning outcomes through scaffolding of SRL processes, and are among the first in-vivo classroom experiments to do so. They suggest that an OLM is especially effective if it is designed to support multiple SRL processes.


Open Learner Model Self-assessment Making problem selection decisions Intelligent tutoring system Learner control Self-regulated learning Classroom experiment 


  1. Aleven, V.: Rule-based cognitive modeling for intelligent tutoring systems. Advances in intelligent tutoring systems. In: Nkambou, R., Bourdeau, J., Mizoguchi, R. (eds.) Studies in Computational Intelligence, vol. 308, pp. 33–62. Springer, Berlin (2010)Google Scholar
  2. Aleven, V., Koedinger, K.R.: Limitations of student control: do students know when they need help? In: Gauthier, G., Frasson, C., VanLehn, K. (eds.) Proceedings of the 5th International Conference on Intelligent Tutoring Systems, pp. 292–303. Springer, Berlin (2000)CrossRefGoogle Scholar
  3. Aleven, V., Koedinger, K.R.: Knowledge component approaches to learner modeling. In: Sottilare, R., Graesser, A., Hu, X., Holden, H. (eds.) Design Recommendations for Adaptive Intelligent Tutoring Systems, pp. 165–182. Orlando, US Army Research Laboratory (2013)Google Scholar
  4. Aleven, V., McLaren, B.M., Sewall, J., Koedinger, K.R.: Example-tracing tutors: a new paradigm for intelligent tutoring systems. Int. J. Artif. Intell. Educ. 19(2), 105–154 (2009)Google Scholar
  5. Aleven, V., McLaren, B.M., Sewall, J., van Velsen, M., Popescu, O., Demi, S., Koedinger, K.R.: Example-tracing tutors: intelligent tutor development for non-programmers. Int. J. Artif. Intell. Educ. 26(1), 224–269 (2016)CrossRefGoogle Scholar
  6. Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive tutors: lessons learned. J. Learn. Sci. 4(2), 167–207 (1995)CrossRefGoogle Scholar
  7. Atkinson, R.C.: Optimizing the learning of a second-language vocabulary. J. Exp. Psychol. 96(1), 124–129 (1972)CrossRefGoogle Scholar
  8. Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Mehranian, H., Fisher, D., Barto, A., Mahadevan, S., Woolf, B.: Repairing disengagement with non invasive interventions. In: Proceedings of the International Conference on Artificial Intelligence in Education, pp. 195–202. Marina del Rey, CA (2007)Google Scholar
  9. Azevedo, R., Aleven, V. (eds.): International Handbook on Metacognition in Computer-Based Learning Environments. Springer, Berlin, DE (2013)Google Scholar
  10. Bandura, A.: Self-efficacy. In: Ramachaudran, V.S. (Ed.) Encyclopedia of Human Behaviour, vol. 4, pp. 71–81. Academic Press, New York (Reprinted in H. Friedman (Ed.). Encyclopedia of mental health, San Diego: Academic Press, 1998) (1994)Google Scholar
  11. Basu, S., Biswas, G., Kinnebrew, J.S.: Learner modeling for adaptive scaffolding in a computational thinking-based science learning environment. User Model. User-Adapt. Interact. J. Personal. Res. 27 (2017) (Special Issue on Impact of Learner Modeling)Google Scholar
  12. Bjork, R.A., Bjork, E.L.: Optimizing treatment and instruction: implications of a new theory of disuse. In: Nilsson, L.G., Ohta, N. (eds.) Memory and Society: Psychological Perspectives, pp. 109–133. Psychology Press, New York (2006)Google Scholar
  13. Brusilovsky, P.: Adaptive navigation support: from adaptive hypermedia to the adaptive web and beyond. Psychol. J. 2(1), 7–23 (2004)Google Scholar
  14. Brusilovsky, P., Schwarz, E., Weber, G.: ELM-ART: an intelligent tutoring system on World Wide Web. In: Frasson, C., Gauthier, G., Lesgold, A. (eds.) Proceedings of Third International Conference on Intelligent Tutoring Systems, ITS-96, pp. 261–269. Springer, Berlin (1996)CrossRefGoogle Scholar
  15. Brusilovsky, P., Sosnovsky, S., Shcherbinina, O.: QuizGuide: Increasing the educational value of individualized self-assessment quizzes with adaptive navigation support. In: Nall, J., Robson, R. (eds.) Proceedings of World Conference on E-Learning, pp. 1806–1813 (2004)Google Scholar
  16. Brusilovsky, P., Somyürek, S., Guerra, J., Hosseini, R. Zadorozhny, V.: The value of social: comparing open student modeling and open social student modeling. In: Proceedings of the 23nd Conference on User Modeling, Adaptation and Personalization (UMAP 2015), June 29–July 3, 2015, Dublin, Ireland (2015)Google Scholar
  17. Bull, S.: Supporting learning with open learner models. In: Proceedings of the 4th Hellenic Conference: Information and Communication Technologies in Education, Athens (2004)Google Scholar
  18. Bull, S., Kay, J.: Metacognition and open learner models. In: Roll, I., Aleven, V. (eds.) Proceedings of Workshop on Metacognition and Self-Regulated Learning in Educational Technologies, International Conference on Intelligent Tutoring Systems, pp. 7–20 (2008)Google Scholar
  19. Bull, S., Kay, J.: Open learner models. Advances in intelligent tutoring systems. In: Nkambou, R., Bourdeau, J., Mizoguchi, R. (eds.) Studies in Computational Intelligence, vol. 308, pp. 301–322. Springer, Berlin (2010)Google Scholar
  20. Bull, S., Kay, J.: SMILI: a framework for interfaces to learning data in Open Learner Models (OLMs), learning analytics and related fields. Int. J. Artif. Intell. Educ. 26(1), 293–331 (2016)CrossRefGoogle Scholar
  21. Bull, S., Jackson, T., Lancaster, M.: Students’ interest in their misconceptions in first year electrical circuits and mathematics courses. Int. J. Electr. Eng. Educ. 47(3), 307–318 (2010)CrossRefGoogle Scholar
  22. Bull, S., Ginon, B., Boscolo, C., Johnson, M.D.: Introduction of learning visualisations and metacognitive support in a persuadable Open Learner Model. In: Gasevic, D., Lynch, G. (eds.) Proceeding of Learning Analytics and Knowledge 2016. ACM (2016)Google Scholar
  23. Clark, C.R., Mayer, E.R.: E-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning. Jossey-Bass, San Francisco (2011)CrossRefGoogle Scholar
  24. Cleary, T., Zimmerman, B.J.: Self-regulation differences during athletic practice by experts, nonexperts, and novices. J. Appl. Sport Psychol. 13, 61–82 (2000)Google Scholar
  25. Corbalan, G., Kester, L., Van Merriënboer, J.J.G.: Selecting learning tasks: effects of adaptation and shared control on efficiency and task involvement. Contemp. Educ. Psychol. 33(4), 733–756 (2008)CrossRefGoogle Scholar
  26. Corbalan, G., Kester, L., van Merriënboer, J.J.G.: Combining shared control with variability over surface features: effects on transfer test performance and task involvement. Comput. Hum. Behav. 25(2), 290–298 (2009)CrossRefGoogle Scholar
  27. Corbett, A.: Cognitive mastery learning in the ACT Programming Tutor. AAAI Technical Report SS-00-01 (2000)Google Scholar
  28. Corbett, A.T., Anderson, J.R.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Model. User-Adapt. Interact. 4(4), 253–278 (1995)CrossRefGoogle Scholar
  29. Corbett, A.T., Bhatnagar, A.: Student modeling in the ACT programming tutor: adjusting a procedural learning model with declarative knowledge. In: Jameson, A., Paris, C., Tasso, C. (eds.) User Modeling. Springer, New York (1997)Google Scholar
  30. Cordova, D.I., Lepper, M.R.: Intrinsic motivation and the process of learning: beneficial effects of contextualization, personalization, and choice. J. Educ. Psychol. 88, 715–730 (1996)CrossRefGoogle Scholar
  31. Desmarais, M.C., Baker, R.S.: A review of recent advances in learner and skill modeling in intelligent learning environments. User Model. User-Adapt. Interact. 22(1–2), 9–38 (2012)CrossRefGoogle Scholar
  32. Dimitrova, V., Brna, P.: From interactive open learner modeling to intelligent mentoring: STyLE-OLM and beyond. Int. J. Artif. Intell. Educ. 26(1), 332–349 (2016)CrossRefGoogle Scholar
  33. Dunlosky, J., Lipko, A.: Metacomprehension: a brief history and how to improve its accuracy. Curr. Dir. Psychol. Sci. 16, 228–232 (2007)CrossRefGoogle Scholar
  34. Flowerday, T., Schraw, G.: Effect of choice on cognitive and affective engagement. J. Educ. Res. 96(4), 207–215 (2003)CrossRefGoogle Scholar
  35. Gong, Y., Beck, J.E., Heffernan, N.T.: How to construct more accurate student models: comparing and optimizing knowledge tracing and performance factor analysis. Int. J. Artif. Intell. Educ. 21(1–2), 27–46 (2011)Google Scholar
  36. Hartley, D., Mitrovic, A.: Supporting learning by opening the student model. In: Cerri, S.A., Gouardères, G., Paraguaçu, F. (eds.) Proceedings of the 6th International Conference on Intelligent Tutoring Systems, pp. 453–462. Springer, Berlin (2002)CrossRefGoogle Scholar
  37. Jameson, A., Schwarzkopf, E.: Pros and cons of controllability: an empirical study. In: De Bra, P., Brusilovsky, P., Conejo, R. (eds.) Adaptive Hypermedia and Adaptive Webbased Systems: Proceedings of AH 2002, pp. 193–202. Springer, Berlin (2002)CrossRefGoogle Scholar
  38. Kapur, M., Bielaczyc, K.: Designing for productive failure. J. Learn. Sci. 21(1), 45–83 (2012)CrossRefGoogle Scholar
  39. Kay, J.: Learner know thyself: student models to give learner control and responsibility. In: Halim, Z., Ottomann, T., Razak, Z. (eds.) Proceeding of the International Conference on Computers in Education. AACE (1997)Google Scholar
  40. Kerly, A., Bull, S.: Children’s interactions with inspectable and negotiated learner models. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S. (eds.) Proceedings of the International Conference on Intelligent Tutoring Systems, pp. 132–141. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  41. Koedinger, K.R., Corbett, A.T.: Cognitive tutors: technology bringing learning science to the classroom. In: Sawyer, K. (ed.) The Cambridge Handbook of the Learning Sciences. Cambridge University Press, Cambridge (2006)Google Scholar
  42. Koedinger, K.R., Aleven, V.: Exploring the assistance dilemma in experiments with cognitive tutors. Educ. Psychol. Rev. 19(3), 239–264 (2007)CrossRefGoogle Scholar
  43. Koedinger, K.R., Corbett, A.T., Perfetti, C.: The Knowledge-Learning-Instruction (KLI) framework: toward bridging the science-practice chasm to enhance robust student learning. Technical report, Carnegie Mellon University, Human Computer Interaction Institute, Pittsburgh (2010)Google Scholar
  44. Kostons, D., van Gog, T., Paas, F.: Self-assessment and task selection in learner-controlled instruction: differences between effective and ineffective learners. Comput. Educ. 54(4), 932–940 (2010)CrossRefGoogle Scholar
  45. Lee, S.J.H., Bull, S.: An open learner model to help parents help their children. Technol. Instr. Cognit. Learn. 6(1), 29–51 (2008)Google Scholar
  46. Long, Y., Aleven, V.: Students’ understanding of their student model. In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds.) Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 179–186. Springer, Berlin (2011)CrossRefGoogle Scholar
  47. Long, Y., Aleven, V.: Active learners?: redesigning an intelligent tutoring system to support self-regulated learning. In: Proceedings of EC-TEL 2013: Scaling up Learning for Sustained Impact, pp. 490–495 (2013a)Google Scholar
  48. Long, Y., Aleven, V.: Supporting students’ self-regulated learning with an open learner model in a linear equation tutor. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education, pp. 219–228 (2013b)Google Scholar
  49. Long, Y., Aleven, V.: Skill diaries: improve student learning in an intelligent tutoring system with periodic self-assessment. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education, pp. 249–258 (2013c)Google Scholar
  50. Long, Y., Aleven, V.: Gamification of joint student/system control over problem selection in a linear equation tutor. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panour-gia, K. (eds.) Proceedings of the 12th International Conference on Intelligent Tutoring Systems, pp. 378–387. Springer, New York (2014)CrossRefGoogle Scholar
  51. Long, Y., Aleven, V. (2016). Mastery-oriented shared student/system control over problem selection in a linear equation tutor. In: Proceedings of the 13th International Conference on Intelligent Tutoring System, pp. 90–100Google Scholar
  52. Mabbott, A., Bull, S.: Comparing student-constructed open learner model presentations to the domain. In: Koedinger, K., Luckin, R., Greer, J. (eds.) Proceedings of the International Conference on Artificial Intelligence in Education. IOS Press, Amsterdam (2007)Google Scholar
  53. Martinez-Maldonado, R., Pardo, A., Mirriahi, N., Yacef, K., Kay, J., Clayphan, A.: The LATUX workflow: designing and deploying awareness tools in technology-enabled learning settings. In: Proceedings of the Fifth International Conference on Learning Analytics and Knowledge, pp. 1–10. ACM (2015)Google Scholar
  54. Metcalfe, J.: Metacognitive judgments and control of study. Curr. Dir. Psychol. Sci. 18(3), 159–163 (2009)CrossRefGoogle Scholar
  55. Metcalfe, J., Kornell, N.: The dynamics of learning and allocation of study time to a region of proximal learning. J. Exp. Psychol. Gen. 132(4), 530 (2003)CrossRefGoogle Scholar
  56. Metcalfe, J., Kornell, N.: A region of proximal learning model of study time allocation. J. Mem. Lang. 52(4), 463–477 (2005)CrossRefGoogle Scholar
  57. Mitrovic, A., Martin, B.: Scaffolding and fading problem selection in SQL-Tutor. In: Hoppe, U., Verdejo, F., Kay, J. (eds.) Proceedings of the 11th International Conference on Artificial Intelligence in Education, pp. 479–481. Springer, Berlin (2003)Google Scholar
  58. Mitrovic, A., Martin, B.: Evaluating the effect of open student models on self-assessment. Int. J. Artif. Intell. Educ. 17(2), 121–144 (2007)Google Scholar
  59. Mitrovic, A., Mayo, M., Suraweera, P., Martin, B.: Constraint-based tutors: A success story. In: Monostori, L., Váncza, J., Ali, M. (eds.) Proceedings 14th International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems, pp. 931–940. Springer, Berlin (2001)Google Scholar
  60. Niemiec, R.P., Sikorski, C., Walberg, H.J.: Learner-control effects: a review of reviews and a meta-analysis. J. Educ. Comput. Res. 15(2), 157–174 (1996)CrossRefGoogle Scholar
  61. Parra, D., Brusilovsky, P.: User-controllable personalization: a case study with SetFusion. Int. J. Hum.-Comput. Stud. 78, 43–67 (2015)CrossRefGoogle Scholar
  62. Pardos, Z. A., Heffernan, N. T.: Modeling individualization in a bayesian networks implementation of knowledge tracing. In: deBra, P., Kobsa, A., Chin, D. (eds.) Proceedings of the 18th International Conference on User Modeling, Adaptation, and Personalization, UMAP 2010, PP. 255–266. Springer, Berlin (2010)Google Scholar
  63. Pardos, Z.A., Bergner, Y., Seaton, D., Pritchard, D.E.: Adapting Bayesian knowledge tracing to a massive open online college course in edX. In: D’Mello, S.K., Calvo, R.A., Olney, A. (eds.) Proceedings of the 6th International Conference on Educational Data Mining (EDM), pp. 137–144. Memphis, TN (2013)Google Scholar
  64. Perez-Marin, D., Alfonseca, E., Rodriguez, P., Pascual-Neito, I.: A study on the possibility of automatically estimating the confidence value of students’ knowledge in generated conceptual models. J. Comput. 2(5), 17–26 (2007)CrossRefGoogle Scholar
  65. Pintrich, P.R.: A conceptual framework for assessing motivation and self-regulated learning in college students. Educ. Psychol. Rev. 16, 385–407 (2004)CrossRefGoogle Scholar
  66. Pintrich, P.R., De Groot, E.V.: Motivational and self-regulated learning components of classroom academic performance. J. Educ. Psychol. 82, 33–40 (1990)CrossRefGoogle Scholar
  67. Schneider, W., Lockl, K.: The development of metacognitive knowledge in children and adolescents. In: Perfect, T.J., Schwartz, B.L. (eds.) Proceedings of Applied metacognition, pp. 224–257. Cambridge University Press, Cambridge (2002)CrossRefGoogle Scholar
  68. Schraw, G.A.: Conceptual analysis of five measures of metacognitive monitoring. Metacognit. Learn. 4(1), 33–45 (2009)CrossRefGoogle Scholar
  69. Schraw, G., Flowerday, T., Reisetter, M.: The role of choice in reader engagement. J. Educ. Psychol. 90, 705–714 (1998)CrossRefGoogle Scholar
  70. Thiede, K.W., Anderson, M.C.M., Therriault, D.: Accuracy of metacognitive monitoring affects learning of texts. J. Educ. Psychol. 95, 66–73 (2003)CrossRefGoogle Scholar
  71. Tongchai, N.: Impact of self-regulation and open learner model on learning achievement in blended learning environment. Int. J. Inf. Educ. Technol 6(5), 343–347 (2016)Google Scholar
  72. University of Rochester: intrinsic motivation inventory (IMI) (1994). Accessed January 2013
  73. Vandewaetere, M., Clarebout, G.: Can instruction as such affect learning? The case of learner control. Comput. Educ. 57(4), 2322–2332 (2011)CrossRefGoogle Scholar
  74. VanLehn, K.: The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ. Psychol. 46(4), 197–221 (2011)CrossRefGoogle Scholar
  75. VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Wintersgill, M.: The Andes physics tutoring system: lessons learned. Int. J. Artif. Intell. Educ. 15(1), 147–204 (2005)Google Scholar
  76. Waalkens, M., Aleven, V., Taatgen, N.: Does supporting multiple student strategies lead to greater learning and motivation? Investigating a source of complexity in the architecture of intelligent tutoring systems. Comput. Educ. 60, 159–171 (2013)CrossRefGoogle Scholar
  77. Winne, P.H., Hadwin, A.E.: Studying as self-regulated learning. In: Hacker, D.J., Dunlosky, J., Graesser, A.C. (eds.) Metacognition in Educational Theory and Practice, pp. 277–304. Lawrence Erlbaum Associates, Mahwah (1998)Google Scholar
  78. Yudelson, M.V., Koedinger, K.R., Gordon, G.J.: Individualized bayesian knowledge tracing models. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) Proceedings of the 16th international conference on artificial intelligence in education, AIED 2013, pp. 171–180. Springer, Berlin (2013)Google Scholar
  79. Zimmerman, B.J., Martinez-Pons, M.: Development of a structured interview for assessing students’ use of self-regulated learning strategies. Am. Educ. Res. J. 23, 614–628 (1986)CrossRefGoogle Scholar
  80. Zimmerman, B.J.: Attaining self-regulation: a social cognitive perspective. In: Boekaerts, M., Pintrich, P., Zeidner, M. (eds.) Handbook of Self-Regulation, pp. 1–39. Academic Press, San Diego (2000)Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  1. 1.Learning Research and Development CenterUniversity of PittsburghPittsburghUSA
  2. 2.Human-Computer Interaction InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations