Skip to main content
SpringerLink
Log in

Enhancing learning outcomes through self-regulated learning support with an Open Learner Model

  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

Open Learner Models (OLMs) have great potential to support students’ Self-Regulated Learning (SRL) in Intelligent Tutoring Systems (ITSs). Yet few classroom experiments have been conducted to empirically evaluate whether and how an OLM can enhance students’ domain level learning outcomes through the scaffolding of SRL processes in an ITS. In two classroom experiments with a total of 302 7th- and 8th-grade students, we investigated the effect of (a) an OLM that supports students’ self-assessment of their equation-solving skills and (b) shared control over problem selection, on students’ equation-solving abilities, enjoyment of learning with the tutor, self-assessment accuracy, and problem selection decisions. In the first, smaller experiment, the hypothesized main effect of the OLM on students’ learning outcomes was confirmed; we found no main effect of shared control of problem selection, nor an interaction. In the second, larger experiment, the hypothesized main effects were not confirmed, but we found an interaction such that the students who had access to the OLM learned significantly better equation-solving skills than their counterparts when shared control over problem selection was offered in the system. Thus, the two experiments support the notion that an OLM can enhance students’ domain-level learning outcomes through scaffolding of SRL processes, and are among the first in-vivo classroom experiments to do so. They suggest that an OLM is especially effective if it is designed to support multiple SRL processes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. The average scores of pre/post-tests are not normally distributed (even with Logarithmic and Square-Root transformations), so we used the learning gains as the dependent variables, which are normally distributed.

References

  • Aleven, V.: Rule-based cognitive modeling for intelligent tutoring systems. Advances in intelligent tutoring systems. In: Nkambou, R., Bourdeau, J., Mizoguchi, R. (eds.) Studies in Computational Intelligence, vol. 308, pp. 33–62. Springer, Berlin (2010)

  • Aleven, V., Koedinger, K.R.: Limitations of student control: do students know when they need help? In: Gauthier, G., Frasson, C., VanLehn, K. (eds.) Proceedings of the 5th International Conference on Intelligent Tutoring Systems, pp. 292–303. Springer, Berlin (2000)

    Chapter  Google Scholar 

  • Aleven, V., Koedinger, K.R.: Knowledge component approaches to learner modeling. In: Sottilare, R., Graesser, A., Hu, X., Holden, H. (eds.) Design Recommendations for Adaptive Intelligent Tutoring Systems, pp. 165–182. Orlando, US Army Research Laboratory (2013)

    Google Scholar 

  • Aleven, V., McLaren, B.M., Sewall, J., Koedinger, K.R.: Example-tracing tutors: a new paradigm for intelligent tutoring systems. Int. J. Artif. Intell. Educ. 19(2), 105–154 (2009)

    Google Scholar 

  • Aleven, V., McLaren, B.M., Sewall, J., van Velsen, M., Popescu, O., Demi, S., Koedinger, K.R.: Example-tracing tutors: intelligent tutor development for non-programmers. Int. J. Artif. Intell. Educ. 26(1), 224–269 (2016)

    Article  Google Scholar 

  • Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive tutors: lessons learned. J. Learn. Sci. 4(2), 167–207 (1995)

    Article  Google Scholar 

  • Atkinson, R.C.: Optimizing the learning of a second-language vocabulary. J. Exp. Psychol. 96(1), 124–129 (1972)

    Article  Google Scholar 

  • Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Mehranian, H., Fisher, D., Barto, A., Mahadevan, S., Woolf, B.: Repairing disengagement with non invasive interventions. In: Proceedings of the International Conference on Artificial Intelligence in Education, pp. 195–202. Marina del Rey, CA (2007)

  • Azevedo, R., Aleven, V. (eds.): International Handbook on Metacognition in Computer-Based Learning Environments. Springer, Berlin, DE (2013)

    Google Scholar 

  • Bandura, A.: Self-efficacy. In: Ramachaudran, V.S. (Ed.) Encyclopedia of Human Behaviour, vol. 4, pp. 71–81. Academic Press, New York (Reprinted in H. Friedman (Ed.). Encyclopedia of mental health, San Diego: Academic Press, 1998) (1994)

  • Basu, S., Biswas, G., Kinnebrew, J.S.: Learner modeling for adaptive scaffolding in a computational thinking-based science learning environment. User Model. User-Adapt. Interact. J. Personal. Res. 27 (2017) (Special Issue on Impact of Learner Modeling)

  • Bjork, R.A., Bjork, E.L.: Optimizing treatment and instruction: implications of a new theory of disuse. In: Nilsson, L.G., Ohta, N. (eds.) Memory and Society: Psychological Perspectives, pp. 109–133. Psychology Press, New York (2006)

    Google Scholar 

  • Brusilovsky, P.: Adaptive navigation support: from adaptive hypermedia to the adaptive web and beyond. Psychol. J. 2(1), 7–23 (2004)

    Google Scholar 

  • Brusilovsky, P., Schwarz, E., Weber, G.: ELM-ART: an intelligent tutoring system on World Wide Web. In: Frasson, C., Gauthier, G., Lesgold, A. (eds.) Proceedings of Third International Conference on Intelligent Tutoring Systems, ITS-96, pp. 261–269. Springer, Berlin (1996)

    Chapter  Google Scholar 

  • Brusilovsky, P., Sosnovsky, S., Shcherbinina, O.: QuizGuide: Increasing the educational value of individualized self-assessment quizzes with adaptive navigation support. In: Nall, J., Robson, R. (eds.) Proceedings of World Conference on E-Learning, pp. 1806–1813 (2004)

  • Brusilovsky, P., Somyürek, S., Guerra, J., Hosseini, R. Zadorozhny, V.: The value of social: comparing open student modeling and open social student modeling. In: Proceedings of the 23nd Conference on User Modeling, Adaptation and Personalization (UMAP 2015), June 29–July 3, 2015, Dublin, Ireland (2015)

  • Bull, S.: Supporting learning with open learner models. In: Proceedings of the 4th Hellenic Conference: Information and Communication Technologies in Education, Athens (2004)

  • Bull, S., Kay, J.: Metacognition and open learner models. In: Roll, I., Aleven, V. (eds.) Proceedings of Workshop on Metacognition and Self-Regulated Learning in Educational Technologies, International Conference on Intelligent Tutoring Systems, pp. 7–20 (2008)

  • Bull, S., Kay, J.: Open learner models. Advances in intelligent tutoring systems. In: Nkambou, R., Bourdeau, J., Mizoguchi, R. (eds.) Studies in Computational Intelligence, vol. 308, pp. 301–322. Springer, Berlin (2010)

  • Bull, S., Kay, J.: SMILI: a framework for interfaces to learning data in Open Learner Models (OLMs), learning analytics and related fields. Int. J. Artif. Intell. Educ. 26(1), 293–331 (2016)

    Article  Google Scholar 

  • Bull, S., Jackson, T., Lancaster, M.: Students’ interest in their misconceptions in first year electrical circuits and mathematics courses. Int. J. Electr. Eng. Educ. 47(3), 307–318 (2010)

    Article  Google Scholar 

  • Bull, S., Ginon, B., Boscolo, C., Johnson, M.D.: Introduction of learning visualisations and metacognitive support in a persuadable Open Learner Model. In: Gasevic, D., Lynch, G. (eds.) Proceeding of Learning Analytics and Knowledge 2016. ACM (2016)

  • Clark, C.R., Mayer, E.R.: E-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning. Jossey-Bass, San Francisco (2011)

    Book  Google Scholar 

  • Cleary, T., Zimmerman, B.J.: Self-regulation differences during athletic practice by experts, nonexperts, and novices. J. Appl. Sport Psychol. 13, 61–82 (2000)

    Google Scholar 

  • Corbalan, G., Kester, L., Van Merriënboer, J.J.G.: Selecting learning tasks: effects of adaptation and shared control on efficiency and task involvement. Contemp. Educ. Psychol. 33(4), 733–756 (2008)

    Article  Google Scholar 

  • Corbalan, G., Kester, L., van Merriënboer, J.J.G.: Combining shared control with variability over surface features: effects on transfer test performance and task involvement. Comput. Hum. Behav. 25(2), 290–298 (2009)

    Article  Google Scholar 

  • Corbett, A.: Cognitive mastery learning in the ACT Programming Tutor. AAAI Technical Report SS-00-01 (2000)

  • Corbett, A.T., Anderson, J.R.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Model. User-Adapt. Interact. 4(4), 253–278 (1995)

    Article  Google Scholar 

  • Corbett, A.T., Bhatnagar, A.: Student modeling in the ACT programming tutor: adjusting a procedural learning model with declarative knowledge. In: Jameson, A., Paris, C., Tasso, C. (eds.) User Modeling. Springer, New York (1997)

    Google Scholar 

  • Cordova, D.I., Lepper, M.R.: Intrinsic motivation and the process of learning: beneficial effects of contextualization, personalization, and choice. J. Educ. Psychol. 88, 715–730 (1996)

    Article  Google Scholar 

  • Desmarais, M.C., Baker, R.S.: A review of recent advances in learner and skill modeling in intelligent learning environments. User Model. User-Adapt. Interact. 22(1–2), 9–38 (2012)

    Article  Google Scholar 

  • Dimitrova, V., Brna, P.: From interactive open learner modeling to intelligent mentoring: STyLE-OLM and beyond. Int. J. Artif. Intell. Educ. 26(1), 332–349 (2016)

    Article  Google Scholar 

  • Dunlosky, J., Lipko, A.: Metacomprehension: a brief history and how to improve its accuracy. Curr. Dir. Psychol. Sci. 16, 228–232 (2007)

    Article  Google Scholar 

  • Flowerday, T., Schraw, G.: Effect of choice on cognitive and affective engagement. J. Educ. Res. 96(4), 207–215 (2003)

    Article  Google Scholar 

  • Gong, Y., Beck, J.E., Heffernan, N.T.: How to construct more accurate student models: comparing and optimizing knowledge tracing and performance factor analysis. Int. J. Artif. Intell. Educ. 21(1–2), 27–46 (2011)

    Google Scholar 

  • Hartley, D., Mitrovic, A.: Supporting learning by opening the student model. In: Cerri, S.A., Gouardères, G., Paraguaçu, F. (eds.) Proceedings of the 6th International Conference on Intelligent Tutoring Systems, pp. 453–462. Springer, Berlin (2002)

    Chapter  Google Scholar 

  • Jameson, A., Schwarzkopf, E.: Pros and cons of controllability: an empirical study. In: De Bra, P., Brusilovsky, P., Conejo, R. (eds.) Adaptive Hypermedia and Adaptive Webbased Systems: Proceedings of AH 2002, pp. 193–202. Springer, Berlin (2002)

    Chapter  Google Scholar 

  • Kapur, M., Bielaczyc, K.: Designing for productive failure. J. Learn. Sci. 21(1), 45–83 (2012)

    Article  Google Scholar 

  • Kay, J.: Learner know thyself: student models to give learner control and responsibility. In: Halim, Z., Ottomann, T., Razak, Z. (eds.) Proceeding of the International Conference on Computers in Education. AACE (1997)

  • Kerly, A., Bull, S.: Children’s interactions with inspectable and negotiated learner models. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S. (eds.) Proceedings of the International Conference on Intelligent Tutoring Systems, pp. 132–141. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  • Koedinger, K.R., Corbett, A.T.: Cognitive tutors: technology bringing learning science to the classroom. In: Sawyer, K. (ed.) The Cambridge Handbook of the Learning Sciences. Cambridge University Press, Cambridge (2006)

    Google Scholar 

  • Koedinger, K.R., Aleven, V.: Exploring the assistance dilemma in experiments with cognitive tutors. Educ. Psychol. Rev. 19(3), 239–264 (2007)

    Article  Google Scholar 

  • Koedinger, K.R., Corbett, A.T., Perfetti, C.: The Knowledge-Learning-Instruction (KLI) framework: toward bridging the science-practice chasm to enhance robust student learning. Technical report, Carnegie Mellon University, Human Computer Interaction Institute, Pittsburgh (2010)

  • Kostons, D., van Gog, T., Paas, F.: Self-assessment and task selection in learner-controlled instruction: differences between effective and ineffective learners. Comput. Educ. 54(4), 932–940 (2010)

    Article  Google Scholar 

  • Lee, S.J.H., Bull, S.: An open learner model to help parents help their children. Technol. Instr. Cognit. Learn. 6(1), 29–51 (2008)

    Google Scholar 

  • Long, Y., Aleven, V.: Students’ understanding of their student model. In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds.) Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 179–186. Springer, Berlin (2011)

    Chapter  Google Scholar 

  • Long, Y., Aleven, V.: Active learners?: redesigning an intelligent tutoring system to support self-regulated learning. In: Proceedings of EC-TEL 2013: Scaling up Learning for Sustained Impact, pp. 490–495 (2013a)

  • Long, Y., Aleven, V.: Supporting students’ self-regulated learning with an open learner model in a linear equation tutor. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education, pp. 219–228 (2013b)

  • Long, Y., Aleven, V.: Skill diaries: improve student learning in an intelligent tutoring system with periodic self-assessment. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education, pp. 249–258 (2013c)

  • Long, Y., Aleven, V.: Gamification of joint student/system control over problem selection in a linear equation tutor. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panour-gia, K. (eds.) Proceedings of the 12th International Conference on Intelligent Tutoring Systems, pp. 378–387. Springer, New York (2014)

    Chapter  Google Scholar 

  • Long, Y., Aleven, V. (2016). Mastery-oriented shared student/system control over problem selection in a linear equation tutor. In: Proceedings of the 13th International Conference on Intelligent Tutoring System, pp. 90–100

  • Mabbott, A., Bull, S.: Comparing student-constructed open learner model presentations to the domain. In: Koedinger, K., Luckin, R., Greer, J. (eds.) Proceedings of the International Conference on Artificial Intelligence in Education. IOS Press, Amsterdam (2007)

    Google Scholar 

  • Martinez-Maldonado, R., Pardo, A., Mirriahi, N., Yacef, K., Kay, J., Clayphan, A.: The LATUX workflow: designing and deploying awareness tools in technology-enabled learning settings. In: Proceedings of the Fifth International Conference on Learning Analytics and Knowledge, pp. 1–10. ACM (2015)

  • Metcalfe, J.: Metacognitive judgments and control of study. Curr. Dir. Psychol. Sci. 18(3), 159–163 (2009)

    Article  Google Scholar 

  • Metcalfe, J., Kornell, N.: The dynamics of learning and allocation of study time to a region of proximal learning. J. Exp. Psychol. Gen. 132(4), 530 (2003)

    Article  Google Scholar 

  • Metcalfe, J., Kornell, N.: A region of proximal learning model of study time allocation. J. Mem. Lang. 52(4), 463–477 (2005)

    Article  Google Scholar 

  • Mitrovic, A., Martin, B.: Scaffolding and fading problem selection in SQL-Tutor. In: Hoppe, U., Verdejo, F., Kay, J. (eds.) Proceedings of the 11th International Conference on Artificial Intelligence in Education, pp. 479–481. Springer, Berlin (2003)

    Google Scholar 

  • Mitrovic, A., Martin, B.: Evaluating the effect of open student models on self-assessment. Int. J. Artif. Intell. Educ. 17(2), 121–144 (2007)

    Google Scholar 

  • Mitrovic, A., Mayo, M., Suraweera, P., Martin, B.: Constraint-based tutors: A success story. In: Monostori, L., Váncza, J., Ali, M. (eds.) Proceedings 14th International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems, pp. 931–940. Springer, Berlin (2001)

    Google Scholar 

  • Niemiec, R.P., Sikorski, C., Walberg, H.J.: Learner-control effects: a review of reviews and a meta-analysis. J. Educ. Comput. Res. 15(2), 157–174 (1996)

    Article  Google Scholar 

  • Parra, D., Brusilovsky, P.: User-controllable personalization: a case study with SetFusion. Int. J. Hum.-Comput. Stud. 78, 43–67 (2015)

    Article  Google Scholar 

  • Pardos, Z. A., Heffernan, N. T.: Modeling individualization in a bayesian networks implementation of knowledge tracing. In: deBra, P., Kobsa, A., Chin, D. (eds.) Proceedings of the 18th International Conference on User Modeling, Adaptation, and Personalization, UMAP 2010, PP. 255–266. Springer, Berlin (2010)

  • Pardos, Z.A., Bergner, Y., Seaton, D., Pritchard, D.E.: Adapting Bayesian knowledge tracing to a massive open online college course in edX. In: D’Mello, S.K., Calvo, R.A., Olney, A. (eds.) Proceedings of the 6th International Conference on Educational Data Mining (EDM), pp. 137–144. Memphis, TN (2013)

    Google Scholar 

  • Perez-Marin, D., Alfonseca, E., Rodriguez, P., Pascual-Neito, I.: A study on the possibility of automatically estimating the confidence value of students’ knowledge in generated conceptual models. J. Comput. 2(5), 17–26 (2007)

    Article  Google Scholar 

  • Pintrich, P.R.: A conceptual framework for assessing motivation and self-regulated learning in college students. Educ. Psychol. Rev. 16, 385–407 (2004)

    Article  Google Scholar 

  • Pintrich, P.R., De Groot, E.V.: Motivational and self-regulated learning components of classroom academic performance. J. Educ. Psychol. 82, 33–40 (1990)

    Article  Google Scholar 

  • Schneider, W., Lockl, K.: The development of metacognitive knowledge in children and adolescents. In: Perfect, T.J., Schwartz, B.L. (eds.) Proceedings of Applied metacognition, pp. 224–257. Cambridge University Press, Cambridge (2002)

    Chapter  Google Scholar 

  • Schraw, G.A.: Conceptual analysis of five measures of metacognitive monitoring. Metacognit. Learn. 4(1), 33–45 (2009)

    Article  Google Scholar 

  • Schraw, G., Flowerday, T., Reisetter, M.: The role of choice in reader engagement. J. Educ. Psychol. 90, 705–714 (1998)

    Article  Google Scholar 

  • Thiede, K.W., Anderson, M.C.M., Therriault, D.: Accuracy of metacognitive monitoring affects learning of texts. J. Educ. Psychol. 95, 66–73 (2003)

    Article  Google Scholar 

  • Tongchai, N.: Impact of self-regulation and open learner model on learning achievement in blended learning environment. Int. J. Inf. Educ. Technol 6(5), 343–347 (2016)

    Google Scholar 

  • University of Rochester: intrinsic motivation inventory (IMI) (1994). http://www.psych.rochester.edu/SDT/measures/IMI_description.php. Accessed January 2013

  • Vandewaetere, M., Clarebout, G.: Can instruction as such affect learning? The case of learner control. Comput. Educ. 57(4), 2322–2332 (2011)

    Article  Google Scholar 

  • VanLehn, K.: The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ. Psychol. 46(4), 197–221 (2011)

    Article  Google Scholar 

  • VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Wintersgill, M.: The Andes physics tutoring system: lessons learned. Int. J. Artif. Intell. Educ. 15(1), 147–204 (2005)

    Google Scholar 

  • Waalkens, M., Aleven, V., Taatgen, N.: Does supporting multiple student strategies lead to greater learning and motivation? Investigating a source of complexity in the architecture of intelligent tutoring systems. Comput. Educ. 60, 159–171 (2013)

    Article  Google Scholar 

  • Winne, P.H., Hadwin, A.E.: Studying as self-regulated learning. In: Hacker, D.J., Dunlosky, J., Graesser, A.C. (eds.) Metacognition in Educational Theory and Practice, pp. 277–304. Lawrence Erlbaum Associates, Mahwah (1998)

    Google Scholar 

  • Yudelson, M.V., Koedinger, K.R., Gordon, G.J.: Individualized bayesian knowledge tracing models. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) Proceedings of the 16th international conference on artificial intelligence in education, AIED 2013, pp. 171–180. Springer, Berlin (2013)

    Google Scholar 

  • Zimmerman, B.J., Martinez-Pons, M.: Development of a structured interview for assessing students’ use of self-regulated learning strategies. Am. Educ. Res. J. 23, 614–628 (1986)

    Article  Google Scholar 

  • Zimmerman, B.J.: Attaining self-regulation: a social cognitive perspective. In: Boekaerts, M., Pintrich, P., Zeidner, M. (eds.) Handbook of Self-Regulation, pp. 1–39. Academic Press, San Diego (2000)

    Google Scholar 

Download references

Acknowledgements

We thank Jonathan Sewall, Borg Lojasiewicz, Octav Popescu, Brett Leber, Gail Kusbit and Emily Zacchero for their kind help with this work. We would also like to thank the participating teachers and students. This work was funded by a National Science Foundation grant to the Pittsburgh Science of Learning Center (NSF Award SBE0354420).

Author information

Authors and Affiliations

  1. Learning Research and Development Center, University of Pittsburgh, 3939 O’Hara Street, Pittsburgh, PA, 15213, USA

    Yanjin Long

  2. Human-Computer Interaction Institute, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA, 15213, USA

    Vincent Aleven

Authors
  1. Yanjin Long
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vincent Aleven
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanjin Long.

Additional information

The paper is based on work that was conducted while the first author was at the Human-Computer Interaction Institute of Carnegie Mellon University. A conference paper based on Classroom Experiment 1 was published in 2013 (Long and Aleven 2013b).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Long, Y., Aleven, V. Enhancing learning outcomes through self-regulated learning support with an Open Learner Model. User Model User-Adap Inter 27, 55–88 (2017). https://doi.org/10.1007/s11257-016-9186-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-016-9186-6

Keywords

Search

Navigation