User Modeling and User-Adapted Interaction

, Volume 21, Issue 3, pp 249–283 | Cite as

Evaluating and improving adaptive educational systems with learning curves

  • Brent Martin
  • Antonija Mitrovic
  • Kenneth R. Koedinger
  • Santosh Mathan
Original Paper


Personalised environments such as adaptive educational systems can be evaluated and compared using performance curves. Such summative studies are useful for determining whether or not new modifications enhance or degrade performance. Performance curves also have the potential to be utilised in formative studies that can shape adaptive model design at a much finer level of granularity. We describe the use of learning curves for evaluating personalised educational systems and outline some of the potential pitfalls and how they may be overcome. We then describe three studies in which we demonstrate how learning curves can be used to drive changes in the user model. First, we show how using learning curves for subsets of the domain model can yield insight into the appropriateness of the model’s structure. In the second study we use this method to experiment with model granularity. Finally, we use learning curves to analyse a large volume of user data to explore the feasibility of using them as a reliable method for fine-tuning a system’s model. The results of these experiments demonstrate the successful use of performance curves in formative studies of adaptive educational systems.


Empirical evaluation Intelligent tutoring systems Student modeling User modelling Domain modelling Learning curves 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Ainsworth S.E., Grimshaw S.: Evaluating the REDEEM authoring tool: can teachers create effective learning environments?. Int. J. Artif. Intell. Educ. 14(3), 279–312 (2004)Google Scholar
  2. Anderson J.R.: Rules of the Mind. Lawrence Erlbaum Associates, Hillsdale, NJ (1993)Google Scholar
  3. Anderson J.R., Corbett A.T., Koedinger K.R., Pelletier R.: Cognitive tutors: lessons learned. J. Learn. Sci. 4(2), 167–207 (1995)CrossRefGoogle Scholar
  4. Baker, R.S.J.D., Habgood, M.P.J., Ainsworth, S.E., Corbett, A.T.: Modeling the acquisition of fluent skill in educational action games. In: Conati, C., McCoy, K., Paliouras, G. (eds.) UM2007, vol. 4511, pp. 17–26. LNCS, Corfu (2007)Google Scholar
  5. Cen H., Koedinger K.R., Junker B.: Learning factors analysis: a general method for cognitive model evaluation and improvement. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, vol. 4053, pp. 164–175. LNCS, Jhongli, Taiwan (2006)Google Scholar
  6. Eagle M., Barnes T.: Intelligent tutoring systems, educational data mining, and the design and evaluation of video games. In: Aleven, V., Kay, J., Mostow, J. (eds) ITS2010, vol. 6094, pp. 215–217. LNCS, Pittsburgh, USA (2010)Google Scholar
  7. Heathcote A., Brown S., Mewhort D.J.: The power law repealed: the case for an exponential law of practice. Psychon. Bull. Rev. 7(2), 185–207 (2000)CrossRefGoogle Scholar
  8. Holt P., Dubs S., Jones M., Greer J.: The state of student modeling. In: Greer, J., McCalla, G. (eds) Student Modeling: The Key to Individualized Knowledge-Based Instruction, pp. 3–39. Springer, New York (1994)Google Scholar
  9. Koedinger, K.R., Mathan, S.: Distinguishing qualitatively different kinds of learning using log files and learning curves. In: ITS 2004 Log Analysis Workshop, Maceio, Brazil, pp. 39–46, (2004)Google Scholar
  10. Koedinger, K.R., Corbett, A.C., Perfetti, C.: The Knowledge-Learning-Instruction (KLI) framework: toward bridging the science-practice chasm to enhance robust student learning. CMU-HCII Tech Report 10–102. (2010)
  11. Martin B.: Constraint-based modelling: representing student knowledge. N. Z. J. Comput. 7(2), 30–38 (1999)Google Scholar
  12. Martin, B., Mitrovic, A.: Authoring web-based tutoring systems with WETAS. In: International Conference on Computers in Education, Auckland, pp. 183–187, (2002a)Google Scholar
  13. Martin, B., Mitrovic, A.: Automatic problem generation in constraint-based tutors. In: Sixth International Conference on Intelligent Tutoring Systems, Biarritz, pp. 388–398, (2002b)Google Scholar
  14. Martin, B., Mitrovic, A.: ITS domain modelling: art or science? In: International Conference on Artificial Intelligence in Education, AIED2003, Sydney, Australia, pp. 183–190, (2003)Google Scholar
  15. Martin B., Mitrovic A.: The effect of adapting feedback generality in ITS. In: Wade, V., Ashman, H., Smyth, B. (eds) AH2006, vol. 4018, pp. 192–202. LNCS, Dublin, Ireland (2006)Google Scholar
  16. Martin, B., Koedinger, K.R., Mitrovic, A., Mathan, S.: On using learning curves to evaluate ITS. In: AIED 2005, Amsterdam, pp. 419–426, (2005)Google Scholar
  17. Mathan, S.: Recasting the feedback debate: benefits of tutoring error detection and correction skills. PhD thesis, School of Computer Science, Human–Computer Interaction Institute. Pittsburgh, PA, Carnegie Mellon University, 130 pp (2003)Google Scholar
  18. Mitrovic A.: An intelligent SQL Tutor on the web. Int. J. Artif. Intell. Educ. 13(2–4), 173–197 (2003)Google Scholar
  19. Mitrovic A., Ohlsson S.: Evaluation of a constraint-based tutor for a database language. Int. J. Artif. Intell. Educ. 10, 238–256 (1999)Google Scholar
  20. Mitrovic A., Martin B., Mayo M.: Using evaluation to shape ITS design: Results and experiences with SQL-Tutor. User Model. User Adapt. Interact. 12(2-3), 243–279 (2002)zbMATHCrossRefGoogle Scholar
  21. Mizoguchi R., Bourdeau J.: Using ontological engineering to overcome common AI-ED problems. Int. J. Artif. Intell. Educ. 11, 107–121 (2000)Google Scholar
  22. Munro A., Johnson M.C., Pizzini Q.A., Surmon D.S., Towne D.M., Wogulis J.L.: Authoring simulation-centred tutors with RIDES. Int. J. Artif. Intell. Educ. 8, 284–316 (1997)Google Scholar
  23. Newell A., Rosenbloom P.S.: Mechanisms of skill acquisition and the law of practice. In: Anderson, J.R. (ed.) Cognitive Skills and Their Acquisition, pp. 1–56. Lawrence Erlbaum Associates, Hillsdale, NJ (1981)Google Scholar
  24. Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R., Weinstein, A.: Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In: AIED2007, Los Angeles, pp. 246–253 (2007)Google Scholar
  25. Ohlsson S.: Constraint-based student modeling. In: Greer, J., McCalla, G. (eds) Student Modeling: The Key to Individualized Knowledge-Based Instruction, pp. 167–189. Springer, New York (1994)Google Scholar
  26. Paramythis A., Weibelzahl S.: A decomposition model for the layered evaluation of interactive adaptive systems. In: Ardissono, L., Brna, P., Mitrovic, A. (eds) 10th International Conference on User Modeling (UM2005), vol. 3538, pp. 438–442. LNCS, Edinburgh, Scotland (2005)Google Scholar
  27. Pavlik P.I. Jr., Cen H., Koedinger K.R.: Learning factors transfer analysis: using learning curve analysis to automatically generate domain models. In: Barnes, T., Desmarais, M., Romero, C., Ventura, S. (eds) Proceedings of the 2nd International Conference on Educational Data Mining, pp. 121–130. Universidad de Cordoba, Cordoba, Spain (2009)Google Scholar
  28. Snoddy G.S.: Learning and stability. J. Appl. Psychol. 10, 1–36 (1926)CrossRefGoogle Scholar
  29. Stevens J.C., Savin H.B.: On the form of learning curves. J. Exp. Anal. Behav. 5(1), 15–18 (1962)CrossRefGoogle Scholar
  30. Suraweera P., Mitrovic A.: An intelligent tutoring system for entity relationship modelling. Int. J. Artif. Intell. Educ. 14(3), 375–417 (2004)Google Scholar
  31. Suraweera P., Mitrovic A., Martin B.: The role of domain ontology in knowledge acquisition for ITS. In: Lester, J.C., Vicari, R., Paraguacu, F. (eds) Seventh International Conference on Intelligent Tutoring Systems, vol. 3220, pp. 207–216. LNCS, Maceio, Brazil (2004)Google Scholar
  32. Uresti J., Du Boulay B.: Expertise, motivation and teaching in learning companion systems. Int. J. Artif. Intell. Educ. 14, 67–106 (2004)Google Scholar
  33. Walker A., Recker M., Lawless K., Wiley D.: Collaborative information filtering: a review and an educational application. Int. J. Artif. Intell. Educ. 14(1), 3–28 (2004)Google Scholar
  34. Witten I.H., Frank E.: Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufman, Seattle (2005)zbMATHGoogle Scholar
  35. Wright T.P.: Factors affecting the cost of airplanes. J. Aeronaut. Sci. 3, 122–128 (1936)Google Scholar
  36. Zapata-Rivera J.D., Greer J.E.: Interacting with inspectable Bayesian student models. Int. J. Artif. Intell. Educ. 14(2), 127–163 (2004)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2010

Authors and Affiliations

  • Brent Martin
    • 1
  • Antonija Mitrovic
    • 1
  • Kenneth R. Koedinger
    • 2
  • Santosh Mathan
    • 3
  1. 1.Intelligent Computer Tutoring Group, Department of Computer Science and Software EngineeringUniversity of CanterburyIlam, ChristchurchNew Zealand
  2. 2.HCI InstituteCarnegie Mellon UniversityPittsburghUSA
  3. 3.Human Centered Systems GroupHoneywell LabsMinneapolisUSA

Personalised recommendations