User Modeling and User-Adapted Interaction

, Volume 18, Issue 4, pp 349–382 | Cite as

A multifactor approach to student model evaluation

  • Michael V. Yudelson
  • Olga P. Medvedeva
  • Rebecca S. Crowley
Original Paper


Creating student models for Intelligent Tutoring Systems (ITS) in novel domains is often a difficult task. In this study, we outline a multifactor approach to evaluating models that we developed in order to select an appropriate student model for our medical ITS. The combination of areas under the receiver-operator and precision-recall curves, with residual analysis, proved to be a useful and valid method for model selection. We improved on Bayesian Knowledge Tracing with models that treat help differently from mistakes, model all attempts, differentiate skill classes, and model forgetting. We discuss both the methodology we used and the insights we derived regarding student modeling in this novel domain.


Student modeling Intelligent tutoring systems Knowledge Tracing Methodology Decision theory Model evaluation Model selection Intelligent medical training systems Machine learning Probabilistic models Bayesian models Hidden Markov Models 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Anderson, J.: Rules of the Mind. Lawrence Erlbaum Associates, Hillsdale, NJ (1993)Google Scholar
  2. Anderson, J., Schunn, C.: Implications of the ACT-R learning theory: no magic bullets. In: Glaser, R. (eds) Advances in Instructional Psychology: Educational Design and Cognitive Science, vol. 5, pp. 1–34. Erlbaum, Mahwah, NJ (2000)Google Scholar
  3. Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive tutors: lessons learned. J. Learn. Sci. 4(2), 167–207 (1995)CrossRefGoogle Scholar
  4. Atkinson, R., Shiffrin, R.: Human memory: a proposed system and its control processes. In: Spence, K.W., Spence, J.T. (eds) The Psychology of Learning and Motivation: Advances in Research and Theory, vol. 2, pp. 742–775. Academic Press, New York (1968)Google Scholar
  5. Beck, J., Sison, J.: Using knowledge tracing to measure student reading proficiencies. In: Proceedings of the 7th International Conference on Intelligent Tutoring Systems, pp. 624–634. Springer-Verlag, Maceio, Brazil (2004)Google Scholar
  6. Brand, M., Oliver, N., Pentland, A.: Coupled hidden Markov models for complex action recognition. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pp. 994–999. San Juan, Puerto Rico (1997)Google Scholar
  7. Byrne, M.D.: Perception and action. In: Anderson, J.R., Lebiére, C. (eds) Atomic Components of Thought, pp. 167–200. Erlbaum, Hillsdale (1998)Google Scholar
  8. Chang, K., Beck, J., Mostow, J., Corbett, A.: Does help help? A Bayes net approach to modeling tutor interventions. In: Proceedings of the 21st Annual Meeting of the American Association for Artificial Intelligence, pp. 41–46. Boston, MA (2006)Google Scholar
  9. Clancey, W.: Methodology for building an intelligent tutoring system. In: Kintsch, W., Miller, H., Poison, P. (eds) Methods and Tactics in Cognitive Science, pp. 51–84. Erlbaum, Hillsdale (1984)Google Scholar
  10. Clancey, W., Letsinger, R.: NEOMYCIN: reconfiguring a rule based expert system for application to teaching. In: Proceedings of the Seventh International Joint Conference on AI, pp. 829–835. Vancouver, BC, Canada (1981)Google Scholar
  11. Conati, C., Zhao, X.: Building and evaluating an intelligent pedagogical agent to improve the effectiveness of an educational game. In: Proceedings of the 9th International Conference on Intelligent User Interface, pp. 6–13. Funchal, Madeira, Portugal (2004)Google Scholar
  12. Conati, C., Gertner, A., VanLehn, K.: Using Bayesian networks to manage uncertainty in student modeling. J. User Model. User-Adap. Interac. 12(4), 371–417 (2002)zbMATHCrossRefGoogle Scholar
  13. Corbett, A., Anderson, J.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Model. User-Adap. Interac. 4, 253–278 (1995)CrossRefGoogle Scholar
  14. Crowley, R., Medvedeva, O.: An intelligent tutoring system for visual classification problem solving. Artif. Intell. Med. 36(1), 85–117 (2006)CrossRefGoogle Scholar
  15. Crowley, R., Naus, G., Stewart, J., Friedman, C.: Development of visual diagnostic expertise in pathology – an information processing study. J. Am. Med. Inform. Assoc. 10(1), 39–51 (2003)CrossRefGoogle Scholar
  16. Crowley, R., Legowski, E., Medvedeva, O., Tseytlin, E., Roh, E., Jukic, D.: Evaluation of an Intelligent Tutoring system in pathology: effects of external representation on performance gains, metacognition, and acceptance. J. Am. Med. Inform. Assoc. 14(2), 182–190 (2007)CrossRefGoogle Scholar
  17. Davis, J., Goadrich, M.: The relationship between Precision-Recall and ROC curves. In: Proceedings of the 23rd International Conference on Machine Learning, vol. 148, pp. 233–240. Pittsburgh, PA (2006)Google Scholar
  18. Ephraim, Y., Roberts, W.: Revisiting autoregressive hidden Markov modeling of speech signals. IEEE Sig. Proc. Lett. 12, 166–169 (2005)CrossRefGoogle Scholar
  19. Fawcett, T.: Graphs: notes and practical considerations for data mining researchers. Tech Reports HPL-2003-4. HP Laboratories, Palo Alto, CA (2003)Google Scholar
  20. Ferguson, K., Arroyo, I., Mahadevan, S., Woolf, B., Barto, A.: Improving intelligent tutoring systems: Using EM to learn student skill levels, Intelligent Tutoring Systems, pp. 453–462. Springer-Verlag, Jhongli, Taiwan (2006)Google Scholar
  21. Fogarty, J., Baker, R.S., Hudson, S.: Case studies in the use of ROC curve analysis for sensor-based estimates in Human Computer Interaction. In: Proceedings of Graphics Interface, pp. 129–136. Victoria, British Columbia, Canada (2005)Google Scholar
  22. Hanley, J.A., McNeil, B.J.: The meaning and use of the area under a Receiver Operating Characteristic (ROC) Curve. Radiology 143, 29–36 (1982)Google Scholar
  23. Jastrzembski, T., Gluck, K., Gunzelmann, G.: Knowledge tracing and prediction of future trainee performance. In: Proceedings of the 2006 Interservice/Industry Training, Simulation, and Education Conference, pp. 1498–1508. National Training Systems Association, Orlando, FL (2006)Google Scholar
  24. Jonsson, A., Johns, J., Mehranian, H., Arroyo, I., Woolf, B., Barto, A., Fisher, D., Mahadevan, S.: Evaluating the feasibility of learning student models from data. In: AAAI05 Workshop on Educational Data Mining, pp. 1–6. Pittsburgh, PA (2005)Google Scholar
  25. Kuenzer, A., Schlick, C., Ohmann, F., Schmidt, L., Luczak, H.: An empirical study of dynamic Bayesian networks for user modeling. In: UM’01 Workshop on Machine Learning for User Modeling, pp. 1–10. Sonthofen, Germany (2001)Google Scholar
  26. Mayo, M., Mitrovic, A.: Optimizing ITS behavior with Bayesian networks and decision theory. Int. J. Artifi. Intell. Educ. 12, 124–153 (2001)Google Scholar
  27. Medvedeva, O., Chavan, G., Crowley, R.: A data collection framework for capturing ITS data based on an agent communication standard. In: Proceedings of the 20th Annual Meeting of the American Association for Artificial Intelligence, pp. 23–30. Pittsburgh, PA (2005)Google Scholar
  28. Moore, D., McCabe, G.: Introduction to the Practice of Statistics. W.H. Freeman and Company, New York (1993)Google Scholar
  29. Murphy, K.: The Bayes Net Toolbox for Matlab. Computing Science and Statistics, vol. 33, pp. 1–20. URL: (Accessed on December 6, 2008) (2001)
  30. Provost, F., Fawcett, T., Kohavi, R.: The case against accuracy estimation for comparing induction algorithms. In: Proceeding of the 15th International Conference on Machine Learning, pp. 445–453. San Francisco, CA (1998)Google Scholar
  31. Reye, J.: Student modeling based on belief networks. Int. J. Artif. Intell. Educ. 14, 1–33 (2004)Google Scholar
  32. Seidemann, E., Meilijson, I., Abeles, M., Bergman, H., Vaadia, E.: Simultaneously recorded single units in the frontal cortex go through sequences of discrete and stable states in monkeys performing a delayed localization task. J. Neurosci. 16(2), 752–768 (1996)Google Scholar
  33. Shang, Y., Shi, H., Chen, S.: An intelligent distributed environment for active learning. J. Educ. Resour. Comput. 1(2), 1–17 (2001)CrossRefGoogle Scholar
  34. VanLehn, K., Niu, Z.: Bayesian student modeling, user interfaces and feedback: a sensitivity analysis. Int. J. Artif. Intell. Educ. 12, 154–184 (2001)Google Scholar
  35. Zukerman, I., Albrecht, D.W., Nicholson, A.E.: Predicting users’ requests on the WWW. In: Proceedings of the Seventh International Conference on User Modeling, (UM-99), pp. 275–284. Banff, Canada (1999)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2008

Authors and Affiliations

  • Michael V. Yudelson
    • 1
    • 2
  • Olga P. Medvedeva
    • 1
  • Rebecca S. Crowley
    • 1
    • 3
    • 4
    • 5
  1. 1.Department of Biomedical InformaticsUniversity of Pittsburgh School of MedicinePittsburghUSA
  2. 2.School of Information SciencesUniversity of PittsburghPittsburghUSA
  3. 3.Intelligent Systems ProgramUniversity of PittsburghPittsburghUSA
  4. 4.Department of PathologyUniversity of Pittsburgh School of MedicinePittsburghUSA
  5. 5.UPMC Cancer PavilionPittsburghUSA

Personalised recommendations