Estimating Student Proficiency Using an Item Response Theory Model

  • Jeff Johns
  • Sridhar Mahadevan
  • Beverly Woolf
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4053)


Item Response Theory (IRT) models were investigated as a tool for student modeling in an intelligent tutoring system (ITS). The models were tested using real data of high school students using the Wayang Outpost, a computer-based tutor for the mathematics portion of the Scholastic Aptitude Test (SAT). A cross-validation framework was developed and three metrics to measure prediction accuracy were compared. The trained models predicted with 72% accuracy whether a student would answer a multiple choice problem correctly.


Mean Square Error Item Response Theory Item Response Theory Model Mean Absolute Error Intelligent Tutoring System 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Anderson, J., Boyle, C., Corbett, A., Lewis, M.: Cognitive Modeling and Intelligent Tutoring. Artificial Intelligence 42(1), 7–49 (1990)CrossRefGoogle Scholar
  2. 2.
    Arroyo, I., Beal, C.R., Murray, T., Walles, R., Park Woolf, B.: Web-based intelligent multimedia tutoring for high stakes achievement tests. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220, pp. 468–477. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  3. 3.
    Arroyo, I., Murray, T., Park Woolf, B., Beal, C.R.: Inferring unobservable learning variables from students’ help seeking behavior. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220, pp. 782–784. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  4. 4.
    Baker, F., Kim, S.-H.: Item Response Theory: Parameter Estimation Techniques. Marcel Dekker, Inc., New York (2004)MATHGoogle Scholar
  5. 5.
    Baker, R., Corbett, A., Koedinger, K., Wagner, A.: Off-Task Behavior in the Cognitive Tutor Classroom: When Students Game the System. In: Proceedings of the ACM CHI 2004 Conference on Human Factors in Computing Systems, pp. 383–390 (2004)Google Scholar
  6. 6.
    Barnes, T.: The Q-Matrix Method of Fault-Tolerant Teaching in Knowledge Assessment and Data Mining. Ph.D. Dissertation. North Carolina State University (2003)Google Scholar
  7. 7.
    Beck, J.: Engagement Tracing: Using Response Times to Model Student Disengagement. In: Proceedings of the International Conference on Artificial Intelligent and Education (2005)Google Scholar
  8. 8.
    Bock, R., Aitkin, M.: Marginal Maximum Likelihood Estimation of Item Parameters: Applications of an EM Algorithm. Psychometrika 46, 443–459 (1981)CrossRefMathSciNetGoogle Scholar
  9. 9.
    Collins, L., Wugalter, S.: Latent Class Models for Stage-Sequential Dynamic Latent Variables. Multivariate Behavioral Research 27(1), 131–157 (1992)CrossRefGoogle Scholar
  10. 10.
    Corbett, A., Anderson, J.: Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge. Journal of User Modeling and User-Adapted Interaction 4, 253–278 (1995)CrossRefGoogle Scholar
  11. 11.
    Dempster, A., Laird, N., Rubin, D.: Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society, Series B 39, 1–38 (1977)MATHMathSciNetGoogle Scholar
  12. 12.
    Embretson, S.: A Multidimensional Latent Trait Model for Measuring Learning and Change. Psychometrika 56, 495–515 (1991)MATHCrossRefGoogle Scholar
  13. 13.
    Jonsson, A., Johns, J., Mehranian, H., Arroyo, I., Woolf, B., Barto, A., Fisher, D., Mahadevan, S.: Evaluating the Feasibility of Learning Student Models from Data. In: American Association for Artificial Intelligence Workshop on Educational Data Mining (2005)Google Scholar
  14. 14.
    Mayo, M., Mitrovic, A.: Optimising ITS Behavior with Bayesian Networks and Decision Theory. International Journal of Artificial Intelligence in Education 12, 124–153 (2001)Google Scholar
  15. 15.
    Rudner, L.: An Evaluation of Measurement Decision Theory,
  16. 16.
    Thissen, D., Wainer, H. (eds.): Test Scoring. Lawrence Erlbaum Associates, Mahwah (2001)Google Scholar
  17. 17.
    van der Linden, W., Hambleton, R. (eds.): Handbook of Modern Item Response Theory. Springer, New York (1997)MATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Jeff Johns
    • 1
  • Sridhar Mahadevan
    • 1
  • Beverly Woolf
    • 1
  1. 1.Computer Science DepartmentUniversity of Massachusetts AmherstAmherstU.S.A.

Personalised recommendations