Does Help Help? Introducing the Bayesian Evaluation and Assessment Methodology

  • Joseph E. Beck
  • Kai-min Chang
  • Jack Mostow
  • Albert Corbett
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5091)


Most ITS have a means of providing assistance to the student, either on student request or when the tutor determines it would be effective. Presumably, such assistance is included by the ITS designers since they feel it benefits the students. However, whether—and how—help helps students has not been a well studied problem in the ITS community. In this paper we present three approaches for evaluating the efficacy of the Reading Tutor’s help: creating experimental trials from data, learning decomposition, and Bayesian Evaluation and Assessment, an approach that uses dynamic Bayesian networks. We have found that experimental trials and learning decomposition both find a negative benefit for help–that is, help hurts! However, the Bayesian Evaluation and Assessment framework finds that help both promotes student long-term learning and provides additional scaffolding on the current problem. We discuss why these approaches give divergent results, and suggest that the Bayesian Evaluation and Assessment framework is the strongest of the three. In addition to introducing Bayesian Evaluation and Assessment, a method for simultaneously assessing students and evaluating tutorial interventions, this paper describes how help can both scaffold the current problem attempt as well as teach the student knowledge that will transfer to later problems.


educational data mining dynamic Bayesian networks assessment evaluation Bayesian Evaluation and Assessment 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Heiner, C., Beck, J.E., Mostow, J.: Improving the help selection policy in a Reading Tutor that listens. In: Proceedings of the InSTIL/ICALL Symposium on NLP and Speech Technologies in Advanced Language Learning Systems, Venice, Italy, pp. 195–198 (2004)Google Scholar
  2. 2.
    Arroyo, I., Beck, J.E., Beal, C.R., Wing, R.E., Woolf, B.P.: Analyzing students’ response to help provision in an elementary mathematics Intelligent Tutoring System in Help Provision and Help Seeking in Interactive Learning Environments. In: Workshop at the Tenth International Conference on Artificial Intelligence in Education, San Antonio (2001)Google Scholar
  3. 3.
    Mostow, J., Aist, G.: Evaluating tutors that listen: An overview of Project LISTEN. In: Feltovich, P. (ed.) Smart Machines in Education, pp. 169–234. MIT/AAAI Press, Menlo Park (2001)Google Scholar
  4. 4.
    Anderson, J.R.: Rules of the mind. Lawrence Erlbaum Associates, Hillsdale (1993)Google Scholar
  5. 5.
    Beck, J.: Using learning decomposition to analyze student fluency development. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053. Springer, Heidelberg (2006)Google Scholar
  6. 6.
    Zhang, X., Mostow, J., Beck, J.E.: All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. In: AIED 2007 Educational Data Mining Workshop, pp. 80–87 (2007)Google Scholar
  7. 7.
    Beck, J.E.: Does learner control affect learning? In: Proceedings of the 13th International Conference on Artificial Intelligence in Education, Los Angeles, pp. 135–142 (2007)Google Scholar
  8. 8.
    Corbett, A.T., Anderson, J.R.: Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 253–278 (1995)CrossRefGoogle Scholar
  9. 9.
    Reye, J.: Student Modelling based on Belief Networks. International Journal of Artificial Intelligence in Education 14, 1–33 (2004)Google Scholar
  10. 10.
    Mostow, J., Beck, J., Bey, J., Cuneo, A., Sison, J., Tobin, B., Valeri, J.: Using automated questions to assess reading comprehension, vocabulary, and effects of tutorial interventions. Technology, Instruction, Cognition and Learning 2, 97–134 (2004)Google Scholar
  11. 11.
    Vygotsky, L.: Play and its role in the mental development of the child. In: Bruner, J., Jolly, A., Sylva, K. (eds.) Play: Its role in development and evolution (1976), pp. 461–463. Penguin Books, New York (1933)Google Scholar
  12. 12.
    Chang, K.-m.K., Beck, J.E., Mostow, J., Corbett, A.: A Bayes Net Toolkit for Student Modeling in Intelligent Tutoring Systems. In: 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan (2006)Google Scholar
  13. 13.
    Murphy, K.: Bayes Net Toolbox for Matlab (1998)Google Scholar
  14. 14.
    Jensen, F.V., Jordan, M., Lauritzen, S.L., Lawless, J.F., Nair, V. (eds.): Bayesian Networks and Decision Graphs. Statistics for Engineering and Information Science. Springer (2001)Google Scholar
  15. 15.
    Beck, J.E., Sison, J.: Using knowledge tracing in a noisy environment to measure student reading proficiencies. International Journal of Artificial Intelligence in Education 16, 129–143 (2006)Google Scholar
  16. 16.
    Hand, D., Mannila, H., Smyth, P.: Principles of Data Mining. MIT Press, Cambridge (2001)Google Scholar
  17. 17.
    Jonsson, A., Johns, J., Mehranian, H., et al.: Evaluating the Feasibility of Learning Student Models from Data. In: Educational Data Mining: Papers from the AAAI Workshop, pp. 1–6. AAAI Press, Pittsburgh (2005)Google Scholar
  18. 18.
    Conati, C., Gertner, A., VanLehn, K.: Using Bayesian Networks to Manage Uncertainty in Student Modeling. User Modeling and User-Adapted Interaction 12(4), 371–417 (2002)CrossRefMATHGoogle Scholar
  19. 19.
    Beck, J.E., Chang, K.-m.: Identifiability: A Fundamental Problem of Student Modeling. In: International Conference on User Modeling, Corfu, Greece, pp. 137–146 (2007)Google Scholar
  20. 20.
    Embretson, S.E., Reise, S.P.: Item Response Theory for Psychologists. In: Harlow, L.L. (ed.) Multivariate Applications, p. 371. Lawrence Erlbaum Associates, Mahwah (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Joseph E. Beck
    • 1
  • Kai-min Chang
    • 2
  • Jack Mostow
    • 2
  • Albert Corbett
    • 2
  1. 1.Computer Science DepartmentWorcester Polytechnic Institute 
  2. 2.School of Computer ScienceCarnegie Mellon University 

Personalised recommendations