Learning What Works in ITS from Non-traditional Randomized Controlled Trial Data

  • Zachary A. Pardos
  • Matthew D. Dailey
  • Neil T. Heffernan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6095)


The traditional, well established approach to finding out what works in education research is to run a randomized controlled trial (RCT) using a standard pretest and posttest design. RCTs have been used in the intelligent tutoring community for decades to determine which questions and tutorial feedback work best. Practically speaking, however, ITS creators need to make decisions on what content to deploy without the benefit of having run an RCT in advance. Additionally, most log data produced by an ITS is not in a form that can easily be evaluated with traditional methods. As a result, there is much data produced by tutoring systems that we would like to learn from but are not. In prior work we introduced a potential solution to this problem: a Bayesian networks method that could analyze the log data of a tutoring system to determine which items were most effective for learning among a set of items of the same skill. The method was validated by way of simulations. In this work we further evaluate the method by applying it to real world data from 11 experiment datasets that investigate the effectiveness of various forms of tutorial help in a web based math tutoring system. The goal of the method was to determine which questions and tutorial strategies cause the most learning. We compared these results with a more traditional hypothesis testing analysis, adapted to our particular datasets. We analyzed experiments in mastery learning problem sets as well as experiments in problem sets that, even though they were not planned RCTs, took on the standard RCT form. We found that the tutorial help or item chosen by the Bayesian method as having the highest rate of learning agreed with the traditional analysis in 9 out of 11 of the experiments. The practical impact of this work is an abundance of knowledge about what works that can now be learned from the thousands of experimental designs intrinsic in datasets of tutoring systems that assign items in a random order.


Knowledge Tracing Item Effect Model Bayesian Networks Randomized Controlled Trials Data Mining 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Pardos, Z.A., Heffernan, N.T.: Detecting the Learning Value of Items in a Randomized Problem Set. In: Dimitrova, Mizoguchi, du Boulay, Graesser (eds.) Proceedings of the 13th International Conference on Artificial Intelligence in Education, pp. 499–506. IOS Press, Amsterdam (2009)Google Scholar
  2. 2.
    Corbett, A.T.: Cognitive computer tutors: solving the two-sigma problem. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) UM 2001. LNCS (LNAI), vol. 2109, pp. 137–147. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  3. 3.
    Corbett, A.T., Anderson, J.R.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4(4), 253–278 (1995)CrossRefGoogle Scholar
  4. 4.
    Koedinger, K.R., Anderson, J.R., Hadley, W.H., Mark, M.A.: Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education 8, 30–43 (1997)Google Scholar
  5. 5.
    Razzaq, L., Heffernan, N.T.: To Tutor or Not to Tutor: That is the Question. In: Dimitrova, Mizoguchi, du Boulay, Graesser (eds.) Proceedings of the 13th International Conference on Artificial Intelligence in Education, pp. 457–464. IOS Press, Amsterdam (2009)Google Scholar
  6. 6.
    Razzaq, L., Heffernan, N.T.: Scaffolding vs. hints in the Assistment system. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053, pp. 635–644. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  7. 7.
    Kim, R., Weitz, R., Heffernan, N., Krach, N.: Tutored Problem Solving vs. “Pure”: Worked Examples. In: Taatgen, N.A., van Rijn, H. (eds.) Proceedings of the 31st Annual Conference of the Cognitive Science Society, Cognitive Science Society, Austin (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Zachary A. Pardos
    • 1
  • Matthew D. Dailey
    • 1
  • Neil T. Heffernan
    • 1
  1. 1.Worcester Polytechnic Institute 

Personalised recommendations