Experimental Evaluation of Automatic Hint Generation for a Logic Tutor

  • John C. Stamper
  • Michael Eagle
  • Tiffany Barnes
  • Marvin Croy
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6738)


In our prior work we showed it was feasible to augment a logic tutor with a data-driven Hint Factory that uses data to automatically generate context-specific hints for an existing computer aided instructional tool. Here we investigate the impact of automatically generated hints on educational outcomes in a robust experiment that shows that hints help students persist in deductive logic courses. Three instructors taught two semester-long courses, each teaching one semester using a logic tutor with hints, and one semester using the tutor without hints, controlling for the impact of different instructors on course outcomes. Our results show that students in the courses using a logic tutor augmented with automatically generated hints attempted and completed significantly more logic proof problems, were less likely to abandon the tutor, and performed significantly better on a post-test implemented within the tutor.


data mining machine learning logic tutor 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aleven, V., McLaren, B.M., Roll, I., Koedinger, K.R.: Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220, pp. 227–239. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  2. 2.
    Barnes, T., Stamper, J.: Toward Automatic Hint Generation for Logic Proof Tutoring Using Historical Student Data. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S. (eds.) ITS 2008. LNCS, vol. 5091, pp. 373–382. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  3. 3.
    Barnes, T., Stamper, J., Lehmann, L., Croy, M.: A Pilot Study on Logic Proof Tutoring Using Hints Generated from Historical Student Data. In: Baker, R., Barnes, T., Beck, J. (eds.) Educational Data Mining (EDM 2008), Montreal, Canada, pp. 197–201 (2008)Google Scholar
  4. 4.
    Beck, J.E., Chang, K.-m., Mostow, J., Corbett, A.T.: Does help help? Introducing the bayesian evaluation and assessment methodology. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S. (eds.) ITS 2008. LNCS, vol. 5091, pp. 383–394. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  5. 5.
    Croy, M., Barnes, T., Stamper, J.: Towards an Intelligent Tutoring System for propositional proof construction. In: Brey, P., Briggle, A., Waelbers, K. (eds.) European Computing and Philosophy Conference, pp. 145–155. IOS Publishers, Amsterdam (2007)Google Scholar
  6. 6.
    Fossati, D., Di Eugenio, B., Ohlsson, S., Brown, c., Chen, L., Cosejo, D.: I learn from you, you learn from me: How to make iList learn from students. In: Dimitrova, V., Mizoguchi, R., Du Boulay, B., Graesser, A. (eds.) Proc. 14th Intl. Conf. on Artificial Intelligence in Education, AIED 2009, pp. 186–195. IOS Press, Brighton (2009)Google Scholar
  7. 7.
    Koedinger, K.R., Baker, R.S.J.d., Cunningham, K., Skogsholm, A., Leber, B., Stamper, J.: A Data Repository for the EDM commuity: The PSLC DataShop. In: Romero, C., Ventura, S., Pechenizkiy, M., Baker, R.S.J.d. (eds.) Handbook of Educational Data Mining. CRC Press, Boca Raton (2010)Google Scholar
  8. 8.
    Mitrovic, A., Koedinger, K., Martin, B.: A comparative analysis of cognitive tutoring and constraint-based modeling. In: User Modeling, pp. 313–322 (2003)Google Scholar
  9. 9.
    McLaren, B., Koedinger, K., Schneider, M., Harrer, A., Bollen, L.: Bootstrapping Novice Data: Semi-automated tutor authoring using student log files. In: Proc. Workshop on Analyzing Student-Tutor Interaction Logs to Improve Educational Outcomes, 7th Intl. Conf. Intelligent Tutoring Systems (IT 2004), Maceió, Brazil (2004)Google Scholar
  10. 10.
    Nkambou, R., Mephu Nguifo, E., Fournier-Viger, P.: Using Knowledge Discovery Techniques to Support Tutoring in an Ill-Defined Domain. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S. (eds.) ITS 2008. LNCS, vol. 5091, pp. 395–405. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  11. 11.
    Razzaq, L., Heffernan, N.T.: Hints: Is It Better to Give or Wait to Be Asked? In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010. LNCS, vol. 6094, pp. 349–358. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  12. 12.
    Stamper, J., Barnes, T., Croy, M.: Enhancing the Automatic Generation of Hints with Expert Seeding. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010. LNCS, vol. 6095, pp. 31–40. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  13. 13.
    Shih, B., Koedinger, K.R., Scheines, R.: A Response Time Model For Bottom-Out Hints as Worked Examples. In: Educational Data Mining 2008, pp. 117–126 (2008)Google Scholar
  14. 14.
    Shute, V.J.: Focus on formative feedback. Review of Educational Research 78(1), 153–189 (2008)CrossRefGoogle Scholar
  15. 15.
    Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • John C. Stamper
    • 1
  • Michael Eagle
    • 2
  • Tiffany Barnes
    • 2
  • Marvin Croy
    • 3
  1. 1.Human-Computer InstituteCarnegie Mellon UniversityUSA
  2. 2.Department of Computer ScienceUniversity of North Carolina at CharlotteUSA
  3. 3.Department of PhilosophyUniversity of North Carolina at CharlotteUSA

Personalised recommendations