Skip to main content
SpringerLink
Log in

The Diagnosing Behaviour of Intelligent Tutoring Systems

  • Conference paper
  • First Online:
Transforming Learning with Meaningful Technologies (EC-TEL 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11722))

Included in the following conference series:

Abstract

Intelligent Tutoring Systems (ITSs) determine the quality of student responses by means of a diagnostic process, and use this information for providing feedback and determining a student’s progress. This paper studies how ITSs diagnose student responses. In a systematic literature review we compare the diagnostic processes of 40 ITSs in various domains. We investigate what kinds of diagnoses are performed and how they are obtained, and how the processes compare across domains. The analysis identifies eight aspects that ITSs diagnose: correctness, difference, redundancy, type of error, common error, order, preference, and time. All ITSs diagnose correctness of a step. Mathematics tutors diagnose common errors more often than programming tutors, and programming tutors diagnose type of error more often than mathematics tutors. We discuss a general model for representing diagnostic processes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aleven, V., McLaren, B.M., Sewall, J.: Scaling up programming by demonstration for intelligent tutoring systems development: an open-access web site for middle school mathematics learning. IEEE Trans. Learn. Technol. 2(2), 64–78 (2009)

    Article  Google Scholar 

  2. Aleven, V., Mclaren, B.M., Sewall, J., Koedinger, K.R.: A new paradigm for intelligent tutoring systems: example-tracing tutors. Int. J. Artif. Intell. Educ. 19(2), 105–154 (2009)

    Google Scholar 

  3. Aleven, V., Popescu, O., Koedinger, K.: Pilot-testing a tutorial dialogue system that supports self-explanation. In: Cerri, S.A., Gouardères, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, pp. 344–354. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-47987-2_38

    Chapter  Google Scholar 

  4. Aleven, V., Popescu, O., Koedinger, K.R.: Towards tutorial dialog to support self-explanation: adding natural language understanding to a cognitive tutor. In: Proceedings of Artificial Intelligence in Education, pp. 246–255. Citeseer (2001)

    Google Scholar 

  5. Anderson, J.R., Boyle, C.F., Reiser, B.J.: Intelligent tutoring systems. Science 228(4698), 456–462 (1985)

    Article  Google Scholar 

  6. Anderson, J.R., Boyle, C.F., Yost, G.: The geometry tutor. In: IJCAI, pp. 1–7 (1985)

    Google Scholar 

  7. Anderson, J.R., Reiser, B.J.: The LISP tutor. Byte 10, 159–175 (1985)

    Google Scholar 

  8. Arends, H., Keuning, H., Heeren, B., Jeuring, J.: An intelligent tutor to learn the evaluation of microcontroller I/O programming expressions. In: Proceedings of the 17th Koli Calling Conference on Computing Education Research, pp. 2–9. ACM (2017)

    Google Scholar 

  9. Arevalillo-Herráez, M., Arnau, D., Marco-Giménez, L.: Domain-specific knowledge representation and inference engine for an intelligent tutoring system. Knowl.-Based Syst. 49, 97–105 (2013)

    Article  Google Scholar 

  10. Arnau, D., Arevalillo-Herráez, M., Puig, L., González-Calero, J.A.: Fundamentals of the design and the operation of an intelligent tutoring system for the learning of the arithmetical and algebraic way of solving word problems. Comput. Educ. 63, 119–130 (2013)

    Article  Google Scholar 

  11. Arnott, E., Hastings, P., Allbritton, D.: Research methods tutor: evaluation of a dialogue-based tutoring system in the classroom. Behav. Res. Methods 40(3), 694–698 (2008)

    Article  Google Scholar 

  12. van der Bent, R.: The diagnosing behaviour of intelligent tutoring systems. Master’s thesis, Universiteit Utrecht (2018)

    Google Scholar 

  13. Blank, G., Parvez, S., Wei, F., Moritz, S.: A web-based ITS for OO design. In: Proceedings of Workshop on Adaptive Systems for Web-based Education at 12th International Conference on Artificial Intelligence in Education (AIED 2005), Amsterdam, the Netherlands, pp. 59–64 (2005)

    Google Scholar 

  14. Brown, J.S., Burton, R.R.: Diagnostic models for procedural bugs in basic mathematical skills. Cogn. Sci. 2(2), 155–192 (1978)

    Article  Google Scholar 

  15. Burton, R.R., Brown, J.S.: A tutoring and student modelling paradigm for gaming environments. ACM SIGCUE Outlook 10(SI), 236–246 (1976)

    Article  Google Scholar 

  16. Chanier, T., Pengelly, M., Twidale, M., Self, J.: Conceptual modelling in error analysis in computer-assisted language learning systems. In: Swartz, M.L., Yazdani, M. (eds.) Intelligent Tutoring Systems for Foreign Language Learning. NATO ASI Series, pp. 125–150. Springer, Heidelberg (1992). https://doi.org/10.1007/978-3-642-77202-3_9

    Chapter  Google Scholar 

  17. Cheung, A.C., Slavin, R.E.: The effectiveness of educational technology applications for enhancing mathematics achievement in k-12 classrooms: a meta-analysis. Educ. Res. Rev. 9, 88–113 (2013)

    Article  Google Scholar 

  18. Corbett, A.T., Anderson, J.R.: Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes. In: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 245–252. ACM (2001)

    Google Scholar 

  19. Corbett, A.T., Anderson, J.R., Patterson, E.G.: Student modeling and tutoring flexibility in the Lisp intelligent tutoring system. In: Gauthier, G., Frasson, C. (eds.) Intelligent Tutoring Systems: at the crossroads of artificial intelligence and education, pp. 83–106. Intellect Ltd. (1990)

    Google Scholar 

  20. Demenko, G., Wagner, A., Cylwik, N.: The use of speech technology in foreign language pronunciation training. Arch. Acoust. 35(3), 309–329 (2010)

    Article  Google Scholar 

  21. El-Kechaï, N., Delozanne, É., Prévit, D., Grugeon, B., Chenevotot, F.: Evaluating the performance of a diagnosis system in school algebra. In: Leung, H., Popescu, E., Cao, Y., Lau, R.W.H., Nejdl, W. (eds.) ICWL 2011. LNCS, vol. 7048, pp. 263–272. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25813-8_28

    Chapter  Google Scholar 

  22. Fossati, D., Di Eugenio, B., Ohlsson, S., Brown, C., Chen, L.: Data driven automatic feedback generation in the ilist intelligent tutoring system. Technol. Instr. Cogn. Learn. 10(1), 5–26 (2015)

    Google Scholar 

  23. Glass, M.: Some phenomena handled by the circsim-tutor version 3 input understander. In: Proceedings of the Tenth Florida Artificial Intelligence Research Symposium, Daytona Beach, pp. 21–25 (1997)

    Google Scholar 

  24. Goguadze, G., Melis, E.: Combining evaluative and generative diagnosis in activemath. In: Proceedings of the 2009 Conference on Artificial Intelligence in Education: Building Learning Systems That Care: From Knowledge Representation to Affective Modelling, pp. 668–670 (2009)

    Google Scholar 

  25. Graesser, A.C., et al.: Autotutor: a tutor with dialogue in natural language. Behav. Res. Methods Instrum. Comput. 36(2), 180–192 (2004)

    Article  Google Scholar 

  26. Graesser, A.C., Wiemer-Hastings, P., Wiemer-Hastings, K., Harter, D., Tutoring Research Group, T.R.G., Person, N.: Using latent semantic analysis to evaluate the contributions of students in autotutor. Interact. Learn. Environ. 8(2), 129–147 (2000)

    Article  Google Scholar 

  27. Grivokostopoulou, F., Perikos, I., Hatzilygeroudis, I.: An educational system for learning search algorithms and automatically assessing student performance. Int. J. Artif. Intell. Educ. 27(1), 207–240 (2017)

    Article  Google Scholar 

  28. Hattie, J., Timperley, H.: The power of feedback. Rev. Educ. Res. 77(1), 81–112 (2007)

    Article  Google Scholar 

  29. Heeren, B., Jeuring, J.: Feedback services for stepwise exercises. Sci. Comput. Program. 88, 110–129 (2014)

    Article  Google Scholar 

  30. Heffernan, N.T., Koedinger, K.R.: An intelligent tutoring system incorporating a model of an experienced human tutor. In: Cerri, S.A., Gouardères, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, pp. 596–608. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-47987-2_61

    Chapter  MATH  Google Scholar 

  31. Hennecke, M.: Online Diagnose in intelligenten mathematischen Lehr-Lern-Systemen. Ph.D. thesis, Hildesheim University (1999). in German

    Google Scholar 

  32. Hong, J.: Guided programming and automated error analysis in an intelligent prolog tutor. Int. J. Hum Comput Stud. 61(4), 505–534 (2004)

    Article  Google Scholar 

  33. Jaques, P.A., et al.: Rule-based expert systems to support step-by-step guidance in algebraic problem solving: the case of the tutor pat2math. Expert Syst. Appl. 40(14), 5456–5465 (2013)

    Article  Google Scholar 

  34. Jeuring, J., Gerdes, A., Heeren, B.: A programming tutor for haskell. In: Zsók, V., Horváth, Z., Plasmeijer, R. (eds.) CEFP 2011. LNCS, vol. 7241, pp. 1–45. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-32096-5_1

    Chapter  Google Scholar 

  35. Jin, W., Barnes, T., Stamper, J., Eagle, M.J., Johnson, M.W., Lehmann, L.: Program representation for automatic hint generation for a data-driven novice programming tutor. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 304–309. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-30950-2_40

    Chapter  Google Scholar 

  36. Jin, W., Corbett, A., Lloyd, W., Baumstark, L., Rolka, C.: Evaluation of guided-planning and assisted-coding with task relevant dynamic hinting. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 318–328. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07221-0_40

    Chapter  Google Scholar 

  37. Johnson, B.G., Phillips, F., Chase, L.G.: An intelligent tutoring system for the accounting cycle: enhancing textbook homework with artificial intelligence. J. Account. Educ. 27(1), 30–39 (2009)

    Article  Google Scholar 

  38. Johnson, S.D., et al.: Application of cognitive theory to the design, development, and implementation of a computer-based troubleshooting tutor (1992)

    Google Scholar 

  39. Johnson, W.L.: Intention-Based Diagnosis of Novice Programming Errors. MorganKaufmann, Los Altos (1986)

    MATH  Google Scholar 

  40. Keuning, H., Heeren, B., Jeuring, J.: Strategy-based feedback in a programming tutor. In: Proceedings of the Computer Science Education Research Conference, pp. 43–54. ACM (2014)

    Google Scholar 

  41. Keuning, H., Jeuring, J., Heeren, B.: A systematic literature review of automated feedback generation for programming exercises. ACM Trans. Comput. Educ. 19(1), 3:1–3:43 (2018)

    Article  Google Scholar 

  42. Kim, N., Evens, M., Michael, J.A., Rovick, A.A.: Circsim-tutor: an intelligent tutoring system for circulatory physiology. In: Maurer, H. (ed.) ICCAL 1989. LNCS, vol. 360, pp. 254–266. Springer, Heidelberg (1989). https://doi.org/10.1007/3-540-51142-3_64

    Chapter  Google Scholar 

  43. Koedinger, K.R., Aleven, V., Heffernan, N., McLaren, B., Hockenberry, M.: Opening the door to non-programmers: authoring intelligent tutor behavior by demonstration. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220, pp. 162–174. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30139-4_16

    Chapter  Google Scholar 

  44. Koedinger, K.R., Anderson, J.R.: Reifying implicit planning in geometry guidelines for model-based intelligent. In: Lajoie, S., Derry, S. (eds.) Computers as Cognitive Tools. Erlbaum, Hillsdale (2013)

    Google Scholar 

  45. Lee, W., de Silva, R., Peterson, E.J., Calfee, R.C., Stahovich, T.F.: Newton’s pen: a pen-based tutoring system for statics. Comput. Graph. 32(5), 511–524 (2008)

    Article  Google Scholar 

  46. Lester, J.C., Stone, B.A., O’Leary, M.A., Stevenson, R.B.: Focusing problem solving in design-centered learning environments. In: Frasson, C., Gauthier, G., Lesgold, A. (eds.) ITS 1996. LNCS, vol. 1086, pp. 475–483. Springer, Heidelberg (1996). https://doi.org/10.1007/3-540-61327-7_146

    Chapter  Google Scholar 

  47. Lester, J.C., Stone, B.A., Stelling, G.D.: Lifelike pedagogical agents for mixed-initiative problem solving in constructivist learning environments. User Model. User-Adap. Inter. 9(1–2), 1–44 (1999)

    Article  Google Scholar 

  48. Lodder, J., Heeren, B., Jeuring, J.: A domain reasoner for propositional logic. J. Univ. Comput. Sci. 22(8), 1097–1122 (2016)

    MathSciNet  Google Scholar 

  49. Looi, C.K.: Automatic debugging of prolog programs in a prolog intelligent tutoring system. Instr. Sci. 20(2–3), 215–263 (1991)

    Article  Google Scholar 

  50. Mitrovic, A., Suraweera, P., Martin, B.: Intelligent tutors for all: the constraint-based approach. IEEE Intell. Syst. 22(4), 38–45 (2007)

    Article  Google Scholar 

  51. Razzaq, L.M., et al.: Blending assessment and instructional assisting. In: AIED, pp. 555–562 (2005)

    Google Scholar 

  52. Rivers, K., Koedinger, K.R.: Data-driven hint generation in vast solution spaces: a self-improving python programming tutor. Int. J. Artif. Intell. Educ. 27(1), 37–64 (2017)

    Article  Google Scholar 

  53. Roll, I., Aleven, V., Koedinger, K.R.: The invention lab: using a hybrid of model tracing and constraint-based modeling to offer intelligent support in inquiry environments. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010. LNCS, vol. 6094, pp. 115–124. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13388-6_16

    Chapter  Google Scholar 

  54. Sangwin, C.: Computer Aided Assessment of Mathematics. Oxford University Press, Oxford (2013)

    Book  MATH  Google Scholar 

  55. Sklavakis, D., Refanidis, I.: An individualized web-based algebra tutor based on dynamic deep model tracing. In: Darzentas, J., Vouros, G.A., Vosinakis, S., Arnellos, A. (eds.) SETN 2008. LNCS (LNAI), vol. 5138, pp. 389–394. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-87881-0_38

    Chapter  Google Scholar 

  56. Sklavakis, D., Refanidis, I.: Mathesis: an intelligent web-based algebra tutoring school. Int. J. Artif. Intell. Educ. 22(4), 191–218 (2013)

    Google Scholar 

  57. Song, J., Hahn, S., Tak, K., Kim, J.: An intelligent tutoring system for introductory C language course. Comput. Educ. 28(2), 93–102 (1997)

    Article  Google Scholar 

  58. Suraweera, P., Mitrovic, A.: KERMIT: a constraint-based tutor for database modeling. In: Cerri, S.A., Gouardères, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, pp. 377–387. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-47987-2_41

    Chapter  Google Scholar 

  59. Sykes, E.R.: Design, development and evaluation of the Java intelligent tutoring system. Technol. Instr. Cogn. Learn. 8(1), 25–65 (2010)

    Google Scholar 

  60. VanLehn, K.: The behavior of tutoring systems. Int. J. Artif. Intell. Educ. 16(3), 227–265 (2006)

    Google Scholar 

  61. VanLehn, K.: The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ. Psychol. 46(4), 197–221 (2011)

    Article  Google Scholar 

  62. VanLehn, K., et al.: The Architecture of Why2-Atlas: A Coach for Qualitative Physics Essay Writing. In: Cerri, S.A., Gouardères, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, pp. 158–167. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-47987-2_20

    Chapter  Google Scholar 

  63. VanLehn, K.: The andes physics tutoring system: lessons learned. Int. J. Artifi. Intell. Educ. 15(3), 147–204 (2005)

    Google Scholar 

  64. Weber, G., Brusilovsky, P.: Elm-art: an adaptive versatile system for web-based instruction. Int. J. Artif. Intell. Educ. (IJAIED) 12, 351–384 (2001)

    Google Scholar 

  65. Weragama, D., Reye, J.: Analysing student programs in the php intelligent tutoring system. Int. J. Artif. Intell. Educ. 24(2), 162–188 (2014)

    Article  Google Scholar 

  66. Wetzel, J., et al.: The design and development of the dragoon intelligent tutoring system for model construction: lessons learned. Interact. Learn. Environ. 25(3), 361–381 (2017)

    Article  MathSciNet  Google Scholar 

  67. Zatarain-Cabada, R., Barrón-Estrada, M.L., Pérez, Y.H., Reyes-García, C.A.: Designing and implementing affective and intelligent tutoring systems in a learning social network. In: Batyrshin, I., Mendoza, M.G. (eds.) MICAI 2012. LNCS (LNAI), vol. 7630, pp. 444–455. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37798-3_39

    Chapter  Google Scholar 

  68. Zinn, C.: Algorithmic debugging to support cognitive diagnosis in tutoring systems. In: Bach, J., Edelkamp, S. (eds.) KI 2011. LNCS (LNAI), vol. 7006, pp. 357–368. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24455-1_35

    Chapter  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers and the members of the Utrecht reading club on educational technology for their helpful suggestions.

Author information

Authors and Affiliations

  1. Department of Information and Computing Sciences, Universiteit Utrecht, Utrecht, The Netherlands

    Renate van der Bent & Johan Jeuring

  2. Faculty of Management, Science & Technology, Open University of the Netherlands, Heerlen, The Netherlands

    Johan Jeuring & Bastiaan Heeren

Authors
  1. Renate van der Bent
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Johan Jeuring
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bastiaan Heeren
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Johan Jeuring .

Editor information

Editors and Affiliations

  1. Open University Netherlands, Heerlen, The Netherlands

    Maren Scheffel

  2. Paul Sabatier University, Toulouse, France

    Julien Broisin

  3. Know-Center GmbH, Graz, Austria

    Viktoria Pammer-Schindler

  4. Cyprus University of Technology, Limassol, Cyprus

    Andri Ioannou

  5. DIPF, Frankfurt/Main, Germany

    Jan Schneider

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

van der Bent, R., Jeuring, J., Heeren, B. (2019). The Diagnosing Behaviour of Intelligent Tutoring Systems. In: Scheffel, M., Broisin, J., Pammer-Schindler, V., Ioannou, A., Schneider, J. (eds) Transforming Learning with Meaningful Technologies. EC-TEL 2019. Lecture Notes in Computer Science(), vol 11722. Springer, Cham. https://doi.org/10.1007/978-3-030-29736-7_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-29736-7_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-29735-0

  • Online ISBN: 978-3-030-29736-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation