Example-Tracing Tutors: Intelligent Tutor Development for Non-programmers

  • Vincent Aleven
  • Bruce M. McLaren
  • Jonathan Sewall
  • Martin van Velsen
  • Octav Popescu
  • Sandra Demi
  • Michael Ringenberg
  • Kenneth R. Koedinger
Article

Abstract

In 2009, we reported on a new Intelligent Tutoring Systems (ITS) technology, example-tracing tutors, that can be built without programming using the Cognitive Tutor Authoring Tools (CTAT). Creating example-tracing tutors was shown to be 4–8 times as cost-effective as estimates for ITS development from the literature. Since 2009, CTAT and its associated learning management system, the Tutorshop, have been extended and have been used for both research and real-world instruction. As evidence that example-tracing tutors are an effective and mature ITS paradigm, CTAT-built tutors have been used by approximately 44,000 students and account for 40 % of the data sets in DataShop, a large open repository for educational technology data sets. We review 18 example-tracing tutors built since 2009, which have been shown to be effective in helping students learn in real educational settings, often with large pre/post effect sizes. These tutors support a variety of pedagogical approaches, beyond step-based problem solving, including collaborative learning, educational games, and guided invention activities. CTAT and other ITS authoring tools illustrate that non-programmer approaches to building ITS are viable and useful and will likely play a key role in making ITS widespread.

Keywords

Intelligent tutoring systems Authoring tools Example-tracing tutors 

References

  1. Adams, D. M., McLaren, B. M., Durkin, K., Mayer, R. E., Rittle-Johnson, B., Isotani, S., & Velsen, M. V. (2014). Using erroneous examples to improve mathematics learning with a web-based tutoring system. Computers in Human Behavior, 36, 401–411. doi:10.1016/j.chb.2014.03.053.CrossRefGoogle Scholar
  2. Adamson, D., Dyke, G., Jang, H., & Rosé, C. P. (2014). Towards an agile approach to adapting dynamic collaboration support to student needs. International Journal of Artificial Intelligence in Education, 24(1), 92–124. doi:10.1007/s40593-013-0012-6.CrossRefGoogle Scholar
  3. Aleven, V. (2010). Rule-Based cognitive modeling for intelligent tutoring systems. In R. Nkambou, J. Bourdeau, & R. Mizoguchi (Eds.), Studies in computational intelligence (Advances in intelligent tutoring systems, Vol. 308, pp. 33–62). Berlin: Springer. doi:10.1007/978-3-642-14363-2_3.Google Scholar
  4. Aleven, V. (2015). A is for Adaptivity, but what is Adaptivity? Re-defining the field of AIED. In K. Porayska-Pomsta, G. McCalla, & B. du Boulay (Eds.). Proceedings of the Workshops at the 17th International Conference on Artificial Intelligence in Education AIED 2015 (Vol. 4, Workshop on Les Contes du Mariage: Should AI stay married to Ed? A workshop examining the current and future identity of the AIED field). Available from: http://adenu.ia.uned.es/publications/aied2015ws/procs/v2/.
  5. Aleven, V., & Koedinger, K. R. (2013). Knowledge component approaches to learner modeling. In R. Sottilare, A. Graesser, X. Hu, & H. Holden (Eds.), Design recommendations for adaptive intelligent tutoring systems (Learner modeling, Vol. I, pp. 165–182). Orlando: US Army Research Laboratory.Google Scholar
  6. Aleven, V., McLaren, B. M., Roll, I., & Koedinger, K. R. (2006a). Toward meta-cognitive tutoring: A model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education, 16, 101–128.Google Scholar
  7. Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. R. (2006b). The Cognitive Tutor Authoring Tools (CTAT): Preliminary evaluation of efficiency gains. In M. Ikeda, K. D. Ashley, & T. W. Chan (Eds.), Proceedings of the 8th International Conference on Intelligent Tutoring Systems, ITS 2006 (pp. 61–70). Berlin: Springer. doi:10.1007/11774303_7.Google Scholar
  8. Aleven, V., Sewall, J., McLaren, B. M., & Koedinger, K. R. (2006c). Rapid authoring of intelligent tutors for real-world and experimental use. In Kinshuk, R. Koper, P. Kommers, P. Kirschner, D. G. Sampson, & W. Didderen (Eds.), Proceedings of the 6th IEEE International Conference on Advanced Learning Technologies, ICALT 2006 (pp. 847–851). Los Alamitos: IEEE Computer Society.Google Scholar
  9. Aleven, V., McLaren, B. M., & Sewall, J. (2009a). Scaling up programming by demonstration for intelligent tutoring systems development: an open-access web site for middle school mathematics learning. IEEE Transactions on Learning Technologies, 2(2), 64–78.CrossRefGoogle Scholar
  10. Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. R. (2009b). A new paradigm for intelligent tutoring systems: Example-tracing tutors. International Journal of Artificial Intelligence in Education, 19(2), 105–154.Google Scholar
  11. Aleven, V., Beal, C. R., & Graesser, A. C. (2013). Introduction to the special issue on advanced learning technologies. Journal of Educational Psychology, 105(4), 929. doi:10.1037/a0034155.CrossRefGoogle Scholar
  12. Aleven, V., Sewall, J., Popescu, O., van Velsen, M., Demi, S., & Leber, B. (2015a). Reflecting on twelve years of ITS authoring tools research with CTAT. In R. Sottilare, A. Graesser, X. Hu, & K. Brawner (Eds.), Design recommendations for adaptive intelligent tutoring systems (Authoring Tools, Vol. III, pp. 263–283). Orlando: US Army Research Laboratory.Google Scholar
  13. Aleven, V., Sewall, J., Popescu, O., Xhakaj, F., Chand, D., Baker, R. S., & Gasevic, D. (2015b). The beginning of a beautiful friendship? Intelligent tutoring systems and MOOCs. In C. Conati, N. Heffernan, A. Mitrovic, & M. F. Verdejo (Eds.), Proceedings of the 17th International Conference on AI in Education, AIED 2015 (pp. 525–528). New York: Springer. doi:10.1007/978-3-319-19773-9_53.Google Scholar
  14. Aleven, V., McLaughlin, E. A., Glenn, R. A., & Koedinger, K. R. (forthcoming). Instruction based on adaptive learning technologies. In R. E. Mayer & P. Alexander (Eds.), Handbook of research on learning and instruction. Routledge.Google Scholar
  15. Anderson, J. R., Conrad, F. G., & Corbett, A. T. (1989). Skill acquisition and the LISP tutor. Cognitive Science, 13(4), 467–505. doi:10.1016/0364-0213(89)90021-9.CrossRefGoogle Scholar
  16. Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4(2), 167–207.CrossRefGoogle Scholar
  17. Arroyo, I., Beck, J., Woolf, B. P., Beal, C. R., & Schultz, K. (2000). Macroadapting animal watch to gender and cognitive differences with respect to hint interactivity and symbolism. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), Proceedings of the 5th International Conference on Intelligent Tutoring Systems, ITS 2000 (pp. 574–583). Berlin: Springer Verlag.Google Scholar
  18. Arroyo, I., Woolf, B. P., Burleson, W., Muldner, K., Rai, D., & Tai, M. (2014). A multimedia adaptive tutoring system for mathematics that addresses cognition, metacognition and affect. International Journal of Artificial Intelligence in Education, 24(4), 387–426. doi:10.1007/s40593-014-0023-y.CrossRefGoogle Scholar
  19. Atkinson, R. K., Renkl, A., & Merrill, M. M. (2003). Transitioning from studying examples to solving problems: Effects of self-explanation prompts and fading worked-out steps. Journal of Educational Psychology, 95(4), 774–783.CrossRefGoogle Scholar
  20. Baghaei, N., Mitrovic, A., & Irwin, W. (2007). Supporting collaborative learning and problem-solving in a constraint-based CSCL environment for UML class diagrams. International Journal of Computer-Supported Collaborative Learning, 2(2–3), 159–190. doi:10.1007/s11412-007-9018-0.CrossRefGoogle Scholar
  21. Blessing, S. B., & Gilbert, S. (2008). Evaluating an authoring tool for model-tracing intelligent tutoring systems. In B. Woolf, E. Aimeur, R. Nkambou, & S. Lajoie (Eds.), ITS ‘08: Proceedings of the 9th International Conference on Intelligent Tutoring Systems (pp. 204–215). Berlin: Springer. doi:10.1007/978-3-540-69132-7_25.CrossRefGoogle Scholar
  22. Blessing, S. B., Gilbert, S. B., Blankenship, L., & Sanghvi, B. (2009). From SDK to xPST: A new way to overlay a tutor on existing software. In Proceedings of the 2009 FLAIRS Conference (pp. 466–467). Association for the Advancement of Artificial Intelligence.Google Scholar
  23. Blessing, S. B., Gilbert, S. B., Ourada, S., & Ritter, S. (2009b). Authoring model-tracing cognitive tutors. International Journal of Artificial Intelligence in Education, 19(2), 189–210.Google Scholar
  24. Blessing, S., Devasani, S., & Gilbert, S. (2011). Evaluation of WebxPST: A browser-based authoring tool for problem-specific tutors. In G. Biswas, S. Bull, J. Kay, & A. Mitrovic (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, AIED 2011 (pp. 423–425). Berlin: Springer.Google Scholar
  25. Booth, J. L., Lange, K. E., Koedinger, K. R., & Newton, K. J. (2013). Using example problems to improve student learning in algebra: Differentiating between correct and incorrect examples. Learning and Instruction, 25, 24–34. doi:10.1016/j.learninstruc.2012.11.002.CrossRefGoogle Scholar
  26. Borek, A., McLaren, B. M., Karabinos, M., & Yaron, D. (2009). How much assistance is helpful to students in discovery learning? In U. Cress, V. Dimitrova, & M. Specht (Eds.), Proceedings 4th European Conference on Technology-Enhanced Learning, EC-TEL 2009 (pp. 391–404). Berlin: Springer. doi:10.1007/978-3-642-04636-0_38.Google Scholar
  27. Bull, S., & Kay, J. (2010). Open learner models. In R. Nkambou, J. Bourdeau, & R. Mizoguchi (Eds.), Studies in Computational Intelligence: Vol. 308. Advances in intelligent tutoring systems (pp. 301-322). Berlin, Heidelberg: Springer. doi:10.1007/978-3-642-14363-2_15.
  28. Chase, C., Marks, J., Bernett, D., Bradley, M., & Aleven, V. (2015a). Towards the development of the invention coach: A naturalistic study of teacher guidance for an exploratory learning task. In C. Conati, N. Heffernan, A. Mitrovic, & M. F. Verdejo (Eds.), Proceedings of the 17th International Conference on Artificial Intelligence in Education, AIED 2015 (pp. 558–561). New York: Springer. doi:10.1007/978-3-319-19773-9_6.Google Scholar
  29. Chase, C., Marks, J., Bernett, D., & Aleven, V. (2015b). The design of an exploratory learning environment to support Invention. Paper presented during the Workshop on Intelligent Support in Exploratory and Open-Ended Learning Environments, held as part of the 17th International Conference on Artificial Intelligence in Education, AIED 2015.Google Scholar
  30. Clark, R. E., Feldon, D., van Merriënboer, J. J. G., Yates, K., & Early, S. (2008). Cognitive task analysis. In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). New York: Macmillan/Gale.Google Scholar
  31. Conati, C., & Vanlehn, K. (2000). Toward computer-based support of meta-cognitive skills: A computational framework to coach self-explanation. International Journal of Artificial Intelligence in Education, 11(4), 389–415.Google Scholar
  32. Cook, R., Kay, J., & Kummerfeld, B. (2015). MOOClm: User modelling for MOOCs. In F. Ricci, K. Bontcheva, O. Conlan, & S. Lawless (Eds.), Proceedings of the 23rd International Conference on User Modeling, Adaptation and Personalization, UMAP 2015 (pp. 80–91). Springer International Publishing. doi:10.1007/978-3-319-20267-9_7.
  33. Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4(4), 253–278.CrossRefGoogle Scholar
  34. Corbett, A., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive Tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81–108.CrossRefGoogle Scholar
  35. Corbett, A., Kauffman, L., MacLaren, B., Wagner, A., & Jones, E. (2010). A Cognitive Tutor for genetics problem solving: Learning gains and student modeling. Journal of Educational Computing Research, 42(2), 219–239.CrossRefGoogle Scholar
  36. de Baker, R. S. J., Corbett, A. T., & Koedinger, K. R. (2007). The difficulty factors approach to the design of lessons in intelligent tutor curricula. International Journal of Artificial Intelligence in Education, 17(4), 341–369.Google Scholar
  37. Devasani, S., Gilbert, S. B., & Blessing, S. B. (2012). Evaluation of two intelligent tutoring system authoring tool paradigms: Graphical user interface-based and text-based. In Proceedings 21st Annual Conference on Behavior Representation in Modeling and Simulation 2012 (BRiMS 2012) (pp. 51–58). Red Hook, NY: Curran Associates, Inc.Google Scholar
  38. Doroudi, S., Holstein, K., Aleven, V., & Brunskill, E. (2015). Towards understanding how to leverage sense-making, induction/refinement and fluency to improve robust learning. In O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, et al. (Eds.), Proceedings of the 8th International Conference on Educational Data Mining, EDM 2015 (pp. 376–379). Worcester: International Educational Data Mining Society.Google Scholar
  39. Earnshaw, Y. (2014). Effects of levels of instructional assistance on learning and mental effort in an intelligent tutoring system: Proportional reasoning and middle school students. Doctoral dissertation. Available from ProQuest Dissertations and Theses database. (UMI No. 3637974).Google Scholar
  40. Forlizzi, J., McLaren, B. M., Ganoe, C., McLaren, P. B., Kihumba, G., & Lister, K. (2014). Decimal point: Designing and developing an educational game to teach decimals to middle school students. In C. Busch (Ed.), 8th European Conference on Games-Based Learning: ECGBL2014 (pp. 128–135). Reading: Academic Conferences and Publishing International.Google Scholar
  41. Goguadze, G., Sosnovsky, S., Isotani, S., & McLaren, B. M. (2011). Evaluating a Bayesian student model of decimal misconceptions. In M. Pechenizkiy, T. Calders, C. Conati, S. Ventura, C. Romero, & J. Stamper (Eds.), Proceedings of the 4th International Conference on Educational Data Mining, EDM 2011 (pp. 301–306). Worcester: International Data Mining Society.Google Scholar
  42. Guan, C. Q., Liu, Y., Chan, D. H. L., Ye, F., & Perfetti, C. A. (2011). Writing strengthens orthography and alphabetic-coding strengthens phonology in learning to read Chinese. Journal of Educational Psychology, 103(3), 509–522. doi:10.1037/a0023730.CrossRefGoogle Scholar
  43. Harpstead, E., MacLellan, C. J., Aleven, V., & Myers, B. A. (2015). Replay analysis in open-ended educational games. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics (pp. 381–399). New York: Springer International Publishing. doi:10.1007/978-3-319-05834-4_17.CrossRefGoogle Scholar
  44. Heffernan, N. T., & Heffernan, C. L. (2014). The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education, 24(4), 470–497. doi:10.1007/s40593-014-0024-x.MathSciNetCrossRefGoogle Scholar
  45. Holmes, N. G., Day, J., Park, A. H., Bonn, D. A., & Roll, I. (2014). Making the failure more productive: Scaffolding the invention process to improve inquiry behaviours and outcomes in productive failure activities. Instructional Science, 42(4), 523–538. doi:10.1007/s11251-013-9300-7.CrossRefGoogle Scholar
  46. Janssen, J., & Bodemer, D. (2013). Coordinated computer-supported collaborative learning: Awareness and awareness tools. Educational Psychologist, 48(1), 40–55.CrossRefGoogle Scholar
  47. Kalyuga, S. (2007). Expertise reversal effect and its implications for learner-tailored instruction. Educational Psychology Review, 19(4), 509–539. doi:10.1007/s10648-007-9054-3.CrossRefGoogle Scholar
  48. Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31. doi:10.1207/S15326985EP3801_4.CrossRefGoogle Scholar
  49. Kay, J., Reimann, P., Diebold, E., & Kummerfeld, B. (2013). MOOCs: So many learners, so much potential. IEEE Intelligent Systems, 3, 70–77. doi:10.1109/MIS.2013.66.CrossRefGoogle Scholar
  50. Kleij, F. M. V. D., Feskens, R. C. W., & Eggen, T. J. H. M. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes. Review of Educational Research. doi:10.3102/0034654314564881.Google Scholar
  51. Kodaganallur, V., Weitz, R., & Rosenthal, D. (2005). A comparison of model-tracing and constraint-based intelligent tutoring paradigms. International Journal of Artificial Intelligence in Education, 15(1), 117–144.Google Scholar
  52. Kodavali, S. K., Gilbert, S., & Blessing, S. (2010). Expansion of the xPST framework to enable non-programmers to create intelligent tutoring systems in 3D game environments. In V. Aleven, J. Kay, & J. Mostow (Eds.), Lecture Notes in Computer Science: Proceedings of the 10th International Conference on Intelligent Tutoring Systems, ITS 2010, vol. 2 (Vol. 6095, pp. 365–367). Berlin: Springer.Google Scholar
  53. Koedinger, K. R. (2002). Toward evidence for instructional design principles: Examples from Cognitive Tutor Math 6. Invited paper. In Proceedings of PME-NA XXXIII (the North American Chapter of the International Group for the Psychology of Mathematics Education (pp. 21–49).Google Scholar
  54. Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19(3), 239–264.CrossRefGoogle Scholar
  55. Koedinger, K. R., & Corbett, A. T. (2006). Cognitive Tutors: Technology bringing learning sciences to the classroom. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 61–78). New York: Cambridge University Press.Google Scholar
  56. Koedinger, K., & Mitrovic, A. (2009). Authoring intelligent tutoring systems: Preface for special issue on authoring intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 19(2), 103–104.Google Scholar
  57. Koedinger, K. R., & Terao, A. (2002). A cognitive task analysis of using pictures to support pre-algebraic reasoning. In W. Gray & C. D. Schunn (Eds.), Proceedings of the Twenty-fourth Annual Conference of the Cognitive Science Society (pp. 542–547). Taylor & Francis Group.Google Scholar
  58. Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8(1), 30–43.Google Scholar
  59. Koedinger, K. R., Suthers, D. D., & Forbus, K. D. (1999). Component-based construction of a science learning space. International Journal of Artificial Intelligence in Education, 10(3), 292–313.Google Scholar
  60. Koedinger, K. R., Aleven, V., Heffernan, N., McLaren, B., & Hockenberry, M. (2004). Opening the door to non-programmers: Authoring intelligent tutor behavior by demonstration. In J. C. Lester, R. M. Vicario, & F. Paraguaçu (Eds.), Proceedings of seventh International Conference on Intelligent Tutoring Systems, ITS 2004 (pp. 162–174). Berlin: Springer.Google Scholar
  61. Koedinger, K. R., de Baker, R. S. J., Cunningham, K., Skogsholm, A., Leber, B., & Stamper, J. (2010). A data repository for the EDM community: The PSLC datashop. In S. Ventura, C. Romero, M. Pechenizkiy, & R. S. J. D. Baker (Eds.), Handbook of educational data mining (pp. 43–55). Boca Raton: CRC Press.Google Scholar
  62. Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012). The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36(5), 757–798. doi:10.1111/j.1551-6709.2012.01245.x.CrossRefGoogle Scholar
  63. Kumar, R., & Kim, J. (2014). Preface to the special issue on intelligent support for learning in groups. International Journal of Artificial Intelligence in Education, 24(1), 1–7. doi:10.1007/s40593-013-0013-5.CrossRefGoogle Scholar
  64. Liew, C. W., & Xhakaj, F. (2015). Teaching a complex process: Insertion in red black trees. In C. Conati, N. Heffernan, A. Mitrovic, & M. F. Verdejo (Eds.), Proceedings of the 17th International Conference on Artificial Intelligence in Education, AIED 2015 (pp. 698–701). New York: Springer International Publishing. doi:10.1007/978-3-319-19773-9_95.Google Scholar
  65. Liu, Y., Wang, M., Perfetti, C. A., Brubaker, B., Wu, S., & MacWhinney, B. (2011). Learning a tonal language by attending to the tone: an in vivo experiment. Language Learning, 61(4), 1119–1141. doi:10.1111/j.1467-9922.2011.00673.x.CrossRefGoogle Scholar
  66. Long, Y., & Aleven, V. (2013a). Active learners: Redesigning an intelligent tutoring system to support self-regulated learning. In D. Hernández-Leo, T. Ley, R. Klamma, & A. Harrer (Eds.), Scaling up learning for sustained impact, Proceedings of the Eighth European Conference on Technology Enhanced Learning (EC-TEL 2013) (pp. 490–495). Berlin: Springer. doi:10.1007/978-3-642-40814-4_44.ShortPaper.CrossRefGoogle Scholar
  67. Long, Y., & Aleven, V. (2013b). Supporting students’ self-regulated learning with an open learner model in a linear equation tutor. In H. C. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Proceedings of the 16th International Conference on Artificial Intelligence in Education, AIED 2013 (pp. 249–258). Berlin: Springer.Google Scholar
  68. Long, Y., & Aleven, V. (2014). Gamification of joint student/system control over problem selection in a linear equation tutor. In S. Trausan-Matu, K. E. Boyer, M. Crosby, & K. Panourgia (Eds.), Proceedings of the 12th International Conference on Intelligent Tutoring Systems, ITS 2014 (pp. 378–387). New York: Springer. doi:10.1007/978-3-319-07221-0_47.Google Scholar
  69. Lovett, M. C. (1998). Cognitive task analysis in the service of intelligent tutoring system design: A case study in statistics. In B. P. Goettle, H. M. Halff, C. L. Redfield, & V. J. Shute (Eds.), Intelligent Tutoring Systems, Proceedings of the Fourth International Conference, ITS 1998 (pp. 234–243). Berlin: Springer Verlag.Google Scholar
  70. MacLellan, C., Koedinger, K. R., & Matsuda, N. (2014). Authoring tutors with SimStudent: An evaluation of efficiency and model quality. In S. Trausan-Matu, K. E. Boyer, M. Crosby, & K. Panourgia (Eds.), Proceedings of the 12th International Conference on Intelligent Tutoring Systems, ITS 2014 (pp. 551–560). Berlin: Springer. doi:10.1007/978-3-319-07221-0_66.Google Scholar
  71. Mathan, S. A., & Koedinger, K. R. (2005). Fostering the intelligent novice: Learning from errors with metacognitive tutoring. Educational Psychologist, 40(4), 257–265.CrossRefGoogle Scholar
  72. Matsuda, N., Cohen, W. W., & Koedinger, K. R. (2015). Teaching the teacher: Tutoring SimStudent leads to more effective Cognitive Tutor authoring. International Journal of Artificial Intelligence in Education, 25(1), 1–34. doi:10.1007/s40593-014-0020-1.CrossRefGoogle Scholar
  73. McKendree, J. (1990). Effective feedback content for tutoring complex skills. Human Computer Interaction, 5(4), 381–413. doi:10.1207/s15327051hci0504_2.CrossRefGoogle Scholar
  74. McLaren, B. M., Lim, S. J., & Koedinger, K. R. (2008). When and how often should worked examples be given to students? New results and a summary of the current state of research. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Meeting of the Cognitive Science Society (pp. 2176–2181). Austin: Cognitive Science Society.Google Scholar
  75. McLaren, B. M., DeLeeuw, K. E., & Mayer, R. E. (2011a). A politeness effect in learning with web-based intelligent tutors. International Journal of Human Computer Studies, 69(1–2), 70–79. doi:10.1016/j.ijhcs.2010.09.001.CrossRefGoogle Scholar
  76. McLaren, B. M., DeLeeuw, K. E., & Mayer, R. E. (2011b). Polite web-based intelligent tutors: Can they improve learning in classrooms? Computers & Education, 56(3), 574–584.CrossRefGoogle Scholar
  77. McLaren, B. M., Adams, D., Durkin, K., Goguadze, G., Mayer, R. E., Rittle-Johnson, B., & Velsen, M. V. (2012). To err is human, to explain and correct is divine: A study of interactive erroneous examples with middle school math students. In A. Ravenscroft, S. Lindstaedt, C. Delgado Kloos, & D. Hernández-Leo (Eds.), 21st-Century Learning for 21st-Century Skills: 7th European Conference on Technology-Enhanced Learning, EC-TEL 2012 (pp. 222–235). Berlin: Springer. doi:10.1007/978-3-642-33263-0_18.CrossRefGoogle Scholar
  78. McLaren, B. M., van Gog, T., Ganoe, C., Yaron, D., & Karabinos, M. (2014). Exploring the assistance dilemma: Comparing instructional support in examples and problems. In S. Trausan-Matu, K. E. Boyer, M. Crosby, & K. Panourgia (Eds.), Proceedings of the 12th International Conference on Intelligent Tutoring Systems, ITS 2014 (pp. 354–361). Berlin: Springer. doi:10.1007/978-3-319-07221-0_66.Google Scholar
  79. McLaren, B. M., Adams, D. M., & Mayer, R. E. (2015a). Delayed learning effects with erroneous examples: a study of learning decimals with a web-based tutor. International Journal of Artificial Intelligence in Education, 25(4), 520–542.CrossRefGoogle Scholar
  80. McLaren, B. M., van Gog, T., Ganoe, C., Yaron, D., & Karabinos, M. (2015b). Worked Examples are more efficient for learning than high-assistance instructional software. In C. Conati, N. Heffernan, A. Mitrovic, & M. F. Verdejo (Eds.), Proceedings of the 17th International Conference on AI in Education, AIED 2015 (pp. 710–713). Berlin: Springer. doi:10.1007/978-3-319-19773-9_98.Google Scholar
  81. McLaren, B. M., van Gog, T., Ganoe, C., Karabinos, M., & Yaron, D. (2016). The efficiency of worked examples compared to erroneous examples, tutored problem solving, and problem solving in classroom experiments. Computers in Human Behavior, 55, 87–99.CrossRefGoogle Scholar
  82. Mitrovic, A., & Ohlsson, S. (1999). Evaluation of a constraint-based tutor for a database language. International Journal of Artificial Intelligence in Education, 10(3–4), 238–256.Google Scholar
  83. Mitrovic, A., Martin, B., Suraweera, P., Zakharov, K., Milik, N., Holland, J., & Mcguigan, N. (2009). ASPIRE: An authoring system and deployment environment for constraint-based tutors. International Journal of Artificial Intelligence in Education, 19(2), 155–188.Google Scholar
  84. Murray, T. (1999). Authoring intelligent tutoring systems: An analysis of the state of the art. International Journal of Artificial Intelligence in Education, 10(1), 98–129.Google Scholar
  85. Murray, T., Blessing, S., & Ainsworth, S. (2003). Authoring tools for advanced technology learning environments: Toward cost-effective adaptive, interactive and intelligent educational software. Amsterdam: Kluwer Academic Publishers.CrossRefGoogle Scholar
  86. Nathan, M. J. (1998). Knowledge and situational feedback in a learning environment for algebra story problem solving. Interactive Learning Environments, 5(1), 135–159.MathSciNetCrossRefGoogle Scholar
  87. Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs: Prentice-Hall.Google Scholar
  88. Newell, A., & Simon, H. A. (1976). Computer science as empirical inquiry: Symbols and search. Communications of the ACM, 19(3), 113–126. doi:10.1145/360018.360022.MathSciNetCrossRefGoogle Scholar
  89. Nye, B. D., Graesser, A. C., & Hu, X. (2014). AutoTutor and family: A review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education, 24(4), 427–469. doi:10.1007/s40593-014-0029-5.
  90. Ogan, A., Aleven, V., & Jones, C. (2009). Advancing development of intercultural competence through supporting predictions in narrative video. International Journal of Artificial Intelligence in Education, 19(3), 267–288.Google Scholar
  91. Olsen, J. K., Belenky, D. M., Aleven, V., & Rummel, N. (2014a). Using an intelligent tutoring system to support collaborative as well as individual learning. In S. Trausan-Matu, K. E. Boyer, M. Crosby, & K. Panourgia (Eds.), Proceedings of the 12th International Conference on Intelligent Tutoring Systems, ITS 2014 (pp. 134–143). Berlin: Springer. doi:10.1007/978-3-319-07221-0_66.Google Scholar
  92. Olsen, J. K., Belenky, D. M., Aleven, V., Rummel, N., Sewall, J., & Ringenberg, M. (2014b). Authoring tools for collaborative intelligent tutoring system environments. In S. Trausan-Matu, K. E. Boyer, M. Crosby, & K. Panourgia (Eds.), Proceedings of the 12th International Conference on Intelligent Tutoring Systems, ITS 2014 (pp. 523–528). Berlin: Springer. doi:10.1007/978-3-319-07221-0_66.Google Scholar
  93. Olsen, J. K., Rummel, N., & Aleven, V. (under review). Investigating effects of embedding collaboration in an intelligent tutoring system for elementary school students.Google Scholar
  94. Paas, F. G. W. C., & Van Merriënboer, J. J. G. (1994). Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of Educational Psychology, 86(1), 122–133.CrossRefGoogle Scholar
  95. Pane, J. F., Griffin, B. A., McCaffrey, D. F., & Karam, R. (2013). Effectiveness of Cognitive Tutor Algebra I at scale. Educational Evaluation and Policy Analysis, 0162373713507480. doi:10.3102/0162373713507480.
  96. Paquette, L., Lebeau, J.-F., & Mayers, A. (2010). Authoring problem-solving tutors: A comparison between ASTUS and CTAT. In R. Nkambou, J. Bourdeau, & R. Mizoguchi (Eds.), Advances in intelligent tutoring systems (pp. 377–405). Berlin: Springer. doi:10.1007/978-3-642-14363-2_19.CrossRefGoogle Scholar
  97. Paquette, L., Lebeau, J.-F., Beaulieu, G., & Mayers, A. (2015). Designing a knowledge representation approach for the generation of pedagogical interventions by MTTs. International Journal of Artificial Intelligence in Education, 25(1), 118–156. doi:10.1007/s40593-014-0030-z.
  98. Pavlik, P. I., Brawner, K., Olney, A., & Mitrovic, A. (2013). A review of student models used in intelligent tutoring systems. In R. Sottilare, A. Graesser, X. Hu, & H. Holden (Eds.), Design recommendations for adaptive intelligent tutoring systems (Learner modeling, Vol. I, pp. 39–68). Orlando: US Army Research Laboratory.Google Scholar
  99. Peterson, J., Pardos, Z., Rau, M. A., Swigart, A., Gerber, C., & McKinsey, J. (2015). Understanding student success in chemistry using gaze tracking and pupillometry. In C. Conati, N. Heffernan, A. Mitrovic, & M. F. Verdejo (Eds.), Proceedings of the 17th International Conference on AI in Education, AIED 2015 (pp. 358–366). New York: Springer International Publishing. doi:10.1007/978-3-319-19773-9_36.Google Scholar
  100. Rau, M. A. (2015). Why do the rich get richer? A structural equation model to test how spatial skills affect learning with representations. In O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, et al. (Eds.), Proceedings of the 8th International Conference on Educational Data Mining, EDM 2015 (pp. 350–359). Worcester: International Educational Data Mining Society.Google Scholar
  101. Rau, M. A., & Wu, S. P. W. (2015). ITS support for conceptual and perceptual processes in learning with multiple graphical representations. In C. Conati, N. Heffernan, A. Mitrovic, & M. F. Verdejo (Eds.), Proceedings of the 17th International Conference on AI in education, AIED 2015 (pp. 398–407). New York: Springer International Publishing. doi:10.1007/978-3-319-19773-9_40.Google Scholar
  102. Rau, M. A., Aleven, V., Rummel, N., & Rohrbach, S. (2012). Sense making alone doesn’t do it: Fluency matters too! ITS support for robust learning with multiple representations. In S. A. Cerri, W. J. Clancey, G. Papadourakis, & K. Panourgia (Eds.), Proceedings of the 11th International Conference on Intelligent Tutoring Systems, ITS 2012 (pp. 174–184). Berlin: Springer.Google Scholar
  103. Rau, M., Aleven, V., & Rummel, N. (2013). Interleaved practice in multi-dimensional learning tasks: Which dimension should we interleave?. Learning and Instruction, 23, 98–114. doi:learninstruc.2012.07.003.Google Scholar
  104. Rau, M. A., Aleven, V., Rummel, N., & Pardos, Z. (2014). How should intelligent tutoring systems sequence multiple graphical representations of fractions? A multi-methods study. International Journal of Artificial Intelligence in Education, 24(1), 125–161. doi:10.1007/s40593-013-0011-7.CrossRefGoogle Scholar
  105. Rau, M. A., Aleven, V., & Rummel, N. (2015a). Successful learning with multiple graphical representations and self-explanation prompts. Journal of Educational Psychology, 107(1), 30–46. doi:10.1037/a0037211.CrossRefGoogle Scholar
  106. Rau, M. A., Michaelis, J. E., & Fay, N. (2015b). Connection making between multiple graphical representations: A multi-methods approach for domain-specific grounding of an intelligent tutoring system for chemistry. Computers & Education, 82, 460–485. doi:10.1016/j.compedu.2014.12.009.CrossRefGoogle Scholar
  107. Razzaq, L., Patvarczki, J., Almeida, S. F., Vartak, M., Feng, M., Heffernan, N. T., & Koedinger, K. R. (2009). The Assistment Builder: Supporting the life cycle of tutoring system content creation. IEEE Transactions on Learning Technologies, 2(2), 157–166.CrossRefGoogle Scholar
  108. Renkl, A. (2013). Toward an instructionally oriented theory of example-based learning. Cognitive Science, 38(1), 1–37. doi:10.1111/cogs.12086.CrossRefGoogle Scholar
  109. Renkl, A., Atkinson, R. K., & Grosse, C. S. (2003). How fading worked solution steps works—a cognitive load perspective. Instructional Science, 32, 1–24.Google Scholar
  110. Rice, W. (2011). Moodle 2.0 e-learning course development: A complete guide to successful learning using Moodle. Birmingham: Packt Publishing Ltd.Google Scholar
  111. Ritter, S., & Koedinger, K. R. (1996). An architecture for plug-in tutor agents. International Journal of Artificial Intelligence in Education, 7(3–4), 315–347.Google Scholar
  112. Ritter, S., Anderson, J. R., Koedinger, K. R., & Corbett, A. (2007). Cognitive Tutor: Applied research in mathematics education. Psychonomic Bulletin & Review, 14(2), 249–255.CrossRefGoogle Scholar
  113. Rittle-Johnson, B., & Koedinger, K. R. (2005). Designing knowledge scaffolds to support mathematical problem solving. Cognition and Instruction, 23(3), 313–349.CrossRefGoogle Scholar
  114. Roll, I., Aleven, V., & Koedinger, K. R. (2010). The Invention Lab: Using a hybrid of model tracing and constraint-based modeling to offer intelligent support in inquiry environments. In V. Aleven, J. Kay, & J. Mostow (Eds.), Lecture Notes in Computer Science: Proceedings of the 10th International Conference on Intelligent Tutoring Systems, ITS 2010 (Vol. I, pp. 115–124). Berlin: Springer.Google Scholar
  115. Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2011). Metacognitive practice makes perfect: Improving students’ self-assessment skills with an intelligent tutoring system. In G. Biswas, S. Bull, J. Kay, & A. Mitrovic (Eds.), Proceedings of the 15th international conference on artificial intelligence in education, AIED 2011 (pp. 288–295). Berlin: Springer. doi:10.1007/978-3-642-21869-9_38.Google Scholar
  116. Rummel, N., Walker, E., & Aleven, V. (under review). Different futures of adaptive collaborative learning support. Manuscript submitted for publication.Google Scholar
  117. Salden, R. J. C. M., Aleven, V., Schwonke, R., & Renkl, A. (2010a). The expertise reversal effect and worked examples in tutored problem solving. Instructional Science, 38(3), 289–307. doi:10.1007/s11251-009-9107-8.CrossRefGoogle Scholar
  118. Salden, R. J. C. M., Koedinger, K. R., Renkl, A., Aleven, V., & McLaren, B. M. (2010b). Accounting for beneficial effects of worked examples in tutored problem solving. Educational Psychology Review, 22(4), 379–392. doi:10.1007/s10648-010-9143-6.CrossRefGoogle Scholar
  119. Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129–184.CrossRefGoogle Scholar
  120. Schwartz, D. L., Chase, C. C., Oppezzo, M. A., & Chin, D. B. (2011). Practicing versus inventing with contrasting cases: The effects of telling first on learning and transfer. Journal of Educational Psychology, 103(4), 759–775. doi:10.1037/a0025140.CrossRefGoogle Scholar
  121. Slavin, R. E. (1996). Research on cooperative learning and achievement: What we know, what we need to know. Contemporary Educational Psychology, 21(1), 43–69.CrossRefGoogle Scholar
  122. Sottilare, R. (2012). Considerations in the development of an ontology for a generalized intelligent framework for tutoring. In I3M defense and homeland security simulation Conference (DHSS 2012). Google Scholar
  123. Sottilare, R., Graesser, A., Hu, X., & Holden, H. (Eds.) (2013). Design recommendations for adaptive intelligent tutoring systems (Vol. I, Learner Modeling). Orlando: US Army Research Laboratory.Google Scholar
  124. Sottilare, R., Graesser, A., Hu, X., & Holden, H. (Eds.) (2014). Design recommendations for adaptive intelligent tutoring systems (Vol. II - Instructional Management). Orlando: US Army Research Laboratory.Google Scholar
  125. Sottilare, R., Graesser, A., Hu, X., & Brawner, K. (Eds.) (2015). Design recommendations for adaptive intelligent tutoring systems (Vol. III - Authoring Tools and Expert Modeling Techniques). Orlando: US Army Research Laboratory.Google Scholar
  126. Stampfer, E., & Koedinger, K. R. (2013). When seeing isn’t believing: Influences of prior conceptions and misconceptions. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society (pp. 916–919). Berlin: Springer. doi:10.1007/978-3-642-39112-5_145.Google Scholar
  127. Stankov, S., Rosić, M., Žitko, B., & Grubišić, A. (2007). TEx-Sys model for building intelligent tutoring systems. Computers & Education, 51(3), 1017–1036. doi:10.1016/j.compedu.2007.10.002.CrossRefGoogle Scholar
  128. Van Gog, T., & Rummel, N. (2010). Example-based learning: Integrating cognitive and social-cognitive research perspectives. Educational Psychology Review, 22(2), 155–174. doi:10.1007/s10648-010-9134-7.CrossRefGoogle Scholar
  129. VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227–265.Google Scholar
  130. VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221.CrossRefGoogle Scholar
  131. VanLehn, K. (this issue). Regulative loops, step loops and task loops. International Journal of Artificial Intelligence in Education, 26(1).Google Scholar
  132. Waalkens, M., Aleven, V., & Taatgen, N. (2013). Does supporting multiple student strategies lead to greater learning and motivation? Investigating a source of complexity in the architecture of intelligent tutoring systems. Computers & Education, 60(1), 159–171. doi:10.1016/j.compedu.2012.07.016.CrossRefGoogle Scholar
  133. Walker, E., Rummel, N., & Koedinger, K. R. (2014). Adaptive intelligent support to improve peer tutoring in algebra. International Journal of Artificial Intelligence in Education, 24(1), 33–61. doi:10.1007/s40593-013-0001-9.CrossRefGoogle Scholar
  134. Weiss, M. A. (2010). Data structures and problem solving using Java (4th Edition). New York: Pearson Education, Inc.Google Scholar
  135. Wiese, E. S., & Koedinger, K.R. (2015). Grounded feedback in a fraction addition tutor. Paper presented as part of the Symposium Multiple Representations and Multimedia: Student Learning and Instruction at the 2015 Annual Meeting of the American Educational Research Association (AERA). Chicago, IL.Google Scholar
  136. Woolf, B. P. (2009). Building intelligent interactive tutors: Student-centered strategies for revolutionizing e-learning. Burlington: Morgan Kaufmann.Google Scholar
  137. Wylie, R., Koedinger, K. R., & Mitamura, T. (2009). Is self-explanation always better? The effects of adding self-explanation prompts to an English grammar tutor. In N. A. Taatgen & H. van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp. 1300–1305). Austin: Cognitive Science Society.Google Scholar
  138. Wylie, R., Koedinger, K. R., & Mitamura, T. (2010a). Analogies, explanations, and practice: Examining how task types affect second language grammar learning. In V. Aleven, J. Kay, & J. Mostow (Eds.), Lecture notes in computer science: Proceedings of the 10th International Conference on Intelligent Tutoring Systems, ITS 2010 (Vol. 6094, pp. 214–223). Berlin: Springer. doi:10.1007/978-3-642-13388-6_26.Google Scholar
  139. Wylie, R., Koedinger, K. R., & Mitamura, T. (2010b). Extending the self-explanation effect to second language grammar learning. In K. Gomez, L. Lyons, & J. Radinsky (Eds.), Learning in the disciplines: Proceedings of the 9th International Conference of the Learning Sciences, ICLS 2010 (Vol. 1, pp. 57–64). Chicago: International Society of the Learning Sciences.Google Scholar
  140. Wylie, R., Sheng, M., Mitamura, T., & Koedinger, K. (2011). Effects of adaptive prompted self-explanation on robust learning of second language grammar. In G. Biswas, S. Bull, J. Kay, & A. Mitrovic (Eds.), Lecture notes in computer science: Artificial intelligence in education (Vol. 6738, pp. 588–590). Berlin: Springer. doi:10.1007/978-3-642-21869-9_110.CrossRefGoogle Scholar
  141. Xhakaj, F. (2015). Intelligent tutors and granularity a new approach to red black trees. Unpublished senior thesis, Department of Computer Science, Lafayette College, Easton Pennsylvania.USA.Google Scholar
  142. Xhakaj, F., & Liew, C. W. (2015). A new approach to teaching red black tree. In V. Dagienė, C. Schulte, & T. Jevsikova (Eds.), Proceedings of the 20th ACM Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ‘15 (pp. 278–283). New York: ACM. doi:10.1145/2729094.2742624.CrossRefGoogle Scholar

Copyright information

© International Artificial Intelligence in Education Society 2016

Authors and Affiliations

  • Vincent Aleven
    • 1
  • Bruce M. McLaren
    • 1
  • Jonathan Sewall
    • 1
  • Martin van Velsen
    • 1
  • Octav Popescu
    • 1
  • Sandra Demi
    • 1
  • Michael Ringenberg
    • 1
  • Kenneth R. Koedinger
    • 1
  1. 1.Human-Computer Interaction InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations