Stupid Tutoring Systems, Intelligent Humans



The initial vision for intelligent tutoring systems involved powerful, multi-faceted systems that would leverage rich models of students and pedagogies to create complex learning interactions. But the intelligent tutoring systems used at scale today are much simpler. In this article, I present hypotheses on the factors underlying this development, and discuss the potential of educational data mining driving human decision-making as an alternate paradigm for online learning, focusing on intelligence amplification rather than artificial intelligence.


Intelligent tutoring system Decision-making Intelligence amplification Automated adaptation 


  1. Aleven, V., Sewall, J., Popescu, O., Xhakaj, F., Chand, D., Baker, R., Wang, Y., Siemens, G., Rosé, C., & Gasevic, D. (2015). The Beginning of a beautiful friendship? Intelligent tutoring systems and MOOCs. Proceedings of the 17th International Conference on Artificial Intelligence in Education, 525–528.Google Scholar
  2. Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: lessons learned. Journal of the Learning Sciences, 4, 167–207.CrossRefGoogle Scholar
  3. Arnold, K. E., Pistilli, M. D. (2012) Course Signals at Purdue: Using learning analytics to increase student success. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge. New York, NY: ACM, pp. 267–270.Google Scholar
  4. Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., Fisher, D., Barto, A., Mahadevan, S., & Woolf, B. P. (2007). Repairing disengagement with non-invasive interventions. Proceedings of the International Conference on Artificial Intelligence in Education, 195–202.Google Scholar
  5. Arroyo, I., Woolf, B. P., Burelson, W., Muldner, K., Rai, D., & Tai, M. (2014). A multimedia adaptive tutoring system for mathematics that addresses cognition, metacognition and affect. International Journal of Artificial Intelligence in Education, 24(4), 387–426.CrossRefGoogle Scholar
  6. Baker, R. S. J. d. (2010). Data mining for education. In B. McGaw, P. Peterson, & E. Baker (Eds.) International Encyclopedia of Education 3rd edn, (pp. 112–118). Oxford, UK: Elsevier.Google Scholar
  7. Baker, R. S. (2014). Big Data And Education. New York, NY: Teachers College, Columbia University.Google Scholar
  8. Baker, R.S. (2016) Stupid Tutoring Systems, Intelligent humans. Manuscript under review.Google Scholar
  9. Baker, R., & Siemens, G. (2014). Educational data mining and learning analytics. In K. Sawyer (Ed.) Cambridge Handbook of the Learning Sciences 2nd edn, (pp. 253–274).Google Scholar
  10. Baker, R. S. J. d., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3–17.Google Scholar
  11. Baker, R. S. J. d., Corbett, A. T., Koedinger, K. R., Evenson, S. E., Roll, I., Wagner, A. Z., Naim, M., Raspat, J., Baker, D. J., & Beck, J. (2006). Adapting to when students game an intelligent tutoring system. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 392–401.Google Scholar
  12. Barber, R., Sharkey, M. (2012) Course correction: using analytics to predict course success. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge, New York, NY: ACM, 259–262.Google Scholar
  13. Beck, J.E. (1997) Modeling the Student with Reinforcement Learning. Proceedings of the Machine learning for User Modeling Workshop at the Sixth International Conference on User Modeling. Google Scholar
  14. Beck, J.E., Mostow, J. (2008) How who should practice: Using learning decomposition to evaluate the efficacy of different types of practice for different types of students. Proceedings of the 9th International Conference on Intelligent Tutoring Systems, 5091, 353–362.Google Scholar
  15. Beck, J.E., Woolf, B.P., Beal, C.R. (2000) ADVISOR: A machine learning architecture for intelligent tutor construction. Proceedings of the 7th National Conference on Artificial Intelligence,New York, NY: ACM, 552–557.Google Scholar
  16. Bergman, P. (under review). Parent-child information frictions and human capital investment: evidence from a field experiment. Manuscript under review. Working paper retrieved online 2/10/2016 from
  17. Brabham, D. C. (2008). Crowdsourcing as a model for problem solving: An introduction and cases. Convergence: The International Journal of Research into New Media Technologies, 14(1), 75–90.Google Scholar
  18. Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press.Google Scholar
  19. Broderick, Z., O’Connor, C., Mulcahy, C., Heffernan, C., & Heffernan, N. (2010). Increasing parent engagement in student learning using an intelligent tutoring system. Journal of Interactive Learning Research, 22(4), 523–550.Google Scholar
  20. Brooks, F. P. (1975). The mythical man-month. Boston, MA: Addison-Wesley.CrossRefGoogle Scholar
  21. Bull, S., Nghiem, T. (2002) Helping learners to understand themselves with a learner model open to students, peers and instructors. Proceedings of the Workshop on Individual and Group Modelling Methods that Help Learners Understand Themselves, 5–13. Biarritz, France.Google Scholar
  22. Campbell, D. T. (1976). Assessing the impact of planned social change. Hanover, NH: Dartmouth College.Google Scholar
  23. Carbonell, J. (1970). AI in CAI: An artificial-intelligence approach to computer-assisted instruction. IEEE Transactions on Man-Machine Systems, 11(4), 190–202.CrossRefGoogle Scholar
  24. Chi, M., VanLehn, K., Litman, D. (2010) Do micro-level tutorial decisions matter: Applying reinforcement learning to induce pedagogical tutoring tactics. Proceedings of the 10th International Conference on Intelligent Tutoring Systems, 184–193.Google Scholar
  25. Clark, R. C., & Mayer, R. E. (2003). E-learning and the science of instruction. San Francisco: Jossey-Bass.Google Scholar
  26. Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, 4(4), 253–278.Google Scholar
  27. Corbett, A. T., & Bhatnagar, A. (1997). Student modeling in the ACT programming tutor: adjusting a procedural learning model with declarative knowledge. User Modeling: Proceedings of the Sixth International Conference, UM97, 243-254.Google Scholar
  28. Corbett, A. T., Koedinger, K. R., & Hadley, W. (2001). Cognitive tutors: from the research classroom to all classrooms. In P. S. Goodman (Ed.), Technology enhanced learning: Opportunities for change (pp. 235–263). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.Google Scholar
  29. Craig, S. D., Hu, X., Graesser, A. C., Bargagliotti, A. E., Sterbinsky, A., Cheney, K. R., & Okwumabua, T. (2013). The impact of a technology-based mathematics after-school program using ALEKS on student’s knowledge and behaviors. Computers & Education, 68, 495–504.CrossRefGoogle Scholar
  30. D’Mello, S., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K. (2010) A time for emoting: when affect-sensitivity is and isn’t effective at promoting deep learning. Proceedings of the 10th International Conference on Intelligent Tutoring Systems, 6094, 245–254.Google Scholar
  31. Darling-Hammond, L. (2000) Teacher Quality and Student Achievement. Education Policy Analysis Archives. Available online at
  32. Dekker, G. W., Pechenizkiy, M., & Vleeshouwers, J. M. (2009). Predicting student drop out: A case study. Proceedings of the 2nd International Conference on Educational Data Mining (EDM'09), 41–50.Google Scholar
  33. Del Soldato, T., & du Boulay, B. (1995). Implementation of motivational tactics in tutoring systems. Journal of Artificial Intelligence in Education, 6(4), 337–378.Google Scholar
  34. Diaz, G., Garcia Loro, F., Castro, M., Tawfik, M., Sancristobal, E., Monteso, S. (2013) Remote electronics lab within a MOOC: Design and preliminary results. Proceedings of Experiment@ International Conference, Redwood, CA: Deepdyve, 89–93.Google Scholar
  35. D'Mello, S. K., & Graesser, A. C. (2012). AutoTutor and affective AutoTutor: learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Transactions on Interactive Intelligent Systems, 2(4), 1–38.CrossRefGoogle Scholar
  36. Falmagne, J., Albert, D., Doble, C., Eppstein, D., & Hu, X. (2013). Knowledge spaces: Applications in education. Berlin-Heidelberg: Springer.CrossRefMATHGoogle Scholar
  37. Farber, B., & Miller, J. (1981). Teacher burnout: A psycho-educational perspective. Teachers College Record, 83(2), 235–243.Google Scholar
  38. Feng, M., & Heffernan, N. T. (2005). Informing teachers live about student learning: reporting in the assistment system. Technology, Instruction, Cognition and Learning, 3, 1–14.Google Scholar
  39. Freund, Y. (2013) Artificial intelligence vs intelligence amplification. California Institute for Telecommunications and Information Technology.
  40. Goldstein, I. J. (1979). The genetic graph: a representation for the evolution of procedural knowledge. International Journal of Man-Machine Studies, 11(1), 51–77.CrossRefGoogle Scholar
  41. Greeno, J. G. (1997). On claims that answer the wrong question. Educational Researcher, 26(1), 5–17.Google Scholar
  42. Hawn, A. (2015). The bridge report: bringing learning analytics to low-income, urban schools. Proceedings of the Fifth International Conference on Learning Analytics And Knowledge. Washington, DC: Association for Computing Machinery, 410–411.Google Scholar
  43. Heffernan, N. T., & Heffernan, C. L. (2014). The ASSISTments ecosystem: building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education, 24(4), 470–497.MathSciNetCrossRefGoogle Scholar
  44. Heffernan, N.T., Koedinger, K.R. (2002) An intelligent tutoring system incorporating a model of an experienced human tutor. Proceedings of the 6th International Conference on Intelligent Tutoring Systems, 2363, 596–608.Google Scholar
  45. Jayaprakash, S. M., Moody, E. W., Lauria, E. J. M., Regan, J. R., & Baron, J. D. (2014). Early alert of academically At-risk students: An open source analytics initiative. Journal of Learning Analytics, 1(1), 6–47.Google Scholar
  46. Jeong, H., Biswas, G. (2008) Mining Student Behavior Models in Learning-by-Teaching Environments. Proceedings of the 1st International Conference on Educational Data Mining, Worcester, MA: International Educational Data Mining Society, 127–136.Google Scholar
  47. Joksimović, S., Kovanović, V., Jovanović, J., Zouaq, A., Gašević, D., & Hatala, M. (2015). What do cMOOC participants talk about in Social Media? A Topic Analysis of Discourse in a cMOOC. Proceedings of the 5th International Learning Analytics and Knowledge (LAK) Conference, New York, NY: ACM, 156–165Google Scholar
  48. Khachatryan, G., Romashov, A., Khachatryan, A., Gaudino, S., Khachatryan, J., Guarian, K., & Yufa, N. (2014). Reasoning mind Genie 2: An Intelligent learning system as a vehicle for international transfer of instructional methods in mathematics. International Journal of Artificial Intelligence in Education, 24(3), 333–382.CrossRefGoogle Scholar
  49. Koedinger, K. R., McLaughlin, E. A., & Heffernan, N. T. (2010). A quasi-experimental evaluation of an on-line formative assessment and tutoring system. Journal of Educational Computing Research, 43(4), 489–510.CrossRefGoogle Scholar
  50. Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012a). The knowledge-learning-instruction framework: bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36, 757–798.CrossRefGoogle Scholar
  51. Koedinger, K. R., McLaughlin, E. A., & Stamper, J. C. (2012b). Automated student model improvement. Proceedings of the 5th International Conference on Educational Data Mining, 17–24.Google Scholar
  52. Koedinger, K.R., Stamper, J., McLaughlin, E., Nixon, T. (2013) Using Data-Driven Discovery of Better Student Models to Improve Student Learning. Proceedings of the International Conference on Artificial Intelligence and Education, 7926, 421–430.Google Scholar
  53. Kovačić, Z. (2010). Early Prediction of Student Success: Mining Students Enrolment Data. Proceedings of Informing Science & IT Education Conference (InSITE), 647–665.Google Scholar
  54. Kulik, C. C., & Kulik, J. A. (1991). Effectiveness of computer-based instruction: an updated analysis. Computers in Human Behavior, 7, 75–95.CrossRefGoogle Scholar
  55. Leelawong, K., & Biswas, G. (2008). Designing learning by teaching agents: the Betty’s brain system. International Journal of Artificial Intelligence in Education, 18(3), 181–208.Google Scholar
  56. Lepper, M. R., Woolverton, M., Mumme, D. L., & Gurtner, J. (1993). Motivational techniques of expert human tutors: lessons for the design of computer-based tutors. Computers as Cognitive Tools, 1993, 75–105.Google Scholar
  57. Matsuda, N., Keiser, V., Raizada, R., Tu, A., Stylianides, G., Cohen, W. (2010) Learning by Teaching SimStudent: Technical Accomplishments and an Initial Use with Students. Proceedings of the 10th International Conference on Intelligent Tutoring Systems, 6094, 317–326.Google Scholar
  58. Maull, K. E., Saldivar, M. G., & Sumner, T. (2011). Online curriculum planning behavior of teachers. Proceedings of the 3rd International Conference on Educational Data Mining, 121–130.Google Scholar
  59. McArthur, D., Stasz, C., & Zmuidzinas, M. (1990). Tutoring techniques in algebra. Cognition and Instruction, 7(3), 197–244.CrossRefGoogle Scholar
  60. McAuley, A., Stewart, B., Siemens, G., Cormier, D. (2010) The MOOC model for digital practice. Available online at
  61. McCaffrey, D. F., Lockwood, J. R., Koretz, D. M., & Hamilton, L. S. (2003). Evaluating value-added models for teacher accountability. Santa Monica, CA: RAND Corporation.CrossRefGoogle Scholar
  62. Merrill, D. C., Reiser, B. J., Ranney, M., & Trafton, J. G. (1992). Effective tutoring techniques: A comparison of human tutors and intelligent tutoring systems. The Journal of the Learning Sciences, 2(3), 277–305.CrossRefGoogle Scholar
  63. Miller, W. L., Baker, R., Labrum, M., Petsche, K., Liu, Y-H., Wagner, A. (2015). Automated detection of proactive remediation by teachers in reasoning mind classrooms. Proceedings of the 5th International Learning Analytics and Knowledge Conference, 290–294.Google Scholar
  64. Ming, N.C., Ming, V.L. (2012) Automated Predictive Assessment from Unstructured Student Writing. Proceedings of the 1st international Conference on Data Analytics. Google Scholar
  65. Mitrovic, A., & Ohlsson, S. (1999). Evaluation of a constraint-based tutor for a database language. International Journal of Artificial Intelligence in Education, 10, 238–256.Google Scholar
  66. Mostow, J. (2008). Experience from a Reading tutor that listens: evaluation purposes, excuses, and methods. In C. K. Kinzer, & L. Verhoeven (Eds.), Interactive Literacy Education: Facilitating Literacy Environments Through Technology (pp. 117–148). New York: Lawrence Erlbaum Associates, Taylor & Francis Group.Google Scholar
  67. Mulqueeny, K., Kostyuk, V., Baker, R. S., & Ocumpaugh, J. (2015). Incorporating effective e-learning principles to improve student engagement in middle-school mathematics. International Journal of STEM Education, 2(15).Google Scholar
  68. Murray, R.C., VanLehn, K. (2005) Effects of Dissuading Unnecessary Help Requests While Providing Proactive Help. Proceedings of the International Conference on Artificial Intelligence in Education, Amsterdam, Netherlands: IOS Press, 887–889Google Scholar
  69. Norvig, P. (2012) Google’s Peter Norvig on Online Education. Public lecture given at Stanford University.
  70. Nye, B. D., Graesser, A. C., & Hu, X. (2014). AutoTutor and family: A review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education, 24(4), 427–469.CrossRefGoogle Scholar
  71. O’Shea, T. (1982). A self-improving quadratic tutor. In D. Sleeman, & J. S. Brown (Eds.), Intelligent Tutoring Systems (pp. 309–336). New York: Academic Press.Google Scholar
  72. Ostashewski, N. (2013) Building for massive scalability: the production case of an astronomy MOOC. Proceedings of the World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, Chesapeake, VA: AACE Google Scholar
  73. Ostrow, K. S., & Heffernan, N. T. (2014). Testing the multimedia principled in the real world: A comparison of video vs. In Text feedback in Authentic Middle School math assignments. Proceedings of the 7th International Conference on Educational Data Mining (pp. 296–299).Google Scholar
  74. Ostrow, K. S., Schultz, S. E., & Arroyo, I. (2014). Promoting Growth Mindset Within Intelligent Tutoring Systems. In CEUR-WS (1183), Gutierrez-Santos, S., & Santos, O.C. (eds.) EDM 2014 Extended Proceedings, 88–93.Google Scholar
  75. Pane, J. F., Griffin, B. A., McCaffrey, D. F., & Karam, R. (2014). Effectiveness of cognitive tutor algebra I at scale. Educational Evaluation and Policy Analysis, 36(2), 127–144.CrossRefGoogle Scholar
  76. Papousek, J., Pelanek, R., & Stanislav, V. (2014). Adaptive practice of facts in domains with varied prior knowledge. Proceedings of the International Conference on Educational Data Mining (EDM), 6–13.Google Scholar
  77. Pardos, Z. A., Baker, R. S. J. D., Gowda, S. M., & Heffernan, N. T. (2011). The sum is greater than the parts: ensembling models of student knowledge in Educational software. SIGKDD Explorations, 13(2), 37–44.CrossRefGoogle Scholar
  78. Pashler, H., Bain, P. M., Bottqe, B. A., Graesser, A., Koedinger, K., McDaniel, M., Metcalfe, J. (2007). Organizing instruction and study to improve student learning. IES practice guide NCER 2007–2004. Washington, DC: US Department of Education, Institute of Education Sciences.Google Scholar
  79. Pavlik, P.I., Cen, H., Koedinger, K.R. (2009) Performance Factors Analysis – A New Alternative to Knowledge Tracing. Proceedings of the International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS Press, 531–538Google Scholar
  80. Quinn, A.J., Bederson, B.B. (2011) Human computation: a survey and taxonomy of a growing field. Proceedings of ACM SIGCHI, 1403–1412, New York, NY: ACMGoogle Scholar
  81. Rafferty, A. N., Brunskill, E., Griffiths, T.L., Shafto, P. (2011) Faster teaching by POMDP planning. Proceedings of the 15th International Conference on Artificial Intelligence in Education, 6738, 280–287.Google Scholar
  82. Raghuveer, V.,Tripathy, B., Singh, T., Khanna, S. (2014) Reinforcement learning approach towards effective content recommendation in mooc environments. Proceedings of IEEE International Conference on MOOC, Innovation and Technology in Education (MITE), 285–289, Patiala: IEEEGoogle Scholar
  83. Razzaq, L., & Heffernan, N. T. (2006). Scaffolding vs. In Hints in the assistment system. Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS 2006) (pp. 635–644).Google Scholar
  84. Rodrigo, M. M. T., Baker, R. S. J. d., Rossi, L. (2013). Student off-task behavior in computer-based learning in the Philippines: comparison to prior research in the USA. Teachers College Record, 115(10), 1–27.Google Scholar
  85. Rodriguez, C. O. (2012). MOOCs and the AI-Stanford like courses: two successful and distinct course formats for massive open online courses. European Journal of Open, Distance and E-Learning, 2012(2).Google Scholar
  86. Schofield, J. (1995). Computers and classroom culture. London: Cambridge University Press.CrossRefGoogle Scholar
  87. Self, J. A. (1990). Theoretical foundations of intelligent tutoring systems. Journal of Artificial Intelligence in Education, 1(4), 3–14.Google Scholar
  88. Self, J. A. (1998). The defining characteristics of intelligent tutoring systems research: ITSs care, precisely. International Journal of Artificial Intelligence in Education (IJAIED), 10, 350–364.Google Scholar
  89. Serrano-Laguna, A., & Fernández-Manjón, B. (2014). Applying learning analytics to simplify serious games deployment in the classroom. In IEEE International Global Engineering Education Conference (EDUCON), Istanbul, 872–877.Google Scholar
  90. Shute, V. (1990). Rose garden promises of intelligent tutoring systems: blossom or thorn? paper presented at the space. Applications and Research (SOAR) Symposium: Operations.Google Scholar
  91. Siemens, G., Baker, R.S.J.d. (2012) Learning Analytics and Educational Data Mining: Towards Communication and Collaboration. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge, New York, NY: ACMGoogle Scholar
  92. Slotta, J. D., Tissenbaum, M., & Lui, M. (2013). Orchestrating of complex inquiry: three roles for learning analytics in a smart classroom infrastructure. In Proceedings of the Third International Conference on Learning Analytics and Knowledge, 270–274. New York, NY: ACM.Google Scholar
  93. Stevens, A., & Collins, A. (1977). The goal structure of a Socratic tutor. Technical Report #3518. Cambridge, MA: Bolt, Beranek, and Newman, Inc..Google Scholar
  94. Van Leeuwen, A., Janssen, J., Erkens, G., & Brekelmans, M. (2014). Supporting teachers in guiding collaborating students: effects of learning analytics in CSCL. Computers & Education, 79, 28–39.CrossRefGoogle Scholar
  95. van Leeuwen, A., Janssen, J., Erkens, G., & Brekelmans, M. (2015). Teacher regulation of cognitive activities during student collaboration: effects of learning analytics. Computers & Education, 90, 80–94.CrossRefGoogle Scholar
  96. VanLehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The. Andes physics tutoring system: Lessons learned, International Journal of Artificial Intelligence in Education, 15(3), 1–47.Google Scholar
  97. Walonoski, J. A., & Heffernan, N. T. (2006). Prevention of off-task gaming behavior in intelligent tutoring systems. Proceedings of the International Conference on Intelligent Tutoring Systems. Berlin-Heidelberg: Springer, 722–724.Google Scholar
  98. Wenger, E. (1987). Artificial Intelligence And Tutoring Systems. Los Altos, CA: Morgan Kauffman.Google Scholar
  99. Winograd, T., & Flores, F. (1986). Understanding computers and cognition. Reading, MA: Addison-Wesley.MATHGoogle Scholar
  100. Wood, D.S., Williams, G. (2013) Data-Driven Factors that Increase Student Course Completion: A Two-Year Study. Proceedings of the 9th Annual Symposium on Student Retention. Google Scholar
  101. Zapata-Rivera, D., & Katz, I. (2014). Keeping your audience in mind: applying audience analysis to the design of interactive score reports. Assessments in Education: Principles, Policy, and Practice, 21(4), 442–463.CrossRefGoogle Scholar

Copyright information

© International Artificial Intelligence in Education Society 2016

Authors and Affiliations

  1. 1.Teachers CollegeColumbia UniversityNew YorkUSA

Personalised recommendations