Recent Meta-reviews and Meta–analyses of AIED Systems

  • Benedict du BoulayEmail author
Letter to the Editor


25th Anniversary Classroom Instruction Small Effect Size Tutoring System Domain Content 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

It is pleasing to see that the 25th Anniversary of IJAIED also brings a certain maturity to the field. In the last 5 years there have been a number useful and positive analyses of the effectiveness of AIED systems designed to tutor one-to-one. For example, VanLehn (2011) found that the effectiveness of the “step-based” intelligent tutoring systems he surveyed were nearly as effective as average human tutors, and that neither of these were as effective as expert human tutors, but certainly both better than providing the domain content without tutoring. “Step-based” means that the tutor evaluates and reacts to each step that the learner makes in a problem-solving session as opposed to evaluating only the final answer given by the learner.

In a meta-analysis, Ma et al. (2014) found similar positive results for step-based ITSs both when compared to a no tutoring condition and when compared to large group human teacher led-instruction, but no differences when compared to small group human tutoring or to one–to–one tutoring. The same authors analysed systems for teaching programming and also found a “a significant advantage of ITSs over teacher-led classroom instruction and non-ITS computer-based instruction” (Nesbit et al. 2014). Likewise Kulik and Fletcher (2015) found similar sized improvements but distinguished between studies that used standardised tests and those where the tests were more specifically tuned to the system providing tuition. Smaller effect sizes were found by Steenbergen-Hu and Cooper (2013) in their meta-analysis of pupils using ITSs in a School setting. They also noted that lower-achievers seemed to do worse with ITSs than did the broad spectrum of school pupils. In a parallel study of university students, Steenbergen-Hu and Cooper (2014) found more positive effects for ITSs as compared to conventional instruction.

Finally in a large-scale study in the USA of the Cognitive Tutors, (Pane et al. 2014) found only limited evidence of the relative effectiveness of these tutors over conventional teaching, though we note that how the tutors were actually used in the classrooms was not strongly controlled. While not specifically addressing ITS effectiveness, Schroeder et al. (2013) found that online pedagogical agents “produced a small but significant effect on learning”.


  1. Kulik, J. A., & Fletcher, J. D. (2015). Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of Educational Research, X(X), 1–37. doi:  10.3102/0034654315581420
  2. Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: a meta-analysis. Journal of Educational Psychology, 106(4), 901–918. doi: 10.1037/a0037123.CrossRefGoogle Scholar
  3. Nesbit, J. C., Adesope, O. O., Liu, Q., & Ma, W. (2014). How Effective are Intelligent Tutoring Systems in Computer Science Education? Paper presented at the IEEE 14th International Conference on Advanced Learning Technologies (ICALT), Athens, Greece.Google Scholar
  4. Pane, J. F., Griffin, B. A., McCaffrey, D. F., & Karam, R. (2014). Effectiveness of cognitive tutor Algebra I at scale. Educational Evaluation and Policy Analysis, 36(2), 127–144. doi: 10.3102/0162373713507480.CrossRefGoogle Scholar
  5. Schroeder, N. L., Adesope, O. O., & Gilbert, R. B. (2013). How effective are pedagogical agents for learning? A meta-analytic review. Journal of Educational Computing Research, 49(1), 1–39. doi: 10.2190/EC.49.1.a.CrossRefGoogle Scholar
  6. Steenbergen-Hu, S., & Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on K–12 students’ mathematical learning. Journal of Educational Psychology, 105(4), 970–987. doi: 10.1037/a0032447.CrossRefGoogle Scholar
  7. Steenbergen-Hu, S., & Cooper, H. (2014). A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning. Journal of Educational Psychology, 106(2), 331–347. doi: 10.1037/a0034752.CrossRefGoogle Scholar
  8. VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221. doi: 10.1080/00461520.2011.611369.CrossRefGoogle Scholar

Copyright information

© International Artificial Intelligence in Education Society 2015

Authors and Affiliations

  1. 1.Human Centred Technology Group, Department of InformaticsUniversity of SussexBrightonUK

Personalised recommendations