Recent Meta-reviews and Meta–analyses of AIED Systems
Keywords25th Anniversary Classroom Instruction Small Effect Size Tutoring System Domain Content
It is pleasing to see that the 25th Anniversary of IJAIED also brings a certain maturity to the field. In the last 5 years there have been a number useful and positive analyses of the effectiveness of AIED systems designed to tutor one-to-one. For example, VanLehn (2011) found that the effectiveness of the “step-based” intelligent tutoring systems he surveyed were nearly as effective as average human tutors, and that neither of these were as effective as expert human tutors, but certainly both better than providing the domain content without tutoring. “Step-based” means that the tutor evaluates and reacts to each step that the learner makes in a problem-solving session as opposed to evaluating only the final answer given by the learner.
In a meta-analysis, Ma et al. (2014) found similar positive results for step-based ITSs both when compared to a no tutoring condition and when compared to large group human teacher led-instruction, but no differences when compared to small group human tutoring or to one–to–one tutoring. The same authors analysed systems for teaching programming and also found a “a significant advantage of ITSs over teacher-led classroom instruction and non-ITS computer-based instruction” (Nesbit et al. 2014). Likewise Kulik and Fletcher (2015) found similar sized improvements but distinguished between studies that used standardised tests and those where the tests were more specifically tuned to the system providing tuition. Smaller effect sizes were found by Steenbergen-Hu and Cooper (2013) in their meta-analysis of pupils using ITSs in a School setting. They also noted that lower-achievers seemed to do worse with ITSs than did the broad spectrum of school pupils. In a parallel study of university students, Steenbergen-Hu and Cooper (2014) found more positive effects for ITSs as compared to conventional instruction.
Finally in a large-scale study in the USA of the Cognitive Tutors, (Pane et al. 2014) found only limited evidence of the relative effectiveness of these tutors over conventional teaching, though we note that how the tutors were actually used in the classrooms was not strongly controlled. While not specifically addressing ITS effectiveness, Schroeder et al. (2013) found that online pedagogical agents “produced a small but significant effect on learning”.
- Kulik, J. A., & Fletcher, J. D. (2015). Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of Educational Research, X(X), 1–37. doi: 10.3102/0034654315581420
- Nesbit, J. C., Adesope, O. O., Liu, Q., & Ma, W. (2014). How Effective are Intelligent Tutoring Systems in Computer Science Education? Paper presented at the IEEE 14th International Conference on Advanced Learning Technologies (ICALT), Athens, Greece.Google Scholar