Authoring Tools for Designing Intelligent Tutoring Systems: a Systematic Review of the Literature

  • Diego DermevalEmail author
  • Ranilson Paiva
  • Ig Ibert Bittencourt
  • Julita Vassileva
  • Daniel Borges


Authoring tools have been broadly used to design Intelligent Tutoring Systems (ITS). However, ITS community still lacks a current understanding of how authoring tools are used by non-programmer authors to design ITS. Hence, the objective of this work is to review how authoring tools have been supporting ITS design for non-programmer authors. In order to meet our goal, we conduct a Systematic Literature Review (SLR) to identify the primary studies on the use of ITS authoring tools, following a pre-defined review protocol. Among the 4622 papers retrieved from seven digital libraries published from 2009 to June 2016, 33 papers are finally included after applying our exclusion and inclusion criteria. We then identify the main ITS components authored, the ITS types designed, the features used to facilitate the authoring process, the technologies used to develop authoring tools and the time at which authoring occurs. We also look for evidence of the benefits of ITS authoring tools. In summary, the main findings of this work are: (1) there is empirical evidence of the benefits (i.e., mainly in terms of effectiveness, efficiency, quality of authored artifacts, and usability) of using ITS authoring tools for non-programmer authors, specially to aid authoring of learning content and to support authoring of model-tracing/cognitive and example-tracing tutors; 2) domain and pedagogical models have been much more targeted by authoring tools; (3) several ITS types have been authored, with an emphasis on model-tracing/cognitive and example-tracing tutors; (4) besides providing features for authoring all four ITS components, current authoring tools are also presenting general features (e.g., view learners’ statistics and reuse tutor design) to create broader authoring tools; (5) a great diversity of technologies, which include AI techniques, software solutions and distributed technologies, are used to develop ITS authoring tools; and (6) authoring tools have been mainly targeting ITS design before students’ instruction, but works are also addressing authoring during and/or post-instruction relying both on human and artificial intelligence. We conclude this work by showing several promising research opportunities that are quite important and interesting but underexplored in current research and practice.


Intelligent tutoring systems Authoring tools Systematic literature review Intelligent tutoring systems design 



This work has been supported by the Brazilian institutions: Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) and Coordenação de Aperfeiçoa-mento de Pessoal de Nível Superior (CAPES).


  1. Achimugu, P., Selamat, A., Ibrahim, R., & Mahrin, M.N. (2014). A systematic literature review of software requirements prioritization research. Information and Software Technology, 56(6), 568–585.CrossRefGoogle Scholar
  2. Aleven, V., Mclaren, B.M., Sewall, J., & Koedinger, K.R. (2009). A new paradigm for intelligent tutoring systems: example-tracing tutors. International Journal of Artificial Intelligence in Education, 19(2), 105–154.Google Scholar
  3. Anderson, J.R. (1983). The architecture of cognition. Cambridge: Harvard University Press.Google Scholar
  4. Baker, R.S. (2016). Stupid tutoring systems, intelligent humans. International Journal of Artificial Intelligence in Education, 26(2), 600–614.CrossRefGoogle Scholar
  5. du Boulay, B. (2016). Recent meta-reviews and meta–analyses of aied systems. International Journal of Artificial Intelligence in Education, 26(1), 536–537.CrossRefGoogle Scholar
  6. Chen, L., Babar, M.A., & Zhang, H. (2010). Towards an evidence-based understanding of electronic data sources. In Proceedings of the 14th international conference on evaluation and assessment in software engineering, British Computer Society, Swinton, UK, EASE’10 (pp. 135–138).Google Scholar
  7. Dermeval, D., Vilela, J., Bittencourt, I.I., Castro, J., Isotani, S., Brito, P., & Silva, A. (2016). Applications of ontologies in requirements engineering: a systematic review of the literature. Requirements Engineering, 21, 1–33.CrossRefGoogle Scholar
  8. Ding, W., Liang, P., Tang, A., & van Vliet, H. (2014). Knowledge-based approaches in software documentation: a systematic literature review. Information and Software Technology, 56(6), 545–567.CrossRefGoogle Scholar
  9. Dyb, T., & Dingsyr, T. (2008). Empirical studies of agile software development: a systematic review. Information and Software Technology, 50(910), 833–859.CrossRefGoogle Scholar
  10. Easterbrook, S., Singer, J., Storey, M.A., & Damian, D. (2008). Selecting empirical methods for software engineering research. In Shull, F., Singer, J., & Sjberg, D. (Eds.) Guide to advanced empirical software engineering (pp. 285–311). London: Springer.Google Scholar
  11. Felder, R.M., & Silverman, L.K. (1988). Learning and teaching styles in engineering education. Engineering Education, 78(7), 674–681.Google Scholar
  12. Garg, A.X., Hackam, D., & Tonelli, M. (2008). Systematic review and meta-analysis: when one study is just not enough. Clinical Journal of the American Society of Nephrology, 3(1), 253–260.CrossRefGoogle Scholar
  13. Hernandes, E.M., Zamboni, A., Fabbri, S., & Thommazo, A.D. (2012). Using gqm and tam to evaluate start - a tool that supports systematic review. CLEI Electron J, 15(1), 3–3.Google Scholar
  14. Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Tech. Rep. EBSE 2007-001, Keele University and Durham University Joint Report.Google Scholar
  15. Koedinger, K.R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19(3), 239–264.CrossRefGoogle Scholar
  16. Koedinger, K.R., Corbett, A.T., & Perfetti, C. (2012). The knowledge-learning-instruction framework: bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36(5), 757–798. Scholar
  17. Kulik, J.A., & Fletcher, J. (2016). Effectiveness of intelligent tutoring systems: a meta-analytic review. Review of Educational Research, 86(1), 42–78.CrossRefGoogle Scholar
  18. LAPES. (2014). Start - state of the art through systematic review tool. Available in Accessed date: October 2013.
  19. Ma, W., Adesope, O.O., Nesbit, J.C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: a meta-analysis. Journal of Educational Psychology, 106, 901–918.CrossRefGoogle Scholar
  20. Mahdavi-Hezavehi, S., Galster, M., & Avgeriou, P. (2013). Variability in quality attributes of service-based software systems: a systematic literature review. Information and Software Technology, 55(2), 320–343. Special Section: Component-Based Software Engineering (CBSE), 2011.CrossRefGoogle Scholar
  21. Murray, T. (1999). Authoring intelligent tutoring systems: an analysis of the state of the art. International Journal of Artificial Intelligence in Education (IJAIED), 10, 98–129.Google Scholar
  22. Murray, T. (2003). An overview of intelligent tutoring system authoring tools: updated analysis of the state of the art. In Authoring tools for advanced technology learning environments (pp. 491–544): Springer.Google Scholar
  23. Sleeman, D., & Brown, J.S. (1982). Intelligent tutoring systems. London: Academic Press.Google Scholar
  24. Sottilare, R., Graesser, A., Hu, X., & Brawner, K. (2015). Design recommendations for intelligent tutoring systems: authoring tools and expert modeling techniques. Robert Sottilare.Google Scholar
  25. Steenbergen-Hu, S., & Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on k–12 students’ mathematical learning. Journal of Educational Psychology, 105(4), 970.CrossRefGoogle Scholar
  26. Steenbergen-Hu, S., & Cooper, H. (2014). A meta-analysis of the effectiveness of intelligent tutoring systems (its) on college students’ academic learning. Journal of Educational Psychology, 106, 331–347.CrossRefGoogle Scholar
  27. Vanlehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227–265.Google Scholar
  28. VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221.CrossRefGoogle Scholar
  29. Wohlin, C., Runeson, P., Höst, M, Ohlsson, M.C., Regnell, B., & Wesslén, A. (2012). Experimentation in software engineering. Springer Science and Business Media.Google Scholar
  30. Woolf, B.P. (2010). Building intelligent interactive tutors: student-centered strategies for revolutionizing e-learning. Morgan Kaufmann. Google Scholar

Systematic Literature Review References

  1. Abbas, M.A., Ahmad, W.F.W., & Kalid, K.S. (2014). Semantic web technologies for pre-school cognitive skills tutoring system. Journal of Information Science and Engineering, 30(3), 835–851.Google Scholar
  2. Alepis, E., & Virvou, M. (2014). Mobile authoring in educational software. Object-Oriented User Interfaces for Personalized Mobile Learning, 64, 31–46.CrossRefGoogle Scholar
  3. Aleven, V., McLaren, B.M., & Sewall, J. (2009). Scaling up programming by demonstration for intelligent tutoring systems development: an open-access web site for middle school mathematics learning. IEEE Transactions on Learning Technologies, 2 (2), 64–78.CrossRefGoogle Scholar
  4. Aleven, V., McLaren, B.M., Sewall, J., van Velsen, M., Popescu, O., Demi, S., Ringenberg, M., & Koedinger, K.R. (2016). Example-tracing tutors: intelligent tutor development for non-programmers. International Journal of Artificial Intelligence in Education, 26(1), 224–269.CrossRefGoogle Scholar
  5. Barron-Estrada, L.M., Zatarain-Cabada, R., Zatarain-Cabada, R., Barbosa-Leon, H., & Reyes-Garcia, C.A. (2010). Building and assessing intelligent tutoring systems with an e-learning 2.0 authoring system. In Proceedings of the Ibero-American conference on artificial intelligence (IBERAMIA) (pp. 1–9).Google Scholar
  6. Barrón-Estrada, M., Zatarain-Cabada, R., Tamayo, P., Tamayo, S., & Pérez-Espinoza, H. (2011). A learning social network with multi-modal affect. In Proceedings of the 10th mexican international conference on artificial intelligence: advances in artificial intelligence and applications, MICAI 2011 (pp. 163–168).Google Scholar
  7. Blessing, S.B., Gilbert, S.B., Ourada, S., & Ritter, S. (2009). Authoring model-tracing cognitive tutors. International Journal of Artificial Intelligence in Education, 19(2), 189–210.Google Scholar
  8. Blessing, S.B., Devasani, S., Gilbert, S.B., & Sinapov, J. (2015). Using conceptgrid as an easy authoring technique to check natural language responses. International Journal of Learning Technology, 10(1), 50–70.CrossRefGoogle Scholar
  9. Brawner, K.W. (2015). Rapid dialogue and branching tutors. In Proceedings of the workshops at the 17th international conference on artificial intelligence in education, AIED 2015, Madrid, Spain, June 22 + 26, 2015.Google Scholar
  10. Chakraborty, S., Roy, D., Kumar Bhowmick, P., & Basu, A. (2010). An authoring system for developing intelligent tutoring system. In Proceedings of the IEE students’ technology symposium (TechSym) (pp. 196–205).Google Scholar
  11. Chou, C.Y., Huang, B.H., & Lin, C.J. (2011). Complementary machine intelligence and human intelligence in virtual teaching assistant for tutoring program tracing. Computers and Education, 57(4), 2303–2312.CrossRefGoogle Scholar
  12. Devasani, S., Gilbert, S., & Blessing, S. (2012). Evaluation of two intelligent tutoring system authoring tool paradigms graphical user interface-based and text-based. In Proceedings of the 21st annual conference on behavior representation in modeling and simulation, BRiMS 2012 (pp. 51– 58).Google Scholar
  13. Escudero, H., & Fuentes, R. (2010). Exchanging courses between different intelligent tutoring systems: a generic course generation authoring tool. Knowledge-Based Systems, 23(8), 864–874.CrossRefGoogle Scholar
  14. Fox, R., Schleifer, M., & Cellio, J. (2011). A tool to generate computer assisted instruction systems through hierarchical classification. In Proceedings of the 2011 international conference on artificial intelligence, ICAI 2011, (Vol. 1 pp. 385–391).Google Scholar
  15. Gilbert, S.B., Blessing, S.B., & Guo, E. (2015). Authoring effective embedded tutors: an overview of the extensible problem specific tutor (xpst) system. International Journal of Artificial Intelligence in Education, 25(3), 428–454.CrossRefGoogle Scholar
  16. Grubisic, A., Stankov, S., & Glavinic, V. (2009). Agent based intelligent courseware generation in e-learning. In Proceeding of the IASTED international conference on computers and advanced technology in education (CATE).Google Scholar
  17. Guin, N., & Lefevre, M. (2013). From a customizable ITS to an adaptive ITS. International Conference on Artificial Intelligence in Education (AIED), 7926 LNAI, 141–150.CrossRefGoogle Scholar
  18. Heffernan, N.T., & Heffernan, C.L. (2014). The assistments ecosystem: building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education, 24(4), 470–497.MathSciNetCrossRefGoogle Scholar
  19. Lane, H.C., Core, M.G., Hays, M.J., Auerbach, D., & Rosenberg, M. (2015). Situated pedagogical authoring: authoring intelligent tutors from a student’s perspective. In International conference artificial intelligence in education (pp. 195–204): Springer International Publishing.Google Scholar
  20. MacLellan C.J., Koedinger K.R., & Matsuda N. (2014). Authoring tutors with simstudent: an evaluation of efficiency and model quality. In International conference on intelligent tutoring systems (pp. 551–560): Springer.Google Scholar
  21. MacLellan, C.J., Harpstead, E., Wiese, E.S., Zou, M., Matsuda, N., Aleven, V., & Koedinger, K.R. (2015). Authoring tutors with complex solutions: a comparative analysis of example tracing and simstudent. In AIED workshops.Google Scholar
  22. Marcus, N., Ben-Naim, D., & Bain, M. (2010). Instructional support for teachers and guided feedback for students in an adaptive elearning environment. In Proceedings of the 8th international conference on information technology: new generations, ITNG 2011 (pp. 626-631).Google Scholar
  23. Matsuda, N., Cohen, W.W., & Koedinger, K.R. (2015). Teaching the teacher: tutoring simstudent leads to more effective cognitive tutor authoring. International Journal of Artificial Intelligence in Education, 25(1), 1–34.CrossRefGoogle Scholar
  24. Mitrovic, A., Martin, B., Suraweera, P., Zakharov, K., Milik, N., Holland, J., & McGuigan, N. (2009). ASPIRE: an authoring system and deployment environment for constraint-based tutors. International Journal of Artificial Intelligence in Education, 19, 155–188.Google Scholar
  25. Olney, A.M., & Cade, W.L. (2015). Authoring intelligent tutoring systems using human computation: designing for intrinsic motivation. In International conference on augmented cognition (pp. 628–639): Springer.Google Scholar
  26. Olsen, J.K., Belenky, D.M., Aleven, V., Rummel, N., Sewall, J., & Ringenberg, M. (2014). Authoring tools for collaborative intelligent tutoring system environments. In International conference on intelligent tutoring systems (pp. 523–528): Springer.Google Scholar
  27. Paquette, L., Lebeau, J.F., & Mayers, A. (2010). Authoring problem-solving tutors: a comparison between astus and ctat. In Advances in intelligent tutoring systems (pp. 377–405): Springer.Google Scholar
  28. Sklavakis, D., & Refanidis, I. (2011). The MATHESIS semantic authoring framework: ontology-driven knowledge engineering for ITS authoring. Knowlege-Based and Intelligent Information and Engineering Systems, pp. 114–123.Google Scholar
  29. Suraweera, P., Mitrovic, A., & Martin, B. (2010). Widening the knowledge acquisition bottleneck for constraint-based tutors. International Journal of Artificial Intelligence in Education, 20(2), 137–173.Google Scholar
  30. Troussas, C., Alepis, E., & Virvou, M. (2014). Mobile authoring in a multiple language learning environment. In The 5th international conference on information, intelligence, systems and applications, IISA 2014 (pp. 405–410): IEEE.Google Scholar
  31. Virvou, M., & Troussas, C. (2011). Knowledge-based authoring tool for tutoring multiple languages. In Intelligent interactive multimedia systems and services (pp. 163–175): Springer.Google Scholar
  32. Wilches, C.O.E., & Palacio, G.V.H. (2014). Development of example-tracing tutors for teaching control systems performance fundamentals. In Proceedings of the 4th world congress on information and communication technologies (pp. 290–295): WICT.Google Scholar
  33. Zatarian-Cabada, R., & Barrón-Estrada, M. (2011). Reyes García, C.A. EDUCA: A web 2.0 authoring tool for developing adaptive and intelligent tutoring systems using a Kohonen network. Expert Systems with Applications, 38(8), 9522–9529.Google Scholar

Copyright information

© International Artificial Intelligence in Education Society 2017

Authors and Affiliations

  1. 1.Penedo Educational Unity, Campus ArapiracaFederal University of AlagoasPenedoBrazil
  2. 2.Computing InstituteFederal University of AlagoasMaceióBrazil
  3. 3.Department of Computer ScienceUniversity of SaskatchewanSaskatoonCanada

Personalised recommendations