Skip to main content

Learning with intelligent tutors and worked examples: selecting learning activities adaptively leads to better learning outcomes than a fixed curriculum

Abstract

The main learning activity provided by intelligent tutoring systems is problem solving, although several recent projects investigated the effectiveness of combining problem solving with worked examples. Previous research has shown that learning from examples is an effective learning strategy, especially for novice learners. A worked example provides step-by-step explanations of how a problem is solved. Many studies have compared learning from examples to unsupported problem solving, and suggested presenting worked examples to students in the initial stages of learning, followed by problem solving once students have acquired enough knowledge. This paper presents a study in which we compare a fixed sequence of alternating worked examples and tutored problem solving with a strategy that adapts learning tasks to students’ needs. The adaptive strategy determines the type of the task (a worked example, a faded example or a problem to be solved) based on how much assistance the student received on the previous problem. The results show that students in the adaptive condition learnt significantly more than their peers who were presented with a fixed sequence of worked examples and problem solving. Novices from the adaptive condition learnt faster than novices from the control group, while the advanced students from the adaptive condition learnt more than their peers from the control group.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Notes

  1. 1.

    Cognitive efficiency \(=\) Performance / Mental Effort Rating.

  2. 2.

    Please note that P is scaled to the range [1,9].

  3. 3.

    Normalised learning gain \(=\) (post test—pre test) / (max score—pre test).

References

  1. Aleven, V., Koedinger, K.R.: An effective metacognitive strategy: learning by doing and explaining with a computer-based Cognitive Tutor. Cognit. Sci. 26(2), 147–179 (2002)

    Article  Google Scholar 

  2. Anthony, L., Mankoff, J., Mitchell, T., Gross, M.: Developing Handwriting-based Intelligent Tutors to Enhance Mathematics Learning. Doctoral Dissertation, Carnegie Mellon University (2008)

  3. Atkinson, R.K., Derry, S.J., Renkl, A., Wortham, D.: Learning from examples: instructional principles from the worked examples research. Rev. Educ. Res. 70(2), 181–214 (2000)

    Article  Google Scholar 

  4. Brown, A.L., Kane, M.J.: Preschool children can learn to transfer: learning to learn and learning from example. Cognit. Psychol. 20(4), 493–523 (1988)

    Article  Google Scholar 

  5. Brusilovsky, P.: Adaptive navigation support in educational hypermedia: the role of student knowledge level and the case for meta-adaptation. Br. J. Educ. Technol. 34(4), 487–497 (2003)

    Article  Google Scholar 

  6. Burow, R., Weber, G.: Example explanation in learning environments. In: Frasson, C., Gauthier, G., Lesgoed, A. (eds.) Proceedings of the 3rd International Conference Intelligent Tutoring Systems, Montreal, Canada, pp. 457–465. Springer, Berlin (1996)

    Google Scholar 

  7. Chandler, P., Sweller, J.: Cognitive load theory and the format of instruction. Cognit. Instr. 8(4), 293–332 (1991)

    Article  Google Scholar 

  8. Chi, M., De Leeuw, N., Chiu, M.H., LaVancher, C.: Eliciting self-explanations improves understanding. Cognit. Sci. 18(3), 439–477 (1994)

    Google Scholar 

  9. Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., Glaser, R.: Self-explanations: how students study and use examples in learning to solve problems. Cognit. Sci. 13(2), 145–182 (1989)

    Article  Google Scholar 

  10. Chi, M.T.H., VanLehn, K.A.: The content of physics self-explanations. J. Learn. Sci. 1(1), 69–105 (1991)

    Article  Google Scholar 

  11. Clark, R.C., Nguyen, F., Sweller, J.: Efficiency in learning: evidence-based guidelines to manage cognitive load. Jossey-Bass, San Francisco (2006)

    Google Scholar 

  12. Corbett, A., Kauffman, L., MacLaren, B., Wagner, A., Jones, E.: A cognitive tutor for genetics problem solving: learning gains and student modeling. J Educ Comput Res. 42, 219–239 (2010)

    Article  Google Scholar 

  13. Corbett, A., MacLaren, B., Wagner, A., Kauffman, L., Mitchell, A., d Baker, R.S.: Differential Impact of Learning Activities Designed to Support Robust Learning in the Genetics Cognitive Tutor. In: Lane, H.C., Yacef, K., Mostow, J. (Eds.) Proceedings of 16th Artificial Intelligence in Education, Memphis, pp. 319–328 (2013)

  14. Davidovic, A., Warren, J., Trichina, E.: Learning benefits of structural example based adaptive tutoring systems. IEEE Trans Educ. 46(2), 241–251 (2003)

    Article  Google Scholar 

  15. Deeks, A.: Web Based Assignments in Structural Analysis. Nanyang Technological University, Deeks, Centre for Educational Development (2000)

    Google Scholar 

  16. Hattie, J.: Visible Learning: A Synthesis of over 800 Meta-Analyses Relating to Achievement. Routledge, New York (2009)

    Google Scholar 

  17. Hausmann, R., Nokes, T., VanLehn, K., Gershman, S.: The design of self-explanation prompts: the fit hypothesis. In: Proceedings of 31st Annual Conference of the Cognitive Science Society, pp. 2626–2631 (2009)

  18. Hilbert, T.S., Renkl, A.: Learning how to use a computer-based concept-mapping tool: self-explaining examples helps. Comput. Hum. Behav. 25(2), 267–274 (2009)

    Article  Google Scholar 

  19. Hsiao, I.H., Bakalov, F., Brusilovsky, P., König-Ries, B.: Progressor: social navigation support through open social student modeling. New Rev. Hypermed. Multimed. 19(2), 112–131 (2013)

    Article  Google Scholar 

  20. Kalyuga, S.: Expertise reversal effect and its implications for learner-tailored instruction. Educ. Psychol. Rev. 19, 509–539 (2007)

    Article  Google Scholar 

  21. Kalyuga, S., Chandler, P., Tuovinen, J., Sweller, J.: When problem solving is superior to studying worked examples. Educ. Psychol. 93(3), 579–588 (2001)

    Article  Google Scholar 

  22. Kalyuga, S., Sweller, J.: Rapid dynamic assessment of expertise to improve the efficiency of adaptive e-learning. Educ. Technol. Res. Dev. 53(3), 83–93 (2005)

    Article  Google Scholar 

  23. Karampiperis, P., Sampson, D.G.: Adaptive Instructional Planning Using Ontologies. In: Proceedings of 4th IEEE International Conference Advanced Learning Technologies Joensuu, Vol. 4, pp. 126-1 (2004)

  24. Kim, R.S., Weitz, R., Heffernan, N.T., Krach, N.: Tutored Problem Solving vs . “ Pure ” Worked Examples. In: Proceedings of 31st Annual Conference of the Cognitive Science Society, pp. 3121–3126. Cognitive Science Society, Austin, TX (2007)

  25. Koedinger, K., Aleven, V.: Exploring the assistance dilemma in experiments with cognitive tutors. Educ. Psychol. Rev. 19(3), 239–264 (2007)

    Article  Google Scholar 

  26. Kostons, D., van Gog, T., Paas, F.: Self-assessment and task selection in learner-controlled instruction: differences between effective and ineffective learners. Comput. Educ. 54(4), 932–940 (2010)

    Article  Google Scholar 

  27. Kyun, S., Kalyuga, S., Sweller, J.: The effect of worked examples when learning to write essays in English literature. J. Exp. Educ 81(3), 385–408

  28. Martin, B., Mitrovic, A.: Using learning curves to mine student models. In: Ardissono, L., Brna, P., Mitrovic, A. (eds.) Proceedings of 10th International Conference User Modeling, pp. 79–88. Springer, Edinburgh (2005)

    Google Scholar 

  29. Masthoff, J.: Design and evaluation of a navigation agent with a mixed locus of control. In: Cerri, S.A., Gouarderes, G., Paraguacu, F. (eds.) Proceedings of International Conference Intelligent Tutoring Systems, pp. 982–991. Springer, Berlin (2002)

    Chapter  Google Scholar 

  30. McLaren, B.M., van Gog, T., Ganoe, C., Karabinos, M., Yaron, D.: The efficiency of worked examples compared to erroneous examples, tutored problem solving, and problem solving in classroom experiments. Comput. Hum. Behav. 55, 87–99 (2016)

    Article  Google Scholar 

  31. McLaren, B., Isotani, S.: When Is It Best to Learn with All Worked Examples? In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds.) Proceedings of 15th International Conference Artificial Intelligence in Education, pp. 222–229. Springer, Berlin (2011)

    Chapter  Google Scholar 

  32. McLaren, B.M., Lim, S.-J., Koedinger, K.R.: When and how often should worked examples be given to students? New results and a summary of the current state of research. In: Proceedings of Conference Cognitive Science Society, pp. 2176–2181. Cognitive Science Society, Austin, TX (2008)

  33. Mitrovic, A.: An intelligent SQL tutor on the web. Int. J. Artif. Intell. Educ. 13(2–4), 173–197 (2003)

    Google Scholar 

  34. Mitrovic, A., Martin, B.: Evaluating the effectiveness of feedback in SQL-Tutor. In: Kinshuk, Jesshope, C., Okamoto, T. (eds.) Proceedings International Workshop on Advanced Learning Technologies, pp. 143–144. Palmerston North (2000)

  35. Mitrovic, A., Martin, B.: Scaffolding and fading problem selection in SQL-Tutor. In: Hoppe, U., Verdejo, F., Kay, J. (eds.) Proceedings of 11th International Conference Artificial Intelligence in Education, pp. 479–481. Australia, Sydney (2003)

    Google Scholar 

  36. Mitrovic, A., Weerasinghe, A.: Revisiting Ill-Definedness and the Consequences for ITSs. In: Dimitrova, V., Mizoguchi, R. (eds.) Proceedings of 14th International Conference on Artificial Intelligence in Education, pp. 375–382. IOS Press, Brighton (2009)

    Google Scholar 

  37. Najar, A.S., Mitrovic, A.: Do novices and advanced students benefit differently from worked examples and ITS? In: Wong, L.H., Liu, C.-C., Hirashima, T., Sumedi, P., Lukman, M. (eds.) Proceedings of International Conference ICCE, pp. 20–29. Bali (2013a)

  38. Najar, A.S., Mitrovic, A.: In: Yacef, K., et al. (eds.) Examples and Tutored Problems: How can Self-Explanation Make a Difference to Learning? Proceedings of 16th International Conference AIED, pp. 339–348. Springer, Heidelberg (2013b)

  39. Najar, A.S., Mitrovic, A.: Examples and tutored problems: is alternating examples and problems the best instructional strategy? Res. Pract. Technol. Enhanc. Learn. 9(3), 439–459 (2014)

    Google Scholar 

  40. Najar, A.S., Mitrovic, A., McLaren, B.M.: Adaptive support versus alternating worked examples and tutored problems: which leads to better learning? In: Dimitrova, V., et al. (eds.) User Modeling, Adaptation, and Personalization, pp. 171–182. Springer International Publishing, Aalborg (2014)

    Google Scholar 

  41. Najar, A.S., Mitrovic, A., McLaren, B.M.: Examples and tutored problems: adaptive support using assistance scores. In: Proceedings of 24th International Joint Conference on Artificial Intelligence Buenos Aires, pp. 4317-4323. AAAI Press (2015)

  42. Paas, F.G.W.C., Van Merrienboer, J.J.G.: The efficiency of instructional conditions: an approach to combine mental effort and performance measures. Hum. Factors 35, 737–743 (1993)

    Google Scholar 

  43. Peachey, D., McCalla, G.: Using planning techniques in intelligent tutoring systems. Int. J. Man-Mach. Stud. 24, 77–98 (1986)

    Article  Google Scholar 

  44. Roediger, H.L., Karpicke, J.D.: Test-enhanced learning taking memory tests improves long-term retention. Psychol. Sci. 17(3), 249–255 (2006)

    Article  Google Scholar 

  45. Rourke, A., Sweller, J.: The worked-example effect using ill-defined problems: learning to recognise designers’ styles. Learn. Instr. 19(2), 185–199 (2009)

    Article  Google Scholar 

  46. Salden, R., Aleven, V., Renkl, A., Schwonke, R.: Worked examples and tutored problem solving: redundant or synergistic forms of support? Topics Cognit. Sci. 1(1), 203–213 (2009)

    Article  Google Scholar 

  47. Salden, R., Koedinger, K., Renkl, A., Aleven, V., McLaren, B.: Accounting for beneficial effects of worked examples in tutored problem solving. Educ. Psychol. Rev. 22(4), 379–392 (2010)

    Article  Google Scholar 

  48. Santos, O.C.: (Ed.) Educational Recommender Systems and Technologies: Practices and Challenges: Practices and Challenges. IGI Global, Hershey (2011)

  49. Schwonke, R., Renkl, A., Krieg, C., Wittwer, J., Aleven, V., Salden, R.: The worked-example effect: not an artefact of lousy control conditions. Comput. Hum. Behav. 25, 258–266 (2009)

    Article  Google Scholar 

  50. Schworm, S., Renkl, A.: Computer-supported example-based learning: when instructional explanations reduce self-explanations. Comput. Educ. 46, 426–445 (2006)

    Article  Google Scholar 

  51. Sosnovsky, S., Brusilovsky, P.: Evaluation of topic-based adaptation and student modeling in QuizGuide. User Model. User-Adapt. Interact. 25(4), 371–424 (2015)

    Article  Google Scholar 

  52. Sweller, J.: The worked example effect and human cognition. Learn. Instr. 16(2), 165–169 (2006)

    Article  Google Scholar 

  53. Sweller, J., Ayres, P., Kalyuga, S.: Cognitive Load Theory. Springer, New York (2011)

    Book  Google Scholar 

  54. Sweller, J., Cooper, G.A.: The use of worked examples as a substitute for problem solving in learning algebra. Cognit. Instr. 2, 59–89 (1985)

    Article  Google Scholar 

  55. Trafton, J. G., Reiser, B. J.: The Contributions of Studying Examples and Solving Problems to Skill Acquisition. In: Proceedings of 15th Annual Conference of the Cognitive Science Society, ACM Press, pp. 1017–1022 (1993)

  56. Van Gog, T.: Effects of identical example-problem and problem-example pairs on learning. Comput. Educ. 57(2), 1775–1779 (2011)

    MathSciNet  Article  Google Scholar 

  57. Van Gog, T., Kester, L., Paas, F.: Effects of worked examples, example-problem, and problem-example pairs on novices’ learning. Contemp. Educ. Psychol. 36(3), 212–218 (2011)

    Article  Google Scholar 

  58. Van Gog, T., Paas, F.: Instructional efficiency: revisiting the original construct in educational research. Educ. Psychol. 43(1), 16–26 (2008)

    Article  Google Scholar 

  59. Van Gog, T., Rummel, N.: Example-based learning: integrating cognitive and social-cognitive research perspectives. Educ. Psychol. Rev. 22(2), 155–174 (2010)

    Article  Google Scholar 

  60. Vassileva, J.: Reactive instructional planning to support interacting teaching strategies. In: Proceedings of 7th Conference Artificial Intelligence in Education, pp. 334–342 (1995)

  61. Webb, N.M.: Peer interaction and learning in small groups. Educ. Res. 13(1), 21–39 (1989)

    Google Scholar 

  62. Weber, G.: Examples and remindings in a case-based help system. In: European Workshop on Advances in Case-Based Reasoning, pp. 165–177. Springer, Berlin (1994)

  63. Weber, G., Brusilovsky, P.: ELM-ART-an interactive and intelligent web-based electronic textbook. Int. J. Artif. Intell. Educ. 26(1), 72–81 (2016)

    Article  Google Scholar 

  64. Weerasinghe, A., Mitrovic, A.: Facilitating deep learning through self-explanation in an open-ended domain. Knowl.-Based Intell. Eng. Syst. 10(1), 3–19 (2006)

    Google Scholar 

  65. Weerasinghe, A., Mitrovic, A., Martin, B.: Towards individualized dialogue support for ill-defined domains. Int. J. Artifici. Intell. Educ. 19(4), 357–379 (2009)

    Google Scholar 

  66. Weerasinghe, A., Mitrovic, A.: Facilitating adaptive tutorial dialogues in EER-tutor. In: International Conference on Artificial Intelligence in Education, pp. 630–631. Springer, Berlin (2011)

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Antonija Mitrovic.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Najar, A.S., Mitrovic, A. & McLaren, B.M. Learning with intelligent tutors and worked examples: selecting learning activities adaptively leads to better learning outcomes than a fixed curriculum. User Model User-Adap Inter 26, 459–491 (2016). https://doi.org/10.1007/s11257-016-9181-y

Download citation

Keywords

  • Intelligent tutoring system
  • Adaptive selection of learning tasks
  • Assistance
  • Self-explanation