Delayed Learning Effects with Erroneous Examples: a Study of Learning Decimals with a Web-Based Tutor

  • Bruce M. McLaren
  • Deanne M. Adams
  • Richard E. Mayer


Erroneous examples – step-by-step problem solutions with one or more errors for students to find and fix – hold great potential to help students learn. In this study, which is a replication of a prior study (Adams et al. 2014), but with a much larger population (390 vs. 208), middle school students learned about decimals either by working with interactive, web-based erroneous examples or with more traditional supported problems to solve. The erroneous examples group was interactively prompted to find, explain, and fix errors in decimal problems, while the problem-solving group was prompted to solve the same decimal problems and explain their solutions. Both groups were given correctness feedback on their work by the web-based program. Although the two groups did not differ on an immediate post-test, the erroneous examples group performed significantly better on a delayed test, given a week after the initial post-test (d = .33, for gain scores), replicating the pattern of the prior study. Interestingly, the problem solving group reported liking the intervention more than the erroneous examples group (d = .21 for liking rating in a questionnaire) and found the user interface easier to interact with (d = .37), suggesting that what students like does not always lead to the best learning outcomes. This result is consistent with that of desirable difficulty studies, in which a more cognitively challenging learning task results in deeper and longer-lasting learning.


Erroneous examples Problem solving Mathematics learning Intelligent tutoring systems 


  1. Adams, D., McLaren, B. M., Durkin, K., Mayer, R.E., Rittle-Johnson, B., Isotani, S., & Van Velsen, M. (2014). Using erroneous examples to improve mathematics learning with a web-based tutoring system. Computers in Human Behavior, 36C (2014), 401–411. Elsevier. doi: 10.1016/j.chb.2014.03.053.
  2. Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. R. (2009). A new paradigm for intelligent tutoring systems: example-tracing tutors. International Journal of Artificial Intelligence in Education, 19(2), 105–154.Google Scholar
  3. Aleven, V., Myers, E., Easterday, M., & Ogan, A. (2010). Toward a framework for the analysis and design of educational games. In: Proceedings of the 2010 I.E. International Conference on Digital Game and Intelligent Toy Enhanced Learning. (pp. 69–76). doi:10.1109/DIGITEL.2010.55.
  4. Anthony, L. (2008). Developing handwriting-based Intelligent Tutors to enhance mathematics learning. Unpublished doctoral dissertation, Carnegie Mellon University, USA.Google Scholar
  5. Atkinson, R. K., Renkl, A., & Merrill, M. M. (2003). Transitioning from studying examples to solving problems: combining fading with prompting fosters learning. Journal of Educational Psychology, 95, 774–783.CrossRefGoogle Scholar
  6. Booth, J. L., Lange, K. E., Koedinger, K. R., & Newton, K. J. (2013). Using example problems to improve student learning in algebra: differentiating between correct and incorrect examples. Learning and Instruction, 25, 24–34.CrossRefGoogle Scholar
  7. Borasi, R. (1996). Reconceiving mathematics instruction: A focus on errors. Ablex Publishing Corporation.Google Scholar
  8. Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999). How people learn: Brain, mind, experience, and school. Washington: National Academy Press.Google Scholar
  9. Brueckner, L. J. (1928). Analysis of difficulties in decimals. Elementary School Journal, 29, 32–41.CrossRefGoogle Scholar
  10. Catrambone, R. (1998). The subgoal learning model: Creating better examples so that students can solve novel problems. Journal of Experimental Psychology: General, 127(4), 355–376.CrossRefGoogle Scholar
  11. Chi, M. T. H. (2000). Self-explaining expository texts: The dual processes of generating inferences and repairing mental models. In R. Glaser (Ed.), Advances in instructional psychology (pp. 161–238). Mahwah: Lawrence Erlbaum Associates, Inc.Google Scholar
  12. Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, R., & Glaser, R. (1989). Self explanations: how students study and used examples in learning to solve problems. Cognitive Science, 13, 145–182.CrossRefGoogle Scholar
  13. Chi, M. T. H., DeLeeuw, N., Chiu, M.-H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 25(4), 471–533.MATHCrossRefGoogle Scholar
  14. College Board (2015). Identifying sentence errors. From the College Board PSAT/NMSQT website:
  15. Dunlowsky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4–58.CrossRefGoogle Scholar
  16. Durkin, K., & Rittle-Johnson, B. (2012). The effectiveness of using incorrect examples to support learning about decimal magnitude. Learning and Instruction, 22, 206–214.CrossRefGoogle Scholar
  17. Fiorella, L., & Mayer, R. E. (2015). Learning as a generative activity: Eight learning strategies that improve understanding. New York: Cambridge University Press.CrossRefGoogle Scholar
  18. Gee, J. P. (2003). What video games have to teach us about learning and literacy (1st ed.). New York: Palgrave Macmillan.Google Scholar
  19. Glasgow, R., Ragan, G., Fields, W. M., Reys, R., & Wasman, D. (2000). The decimal dilemma. Teaching Children Mathematics, 7(2), 89–93.Google Scholar
  20. Graeber, A., & Tirosh, D. (1988). Multiplication and division involving decimals: preservice elementary teachers’ performance and beliefs. Journal of Mathematics Behavior, 7, 263–280.Google Scholar
  21. Groβe, C. S., & Renkl, A. (2007). Finding and fixing errors in worked examples: can this foster learning outcomes? Learning and Instruction, 17(6), 612–634.CrossRefGoogle Scholar
  22. Gunderman, R. B., & Burdick, E. J. (2007). Error and opportunity. American Journal of Roentgenology, 188(4), 901–903.CrossRefGoogle Scholar
  23. Guthrie, E. R. (1952). The psychology of learning. New York: Harper & Brothers.Google Scholar
  24. Hausmann, R. G. M., & Chi, M. T. H. (2002). Can a computer interface support self-explanation? International Journal of Cognitive Technology, 7, 4–14.Google Scholar
  25. Hiebert, J. (1992). Mathematical, cognitive, and instructional analyses of decimal fractions. Chapter 5 in Analysis of arithmetic for mathematics teaching, pp 283–322. Lawrence Erlbaum.Google Scholar
  26. Hiebert, J., & Wearne, D. (1985). A model of students’ decimal computation procedures. Cognition and Instruction, 2, 175–205.CrossRefGoogle Scholar
  27. Huang, T.-H., Liu, Y.-C., & Shiu, C.-Y. (2008). Construction of an online learning system for decimal numbers through the use of cognitive conflict strategy. Computers & Education, 50, 61–76.CrossRefGoogle Scholar
  28. Hull, C. L. (1952). A behavior system: An introduction to behavior theory concerning the individual organism. New Haven: Yale University Press.Google Scholar
  29. Irwin, K. C. (2001). Using everyday knowledge of decimals to enhance understanding. Journal for Research in Mathematics Education, 32(4), 399–420.MathSciNetCrossRefGoogle Scholar
  30. Isotani, S., McLaren, B. M., & Altman, M. (2010). Towards intelligent tutoring with erroneous examples: A taxonomy of decimal misconceptions. In V. Aleven, J. Kay, & J. Mostow (Eds.), Proceedings of the 10th International Conference on Intelligent Tutoring Systems (ITS-10), Lecture Notes in Computer Science, 6094 (pp. 346–348). Berlin: Springer.Google Scholar
  31. Johnson, C. I., & Mayer, R. E. (2010). Applying the self-explanation principle to multimedia learning in a computer-based game-like environment. Computers in Human Behavior, 26, 1246–1252.CrossRefGoogle Scholar
  32. Kalyuga, S., Chandler, P., Tuovinen, J., & Sweller, J. (2001). When problem solving is superior to studying worked examples. Journal of Educational Psychology, 93, 579–588.CrossRefGoogle Scholar
  33. King, A. (1994). Guiding knowledge construction in the classroom: effects of teaching children how to question and how to explain. American Educational Research Journal, 31(2), 338–368.CrossRefGoogle Scholar
  34. Lomas, J. D., Patel, K., Forlizzi, J., & Koedinger, K. (2013). Optimizing challenge in an educational game using large-scale design experiments. Proceedings of CHI2013. New York: ACM Press.Google Scholar
  35. Mayer, R. E., & Johnson, C. I. (2010). Adding instructional features that promote learning in a game-like environment. Journal of Educational Computing Research, 42, 241–265.CrossRefGoogle Scholar
  36. McLaren, B. M., Lim, S., & Koedinger, K. R. (2008). When and how often should worked examples be given to students? New results and a summary of the current state of research. In Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 2176–2181). Austin: Cognitive Science Society.Google Scholar
  37. Moreno, R., & Park, B. (2010). Cognitive load theory: Historical development and relation to other theories. In J. L. Plass, R. Moreno, & R. Brünken (Eds.), Cognitive Load Theory. Cambridge: Cambridge University Press.Google Scholar
  38. National Health Care, U.K. (2013), Intrathecal injection error video:
  39. National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National Mathematics Advisory Panel. Washington: U.S. Department of Education.Google Scholar
  40. Paas, F., & van Merriënboer, J. (1994). Variability of worked examples and transfer of geometrical problem-solving skills: a cognitive-load approach. Journal of Educational Psychology, 86(1), 122–133.CrossRefGoogle Scholar
  41. Putt, I. J. (1995). Preservice teachers ordering of decimal numbers: when more is smaller and less is larger! Focus on Learning Problems in Mathematics, 17(3), 1–15.Google Scholar
  42. Renkl, A. (2002). Worked-out examples: instructional explanations support learning by self explanation. Learning and Instruction, 12, 529–556.CrossRefGoogle Scholar
  43. Renkl, A. (2014). The worked examples principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 391–412). New York: Cambridge University Press.Google Scholar
  44. Renkl, A., & Atkinson, R. K. (2010). Learning from worked-out examples and problem solving. In J. L. Plass, R. Moreno, & R. Brünken (Eds.), Cognitive Load Theory. Cambridge: Cambridge University Press.Google Scholar
  45. Resnick, L. B., Nesher, P., Leonard, F., Magone, M., Omanson, S., & Peled, I. (1989). Conceptual bases of arithmetic errors: the case of decimal fractions. Journal for Research in Mathematics Education, 20(1), 8–27.CrossRefGoogle Scholar
  46. Rittle-Johnson, B., Siegler, R. S., & Alibali, M. W. (2001). Developing conceptual understanding and procedural skill in mathematics: an iterative process. Journal of Educational Psychology, 93, 346–362.CrossRefGoogle Scholar
  47. Sackur-Grisvard, C., & Léonard, F. (1985). Intermediate cognitive organizations in the process of learning a mathematical concept: the order of positive decimal numbers. Cognition and Instruction, 2, 157–174.CrossRefGoogle Scholar
  48. Salden, R. J. C. M., Aleven, V., Schwonke, R., & Renkl, A. (2010). The expertise reversal effect and worked examples in tutored problem solving. Instructional Science, 38, 289–307.CrossRefGoogle Scholar
  49. Schwonke, R., Renkl, A., Krieg, C., Wittwer, J., Aleven, V., & Salden, R. (2009). The worked-example effect: not an artefact of lousy control conditions. Computers in Human Behavior, 25(2009), 258–266.CrossRefGoogle Scholar
  50. Shoebottom, P. (2015). Error correction. From the Frankfurt International School website:
  51. Siegler, R. S. (2002). Microgenetic studies of self-explanation. In N. Granott & J. Parziale (Eds.), Microdevelopment, Transition Processes in Development and Learning (pp. 31–58). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  52. Skinner, B. F. (1938). The behavior of organisms: An experimental analysis. New York: Appleton-Century.Google Scholar
  53. Stacey, K. (2005). Travelling the road to expertise: A longitudinal study of learning. In. H. Chick & J. Vincent (Eds.), Proceedings of the 29th Conference of the International Group for the Psychology of Mathematics Education (vol 1, pp.19–36). University of Melbourne: PME.Google Scholar
  54. Stacey, K., Helme, S., & Steinle, V. (2001). Confusions between decimals, fractions and negative numbers: A consequence of the mirror as a conceptual metaphor in three different ways. In M. v. d. Heuvel-Panhuizen (Ed.), Proceedings of the 25th Conference of the International Group for the Psychology of Mathematics Education (Vol. 4, pp. 217–224). Utrecht: PME.Google Scholar
  55. Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2, 59–89.CrossRefGoogle Scholar
  56. Sweller, J., Van Merriënboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251–296.CrossRefGoogle Scholar
  57. Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. New York: Springer.CrossRefGoogle Scholar
  58. Swigger, K. M., & Wallace, L. F. (1988). A discussion of past programming errors and their effect on learning Assembly language. The Journal of Systems and Software, 8, 395–399.CrossRefGoogle Scholar
  59. The Doctor’s Company (2013). Video on learning from errors.
  60. Tsamir, P. & Tirosh, D. (2003). In-service mathematics teachers’ views of errors in the classroom. In International Symposium: Elementary Mathematics Teaching, Prague.Google Scholar
  61. Tsovaltzi, D., Melis, E., & McLaren, B. M. (2012). Erroneous examples: Effects on learning fractions in a web-based setting. International Journal of Technology Enhanced Learning (IJTEL). V4 N3/4 2012 pp 191–230.Google Scholar
  62. WHO (2014). World Health Organization (WHO): “Learning from Errors to Prevent Harm” workshop.
  63. Widjaja, W., Stacey, K., & Steinle, V. (2011). Locating Negative Decimals on the Number Line: Insights into the Thinking of Pre-service Primary Teachers. Journal of Mathematical Behavior. 30, 80–91.
  64. Wylie, R., & Chi, M. T. H. (2014). The self-explanation principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 413–432). New York: Cambridge University Press.Google Scholar
  65. Yue, C. L., Bjork, E. L., & Bjork, R. A. (2013). Reducing verbal redundancy in multimedia learning: an undesired desirable difficulty. Journal of Educational Psychology, 105, 266–277.CrossRefGoogle Scholar
  66. Zhu, X., & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137–66.CrossRefGoogle Scholar

Copyright information

© International Artificial Intelligence in Education Society 2015

Authors and Affiliations

  • Bruce M. McLaren
    • 1
  • Deanne M. Adams
    • 2
  • Richard E. Mayer
    • 3
  1. 1.Human-Computer Interaction InstituteCarnegie Mellon UniversityPittsburghUSA
  2. 2.Department of PsychologyUniversity of Notre DameSouth BendUSA
  3. 3.Department of Psychological and Brain SciencesUniversity of CaliforniaSanta BarbaraUSA

Personalised recommendations