Educational Psychology Review

, Volume 27, Issue 2, pp 317–326 | Cite as

The Testing Effect Is Alive and Well with Complex Materials

  • Jeffrey D. Karpicke
  • William R. Aue


Van Gog and Sweller (2015) claim that there is no testing effect—no benefit of practicing retrieval—for complex materials. We show that this claim is incorrect on several grounds. First, Van Gog and Sweller’s idea of “element interactivity” is not defined in a quantitative, measurable way. As a consequence, the idea is applied inconsistently in their literature review. Second, none of the experiments on retrieval practice with worked-example materials manipulated element interactivity. Third, Van Gog and Sweller’s literature review omitted several studies that have shown retrieval practice effects with complex materials, including studies that directly manipulated the complexity of the materials. Fourth, the experiments that did not show retrieval practice effects, which were emphasized by Van Gog and Sweller, either involved retrieval of isolated words in individual sentences or required immediate, massed retrieval practice. The experiments failed to observe retrieval practice effects because of the retrieval tasks, not because of the complexity of the materials. Finally, even though the worked-example experiments emphasized by Van Gog and Sweller have methodological problems, they do not show strong evidence favoring the null. Instead, the data provide evidence that there is indeed a small positive effect of retrieval practice with worked examples. Retrieval practice remains an effective way to improve meaningful learning of complex materials.


Testing effect Retrieval practice Worked examples Complex learning Element interactivity 



We thank James Nairne, the Purdue Cognition and Learning Laboratory, and Roddy Roediger and the Washington University Memory Laboratory for the comments on the manuscript. We also thank Mario de Jonge for providing materials.


  1. Agarwal, P. K., Bain, P. M., & Chamberlain, R. W. (2012). The value of applied research: retrieval practice improves classroom learning and recommendations from a teacher, a principal, and a scientist. Educational Psychology Review, 24, 437–448.Google Scholar
  2. Agarwal, P. K., Karpicke, J. D., Kang, S. H., Roediger, H. L., & McDermott, K. B. (2008). Examining the testing effect with open and closed book tests. Applied Cognitive Psychology, 22(7), 861–876.CrossRefGoogle Scholar
  3. Blunt, J. R., & Karpicke, J. D. (2014). Learning with retrieval-based concept mapping. Journal of Educational Psychology, 106(3), 849–858.CrossRefGoogle Scholar
  4. Butler, A. C. (2010). Repeated testing produces superior transfer of learning relative to repeated studying. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(5), 1118–1133.Google Scholar
  5. Butler, A. C., & Roediger, H. L., III. (2007). Testing improves long-term retention in a simulated classroom setting. European Journal of Cognitive Psychology, 19(4–5), 514–527.CrossRefGoogle Scholar
  6. Butler, A. C., Marsh, E. J., Slavinsky, J. P., & Baraniuk, R. G. (2014). Integrating cognitive science and technology improves learning in a STEM classroom. Educational Psychology Review, 26, 331–340.Google Scholar
  7. Carpenter, S. K. (2012). Testing enhances the transfer of learning. Current Directions in Psychological Science, 21(5), 279–283.Google Scholar
  8. Carpenter, S. K., & DeLosh, E. L. (2005). Application of the testing and spacing effects to name learning. Applied Cognitive Psychology, 19(5), 619–636.CrossRefGoogle Scholar
  9. Carpenter, S. K., & Kelly, J. W. (2012). Tests enhance retention and transfer of spatial learning. Psychonomic Bulletin & Review, 19(3), 443–448.CrossRefGoogle Scholar
  10. Chan, J. C. (2009). When does retrieval induce forgetting and when does it induce facilitation? Implications for retrieval inhibition, testing effect, and text processing. Journal of Memory and Language, 61(2), 153–170.CrossRefGoogle Scholar
  11. Chan, J. C. (2010). Long-term effects of testing on the recall of nontested materials. Memory, 18(1), 49–57.CrossRefGoogle Scholar
  12. Chan, J. C., McDermott, K. B., & Roediger, H. L., III. (2006). Retrieval-induced facilitation: initially nontested material can benefit from prior testing of related material. Journal of Experimental Psychology: General, 135(4), 553.CrossRefGoogle Scholar
  13. Cook, L. K., & Mayer, R. E. (1988). Teaching readers about the structure of scientific text. Journal of Educational Psychology, 80(4), 448–456.CrossRefGoogle Scholar
  14. de Jonge, M., Tabbers, H. K., & Rikers, M. J. P. (2015). The effect of testing on the retention of coherent and incoherent text materials. Educational Psychology Review. doi: 10.1007/s10648-015-9300-z
  15. Delaney, P. F., Verkoeijen, P. P., & Spirgel, A. (2010). Spacing and testing effects: a deeply critical, lengthy, and at times discursive review of the literature. Psychology of Learning and Motivation, 53, 63–147.Google Scholar
  16. Dobson, J. L., & Linderholm, T. (2015). Self-testing promotes superior retention of anatomy and physiology information. Advances in Health Sciences Education, 20, 149–161.Google Scholar
  17. Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4–58.CrossRefGoogle Scholar
  18. Foltz, P. W., Kintsch, W., & Landauer, T. K. (1998). The measurement of textual coherence with latent semantic analysis. Discourse Processes, 25(2–3), 285–307.CrossRefGoogle Scholar
  19. Graesser, A. C., McNamara, D. S., & Kulikowich, J. M. (2011). Coh-metrix providing multilevel analyses of text characteristics. Educational Researcher, 40(5), 223–234.CrossRefGoogle Scholar
  20. Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-metrix: analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36(2), 193–202.CrossRefGoogle Scholar
  21. Hinze, S. R., & Wiley, J. (2011). Testing the limits of testing effects using completion tests. Memory, 19(3), 290–304.CrossRefGoogle Scholar
  22. Jensen, J. L., McDaniel, M. A., Woodard, S. M., & Kummer, T. A. (2014). Teaching to the test… or testing to teach: exams requiring higher order thinking skills encourage greater conceptual understanding. Educational Psychology Review, 26(2), 307–329.CrossRefGoogle Scholar
  23. Johnson, C. I., & Mayer, R. E. (2009). A testing effect with multimedia learning. Journal of Educational Psychology, 101(3), 621–629.CrossRefGoogle Scholar
  24. Kang, S. H., McDermott, K. B., & Roediger, H. L., III. (2007). Test format and corrective feedback modify the effect of testing on long-term retention. European Journal of Cognitive Psychology, 19(4–5), 528–558.CrossRefGoogle Scholar
  25. Karpicke, J. D. (2012). Retrieval-based learning active retrieval promotes meaningful learning. Current Directions in Psychological Science, 21(3), 157–163.CrossRefGoogle Scholar
  26. Karpicke, J. D., & Bauernschmidt, A. (2011). Spaced retrieval: absolute spacing enhances learning regardless of relative spacing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(5), 1250–1257.Google Scholar
  27. Karpicke, J. D., & Blunt, J. R. (2011). Retrieval practice produces more learning than elaborative studying with concept mapping. Science, 331, 772–775.CrossRefGoogle Scholar
  28. Karpicke, J. D., Lehman, M., & Aue, W. R. (2014). Retrieval-based learning: an episodic context account. Psychology of Learning and Motivation, 61, 237–284.Google Scholar
  29. Karpicke, J. D., & Roediger, H. L. (2007). Repeated retrieval during learning is the key to long-term retention. Journal of Memory and Language, 57(2), 151–162.CrossRefGoogle Scholar
  30. Karpicke, J. D., & Zaromb, F. M. (2010). Retrieval mode distinguishes the testing effect from the generation effect. Journal of Memory and Language, 62(3), 227–239.CrossRefGoogle Scholar
  31. Larsen, D. P., Butler, A. C., Lawson, A. L., & Roediger, H. L., III. (2013). The importance of seeing the patient: test-enhanced learning with standardized patients and written tests improves clinical application of knowledge. Advances in Health Sciences Education, 18(3), 409–425.CrossRefGoogle Scholar
  32. Leahy, W., Hanham, J., & Sweller, J. (2015). High element interactivity information during problem solving may lead to failure to obtain the testing effect. Educational Psychology Review. doi: 10.1007/s10648-015-9296-4
  33. Lehman, M., Smith, M. A., & Karpicke, J. D. (2014). Toward an episodic context account of retrieval-based learning: dissociating retrieval practice and elaboration. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(6), 1787–1794.Google Scholar
  34. Lyle, K. B., & Crawford, N. A. (2011). Retrieving essential material at the end of lectures improves performance on statistics exams. Teaching of Psychology, 38(2), 94–97.Google Scholar
  35. McDaniel, M. A., Anderson, J. L., Derbish, M. H., & Morrisette, N. (2007a). Testing the testing effect in the classroom. European Journal of Cognitive Psychology, 19(4–5), 494–513.CrossRefGoogle Scholar
  36. McDaniel, M. A., & Einstein, G. O. (1989). Material-appropriate processing: a contextualist approach to reading and studying strategies. Educational Psychology Review, 1(2), 113–145.CrossRefGoogle Scholar
  37. McDaniel, M. A., Howard, D. C., & Einstein, G. O. (2009). The read-recite-review study strategy effective and portable. Psychological Science, 20(4), 516–522.CrossRefGoogle Scholar
  38. McDaniel, M. A., Roediger, H. L., & McDermott, K. B. (2007b). Generalizing test-enhanced learning from the laboratory to the classroom. Psychonomic Bulletin & Review, 14(2), 200–206.CrossRefGoogle Scholar
  39. McDaniel, M. A., Thomas, R. C., Agarwal, P. K., McDermott, K. B., & Roediger, H. L. (2013). Quizzing in middle‐school science: successful transfer performance on classroom exams. Applied Cognitive Psychology, 27(3), 360–372.CrossRefGoogle Scholar
  40. McDermott, K. B., Agarwal, P. K., D’Antonio, L., Roediger III, H. L., & McDaniel, M. A. (2014). Both multiple-choice and short-answer quizzes enhance later exam performance in middle and high school classes. Journal of Experimental Psychology: Applied, 20(1), 3.Google Scholar
  41. Meyer, B. J. F. (1975). The organization of prose and its effects on memory. North-Holland Pub. CoGoogle Scholar
  42. Morey, R. D. & Rouder, J. N. (2015). BayesFactor: computation of Bayes factors for common designs. R package version 0.9.10-2.
  43. Pyc, M. A., & Rawson, K. A. (2007). Examining the efficiency of schedules of distributed retrieval practice. Memory & Cognition, 35(8), 1917–1927.CrossRefGoogle Scholar
  44. R Core Team (2014). R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL
  45. Rohrer, D., Taylor, K., & Sholar, B. (2010). Tests enhance the transfer of learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(1), 233–239.Google Scholar
  46. Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning taking memory tests improves long-term retention. Psychological Science, 17(3), 249–255.CrossRefGoogle Scholar
  47. Roediger, H. L., & Pyc, M. A. (2012). Inexpensive techniques to improve education: applying cognitive psychology to enhance educational practice. Journal of Applied Research in Memory and Cognition, 1(4), 242–248.CrossRefGoogle Scholar
  48. Roediger III, H. L., Agarwal, P. K., McDaniel, M. A., & McDermott, K. B. (2011). Test-enhanced learning in the classroom: long-term improvements from quizzing. Journal of Experimental Psychology: Applied, 17(4), 382.Google Scholar
  49. Rouder, J. N., & Morey, R. D. (2011). A Bayes factor meta-analysis of Bem’s ESP claim. Psychonomic Bulletin & Review, 18(4), 682–689.Google Scholar
  50. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225–237.CrossRefGoogle Scholar
  51. Tran, R., Rohrer, D., & Pashler, H. (2015). Retrieval practice: the lack of transfer to deductive inferences. Psychonomic Bulletin & Review, 22, 135–140.CrossRefGoogle Scholar
  52. Van Gog, T., & Kester, L. (2012). A test of the testing effect: acquiring problem‐solving skills from worked examples. Cognitive Science, 36(8), 1532–1541.CrossRefGoogle Scholar
  53. Van Gog, T., & Sweller, J. (2015). Not new, but nearly forgotten: the testing effect decreases or even disappears as the complexity of learning materials increases. Educational Psychology Review. doi: 10.1007/s10648-015-9310-x
  54. Van Gog, T., Kester, L., Dirkx, K., Hoogerheide, V., Boerboom, J., & Verkoeijen, P. P. J. L. (2015). Testing after worked example study does not enhance delayed problem-solving performance compared to restudy. Educational Psychology Review. doi: 10.1007/s10648-015-9297-3
  55. Weinstein, Y., McDermott, K. B., & Roediger, H. L., III. (2010). A comparison of study strategies for passages: rereading, answering questions, and generating questions. Journal of Experimental Psychology: Applied, 16(3), 308–316.Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Department of Psychological Sciences, Purdue UniversityWest LafayetteUSA

Personalised recommendations