Abstract
Prior testing can facilitate subsequent learning, a phenomenon termed the forward testing effect (FTE). We examined a metacognitive account of this effect, which proposes that the FTE occurs because retrieval leads to strategy optimizations during later learning. One prediction of this account is that tests that require less retrieval effort (e.g., multiple-choice relative to cued-recall) should lead to a smaller benefit on new learning. We examined the impact of interpolated multiple-choice or cued-recall testing (relative to no prior testing) on new learning of a four-section STEM text passage. The effect sizes associated with the FTE were numerically, though not significantly larger when the prior tests were cued-recall than multiple-choice, but only when interpolated judgments of learning were not queried. Further, when multiple-choice tests were made more difficult through lure similarity, the FTE was similarly increased. Finally, the FTE was eliminated entirely when participants provided four JOLs after reading each text section. We believe this elimination of the FTE stemmed from an increase in performance for the control participants induced by reactivity from repeated metacognitive queries requiring deep metacognitive reflection. Taken together, these experiments support a metacognitive account of FTE and have important implications for how educators and students should employ retrieval practice and leverage the benefits of metacognitive reflection to improve new learning.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Note that Kubik et al. (2022) also examined JOLs in an FTE paradigm. However, their method employed item-by-item JOLs, which can recruit covert retrieval processes that mimic overt retrieval. In the case of the present study, we were interested primarily in aggregate judgments, which are less likely to elicit covert retrieval, particularly for complex material.
The experiments reported here were not publicly pre-registered, but the methodology was presented in a dissertation proposal by S.D. to her doctoral program committee members, which include J.C.K. Importantly, the dissertation process served a similar function to a pre-registration, because S.D. was expected to adhere to the proposed methodology. Experiments 1a, 1b, and 2b were included in this proposal, and Experiment 2a was designed using the same sample size targets to ensure consistency between experiments.
We conducted a 2 (Sample: Online vs. Lab) × 4 (Experiment) × 3 (Interpolated Test Condition: Cued-Recall, Easy Multiple-Choice, or No Test) × 2 (Criterial Test Condition: Cued-Recall vs. Multiple Choice) to determine the impact of sample on the primary dependent measure. The main effect of sample was not significant, nor were any interactions involving the Sample variable, F’s < 1.87.
articipants were never given a delayed multiple-choice test in actuality, but we included questions regarding a delayed cued-recall and multiple-choice test to be consistent with the questions for the immediate tests.
Further, the most optimal strategy that a learner can use likely varies tremendously based on the difficulty of the to-be-studied material, the learner’s motivation, time constraints, and the learner’s ultimate goals.
We would be remiss if we did not add the caveat that the JOL effects that we discuss here were only observed cross-experimentally. In fact, we did not anticipate the JOL reactivity effect at the outset of these experiments, given that there is evidence that reactivity effects are limited in complex materials (Tauber et al., 2015).
References
Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017). Rethinking the use of tests: A meta-analysis of practice testing. Review of Educational Research, 87(3), 659–701. https://doi.org/10.3102/0034654316689306
Abel, M., Bäuml, T., & K. H. (2016). Retrieval practice can eliminate list method directed forgetting. Memory & Cognition, 44, 15–23. https://doi.org/10.3758/s13421-015-0539-x
Ahn, D., & Chan, J. C. K. (2022). Does testing enhance new learning because it insulates against proactive interference? Memory & Cognition, 50(8), 1664–1682. https://doi.org/10.3758/s13421-022-01273-7
Ahn, D., & Chan, J.C.K. (in press). Does testing enhance new learning because it enables learners to use better strategies? Journal of Experimental Psychology: Learning, Memory, and Cognition.
Ariel, R., Karpicke, J. D., Witherby, A. E., & Tauber, S. K. (2021). Do judgments of learning directly enhance learning of educational materials? Educational Psychology Review, 33, 693–712. https://doi.org/10.1007/s10648-020-09556-8
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417–444.
Carpenter, S. K., & DeLosh, E. L. (2006). Impoverished cue support enhances subsequent retention: Support for the elaborative retrieval explanation of the testing effect. Memory & Cognition, 34(2), 268–276. https://doi.org/10.3758/BF03193405
Chan, J. C. K., Manley, K. D., Davis, S. D., & Szpunar, K. K. (2018a). Testing potentiates new learning across a retention interval and a lag: A strategy change perspective. Journal of Memory and Language, 102, 83–96. https://doi.org/10.1016/j.jml.2018.05.007
Chan, J. C. K., Meissner, C. A., & Davis, S. D. (2018b). Test-potentiated new learning: A meta-analytic review. Psychological Bulletin, 114(11), 1111–1146. https://doi.org/10.1037/bul0000166
Chan, J. C. K., Manley, K. D., & Ahn, D. (2020). Does retrieval potentiate new learning when retrieval stops but new learning continues? Journal of Memory and Language, 115, 1–19. https://doi.org/10.1016/j.jml.2020.104150
Chan, J. C., O’Donnell, R., & Manley, K. D. (2022). Warning weakens retrieval-enhanced suggestibility only when it is given shortly after misinformation: The critical importance of timing. Journal of Experimental Psychology: Applied, 28(4), 694–716. https://doi.org/10.1037/xap0000394
Cho, K. W., & Powers, A. (2019). Testing enhances both memorization and conceptual learning of categorical materials. Journal of Applied Research in Memory and Cognition, 8(2), 166–177. https://doi.org/10.1016/j.jarmac.2019.01.003
Choi, H., & Lee, H. (2020). Knowing Is not half the battle: The role of actual test experience in the forward testing effect. Educational Psychology Review, 32(3), 765–789. https://doi.org/10.1007/s10648-020-09518-0
Craik, F. I. (1986). A functional account of age differences in memory. Human Memory and Cognitive Capabilities: Mechanisms and Performances, 5, 409–422.
Davis, S. D., Chan, J. C. K., & Wilford, M. M. (2017). The dark side of interpolated testing: Frequent switching between retrieval and encoding impairs new learning. Journal of Applied Research on Memory and Cognition, 6, 434–441.
Davis, S. D., & Peterson, D. J. (2019). Reducing the number of retrieval opportunities reduces the magnitude of the forward testing effect. Poster presented at the 60th annual meeting of the Psychonomic Society, Montreal, QC, Canada.
Don, H. J., Yang, C., Boustani, S., & Shanks, D. R. (2022). Do partial and distributed tests enhance new learning? Journal of Experimental Psychology: Applied, 29(2), 358–373. https://doi.org/10.1037/xap0000440
Double, K. S., Birney, D. P., & Walker, S. A. (2018). A meta-analysis and systematic review of reactivity to judgments of learning. Memory, 26(6), 741–750. https://doi.org/10.1080/09658211.2017.1404111
Dougherty, M. R., Scheck, P., Nelson, T. O., & Narens, L. (2005). Using the past to predict the future. Memory & Cognition, 33(6), 1096–1115. https://doi.org/10.3758/BF03193216
Dougherty, M. R., Robey, A. M., & Buttaccio, D. (2018). Do metacognitive judgments alter memory performance beyond the benefits of retrieval practice? A comment on and replication attempt of Dougherty, Scheck, Nelson, and Narens (2005). Memory & Cognition, 46, 558–565. https://doi.org/10.3758/s13421-018-0791-y
Dunlosky, J., & Rawson, K. A. (2012). Overconfidence produces underachievement: Inaccurate self evaluations undermine students’ learning and retention. Learning and Instruction, 22(4), 271–280. https://doi.org/10.1016/j.learninstruc.2011.08.003
Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4–58. https://doi.org/10.1177/1529100612453266
Dunn, P. K. (2021). Scientific research and methodology: An introduction to quantitative research in science and health. https://bookdown.org/pkaldunn/Book
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146
Finn, B., & Roediger, H. L., III. (2013). Interfering effects of retrieval in learning new information. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39(6), 1665–1681. https://doi.org/10.1037/a0032377
Finn, B. (2017). A framework of episodic updating: An account of memory updating after retrieval. In Psychology of learning and motivation. Academic Press, 67, 173–211.
Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87(6), 1082–1116. https://doi.org/10.3102/0034654317726529
Hayslip, B., Jr., & Kennelly, K. J. (1982). Short-term memory and crystallized-fluid intelligence in adulthood. Research on Aging, 4(3), 314–332. https://doi.org/10.1177/0164027582004003003
Janes, J. L., Rivers, M. L., & Dunlosky, J. (2018). The influence of making judgments of learning on memory performance: Positive, negative, or both? Psychonomic Bulletin & Review, 25(6), 2356–2364.
JASP Team, (2023). JASP (Version 0.17). [Computer software].
Jing, H. G., Szpunar, K. K., & Schacter, D. L. (2016). Interpolated testing influences focused attention and improves integration of information during a video-recorded lecture. Journal of Experimental Psychology: Applied, 22, 305–318. https://doi.org/10.1037/xap0000087
King, J. F., Zechmeister, E. B., & Shaughnessy, J. J. (1980). Judgments of knowing: The influence of retrieval practice. The American Journal of Psychology, 329–343.
Kliegl, O., & Bäuml, K.-H.T. (2021). When retrieval practice promotes new learning – The critical role of study material. Journal of Memory and Language, 120, 104253. https://doi.org/10.1016/j.jml.2021.104253
Koriat, A., Sheffer, L., & Ma’ayan, H. (2002). Comparing objective and subjective learning curves: Judgments of learning exhibit increased underconfidence with practice. Journal of Experimental Psychology: General, 131(2), 147–162.
Kornell, N., & Metcalfe, J. (2006). Study efficacy and the region of proximal learning framework. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32(3), 609–622. https://doi.org/10.1037/0278-7393.32.3.609
Kornell, N., & Bjork, R. A. (2007). The promise and perils of self-regulated study. Psychonomic bulletin & review, 14(2), 219–224. https://doi.org/10.3758/BF03194055
Kubik, V., Koslowski, K., Schubert, T., & Aslan, A. (2022). Metacognitive judgments can potentiate new learning: The role of covert retrieval. Metacognition and Learning, 17, 1057–1077. https://doi.org/10.1007/s11409-022-09307-w
Lee, H. S., & Ha, H. (2019). Metacognitive judgments of prior material facilitate the learning of new material: The forward effect of metacognitive judgments in inductive learning. Journal of Educational Psychology, 111(7), 1189–1201. https://doi.org/10.1037/edu0000339
Little, J. L., Bjork, E. L., Bjork, R. A., & Angello, G. (2012). Multiple-choice tests exonerated, at least of some charges: Fostering test-induced learning and avoiding test-induced forgetting. Psychological Science, 23(11), 1337–1344. https://doi.org/10.1177/0956797612443370
Mazzoni, G., Cornoldi, C., & Marchitelli, G. (1990). Do memorability ratings affect study-time allocation? Memory & Cognition, 18(2), 196–204. https://doi.org/10.3758/BF03197095
Metcalfe, J., & Kornell, N. (2005). A region of proximal learning model of study time allocation. Journal of memory and language, 52(4), 463–477. https://doi.org/10.1016/j.jml.2004.12.001
McDaniel, M. A., Anderson, J. L., Derbish, M. H., & Morrisette, N. (2007). Testing the testing effect in the classroom. European Journal of Cognitive Psychology, 19(4–5), 494–513. https://doi.org/10.1080/09541440701326154
McDermott, K. B. (2021). Practicing retrieval facilitates learning. Annual Review of Psychology, 72, 609–633. https://doi.org/10.1146/annurev-psych-010419-051019
Mitchum, A. L., Kelley, C. M., & Fox, M. C. (2016). When asking the question changes the ultimate answer: Metamemory judgments change memory. Journal of Experimental Psychology: General, 145(2), 200–219. https://doi.org/10.1037/a0039923
Morris, C. D., Bransford, J. D., & Franks, J. J. (1977). Levels of processing versus transfer appropriate processing. Journal of Verbal Learning and Verbal Behavior, 16(5), 519–533. https://doi.org/10.1016/S0022-5371(77)80016-9
Myers, S. J., Rhodes, M. G., & Hausman, H. E. (2020). Judgments of learning (JOLs) selectively improve memory depending on the type of test. Memory and Cognition, 48, 745–758. https://doi.org/10.3758/s13421-020-01025-5
Panadero, E., Jonsson, A., & Botella, J. (2017). Effects of self-assessment on self-regulated learning and self-efficacy: Four meta-analyses. Educational Research Review, 22, 74–98. https://doi.org/10.1016/j.edurev.2017.08.004
Pastötter, B., & Bäuml, K. T. (2014). Retrieval practice enhances new learning: The forward effect of testing. Frontiers in Psychology, 5, 5.
Pastötter, B., Schicker, S., Niedernhuber, J., & Bäuml, K. H. T. (2011). Retrieval during learning facilitates subsequent memory encoding. Journal of Experimental Psychology: Learning, Memory, & Cognition, 37, 287–297. https://doi.org/10.1037/a0021801
Pastötter, B., Weber, J., & Bäuml, K. H. T. (2013). Using testing to improve learning after severe traumatic brain injury. Neuropsychology, 27(2), 280–285. https://doi.org/10.1037/a0031797
Pastötter, B., Engel, M., & Frings, C. (2018). The forward effect of testing: Behavioral evidence for the reset-of-encoding hypothesis using serial position analysis. Frontiers in Psychology, 9, 1197. https://doi.org/10.3389/fpsyg.2018.01197
Rawson, K. A. (2015). The status of the testing effect for complex materials: Still a winner. Educational Psychology Review, 27(2), 327–331. https://doi.org/10.1007/s10648-015-9308-4
Rhodes, M. G. (2016). Judgments of learning: Methods, data, and theory. In J. Dunlosky, & S. K. Tauber (Eds.), The Oxford Handbook of Metamemory; (pp. 65–80, Chapter xv, 574 Pages) Oxford University Press, New York, NY.
Roediger, H. L., III., & Marsh, E. J. (2005). The positive and negative consequences of multiple-choice testing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(5), 1155–1159. https://doi.org/10.1037/0278-7393.31.5.1155
Rowland, C. A. (2014). The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychological Bulletin, 140(6), 1432–1463. https://doi.org/10.1037/a0037559
Sahakyan, L., Delaney, P. F., & Kelley, C. M. (2004). Self-evaluation as a moderating factor of strategy change in directed forgetting benefits. Psychonomic Bulletin & Review, 11(1), 131–136. https://doi.org/10.3758/BF03206472
Shi, A., Xu, C., Zhao, W., Shanks, D. R., Hu, X., Luo, L., & Yang, C. (2023). Judgments of learning reactively facilitate visual memory by enhancing learning engagement. Psychonomic Bulletin & Review, 30(2), 676–687. https://doi.org/10.3758/s13423-022-02174-1
Smith, M. A., & Karpicke, J. D. (2014). Retrieval practice with short-answer, multiple-choice, and hybrid tests. Memory, 22(7), 784–802. https://doi.org/10.1080/09658211.2013.831454
Soderstrom, N. C., Clark, C. T., Halamish, V., & Bjork, E. L. (2015). Judgments of learning as memory modifiers. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41(2), 553–558. https://doi.org/10.1037/a0038388
Spellman, B. A., & Bjork, R. A. (1992). When predictions create reality: Judgments of learning may alter what they are intended to assess. Psychological Science, 3(5), 315–317. https://doi.org/10.1111/j.1467-9280.1992.tb00680.x
Szpunar, K. K., McDermott, K. B., Roediger, H. L., & III. (2008). Testing during study insulates against the buildup of proactive interference. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34(6), 1392–1399. https://doi.org/10.1037/a0013082
Szpunar, K. K., Khan, N. Y., & Schacter, D. L. (2013). Interpolated memory tests reduce mind wandering and improve learning of online lectures. Proceedings of the National Academy of Sciences of the United States of America, 110, 6313–6317.
Szpunar, K. K., Jing, H. G., & Schacter, D. L. (2014). Overcoming overconfidence in learning from video-recorded lectures: Implications of interpolated testing for online education. Journal of Applied Research in Memory and Cognition, 3, 161–164.
Tauber, S. K. U., Dunlosky, J., & Rawson, K. A. (2015). The influence of retrieval practice versus delayed judgments of learning on memory: Resolving a memory-metamemory paradox. Experimental Psychology, 62(4), 254–263. https://doi.org/10.1027/1618-3169/a000296
Thiede, K. W., & Dunlosky, J. (1994). Delaying students’ metacognitive monitoring improves their accuracy in predicting their recognition performance. Journal of Educational Psychology, 86(2), 290–302. https://doi.org/10.1037/0022-0663.86.2.290
Weinstein, Y., Gilmore, A. W., Szpunar, K. K., & McDermott, K. B. (2014). The role of test expectancy in the build-up of proactive interference in long-term memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(4), 1039–1048. https://doi.org/10.1037/a0036164
Wissman, K. T., & Rawson, K. A. (2015). Grain size of recall practice for lengthy text material: Fragile and mysterious effects on memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41, 439–455. https://doi.org/10.1037/xlm0000047
Wissman, K. T., Rawson, K. A., & Pyc, M. A. (2011). The interim test effect: Testing prior material can facilitate the learning of new material. Psychonomic Bulletin & Review, 18, 1140–1147.
Woodford, C. (2021). Lasers. Explain that Stuff. Retrieved January 24, 2023 from https://www.explainthatstuff.com/lasers.html
Yang, C., Potts, R., & Shanks, D. R. (2017). The forward testing effect on self-regulated study time allocation and metamemory monitoring. Journal of Experimental Psychology: Applied, 23(3), 263–277. https://doi.org/10.1037/xap0000122
Yang, C., Potts, R., & Shanks, D. R. (2018). Enhancing learning and retrieval of new information: a review of the forward testing effect. NPJ science of learning, 3(1), 8. https://doi.org/10.1038/s41539-018-0024-y
Yang, C., Chew, S. J., Sun, B., & Shanks, D. R. (2019). The forward effects of testing transfer to different domains of learning. Journal of Educational Psychology, 111(5), 809–826. https://doi.org/10.1037/edu0000320
Yang, C., Zhao, W., Luo, L., Sun, B., Potts, R., & Shanks, D. R. (2022). Testing potential mechanisms underlying test-potentiated new learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 48(8), 1127–1143. https://doi.org/10.1037/xlm0001021
Zhao, W., Li, B., Shanks, D. R., Zhao, W., Zheng, J., Hu, X., Su, N., Fan, T., Yin, Y., Luo, L., & Yang, C. (2021). When judging what you know changes what you really know: Soliciting metamemory judgments reactively enhances children’s learning. Child Development, 93(2), 405–417. https://doi.org/10.1111/cdev.13689
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
All materials as well as data used for analyses can be found at the Open Science Framework at https://osf.io/ezjuv/.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Davis, S.D., Chan, J.C.K. Effortful Tests and Repeated Metacognitive Judgments Enhance Future Learning. Educ Psychol Rev 35, 86 (2023). https://doi.org/10.1007/s10648-023-09803-8
Accepted:
Published:
DOI: https://doi.org/10.1007/s10648-023-09803-8