Journal of General Internal Medicine

, Volume 23, Issue 8, pp 1164–1171 | Cite as

Knowledge Retention after an Online Tutorial: A Randomized Educational Experiment among Resident Physicians

  • Douglas S. Bell
  • Charles E. Harless
  • Jerilyn K. Higa
  • Elizabeth L. Bjork
  • Robert A. Bjork
  • Mohsen Bazargan
  • Carol M. Mangione
Original Article



The time course of physicians’ knowledge retention after learning activities has not been well characterized. Understanding the time course of retention is critical to optimizing the reinforcement of knowledge.


Educational follow-up experiment with knowledge retention measured at 1 of 6 randomly assigned time intervals (0–55 days) after an online tutorial covering 2 American Diabetes Association guidelines.


Internal and family medicine residents.


Multiple-choice knowledge tests, subject characteristics including critical appraisal skills, and learner satisfaction.


Of 197 residents invited, 91 (46%) completed the tutorial and were randomized; of these, 87 (96%) provided complete follow-up data. Ninety-two percent of the subjects rated the tutorial as “very good” or “excellent.” Mean knowledge scores increased from 50% before the tutorial to 76% among those tested immediately afterward. Score gains were only half as great at 3–8 days and no significant retention was measurable at 55 days. The shape of the retention curve corresponded with a 1/4-power transformation of the delay interval. In multivariate analyses, critical appraisal skills and participant age were associated with greater initial learning, but no participant characteristic significantly modified the rate of decline in retention.


Education that appears successful from immediate posttests and learner evaluations can result in knowledge that is mostly lost to recall over the ensuing days and weeks. To achieve longer-term retention, physicians should review or otherwise reinforce new learning after as little as 1 week.


knowledge retention online tutorial randomized educational experiment resident physicians educational technology learning theory 

Supplementary material

11606_2008_604_MOESM1_ESM.pdf (30 kb)
Appendix 1.(PDF 30 KB)


  1. 1.
    Davis D, O’Brien MA, Freemantle N, Wolf FM, Mazmanian P,Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. 1999;282(9):867–74.PubMedCrossRefGoogle Scholar
  2. 2.
    Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142(4):260–73.PubMedGoogle Scholar
  3. 3.
    Eisenberg JM. An educational program to modify laboratory use by house staff. J Med Educ. 1977;52(7):578–81.PubMedGoogle Scholar
  4. 4.
    Evans CE, Haynes RB, Birkett NJ, Gilbert JR, Taylor DW, Sackett DL, et al. Does a mailed continuing education program improve physician performance? Results of a randomized trial in antihypertensive care. JAMA. 1986;255(4):501–4.PubMedCrossRefGoogle Scholar
  5. 5.
    Marinopoulos SS, Dorman T, Ratanawongsa N, Wilson LM, Ashar BH, Magaziner JL, et al. Effectiveness of Continuing Medical Education. Rockville, MD: Agency for Healthcare Research and Quality; 2007. Evidence Report AHRQ Publication No. 07-E006, January.Google Scholar
  6. 6.
    Bjork EL, Bjork RA. On the adaptive aspects of retrieval failure in autobiographical memory. In: Grueneberg MM, Morris PE, Sykes RN, eds. Practical Aspects of Memory: Current Research and Issues: Vol 1 Memory in Everyday Life. New York: Wiley; 1988.Google Scholar
  7. 7.
    Bjork RA, Bjork EL. A new theory of disuse and an old theory of stimulus fluctuation. In: Healy A, Kosslyn S, Shiffrin R, eds. From Learning Processes to Cognitive Processes: Essays in honor of William K Estes. Hillsdale, NJ: Lawrence; 1992.Google Scholar
  8. 8.
    Schmidt RA, Bjork RA. New conceptualizations of practice: common principles in three paradigms suggest new concepts for training. Psychol Sci. 1992;3(4):207–17.CrossRefGoogle Scholar
  9. 9.
    Anderson JR, editor. Cognitive psychology and its implications. New York: Freeman; 1995.Google Scholar
  10. 10.
    Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191–215.PubMedCrossRefGoogle Scholar
  11. 11.
    Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: sound educational method or hype? A review of the evaluation literature. Acad Med. 2002;77(10 Suppl):S86–93.PubMedCrossRefGoogle Scholar
  12. 12.
    Cook DA, Dupras DM. A practical guide to developing effective web-based learning. J Gen Intern Med. 2004;19(6):698–707.PubMedCrossRefGoogle Scholar
  13. 13.
    Embi PJ, Bowen JL, Singer E. A Web-based curriculum to improve residents’ education in outpatient medicine. Acad Med. 2001;76(5):545.PubMedCrossRefGoogle Scholar
  14. 14.
    Sisson SD, Hughes MT, Levine D, Brancati FL. Effect of an internet-based curriculum on postgraduate education. A multicenter intervention. J Gen Intern Med. 2004;19(5 Pt 2):505–9.PubMedCrossRefGoogle Scholar
  15. 15.
    Cook DA, Dupras DM, Thompson WG, Pankratz VS. Web-based learning in residents’ continuity clinics: a randomized, controlled trial. Acad Med. 2005;80(1):90–7.PubMedCrossRefGoogle Scholar
  16. 16.
    Bennett NL, Casebeer LL, Kristofco RE, Strasser SM. Physicians’ internet information-seeking behaviors. J Contin Educ Health Prof. 2004;24(1):31–8.PubMedCrossRefGoogle Scholar
  17. 17.
    Accreditation Council for Continuing Medical Education. ACCME Annual Report Data 2004; 2005.Google Scholar
  18. 18.
    van Braak JP, Goeman K. Differences between general computer attitudes and perceived computer attributes: development and validation of a scale. Psychol Rep. 2003;92(2):655–60.PubMedCrossRefGoogle Scholar
  19. 19.
    Chin MH, Cook S, Jin L, Drum ML, Harrison JF, Koppert J, et al. Barriers to providing diabetes care in community health centers. Diabetes Care. 2001;24(2):268–74.PubMedCrossRefGoogle Scholar
  20. 20.
    Cabana MD, Rand C, Slish K, Nan B, Davis MM, Clark N. Pediatrician self-efficacy for counseling parents of asthmatic children to quit smoking. Pediatrics. 2004;113(1 Pt 1):78–81.PubMedCrossRefGoogle Scholar
  21. 21.
    Arauz-Pacheco C, Parrott MA, Raskin P. Hypertension management in adults with diabetes. Diabetes Care. 2004;27(Suppl 1):S65–7.PubMedGoogle Scholar
  22. 22.
    Haffner SM. Dyslipidemia management in adults with diabetes. Diabetes Care. 2004;27(Suppl 1):S68–71.PubMedGoogle Scholar
  23. 23.
    Kern DE, Thomas PA, Howard DM, Bass EB. Goals and Objectives. Curriculum Development for Medical Education: A Six Step Approach. Baltimore: Johns Hopkins University Press; 1998.Google Scholar
  24. 24.
    Bell DS, Fonarow GC, Hays RD, Mangione CM. Self-study from web-based and printed guideline materials. A randomized, controlled trial among resident physicians. Ann Intern Med. 2000;132(12):938–46.PubMedGoogle Scholar
  25. 25.
    Matts JP, Lachin JM. Properties of permuted-block randomization in clinical trials. Control Clin Trials. 1988;9(4):327–44.PubMedCrossRefGoogle Scholar
  26. 26.
    Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297.CrossRefGoogle Scholar
  27. 27.
    Rubin DC, Wenzel AE. One hundred years of forgetting: a quantitative description of retention. Psychol Rev. 1996;103(4):734–60.CrossRefGoogle Scholar
  28. 28.
    Bjork RA. Memory and metamemory considerations in the training of human beings. In: Metcalfe J, Shimamura A, eds. Metacognition: Knowing about Knowing. Cambridge, MA: MIT; 1994:185–205.Google Scholar
  29. 29.
    Chung S, Mandl KD, Shannon M, Fleisher GR. Efficacy of an educational Web site for educating physicians about bioterrorism. Acad Emerg Med. 2004;11(2):143–8.PubMedGoogle Scholar
  30. 30.
    Beal T, Kemper KJ, Gardiner P, Woods C. Long-term impact of four different strategies for delivering an on-line curriculum about herbs and other dietary supplements. BMC Med Educ. 2006;6:39.PubMedCrossRefGoogle Scholar
  31. 31.
    Fordis M, King JE, Ballantyne CM, Jones PH, Schneider KH, Spann SJ, et al. Comparison of the instructional efficacy of internet-based CME with live interactive CME workshops: a randomized controlled trial. JAMA. 2005;294(9):1043–51.PubMedCrossRefGoogle Scholar
  32. 32.
    Gais S, Born J. Declarative memory consolidation: mechanisms acting during human sleep. Learn Mem. 2004;11(6):679–85.PubMedCrossRefGoogle Scholar
  33. 33.
    Chen FM, Bauchner H, Burstin H. A call for outcomes research in medical education. Acad Med. 2004;79(10):955–60.PubMedCrossRefGoogle Scholar
  34. 34.
    Institute of Medicine Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.Google Scholar

Copyright information

© Society of General Internal Medicine 2008

Authors and Affiliations

  • Douglas S. Bell
    • 1
  • Charles E. Harless
    • 1
  • Jerilyn K. Higa
    • 1
  • Elizabeth L. Bjork
    • 2
  • Robert A. Bjork
    • 2
  • Mohsen Bazargan
    • 3
  • Carol M. Mangione
    • 1
    • 4
  1. 1.Division of General Internal Medicine and Health Services ResearchDepartment of Medicine, David Geffen School of Medicine at UCLALos AngelesUSA
  2. 2.Department of PsychologyUniversity of CaliforniaLos AngelesUSA
  3. 3.Department of Family MedicineCharles R. Drew UniversityLos AngelesUSA
  4. 4.Department of Health ServicesUCLA School of Public HealthLos AngelesUSA

Personalised recommendations