Advances in Health Sciences Education

, Volume 15, Issue 5, pp 755–770 | Cite as

Time and learning efficiency in Internet-based learning: a systematic review and meta-analysis

  • David A. Cook
  • Anthony J. Levinson
  • Sarah Garside


Authors have claimed that Internet-based instruction promotes greater learning efficiency than non-computer methods. Objectives Determine, through a systematic synthesis of evidence in health professions education, how Internet-based instruction compares with non-computer instruction in time spent learning, and what features of Internet-based instruction are associated with improved learning efficiency. Data sources We searched databases including MEDLINE, CINAHL, EMBASE, and ERIC from 1990 through November 2008. Study selection and data abstraction We included all studies quantifying learning time for Internet-based instruction for health professionals, compared with other instruction. Reviewers worked independently, in duplicate, to abstract information on interventions, outcomes, and study design. Results We identified 20 eligible studies. Random effects meta-analysis of 8 studies comparing Internet-based with non-Internet instruction (positive numbers indicating Internet longer) revealed pooled effect size (ES) for time −0.10 (p = 0.63). Among comparisons of two Internet-based interventions, providing feedback adds time (ES 0.67, p = 0.003, two studies), and greater interactivity generally takes longer (ES 0.25, p = 0.089, five studies). One study demonstrated that adapting to learner prior knowledge saves time without significantly affecting knowledge scores. Other studies revealed that audio narration, video clips, interactive models, and animations increase learning time but also facilitate higher knowledge and/or satisfaction. Across all studies, time correlated positively with knowledge outcomes (r = 0.53, p = 0.021). Conclusions On average, Internet-based instruction and non-computer instruction require similar time. Instructional strategies to enhance feedback and interactivity typically prolong learning time, but in many cases also enhance learning outcomes. Isolated examples suggest potential for improving efficiency in Internet-based instruction.


Internet Educational technology Education, professional Computer-assisted instruction Distance education Learning efficiency 



The authors thank Denise M. Dupras, MD, PhD, Patricia J. Erwin, MLS, and Victor M. Montori, MD, MSc for their role in study identification and data abstraction. This work was supported by intramural funds and a Mayo Foundation Education Innovation award. Dr. Levinson is supported in part through the John R. Evans Chair in Health Sciences Education Research. Dr. Cook had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.


  1. Bell, D. S., Fonarow, G. C., Hays, R. D., & Mangione, C. M. (2000). Self-study from web-based and printed guideline materials: A randomized, controlled trial among resident physicians. Annals of Internal Medicine, 132, 938–946.Google Scholar
  2. Berman, N. B., Fall, L. H., Maloney, C. G., & Levine, D. A. (2008). Computer-assisted instruction in clinical education: A roadmap to increasing CAI implementation. Advances in Health Sciences Education, 13, 373–383.CrossRefGoogle Scholar
  3. Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A., et al. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79, 1243–1289.CrossRefGoogle Scholar
  4. Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., et al. (2004). How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74, 379–439.CrossRefGoogle Scholar
  5. Blackmore, C., Tantam, D., & Van Deurzen, E. (2006). The role of the eTutor—evaluating tutor input in a virtual learning community for psychotherapists and psychologists across Europe. International Journal of Psychotherapy, 10(2), 35–46.Google Scholar
  6. Chumley-Jones, H. S., Dobbie, A., & Alford, C. L. (2002). Web-based learning: Sound educational method or hype? A review of the evaluation literature. Academic Medicine, 77(10 suppl), S86–S93.CrossRefGoogle Scholar
  7. Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53, 445–459.Google Scholar
  8. Clark, D. (2002). Psychological myths in e-learning. Medical Teacher, 24(6), 598–604.CrossRefGoogle Scholar
  9. Clark, R. C., & Mayer, R. E. (2007). E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning (2nd ed.). San Francisco: Pfeiffer.Google Scholar
  10. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  11. Cohen, P. A., & Dacanay, L. D. (1992). Computer-based instruction and health professions education: A meta-analysis of outcomes. Evaluation and the Health Professions, 15, 259–281.CrossRefGoogle Scholar
  12. Cohen, P. A., & Dacanay, L. D. (1994). A meta-analysis of computer-based instruction in nursing education. Computers in Nursing, 12(2), 89–97.Google Scholar
  13. Cook, D. A. (2005). The research we still are not doing: An agenda for the study of computer-based learning. Academic Medicine, 80, 541–548.CrossRefGoogle Scholar
  14. Cook, D. A. (2009). The failure of e-learning research to inform educational practice, and what we can do about it. Medical Teacher, 31, 158–162.CrossRefGoogle Scholar
  15. Cook, D. A., Beckman, T. J., Thomas, K. G., & Thompson, W. G. (2008a). Adapting Web-based instruction to residents’ knowledge improves learning efficiency: A randomized controlled trial. Journal of General Internal Medicine, 23, 985–990.CrossRefGoogle Scholar
  16. Cook, D. A., Beckman, T. J., Thomas, K. G., & Thompson, W. G. (2008b). Introducing resident physicians to complexity in ambulatory medicine. Medical Education, 42, 838–848.CrossRefGoogle Scholar
  17. Cook, D. A., & Dupras, D. M. (2004). A practical guide to developing effective web-based learning. Journal of General Internal Medicine, 19, 698–707.CrossRefGoogle Scholar
  18. Cook, D. A., Dupras, D. M., Thompson, W. G., & Pankratz, V. S. (2005). Web-based learning in residents’ continuity clinics: A randomized, controlled trial. Academic Medicine, 80, 90–97.CrossRefGoogle Scholar
  19. Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M. (2008c). Internet-based learning in the health professions: A meta-analysis. JAMA, 300, 1181–1196.CrossRefGoogle Scholar
  20. Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M. (2010). Instructional design variations in internet-based learning for health professions education: A systematic review and meta-analysis. Academic Medicine, 85, 909–922.Google Scholar
  21. Cook, D. A., Thompson, W. G., & Thomas, K. G. (2009a). Case-based or non-case-based questions for teaching postgraduate physicians: A randomized crossover trial. Academic Medicine, 84, 1419–1425.CrossRefGoogle Scholar
  22. Cook, D. A., Thompson, W. G., Thomas, K. G., & Thomas, M. R. (2009b). Lack of interaction between sensing–intuitive learning styles and problem-first versus information-first instruction: A randomized crossover trial. Advances in Health Sciences Education, 14, 79–90.CrossRefGoogle Scholar
  23. Cook, D. A., Thompson, W. G., Thomas, K. G., Thomas, M. R., & Pankratz, V. S. (2006). Impact of self-assessment questions and learning styles in web-based learning: A randomized, controlled, crossover trial. Academic Medicine, 81, 231–238.CrossRefGoogle Scholar
  24. Cooper, E. H., & Pantle, A. J. (1967). The total-time hypothesis in verbal learning. Psychological Bulletin, 68, 221–234.CrossRefGoogle Scholar
  25. Craik, F. I. M., & Tulving, E. (1975). Depth of processing and the retention of words in episodic memory. Journal ol Experimental Psychology: General, 104, 268–294.CrossRefGoogle Scholar
  26. Cronbach, L. J., & Snow, R. E. (1977). Aptitudes and instructional methods: A handbook for research on interactions. New York: Irvington Publishers.Google Scholar
  27. Curtin, F., Altman, D. G., & Elbourne, D. (2002). Meta-analysis combining parallel and cross-over clinical trials. I: Continuous outcomes. Statistics in Medicine, 21, 2131–2144.CrossRefGoogle Scholar
  28. Dennis, J. K. (2003). Problem-based learning in online vs. face-to-face environments. Education for Health, 16(2), 198–209.CrossRefGoogle Scholar
  29. Effective Use of Educational Technology in Medical Education: Summary Report of the 2006 AAMC Colloquium on Educational Technology. (2007). Washington, DC: Association of American Medical Colleges.Google Scholar
  30. Fletcher, J. D. (2009). Education and training technology in the military. Science, 323, 72–75.CrossRefGoogle Scholar
  31. Fordis, M., King, J. E., Ballantyne, C. M., Jones, P. H., Schneider, K. H., Spann, S. J., et al. (2005). Comparison of the instructional efficacy of internet-based CME with live interactive CME workshops: A randomized controlled trial. JAMA, 294, 1043–1051.CrossRefGoogle Scholar
  32. Fredrick, W. C., & Walberg, H. J. (1980). Learning as a function of time. Journal of Educational Research, 73, 183–194.Google Scholar
  33. Friedl, R., Hoppler, H., Ecard, K., Scholz, W., Hannekum, A., Ochsner, W., et al. (2006a). Multimedia-driven teaching significantly improves students’ performance when compared with a print medium. Annals of Thoracic Surgery, 81, 1760–1766.CrossRefGoogle Scholar
  34. Friedl, R., Hoppler, H., Ecard, K., Scholz, W., Hannekum, A., Oechsner, W., et al. (2006b). Comparative evaluation of multimedia driven, interactive, and case-based teaching in heart surgery. Annals of Thoracic Surgery, 82, 1790–1795.CrossRefGoogle Scholar
  35. Gijselaers, W. H., & Schmidt, H. G. (1995). Effects of quantity of instruction on time spent on learning and achievement. Educational Research and Evaluation, 1(2), 183–201.CrossRefGoogle Scholar
  36. Goitein, L., Shanafelt, T. D., Wipf, J. E., Slatore, C. G., & Back, A. L. (2005). The effects of work-hour limitations on resident well-being, patient care, and education in an internal medicine residency program. Archives of Internal Medicine, 165, 2601–2606.CrossRefGoogle Scholar
  37. Greenhalgh, T. (2001). Computer assisted learning in undergraduate medical education. BMJ, 322, 40–44.CrossRefGoogle Scholar
  38. Grundman, J., Wigton, R., & Nickol, D. (2000). A controlled trial of an interactive, web-based virtual reality program for teaching physical diagnosis skills to medical students. Academic Medicine, 75(10 Suppl), S47–S49.CrossRefGoogle Scholar
  39. Higgins, J. P. T., & Green, S. (2008). Cochrane handbook for systematic reviews of interventions 5.0.1 [updated September 2008]. Retrieved May 11, 2009, from
  40. Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ, 327, 557–560.CrossRefGoogle Scholar
  41. Kopp, V., Stark, R., & Fischer, M. R. (2008). Fostering diagnostic knowledge through computer-supported, case-based worked examples: Effects of erroneous examples and feedback. Medical Education, 42, 823–829.CrossRefGoogle Scholar
  42. Kulik, C.-L. C., Kulik, J. A., & Shwalb, B. J. (1986). The effectiveness of computer-based adult education: A meta-analysis. Journal of Educational Computing Research, 2, 235–252.CrossRefGoogle Scholar
  43. Larreamendy-Joerns, J., & Leinhardt, G. (2006). Going the distance with online education. Review of Educational Research, 76(4), 567–605.CrossRefGoogle Scholar
  44. Leong, S. L., Baldwin, C. D., & Adelman, A. M. (2003). Integrating web-based computer cases into a required clerkship: Development and evaluation. Academic Medicine, 78, 295–301.CrossRefGoogle Scholar
  45. Lewis, M. J., Davies, R., Jenkins, D., & Tait, M. I. (2001). A review of evaluative studies of computer-based learning in nursing education. Nurse Education Today, 21, 26–37.CrossRefGoogle Scholar
  46. Mattheos, N., Nattestad, A., Christersson, C., Jansson, H., & Attstrom, R. (2004). The effects of an interactive software application on the self-assessment ability of dental students. European Journal of Dental Education, 8(3), 97–104.Google Scholar
  47. McKimm, J., Jollie, C., & Cantillon, P. (2003). ABC of learning and teaching in medicine: Web based learning. BMJ, 326(7394), 870–873.CrossRefGoogle Scholar
  48. Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999). Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Quality of reporting of meta-analyses. Lancet, 354, 1896–1900.CrossRefGoogle Scholar
  49. Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA Statement. Annals of Internal Medicine, 151, 264–269.Google Scholar
  50. Nicholson, D. T., Chalk, C., Funnell, W. R. J., & Daniel, S. J. (2006). Can virtual reality improve anatomy education? A randomised controlled study of a computer-generated three-dimensional anatomical ear model. Medical Education, 40(1), 1081–1087.CrossRefGoogle Scholar
  51. Papa, F. J., Aldrich, D., & Schumacker, R. E. (1999). The effects of immediate online feedback upon diagnostic performance. Academic Medicine, 74(10 suppl), S16–S18.CrossRefGoogle Scholar
  52. Papa, F. J., Oglesby, M. W., Aldrich, D. G., Schaller, F., & Cipher, D. J. (2007). Improving diagnostic capabilities of medical students via application of cognitive sciences-derived learning principles. Medical Education, 41, 419–425.CrossRefGoogle Scholar
  53. Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 231–244). New York: Russell Sage Foundation.Google Scholar
  54. Ruiz, J. G., Mintzer, M. J., & Leipzig, R. M. (2006). The impact of e-learning in medical education. Academic Medicine, 81, 207–212.CrossRefGoogle Scholar
  55. Ryan, J. (2005). Unintended consequences: The accreditation council for graduate medical education work-hour rules in practice. Annals of Internal Medicine, 143, 82–83.Google Scholar
  56. Schittek Janda, M., Tani Botticelli, A., Mattheos, N., Nebel, D., Wagner, A., Nattestad, A., et al. (2005). Computer-mediated instructional video: A randomised controlled trial comparing a sequential and a segmented instructional video in surgical hand wash. European Journal of Dental Education, 9(2), 53–58.CrossRefGoogle Scholar
  57. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.CrossRefGoogle Scholar
  58. Spickard, A. III, Alrajeh, N., Cordray, D., & Gigante, J. (2002). Learning about screening using an online or live lecture: Does it matter? Journal of General Internal Medicine, 17, 540–545.CrossRefGoogle Scholar
  59. Spickard, A. III., Smithers, J., Cordray, D., Gigante, J., & Wofford, J. L. (2004). A randomised trial of an online lecture with and without audio. Medical Education, 38, 787–790.CrossRefGoogle Scholar
  60. Stroup, D. F., Berlin, J. A., Morton, S. C., Olkin, I., Williamson, G. D., Rennie, D., et al. (2000). Meta-analysis of observational studies in epidemiology: A proposal for reporting. JAMA, 283, 2008–2012.CrossRefGoogle Scholar
  61. Tallent-Runnels, M. K., Thomas, J. A., Lan, W. Y., Cooper, S., Ahern, T. C., Shaw, S. M., et al. (2006). Teaching courses online: A review of the research. Review of Educational Research, 76(1), 93–135.CrossRefGoogle Scholar
  62. Tunuguntla, R., Rodriguez, O., Ruiz, J. G., Qadri, S. S., Mintzer, M. J., Van Zuilen, M. H., et al. (2008). Computer-based animations and static graphics as medical student aids in learning home safety assessment: A randomized controlled trial. Medical Teacher, 30, 815–817.CrossRefGoogle Scholar
  63. Wells, G. A., Shea, B., O’Connell, D., Peterson, J., Welch, V., Losos, M., et al. (2007). The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Retrieved March 20, 2010, from
  64. Wutoh, R., Boren, S. A., & Balas, E. A. (2004). eLearning: A review of internet-based continuing medical education. Journal of Continuing Education in the Health Professions, 24, 20–30.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2010

Authors and Affiliations

  • David A. Cook
    • 1
  • Anthony J. Levinson
    • 2
  • Sarah Garside
    • 2
  1. 1.Division of General Internal Medicine and Office of Education ResearchMayo Clinic College of MedicineRochesterUSA
  2. 2.Division of e-Learning InnovationMcMaster UniversityHamiltonCanada

Personalised recommendations