Patient Outcomes in Simulation-Based Medical Education: A Systematic Review
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
Evaluating the patient impact of health professions education is a societal priority with many challenges. Researchers would benefit from a summary of topics studied and potential methodological problems. We sought to summarize key information on patient outcomes identified in a comprehensive systematic review of simulation-based instruction.
Systematic search of MEDLINE, EMBASE, CINAHL, PsychINFO, Scopus, key journals, and bibliographies of previous reviews through May 2011.
Original research in any language measuring the direct effects on patients of simulation-based instruction for health professionals, in comparison with no intervention or other instruction.
APPRAISAL and SYNTHESIS
Two reviewers independently abstracted information on learners, topics, study quality including unit of analysis, and validity evidence. We pooled outcomes using random effects.
From 10,903 articles screened, we identified 50 studies reporting patient outcomes for at least 3,221 trainees and 16,742 patients. Clinical topics included airway management (14 studies), gastrointestinal endoscopy (12), and central venous catheter insertion (8). There were 31 studies involving postgraduate physicians and seven studies each involving practicing physicians, nurses, and emergency medicine technicians. Fourteen studies (28 %) used an appropriate unit of analysis. Measurement validity was supported in seven studies reporting content evidence, three reporting internal structure, and three reporting relations with other variables. The pooled Hedges’ g effect size for 33 comparisons with no intervention was 0.47 (95 % confidence interval [CI], 0.31–0.63); and for nine comparisons with non-simulation instruction, it was 0.36 (95 % CI, −0.06 to 0.78).
Focused field in education; high inconsistency (I2 > 50 % in most analyses).
Simulation-based education was associated with small-moderate patient benefits in comparison with no intervention and non-simulation instruction, although the latter did not reach statistical significance. Unit of analysis errors were common, and validity evidence was infrequently reported.
- Chen FM, Bauchner H, Burstin H. A call for outcomes research in medical education. Acad Med. 2004;79:955–60. CrossRef
- Dauphinee WD. Educators must consider patient outcomes when assessing the impact of clinical training. Med Educ. 2012;46:13–20. CrossRef
- Shea JA. Mind the gap: some reasons why medical education research is different from health services research. Med Educ. 2001;35:319–20. CrossRef
- McGaghie WC. Medical education research as translational science. Sci Transl Med. 2010;2:19cm18. CrossRef
- Prystowsky JB, Bordage G. An outcomes research perspective on medical education: the predominance of trainee assessment and satisfaction. Med Educ. 2001;35:331–6. CrossRef
- Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298:1002–9. CrossRef
- Baernstein A, Liss HK, Carney PA, Elmore JG. Trends in study methods used in undergraduate medical education research, 1969–2007. JAMA. 2007;298:1038–45. CrossRef
- Reed DA, Beckman TJ, Wright SM, Levine RB, Kern DE, Cook DA. Predictive validity evidence for medical education research study quality instrument scores: quality of submissions to JGIM’s medical education special issue. J Gen Intern Med. 2008;23:903–7. CrossRef
- Cook DA, Levinson AJ, Garside S. Method and reporting quality in health professions education research: a systematic review. Med Educ. 2011;45:227–38. CrossRef
- Issenberg SB, McGaghie WC, Hart IR, Mayer JW, Felner JM, Petrusa ER, et al. Simulation technology for health care professional skills training and assessment. JAMA. 1999;282:861–6. CrossRef
- Gaba DM. The future vision of simulation in healthcare. Simul Healthc. 2007;2:126–35. CrossRef
- McGaghie WC, Issenberg SB, Petrusa ER, Scalese RJ. A critical review of simulation-based medical education research: 2003–2009. Med Educ. 2010;44:50–63. CrossRef
- Cook DA, Hatala R, Brydges R, Zendejas B, Szostek JH, Wang AT, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011;306:978–88. CrossRef
- Cook DA, Brydges R, Hamstra S, Zendejas B, Szostek JH, Wang AT, et al. Comparative Effectiveness of Technology-Enhanced Simulation vs. Other Instructional Methods: A Systematic Review and Meta-Analysis. Simul Healthc. 2012;7:308–20.
- Cook DA, Hamstra S, Brydges R, Zendejas B, Szostek JH, Wang AT, et al. Comparative Effectiveness of Instructional Design Features in Simulation-based Education: Systematic Review and Meta-analysis. Med Teach. 2012; Online early (doi:10.3109/0142159X.2012.714886).
- Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ. 2003;37:830–7. CrossRef
- Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119:166.e7–16. CrossRef
- Beckman TJ, Cook DA, Mandrekar JN. What is the validity evidence for assessments of clinical teaching? J Gen Intern Med. 2005;20:1159–64. CrossRef
- Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009;302:1316–26. CrossRef
- Ratanawongsa N, Thomas PA, Marinopoulos SS, Dorman T, Wilson LM, Ashar BH, et al. The reported validity and reliability of methods for evaluating continuing medical education: a systematic review. Acad Med. 2008;83:274–83. CrossRef
- Reed DA, Fletcher KE, Arora VM. Systematic review: association of shift length, protected sleep time, and night float with patient care, residents’ health, and education. Ann Intern Med. 2010;153:829–42. CrossRef
- Kalet AL, Gillespie CC, Schwartz MD, Holmboe ES, Ark TK, Jay M, et al. New measures to establish the evidence base for medical education: identifying educationally sensitive patient outcomes. Acad Med. 2010;85:844–51. CrossRef
- Bland JM, Kerry SM. Trials randomised in clusters (statistics notes). BMJ. 1997;315:600. CrossRef
- Donner A. An empirical study of cluster randomization. Int J Epidemiol. 1982;11:283–6. CrossRef
- Whiting-O'Keefe QE, Henke C, Simborg DW. Choosing the correct unit of analysis in medical care experiments. Med Care. 1984;22:1101–14. CrossRef
- Divine GW, Brown JT, Frazier LM. The unit of analysis error in studies about physicians’ patient care behavior. J Gen Intern Med. 1992;7:623–9. CrossRef
- Thomas RE, Ramsay CR, McAuley L, Grimshaw JM. Unit of analysis errors should be clarified in meta-analyses. BMJ. 2003;326:397. CrossRef
- Calhoun AW, Guyatt GH, Cabana MD, Lu D, Turner DA, Valentine S, et al. Addressing the unit of analysis in medical care studies: a systematic review. Med Care. 2008;46:635–43. CrossRef
- Eldridge S, Ashby D, Bennett C, Wakelin M, Feder G. Internal and external validity of cluster randomised trials: systematic review of recent trials. BMJ. 2008;336:876–80. CrossRef
- Kerry SM, Bland JM. Analysis of a trial randomised in clusters (Statistics notes). BMJ. 1998;316:54. CrossRef
- Campbell MK, Elbourne DR, Altman DG. CONSORT statement: extension to cluster randomised trials. BMJ. 2004;328:702–8. CrossRef
- Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151:264–9. CrossRef
- Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: a meta-analysis. JAMA. 2008;300:1181–96. CrossRef
- Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. 2005;27:10–28. CrossRef
- Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327:557–60. CrossRef
- Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum; 1988.
- Swanson ER, Fosnocht DE. Effect of an airway education program on prehospital intubation. Air Med J. 2002;21:28–31.
- Di Giulio E, Fregonese D, Casetti T, Cestari R, Chilovi F, D'Ambra G, et al. Training with a computer-based simulator achieves basic manual skills required for upper endoscopy: a randomized controlled trial. Gastrointest Endosc. 2004;60:196–200. CrossRef
- Sedlack RE, Kolars JC. Computer simulator training enhances the competency of gastroenterology fellows at colonoscopy: results of a pilot study. Am J Gastroenterol. 2004;99:33–7. CrossRef
- Sedlack RE, Kolars JC, Alexander JA. Computer simulation training enhances patient comfort during endoscopy. Clin Gastroenterol Hepatol. 2004;2:348–52. CrossRef
- Ahlberg G, Hultcrantz R, Jaramillo E, Lindblom A, Arvidsson D. Virtual reality colonoscopy simulation: a compulsory practice for the future colonoscopist? Endoscopy. 2005;37:1198–204. CrossRef
- Hochberger J, Matthes K, Maiss J, Koebnick C, Hahn EG, Cohen J. Training with the compactEASIE biologic endoscopy simulator significantly improves hemostatic technical skill of gastroenterology fellows: a randomized controlled comparison with clinical endoscopy training alone. Gastrointest Endosc. 2005;61:204–15. CrossRef
- Cohen J, Cohen SA, Vora KC, Xue X, Burdick JS, Bank S, et al. Multicenter, randomized, controlled trial of virtual-reality simulator training in acquisition of competency in colonoscopy. Gastrointest Endosc. 2006;64:361–8. CrossRef
- Thomson M, Heuschkel R, Donaldson N, Murch S, Hinds R. Acquisition of competence in paediatric ileocolonoscopy with virtual endoscopy training. J Pediatr Gastroenterol Nutr. 2006;43:699–701. CrossRef
- Ahlberg G, Enochsson L, Gallagher AG, Hedman L, Hogman C, McClusky DA III, et al. Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg. 2007;193:797–804. CrossRef
- Park J, MacRae H, Musselman LJ, Rossos P, Hamstra SJ, Wolman S, et al. Randomized controlled trial of virtual reality simulator training: transfer to live patients. Am J Surg. 2007;194:205–11. CrossRef
- Draycott TJ, Crofts JF, Ash JP, Wilson LV, Yard E, Sibanda T, et al. Improving neonatal outcome through practical shoulder dystocia training. Obstet Gynecol. 2008;112:14–20. CrossRef
- Wayne DB, Didwania A, Feinglass J, Fudala MJ, Barsuk JH, McGaghie WC. Simulation-based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case-control study. Chest. 2008;133:56–61. CrossRef
- Yi SY, Ryu KH, Na YJ, Woo HS, Ahn W, Kim WS, et al. Improvement of colonoscopy skills through simulation-based training. Stud Health Technol Inform. 2008;132:565–7.
- Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Use of simulation-based education to reduce catheter-related bloodstream infections. Arch Intern Med. 2009;169:1420–3. CrossRef
- Barsuk JH, McGaghie WC, Cohen ER, Balachandran JS, Wayne DB. Use of simulation-based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit. J Hosp Med. 2009;4:397–403. CrossRef
- Barsuk JH, McGaghie WC, Cohen ER, O'Leary KJ, Wayne DB. Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit. Crit Care Med. 2009;37:2697–701. CrossRef
- Britt RC, Novosel TJ, Britt LD, Sullivan M. The impact of central line simulation before the ICU experience. Am J Surg. 2009;197:533–6. CrossRef
- Duncan DR, Morgenthaler TI, Ryu JH, Daniels CE. Reducing iatrogenic risk in thoracentesis: establishing best practice via experiential training in a zero-risk environment. Chest. 2009;135:1315–20. CrossRef
- Gaies MG, Morris SA, Hafler JP, Graham DA, Capraro AJ, Zhou J, et al. Reforming procedural skills training for pediatric residents: a randomized, interventional trial. Pediatrics. 2009;124:610–9. CrossRef
- Lubin J, Carter R. The feasibility of daily mannequin practice to improve intubation success. Air Med J. 2009;28:195–7. CrossRef
- Siassakos D, Hasafa Z, Sibanda T, Fox R, Donald F, Winter C, et al. Retrospective cohort study of diagnosis—delivery interval with umbilical cord prolapse: the effect of team training. BJOG. 2009;116:1089–96. CrossRef
- Nishisaki A, Donoghue AJ, Colborn S, Watson C, Meyer A, Brown CA III, et al. Effect of just-in-time simulation training on tracheal intubation procedure safety in the pediatric intensive care unit. Anesthesiology. 2010;113:214–23. CrossRef
- Smith CC, Huang GC, Newman LR, Clardy PF, Feller-Kopman D, Cho M, et al. Simulation training and its effect on long-term resident performance in central venous catheterization. Simul Healthc. 2010;5:146–51. CrossRef
- Hosking EJ. Does practising intubation on a manikin improve both understanding and clinical performance of the task by medical students? Anaesthesia Points West. 1998;31:25–8.
- Weidman EK, Bell G, Walsh D, Small S, Edelson DP. Assessing the impact of immersive simulation on clinical performance during actual in-hospital cardiac arrest with CPR-sensing technology: a randomized feasibility study. Resuscitation. 2010;81:1556–61. CrossRef
- Gómez LM, Calderón M, Sáenz X, Reyes G, Moreno MA, Ramírez LJ, et al. Effect and benefit of clinical simulation in the development of psychomotor competencies in anesthesia: a random double-blind clinical trial [Spanish]. Revista Colombiana de Anestesiologia. 2008;36:93–107. CrossRef
- Andreatta P, Saxton E, Thompson M, Annich G. Simulation-based mock codes significantly correlate with improved pediatric patient cardiopulmonary arrest survival rates. Pediatr Crit Care Med. 2010;12:33–8. CrossRef
- Evans LV, Dodge KL, Shah TD, Kaplan LJ, Siegel MD, Moore CL, et al. Simulation training in central venous catheter insertion: improved performance in clinical practice. Acad Med. 2010;85:1462–9. CrossRef
- Capella J, Smith S, Philp A, Putnam T, Gilbert C, Fry W, et al. Teamwork training improves the clinical care of trauma patients. J Surg Educ. 2010;67:439–43. CrossRef
- Ferlitsch A, Schoefl R, Puespoek A, Miehsler W, Schoeniger-Hekele M, Hofer H, et al. Effect of virtual endoscopy simulator training on performance of upper gastrointestinal endoscopy in patients: a randomized controlled trial. Endoscopy. 2010;42:1049–56. CrossRef
- Kallstrom R, Hjertberg H, Svanvik J. Impact of virtual reality-simulated training on urology residents’ performance of transurethral resection of the prostate. J Endourol. 2010;24:1521–8. CrossRef
- Khouli H, Jahnes K, Shapiro J, Rose K, Mathew J, Gohil A, et al. Performance of medical residents in sterile techniques during central vein catheterization: randomized trial of efficacy of simulation-based training. Chest. 2011;139:80–7. CrossRef
- Tongprasert F, Wanapirak C, Sirichotiyakul S, Piyamongkol W, Tongsong T. Training in cordocentesis: the first 50 case experience with and without a cordocentesis training model. Prenat Diagn. 2010;30:467–70.
- Zamora Z, Clark MJ, Winslow B, Schatzschneider M, Burkard J. Orthotopic neobladder irrigation: competency assessment through simulation. Urol Nurs. 2011;31:113–20.
- Lefcoe DL, Green ML. Simulated models: a mode for instruction in root planing procedures. Educ Dir Dent Aux. 1979;3:20–4.
- Ovassapian A, Yelich SJ, Dykes MH, Golman ME. Learning fibreoptic intubation: use of simulators v. traditional teaching. Br J Anaesth. 1988;61:217–20. CrossRef
- Limpaphayom K, Ajello C, Reinprayoon D, Lumbiganon P, Graffikin L. The effectiveness of model-based training in accelerating IUD skill acquisition. A study of midwives in Thailand. Br J Fam Plann. 1997;23:58–61.
- Gerson LB, Van Dam J. A prospective randomized trial comparing a virtual reality simulator to bedside teaching for training in sigmoidoscopy. Endoscopy. 2003;35:569–75. CrossRef
- Sotto JAR, Ayuste EC Jr, Bowyer MW, Almonte JR, Dofitas RB, Lapitan MCM, et al. Exporting simulation technology to the Philippines: a comparative study of traditional versus simulation methods for teaching intravenous cannulation. Stud Health Technol Inform. 2009;142:346–51.
- Haycock A, Koch AD, Familiari P, van Delft F, Dekker E, Petruzziello L, et al. Training and transfer of colonoscopy skills: a multinational, randomized, blinded, controlled trial of simulator versus bedside training. Gastrointest Endosc. 2010;71:298–307. CrossRef
- Naik VN, Matsumoto ED, Houston PL, Hamstra SJ, Yeung RY-M, Mallon JS, et al. Fiberoptic orotracheal intubation on anesthetized patients: do manipulation skills learned on a simple model transfer into the operating room? Anesthesiology. 2001;95:343–8. CrossRef
- Velmahos GC, Toutouzas KG, Sillin LF, Chan L, Clark RE, Theodorou D, et al. Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory. Am J Surg. 2004;187:114–9. CrossRef
- Campos JH, Hallam EA, Ueda K. Training in placement of the left-sided double-lumen tube among non-thoracic anaesthesiologists: intubation model simulator versus computer-based digital video disc, a randomised controlled trial. Eur J Anaesthesiol. 2011;28:169–74. CrossRef
- Stewart RD, Paris PM, Pelton GH, Garretson D. Effect of varied training techniques on field endotracheal intubation success rates. Ann Emerg Med. 1984;13:1032–6. CrossRef
- Stratton SJ, Kane G, Gunter CS, Wheeler NC, Ableson-Ward C, Reich E, et al. Prospective study of manikin-only versus manikin and human subject endotracheal intubation training of paramedics. Ann Emerg Med. 1991;20:1314–8. CrossRef
- Trooskin SZ, Rabinowitz S, Eldridge C, McGowan DE, Flancbaum L. Teaching endotracheal intubation using animals and cadavers. Prehospital Disaster Med. 1992;7:179–82.
- Chang KK, Chung JW, Wong TK. Learning intravenous cannulation: a comparison of the conventional method and the CathSim intravenous training system. J Clin Nurs. 2002;11:73–8. CrossRef
- Rumball C, Macdonald D, Barber P, Wong H, Smecher C. Endotracheal intubation and esophageal tracheal Combitube insertion by regular ambulance attendants: a comparative trial. Prehosp Emerg Care. 2004;8:15–22. CrossRef
- Davis DP, Buono C, Ford J, Paulson L, Koenig W, Carrison D. The effectiveness of a novel, algorithm-based difficult airway curriculum for air medical crews using human patient simulators. Prehosp Emerg Care. 2007;11:72–9. CrossRef
- Chandra DB, Savoldelli GL, Joo HS, Weiss ID, Naik VN. Fiberoptic oral intubation: the effect of model fidelity on training for transfer to patient care. Anesthesiology. 2008;109:1007–13. CrossRef
- Haan CK, Edwards FH, Poole B, Godley M, Genuardi FJ, Zenni EA. A model to begin to use clinical outcomes in medical education. Acad Med. 2008;83:574–80. CrossRef
- Swing SR, Schneider S, Bizovi K, Chapman D, Graff LG, Hobgood C, et al. Using patient care quality measures to assess educational outcomes. Acad Emerg Med. 2007;14:463–73.
- Cook DA. If you teach them, they will learn: why medical education needs comparative effectiveness research. Adv Health Sci Educ Theory Pract. 2012;17:305–10. CrossRef
- Cook DA, West CP. Reconsidering the focus on “outcomes research” in medical education: a cautionary note. Acad Med. 2013; Accepted (in press).
- Patient Outcomes in Simulation-Based Medical Education: A Systematic Review
Journal of General Internal Medicine
Volume 28, Issue 8 , pp 1078-1089
- Cover Date
- Print ISSN
- Online ISSN
- Springer US
- Additional Links
- medical education
- outcomes research
- educational technology
- program evaluation
- quantitative research methods
- Industry Sectors
- Author Affiliations
- 1. Department of Surgery, Mayo Clinic College of Medicine, Rochester, MN, USA
- 2. Department of Medicine, University of Toronto, Toronto, ON, Canada
- 3. Division of General Internal Medicine, Mayo Clinic College of Medicine, Mayo 17, 200 First Street SW, Rochester, MN, 55905, USA
- 4. Office of Education Research, Mayo Medical School, Rochester, MN, USA