The internal validity of arthroscopic simulators and their effectiveness in arthroscopic education

  • Jesse Alan Slade Shantz
  • Jeff R. S. Leiter
  • Tania Gottschalk
  • Peter Benjamin MacDonald
Sports Medicine

Abstract

Purpose

The purpose of this systematic review was to identify standard procedures for the validation of arthroscopic simulators and determine whether simulators improve the surgical skills of users.

Methods

Arthroscopic simulator validation studies and randomized trials assessing the effectiveness of arthroscopic simulators in education were identified from online databases, as well as, grey literature and reference lists. Only validation studies and randomized trials were included for review. Study heterogeneity was calculated and where appropriate, study results were combined employing a random effects model.

Results

Four hundred and thirteen studies were reviewed. Thirteen studies met the inclusion criteria assessing the construct validity of simulators. A pooled analysis of internal validation studies determined that simulators could discriminate between novice and experts, but not between novice and intermediate trainees on time of completion of a simulated task. Only one study assessed the utility of a knee simulator in training arthroscopic skills directly and demonstrated that the skill level of simulator-trained residents was greater than non-simulator-trained residents.

Conclusions

Excessive heterogeneity exists in the literature to determine the internal and transfer validity of arthroscopic simulators currently available. Evidence suggests that simulators can discriminate between novice and expert users, but discrimination between novice and intermediate trainees in surgical education should be paramount. International standards for the assessment of arthroscopic simulator validity should be developed to increase the use and effectiveness of simulators in orthopedic surgery.

Level of evidence

Diagnostic study, Level III.

Keywords

Arthroscopic simulators Virtual reality Validation Surgical education 

Supplementary material

167_2012_2228_MOESM1_ESM.docx (125 kb)
Supplementary material 1 (DOCX 124 kb)

References

  1. 1.
    Ahlberg G, Enochsson L, Gallagher AG, Hedman L, Hogman C, McClusky DA III, Ramel S, Smith CD, Arvidsson D (2007) Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg 193(6):797–804PubMedCrossRefGoogle Scholar
  2. 2.
    Atesok K, Mabrey JD, Jazrawi LM, Egol KA (2012) Surgical simulation in orthopaedic skills training. J Am Acad Orthop Surg 20(7):410–422PubMedCrossRefGoogle Scholar
  3. 3.
    Babineau TJ, Becker J, Gibbons G, Sentovich S, Hess D, Robertson S, Stone M (2004) The “cost” of operative training for surgical residents. Arch Surg 139(4):366–369PubMedCrossRefGoogle Scholar
  4. 4.
    Barnes RW (1987) Surgical handicraft: teaching and learning surgical skills. Am J Surg 153(5):422–427PubMedCrossRefGoogle Scholar
  5. 5.
    Berwick DM, Leape LL (1999) Reducing errors in medicine. BMJ 319(7203):136–137PubMedCrossRefGoogle Scholar
  6. 6.
    Bridges M, Diamond DL (1999) The financial impact of teaching surgical residents in the operating room. Am J Surg 177(1):28–32PubMedCrossRefGoogle Scholar
  7. 7.
    Cannon WD, Eckhoff DG, Garrett WE Jr, Hunter RE, Sweeney HJ (2006) Report of a group developing a virtual reality simulator for arthroscopic surgery of the knee joint. Clin Orthop Relat Res 442:21–29PubMedCrossRefGoogle Scholar
  8. 8.
    Ceponis PJ, Chan D, Boorman RS, Hutchison C, Mohtadi NG (2007) A randomized pilot validation of educational measures in teaching shoulder arthroscopy to surgical residents. Can J Surg 50(5):387–393PubMedCentralPubMedGoogle Scholar
  9. 9.
    Deziel DJ, Millikan KW, Economou SG, Doolas A, Ko ST, Airan MC (1993) Complications of laparoscopic cholecystectomy: a national survey of 4,292 hospitals and an analysis of 77,604 cases. Am J Surg 165(1):9–14PubMedCrossRefGoogle Scholar
  10. 10.
    Farnworth LR, Lemay DE, Wooldridge T, Mabrey JD, Blaschak MJ, DeCoster TA, Wascher DC, Schenck RC Jr (2001) A comparison of operative times in arthroscopic ACL reconstruction between orthopaedic faculty and residents: the financial impact of orthopaedic surgical training in the operating room. Iowa Orthop J 21:31–35PubMedCentralPubMedGoogle Scholar
  11. 11.
    Gomoll AH, O’Toole RV, Czarnecki J, Warner JJ (2007) Surgical experience correlates with performance on a virtual reality simulator for shoulder arthroscopy. Am J Sports Med 35(6):883–888PubMedCrossRefGoogle Scholar
  12. 12.
    Gomoll AH, Pappas G, Forsythe B, Warner JJ (2008) Individual skill progression on a virtual reality simulator for shoulder arthroscopy: a 3-year follow-up study. Am J Sports Med 36(6):1139–1142PubMedCrossRefGoogle Scholar
  13. 13.
    Hariri S, Rawn C, Srivastava S, Youngblood P, Ladd A (2004) Evaluation of a surgical simulator for learning clinical anatomy. Med Educ 38(8):896–902PubMedCrossRefGoogle Scholar
  14. 14.
    Howells NR, Brinsden MD, Gill RS, Carr AJ, Rees JL (2008) Motion analysis: a validated method for showing skill levels in arthroscopy. Arthroscopy 24(3):335–342PubMedCrossRefGoogle Scholar
  15. 15.
    Howells NR, Gill HS, Carr AJ, Price AJ, Rees JL (2008) Transferring simulated arthroscopic skills to the operating theatre: a randomised blinded study. J Bone Joint Surg Br 90(4):494–499PubMedCrossRefGoogle Scholar
  16. 16.
    Insel A, Carofino B, Leger R, Arciero R, Mazzocca AD (2009) The development of an objective model to assess arthroscopic performance. J Bone Joint Surg Am 91(9):2287–2295PubMedCrossRefGoogle Scholar
  17. 17.
    Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ (1996) Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 17(1):1–12PubMedCrossRefGoogle Scholar
  18. 18.
    James PB, Hope SH-B, Mark WS (2005) Determining the efficacy of an immersive trainer for arthroscopy skills. In: MMVR13, Amsterdam, Netherlands, 1999. Medicine meets virtual reality. The magical next becomes the medical now. IOS press, pp 54–56Google Scholar
  19. 19.
    Kauvar DS, Braswell A, Brown BD, Harnisch M (2006) Influence of resident and attending surgeon seniority on operative performance in laparoscopic cholecystectomy. J Surg Res 132(2):159–163PubMedCrossRefGoogle Scholar
  20. 20.
    Mabrey JD, Reinig KD, Cannon WD (2010) Virtual reality in orthopaedics: is it a reality? Clin Orthop Relat Res 468:2586–2591PubMedCrossRefGoogle Scholar
  21. 21.
    McCarthy A, Harley P, Smallwood R Virtual arthroscopy training: do the virtual skills developed match the real skills required? In: MMVR 7, Amsterdam, Netherlands, 1999. Medicine meets virtual reality. Convergence of physical and informational technologies: Options for a New Era in Healthcare. IOS Press, pp 221–227Google Scholar
  22. 22.
    McCarthy AD, Moody L, Waterworth AR, Bickerstaff DR (2006) Passive haptics in a knee arthroscopy simulator: is it valid for core skills training? Clin Orthop Relat Res 442:13–20PubMedCrossRefGoogle Scholar
  23. 23.
    Megali G, Tonet O, Mazzoni M, Dario P, Vascellari A, Marcacci M (2002) A new tool for surgical training in knee arthroscopy. In: Proceedings of the part II, Berlin, Germany, 2002. Medical image computing and computer-assisted intervention—MICCAI 2002. 5th international conference. Springer, pp 170–177Google Scholar
  24. 24.
    Modi CS, Morris G, Mukherjee R (2010) Computer-simulation training for knee and shoulder arthroscopic surgery. Arthroscopy 26(6):832–840PubMedCrossRefGoogle Scholar
  25. 25.
    Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. BMJ 327(7422):1032–1037PubMedCrossRefGoogle Scholar
  26. 26.
    Pedowitz RA, Esch J, Snyder S (2002) Evaluation of a virtual reality simulator for arthroscopy skills development. Arthroscopy 18(6):E29PubMedCrossRefGoogle Scholar
  27. 27.
    Review Manager (RevMan) [Computer program] (2011) Version 5.1. Copenhagen: The Nordic Cochrane Centre, The cochrane collaborationGoogle Scholar
  28. 28.
    Schulz KF, Altman DG, Moher D (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 340:c332PubMedCentralPubMedCrossRefGoogle Scholar
  29. 29.
    Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236(4):458–463PubMedCrossRefGoogle Scholar
  30. 30.
    Sherman KP, Ward JW, Wills DP, Sherman VJ, Mohsen AM (2001) Surgical trainee assessment using a VE knee arthroscopy training system (VE-KATS): experimental results. Stud Health Technol Inform. 81:465–470PubMedGoogle Scholar
  31. 31.
    Sidhu RS, Park J, Brydges R, MacRae HM, Dubrowski A (2007) Laboratory-based vascular anastomosis training: a randomized controlled trial evaluating the effects of bench model fidelity and level of training on skill acquisition. J Vasc Surg 45(2):343–349PubMedCrossRefGoogle Scholar
  32. 32.
    Smith S, Wan A, Taffinder N, Read S, Emery R, Darzi A (1999) Early experience and validation work with Procedicus VA—the Prosolvia virtual reality shoulder arthroscopy trainer. Studies In Health Technology And Informatics 62:337–343PubMedGoogle Scholar
  33. 33.
    Srivastava S, Youngblood PL, Rawn C, Hariri S, Heinrichs WL, Ladd AL (2004) Initial evaluation of a shoulder arthroscopy simulator: establishing construct validity. J Shoulder Elbow Surg 13(2):196–205PubMedCrossRefGoogle Scholar
  34. 34.
    Strom P, Kjellin A, Hedman L, Johnson E, Wredmark T, Fellander-Tsai L (2003) Validation and learning in the Procedicus KSA virtual reality surgical simulator. Surg Endosc 17(2):227–231PubMedCrossRefGoogle Scholar
  35. 35.
    Strom P, Kjellin A, Hedman L, Wredmark T, Fellander-Tsai L (2004) Training in tasks with different visual-spatial components does not improve virtual arthroscopy performance. Surg Endosc 18(1):115–120PubMedCrossRefGoogle Scholar
  36. 36.
    Tashiro Y, Miura H, Nakanishi Y, Okazaki K, Iwamoto Y (2009) Evaluation of skills in arthroscopic training based on trajectory and force data. Clin Orthop Relat Res 467(2):546–552PubMedCrossRefGoogle Scholar
  37. 37.
    Tuijthof GJM, van Sterkenburg MN, Sierevelt IN, van Oldenrijk J, Van Dijk CN, Kerkhoffs GMMJ (2010) First validation of the PASSPORT training environment for arthroscopic skills. Knee Surg Sports Traumatol Arthrosc 18(2):218–224PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Jesse Alan Slade Shantz
    • 1
    • 5
    • 6
  • Jeff R. S. Leiter
    • 2
    • 3
    • 5
  • Tania Gottschalk
    • 4
  • Peter Benjamin MacDonald
    • 2
    • 5
  1. 1.Department of Orthopaedic SurgeryUniversity of CaliforniaSan FranciscoUSA
  2. 2.Department of Surgery, Section of OrthopedicsUniversity of ManitobaWinnipegCanada
  3. 3.Department of Human Anatomy and Cell ScienceUniversity of ManitobaWinnipegCanada
  4. 4.Neil John Maclean Health Sciences LibraryUniversity of ManitobaWinnipegCanada
  5. 5.Pan Am ClinicWinnipegCanada
  6. 6.San Francisco General HospitalOrthopaedic Trauma InstituteSan FranciscoUSA

Personalised recommendations