Consensus guidelines for validation of virtual reality surgical simulators

  • F. J. Carter
  • M. P. Schijven
  • R. Aggarwal
  • T. Grantcharov
  • N. K. Francis
  • G. B. Hanna
  • J. J. Jakimowicz


The Work Group for Evaluation and Implementation of Simulators and Skills Training Programmes is a newly formed sub-group of the European Association of Endoscopic Surgeons (EAES). This work group undertook a review of validation evidence for surgical simulators and the resulting consensus is presented in this article. Using clinical guidelines criteria, the evidence for validation for six different simulators was rated and subsequently translated to a level of recommendation for each system. The simulators could be divided into two basic types; systems for laparoscopic general surgery and flexible gastrointestinal endoscopy. Selection of simulators for inclusion in this consensus was based on their availability and relatively widespread usage as of July 2004. Whilst level 2 recommendations were achieved for a few systems, it was clear that there was an overall lack of published validation studies with rigorous experimental methodology. Since the consensus meeting, there have been a number of new articles, system upgrades and new devices available. The work group intends to update these consensus guidelines on a regular basis, with the resulting article available on the EAES website ( ).


Virtual reality Surgical training Validation Simulation Consensus guidelines 


  1. 1.
    Schijven M, Jakimowicz J, Broeders I, Tseng L (2005) The Eindhoven laparoscopic cholecystectomy training course: improving operating room performance using virtual reality training. Results from the first EAES-accredited virtual reality training curriculum. Surg Endosc, Accessed online 18th October 2005 at Google Scholar
  2. 2.
    Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J (2004) Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg 91: 146–150CrossRefPubMedGoogle Scholar
  3. 3.
    Seymour NE, Gallagher AG, Roman SA, O’Brian MK, Bansal VK (2002) Virtual reality improves operating room performance: results of a randomized, double-blind study. Ann Surg 236: 458–463CrossRefPubMedGoogle Scholar
  4. 4.
    Hyltander A, Liljegren E, Rhodin PH, Lonroth H (2002) The transfer of basic skills learned in a laparoscopic simulator to the operating room. Surg Endosc 16: 1324–1328CrossRefPubMedGoogle Scholar
  5. 5.
    Hamilton EC, Scott DJ, Fleming RV, Rege RV, Laycock R (2002) Comparison of video trainer and virtual reality training systems on acquisition of laparoscopic skill. Surg Endosc 16: 406–411CrossRefPubMedGoogle Scholar
  6. 6.
    Ahlberg G, Heikkinen T, Iselius L, Leijonmarck CE, Rutqvist J (2002) Does training in a virtual reality simulator improve surgical performance? Surg Endosc 16: 126–129CrossRefPubMedGoogle Scholar
  7. 7.
    Fry H, Ketteridge S, Marshall S (1999) Understanding student learning. In: Fry H, Ketteridge S, Marshall S (eds) A handbook for teaching and learning in higher education. Kogan Page, LondonGoogle Scholar
  8. 8.
    Kolb DA (1984) Experiential learning. Prentice-Hall, Englewood Cliffs, New JerseyGoogle Scholar
  9. 9.
    Murphy KR, Davidshofer CO (1998) Psychological testing: principle and applications. 4th ed. Prentice-Hall, Upper Saddle River, New JerseyGoogle Scholar
  10. 10.
    Ayodeji ID, Schijven MP, Jakimowicz JJ, Bonjer HJ (2004) Face validation of the LapMentor laparoscopy trainer (Abstract). Fall meeting of the Dutch Society of Surgery, Ede, The NetherlandsGoogle Scholar
  11. 11.
    Prystowski JB, Regehr G, Rogers DA, Loan JP, Hiemenz LL, Smith KM (1999) A virtual reality module for intravenous catheter placement. Am J Surg 177: 171–175Google Scholar
  12. 12.
    Schijven M, Jakimowicz J (2003) Construct validity: experts and novices performing on the Xitact LS500 laparoscopy simulator. Surg Endosc 17: 803–810PubMedGoogle Scholar
  13. 13.
    Schijven M (2005) Virtual reality simulation for laparoscopic cholecystectomy: the process of validation and implementation in the surgical curriculum outlined. Thesis, ISBN 90-9019048-1. University of Leiden, Leiden, the NetherlandsGoogle Scholar
  14. 14.
    Andrews EJ, Redmond HP (2004) A review of clinical guidelines. Br J Surg 91: 956–964CrossRefPubMedGoogle Scholar
  15. 15.
    Neugebauer E, Sauerland S (2000) Recommendations for evidence-based endoscopic surgery. Springer-Verlag, Paris, FranceGoogle Scholar
  16. 16.
    Fink A, Kosecoff J, Chassin M, Brook RH (1984) Consensus methods: characteristics and guidelines for use. Am J Public Health 74: 979–983PubMedCrossRefGoogle Scholar
  17. 17.
    (NABON), N.B.O.N. and N.B.C.C.t. Netherlands (1992) Richtlijn behandeling van het mammacarcinoom (Guidelines for treatment of breast cancer). van Zuiden communications, Alphen aan de Rijn.Google Scholar
  18. 18.
    Datta VK, Mandalia M, Mackay SD, Darzi AW (2001) Evaluation and validation of a virtual reality–based flexible sigmoidoscopy trainer (Abstract). Gut 48((Suppl 1)): A97–A98Google Scholar
  19. 19.
    Garuda S, Keshavarzian A, Losurdo J, Brown MD (2002) Efficacy of a computer-assisted endoscopic simulator in training residents in flexible endoscopy (Poster). Proceedings of ACG 2002, Seattle, WAGoogle Scholar
  20. 20.
    Carter FJ, Steele RJC, Kennovin MF, Cuschieri A (2003) Validation of a virtual reality colonoscopy simulator using subjects of differing experience (Abstract). Proc. 1st European Endoscopic Surgical Week and 11th EAES Congress, 15th–18th June 2003, Glasgow, UKGoogle Scholar
  21. 21.
    Sedlack RE, Kolars JC (2003) Validation of a computer-based colonoscopy simulator. Gastrointest Endosc 57: 214–218CrossRefPubMedGoogle Scholar
  22. 22.
    Mahmood T, Darzi A (2003) A study to validate the colonoscopy simulator: it is usefully discriminatory for more than one measurable outcome. Surg Endosc 17: 1583–1589CrossRefPubMedGoogle Scholar
  23. 23.
    Thomas-Gibson S, Vance ME, Saunders BP (2003) Can a colonoscopy computer simulator differentiate between a novice and expert? (Abstract) Gut 52(Suppl 1): A73Google Scholar
  24. 24.
    Sedlack RE, Kolars JC (2004) Computer simulator training enhances the competency of gastroenterology fellows at colonoscopy: results of a pilot study. Am J Gastroenterol 99: 33–37CrossRefPubMedGoogle Scholar
  25. 25.
    Sedlack RE, Kolars JC (2003) Direct comparison of ERCP teaching models. Gastrointest Endosc 57: 886–890CrossRefPubMedGoogle Scholar
  26. 26.
    Aalbakken L, Adamsen A, Kruse A (2000) Performance of a colonoscopy simulator: experience from a hands-on endoscopy course. Endoscopy 32: 898–900Google Scholar
  27. 27.
    Grantcharov TP, Everbusch MD, Funch-Jensen P (2004) Teaching and testing surgical skills on a VR endoscopy simulator: learning curves and impact of psychomotor training on performance in simulated colonoscopy (Poster). SAGES meeting Denver, Colorado, March–April, 2004, P179Google Scholar
  28. 28.
    Yousfi MM, Darius Sorbi MD, Baron T, Fleischer DE (2002) Flexible sigmoidoscopy: assessing endoscopic skills using a computer-based simulator (Abstract). ACG meeting, Seattle, Washington, 2002Google Scholar
  29. 29.
    Ritter EM, McClusky DA III, Lederman AB, Gallagher AG, Smith CD (2003) Objective psychomotor skills assessment of experienced and novice flexible endoscopists with a virtual reality simulator. J Gastrointest Surg 7: 871–878CrossRefPubMedGoogle Scholar
  30. 30.
    Adamsen S, Funch-Jensen PM, Drewes AM, Rosenberg J, Grantcharov TP (2005) A comparative study of skills in virtual laparoscopy and endoscopy. Surg Endosc 19: 229–234CrossRefPubMedGoogle Scholar
  31. 31.
    Ferlitsch A, Glauninger P, Gupper A, Schillinger M, Haefner M, Gangi A, Schoefl R (2002) Evaluation of a virtual endoscopy simulator for training in gastrointestinal endoscopy. Endoscopy 34: 698–702PubMedGoogle Scholar
  32. 32.
    Moorthy K, Munz Y, Jiwanji M, Bahn S, Chang A, Darzi A (2004) Validity and reliability of a virtual reality upper gastrointestinal simulator and cross validation using structured assessment of individual performance with video playback. Surg. Endosc 28: 328–333Google Scholar
  33. 33.
    Enochson L, Isaksson B, Tour R, Kjellin A, Hedman L, Wredmark T, Tsai-Fellander L (2004) Visuospatial skills and computer game experience influence the performance of virtual endoscopy. J Gastrointest Surg 8: 874–880Google Scholar
  34. 34.
    Fanelli RD, Mainella MT, Justin RC, Gersin KS (2003) Initial experience using an endoscopic simulator to train residents in flexible endoscopy in a community medical center–based residency program. Flexible diagnostic and therapeutic endoscopy (Poster). Proceedings of the SAGES meeting, 12–15 March 2003, Los Angeles, California P233Google Scholar
  35. 35.
    Taffinder N, Russell RCG, McManus IC, Jansen J, Darzi A (1998) An objective assessment of surgeons’ psychomotor skills: validation of the MIST-VR laparoscopic simulator. Br J Surg 85(Supp 1): 75Google Scholar
  36. 36.
    Grantcharov TP, Rosenberg J, Pahle E, Funch-Jensen P (2001) Virtual reality computer simulation: an objective method for the evaluation of laparoscopic surgical skills. Surg Endosc 15: 242–244CrossRefPubMedGoogle Scholar
  37. 37.
    McNatt SS, Daniel Smith C (2001) A computer-based laparoscopic skills assessment device differentiates experienced from novice laparoscopic surgeons. Surg Endosc 15: 1085–1089CrossRefPubMedGoogle Scholar
  38. 38.
    Gallagher HJ, Allan JD, Tolley DA (2001) Spatial awareness in urologists: are they different?’ BJU Int 88: 666–670CrossRefPubMedGoogle Scholar
  39. 39.
    Gallagher AG, Richie K, McClure N, McGuigan J (2001) Objective psychomotor skills assessment of experienced, junior, and novice laparoscopists with virtual reality. World J Surg 25: 1478–1483PubMedGoogle Scholar
  40. 40.
    Gallagher AG, Satava RM (2002) Virtual reality as a metric for the assessment of laparoscopic psychomotor skills: learning curves and reliability measures. Surg Endosc 16: 1746–1752CrossRefPubMedGoogle Scholar
  41. 41.
    Grantcharov TP, Bardram L, Funch-Jensen P, Rosenberg J (2003) Learning curves and impact of previous operative experience on performance on a virtual reality simulator to test laparoscopic surgical skills. Am J Surg 185: 146–149CrossRefPubMedGoogle Scholar
  42. 42.
    Gallagher AG, Lederman AB, McGlade K, Satava RM, Smith CD (2004) Discriminative validity of the Minimally Invasive Surgical Trainer in Virtual Reality (MIST-VR) using criteria levels based on expert performance. Surg Endosc 18: 660–665CrossRefPubMedGoogle Scholar
  43. 43.
    Carter FJ, Francis NK, Tang B, Martindale JP, Cuschieri A (2003) Validation of a virtual reality simulator with error analysis of a simulated surgical procedure (Abstract). Surg. Endosc 17(Suppl 1): S61Google Scholar
  44. 44.
    Lonroth H, Olsson JH, Johnsson E, Pazooki D (2003) Can virtual reality training tools differentiate between different levels of expertise and experience? (Abstract) Surg. Endosc 17(Suppl 1): S63Google Scholar
  45. 45.
    Duffy AJ, Hogle NJ, McCarthy H, Lew JI, Egan A, Christos PJ, Fowler DL (2003). Construct validity for the lapsim laparoscopic surgical simulator (Abstract). Surg. Endosc 17(Suppl 1): S230Google Scholar
  46. 46.
    Broe D, Ridgway PF, Johnson S, Tierney C, Conlon KC (2004) Validation of a novel hybrid surgical simulator. Proc. 12th EAES Congress 9th–12th June 2004, Barcelona, SpainGoogle Scholar
  47. 47.
    Hance J, Aggarwal R, Undre S, Patel H, Darzi A (2004) Evaluation of a laparoscopic video trainer with built in measures of performance. Proc. 13th SLS meeting and EndoExpo 29th September–2nd October 2004, New YorkGoogle Scholar
  48. 48.
    McClusky DAM, Van Sickle K, Gallagher AG (2004) Relationship between motion analysis, time, accuracy, and errors during performance of a laparoscopic suturing task on an augmented reality simulator’. Proc. 12th EAES Congress 9th–12th June 2004, Barcelona, SpainGoogle Scholar

Copyright information

© Springer Science+Business Media, Inc. 2005

Authors and Affiliations

  • F. J. Carter
    • 1
  • M. P. Schijven
    • 2
  • R. Aggarwal
    • 3
  • T. Grantcharov
    • 4
  • N. K. Francis
    • 5
  • G. B. Hanna
    • 3
  • J. J. Jakimowicz
    • 6
  1. 1.The Cuschieri Skills CentreNinewells Hospital, University of DundeeDundee
  2. 2.Department of SurgeryIJsselland HospitalThe Netherlands
  3. 3.Department of Surgical Oncology and TechnologyImperial College LondonLondon
  4. 4.Glostrup HospitalCopenhagenDenmark
  5. 5.Ward 7, Directorate of SurgeryTUHT, Ninewells HospitalDundee
  6. 6.Department of SurgeryCatharina HospitalMichelangelolaanThe Netherlands

Personalised recommendations