Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure

  • 255 Accesses

  • 3 Citations



The use of robotic surgery for minimally invasive procedures has increased considerably over the last decade. Robotic surgery has potential advantages compared to laparoscopic surgery but also requires new skills. Using virtual reality (VR) simulation to facilitate the acquisition of these new skills could potentially benefit training of robotic surgical skills and also be a crucial step in developing a robotic surgical training curriculum. The study's objective was to establish validity evidence for a simulation-based test for procedural competency for the vaginal cuff closure procedure that can be used in a future simulation-based, mastery learning training curriculum.


Eleven novice gynaecological surgeons without prior robotic experience and 11 experienced gynaecological robotic surgeons (> 30 robotic procedures) were recruited. After familiarization with the VR simulator, participants completed the module ‘Guided Vaginal Cuff Closure’ six times. Validity evidence was investigated for 18 preselected simulator metrics. The internal consistency was assessed using Cronbach’s alpha and a composite score was calculated based on metrics with significant discriminative ability between the two groups. Finally, a pass/fail standard was established using the contrasting groups’ method.


The experienced surgeons significantly outperformed the novice surgeons on 6 of the 18 metrics. The internal consistency was 0.58 (Cronbach’s alpha). The experienced surgeons’ mean composite score for all six repetitions were significantly better than the novice surgeons’ (76.1 vs. 63.0, respectively, p < 0.001). A pass/fail standard of 75/100 was established. Four novice surgeons passed this standard (false positives) and three experienced surgeons failed (false negatives).


Our study has gathered validity evidence for a simulation-based test for procedural robotic surgical competency in the vaginal cuff closure procedure and established a credible pass/fail standard for future proficiency-based training.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4


  1. 1.

    Schreuder HW, Verheijen RH (2009) Robotic surgery. BJOG 116(2):198–213. https://doi.org/10.1111/J.1471-0528.2008.02038.X

  2. 2.

    Weinberg L, Rao S, Escobar P (2011) Robotic surgery in gynecology: an updated systematic review. Obstet Gynecol Int 2011:852061. https://doi.org/10.1155/2011/852061

  3. 3.

    Yates DR, Vaessen C, Roupret M (2011) From Leonardo to da Vinci: the history of robot-assisted surgery in urology. BJU Int 108(11):1708–1713. https://doi.org/10.1111/J.1464-410x.2011.10576.X (discussion 1714)

  4. 4.

    Maan ZN, Gibbins N, Al-Jabri T, D’souza AR (2012) The use of robotics in otolaryngology-head and neck surgery: a systematic review. Am J Otolaryngol 33(1):137–146. https://doi.org/10.1016/J.Amjoto.2011.04.003

  5. 5.

    Veronesi G (2013) Robotic surgery for the treatment of early-stage lung cancer. Curr Opin Oncol 25(2):107–114. https://doi.org/10.1097/Cco.0b013e32835daf4f

  6. 6.

    Baek SK, Carmichael JC, Pigazzi A (2013) Robotic surgery: colon and rectum. Cancer J 19(2):140–146. https://doi.org/10.1097/Ppo.0b013e31828ba0fd

  7. 7.

    Liu Z, Li X, Tian S, Zhu T, Yao Y, Tao Y (2017) Superiority of robotic surgery for cervical cancer in comparison with traditional approaches: a systematic review and meta-analysis. Int J Surg 40:145–154. https://doi.org/10.1016/J.Ijsu.2017.02.062

  8. 8.

    Xie W, Cao D, Yang J, Shen K, Zhao L (2016) Robot-assisted surgery versus conventional laparoscopic surgery for endometrial cancer: a systematic review and meta-analysis. J Cancer Res Clin Oncol 142(10):2173–2183. https://doi.org/10.1007/S00432-016-2180-X

  9. 9.

    Foote JR, Valea FA (2016) Robotic surgical training: where are we? Gynecol Oncol 143(1):179–183. https://doi.org/10.1016/J.Ygyno.2016.05.031

  10. 10.

    Bric J, Connolly M, Kastenmeier A, Goldblatt M, Gould J (2014) Proficiency training on a virtual reality robotic surgical skills curriculum. Surg Endosc 28(12):3343–3348. https://doi.org/10.1007/S00464-014-3624-5

  11. 11.

    Sandadi S, Gadzinski JA, Lee S, Chi DS, Sonoda Y, Jewell EL, Brown CL, Gardner GJ, Barakat RR, Leitao MM (2014) Fellowship learning curve associated with completing a robotic assisted total laparoscopic hysterectomy. Gynecol Oncol 132(1):102–106. https://doi.org/10.1016/J.Ygyno.2013.11.017

  12. 12.

    Seamon LG, Fowler JM, Richardson DL, Carlson MJ, Valmadre S, Phillips GS, Cohn DE (2009) A detailed analysis of the learning curve: robotic hysterectomy and pelvic-aortic lymphadenectomy for endometrial cancer. Gynecol Oncol 114(2):162–167. https://doi.org/10.1016/J.Ygyno.2009.04.017

  13. 13.

    Balasundaram I, Aggarwal R, Darzi A (2008) Short-phase training on a virtual reality simulator improves technical performance in tele-robotic surgery. Int J Med Robot 4(2):139–145. https://doi.org/10.1002/Rcs.181

  14. 14.

    Schreuder HW, Wolswijk R, Zweemer RP, Schijven MP, Verheijen RH (2012) Training and learning robotic surgery, time for a more structured approach: a systematic review. BJOG 119(2):137–149. https://doi.org/10.1111/J.1471-0528.2011.03139.X

  15. 15.

    Andersen SA, Konge L, Cayé-Thomasen P, Sørensen MS (2015) Learning curves of virtual mastoidectomy in distributed and massed practice. JAMA Otolaryngol 141(10):913–918. https://doi.org/10.1001/Jamaoto.2015.1563

  16. 16.

    Lendvay TS, Brand TC, White L, Kowalewski T, Jonnadula S, Mercer LD, Khorsand D, Andros J, Hannaford B, Satava R (2013) Virtual reality robotic surgery warm-up improves task performance in a dry laboratory environment: a prospective randomized controlled study. J Am Coll Surg 216(6):1181–1192. https://doi.org/10.1016/J.Jamcollsurg.2013.02.012

  17. 17.

    Larsen CR, Soerensen JL, Grantcharov TP, Dalsgaard T, Schouenborg L, Ottosen C, Schroeder TV, Ottesen BS (2009) Effect of virtual reality training on laparoscopic surgery: randomised controlled trial. BMJ 338:B1802. https://doi.org/10.1136/Bmj.B1802

  18. 18.

    Larsen CR, Oestergaard J, Ottosen C, Soerensen JL, (2012) The efficacy of virtual reality simulation training in laparoscopy: a systematic review of randomized trials. Acta Obstet Gynecol Scand 91(9):1015–1028. https://doi.org/10.1111/J.1600-0412.2012.01482.X

  19. 19.

    Culligan P, Gurshumov E, Lewis C, Priestley J, Komar J, Salamon C (2014) Predictive validity of a training protocol using a robotic surgery simulator. Female Pelvic Med Reconstr Surg 20(1):48–51. https://doi.org/10.1097/Spv.0000000000000045

  20. 20.

    Kiely DJ, Gotlieb WH, Lau S, Zeng X, Samouelian V, Ramanakumar AV, Zakrzewski H, Brin S, Fraser SA, Korsieporn P, Drudi L (2015) Virtual reality robotic surgery simulation curriculum to teach robotic suturing: a randomized controlled trial. J Robot Surg 9(3):179–186. https://doi.org/10.1007/S11701-015-0513-4

  21. 21.

    Hung AJ, Jayaratna IS, Teruya K, Desai MM, Gill IS, Goh AC (2013) Comparative assessment of three standardized robotic surgery training methods. BJU Int 112(6):864–871. https://doi.org/10.1111/Bju.12045

  22. 22.

    Kenney PA, Wszolek MF, Gould JJ, Libertino JA, Moinzadeh A (2009) Face, content, and construct validity of dV-trainer, a novel virtual reality simulator for robotic surgery. Urology 73(6):1288–1292. https://doi.org/10.1016/J.Urology.2008.12.044

  23. 23.

    Raza SJ, Froghi S, Chowriappa A, Ahmed K, Field E, Stegemann AP, Rehman S, Sharif M, Shi Y, Wilding GE, Kesavadas T (2014) Construct validation of the key components of fundamental skills of robotic surgery (FSRS) curriculum—a multi-institution prospective study. J Surg Educ 71(3):316–324. https://doi.org/10.1016/J.Jsurg.2013.10.006

  24. 24.

    Whittaker G, Aydin A, Raison N, Kum F, Challacombe B, Khan MS, Dasgupta P, Ahmed K (2016) Validation of the robotix mentor robotic surgery simulator. J Endourol 30(3):338–346. https://doi.org/10.1089/End.2015.0620

  25. 25.

    Simbionix (2017) http://simbionix.com/simulators/robotix-mentor/robotix-library-of-modules/. Accessed 21 Feb 2017

  26. 26.

    Fisher RA, Dasgupta P, Mottrie A, Volpe A, Khan MS, Challacombe B, Ahmed K (2015) An over-view of robot assisted surgery curricula and the status of their validation. Int J Surg 13:115–123. https://doi.org/10.1016/J.Ijsu.2014.11.033

  27. 27.

    Bric JD, Lumbard DC, Frelich MJ, Gould JC (2016) current state of virtual reality simulation in robotic surgery training: a review. Surg Endosc 30(6):2169–2178. https://doi.org/10.1007/S00464-015-4517-Y

  28. 28.

    Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R (2013) Mastery learning for health professionals using technology-enhanced simulation: a systematic review and meta-analysis. Acad Med 88(8):1178–1186. https://doi.org/10.1097/Acm.0b013e31829a365d

  29. 29.

    Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R (2014) What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract 19(2):233–250. https://doi.org/10.1007/S10459-013-9458-4

  30. 30.

    Ghaderi I, Manji F, Park YS, Juul D, Ott M, Harris I, Farrell T (2015) Technical skills assessment toolbox: a review using the unitary framework of validity. Ann Surg 261(2):251–262. https://doi.org/10.1097/Sla.0000000000000520

  31. 31.

    Larsen CR, Grantcharov T, Aggarwal R, Tully A, Sorensen JL, Dalsgaard T, Ottesen B (2006) objective assessment of gynecologic laparoscopic skills using the LapSimGyn virtual reality simulator. Surg Endosc 20(9):1460–1466. https://doi.org/10.1007/S00464-005-0745-X

  32. 32.

    Tulandi T, Einarsson J (2014) The use of barbed suture for laparoscopic hysterectomy and myomectomy: a systematic review and meta-analysis. J Minim Invasive Gynecol 21(2):210–216. https://doi.org/10.1016/J.Jmig.2013.09.014

  33. 33.

    American Educational Research Association APA, National Council on Measurement in Education, Joint Committee on Standards for Educational and Psychological Testing (2014) Standards for educational and psychological testing. American Educational Research Association, Washington, DC

  34. 34.

    Bloch R, Norman G (2012) Generalizability theory for the perplexed: a practical introduction and guide: AMEE Guide No. 68. Med Teach 34(11):960–992. https://doi.org/10.3109/0142159x.2012.703791

  35. 35.

    Downing SM, Yudkowsky R (2009) Assessment in Health Professions Education. Routledge, New York

  36. 36.

    Dyre L, Norgaard LN, Tabor A, Madsen ME, Sorensen JL, Ringsted C, Tolsgaard M (2016) Collecting validity evidence for the assessment of mastery learning in simulation-based ultrasound training. Ultraschall Med 37(4):386–392. https://doi.org/10.1055/S-0041-107976

  37. 37.

    Madsen ME, Konge L, Norgaard LN, Tabor A, Ringsted C, Klemmensen AK, Ottesen B, Tolsgaard M (2014) Assessment of performance measures and learning curves for use of a virtual-reality ultrasound simulator in transvaginal ultrasound examination. Ultrasound Obstet Gynecol 44(6):693–699. https://doi.org/10.1002/Uog.13400

  38. 38.

    Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP, Moses G, Smith CD, Satava RM (2005) Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg 241(2):364–372

  39. 39.

    Connolly M, Seligman J, Kastenmeier A, Goldblatt M, Gould J (2014) Validation of a virtual reality-based robotic surgical skills curriculum. Surg Endosc 28(5):1691–1694. https://doi.org/10.1007/S00464-013-3373-X

  40. 40.

    Lee JY, Mucksavage P, Kerbl DC, Huynh VB, Etafy M, McDougall EM (2012) Validation study of a virtual reality robotic simulator—role as an assessment tool? J Urol 187(3):998–1002. https://doi.org/10.1016/J.Juro.2011.10.160

  41. 41.

    Sethi AS, Peine WJ, Mohammadi Y, Sundaram CP (2009) Validation of a novel virtual reality robotic simulator. J Endourol 23(3):503–508. https://doi.org/10.1089/End.2008.0250

  42. 42.

    Yule S, Flin R, Paterson-Brown S, Maran N (2006) Non-technical skills for surgeons in the operating room: a review of the literature. Surgery 139(2):140–149. https://doi.org/10.1016/J.Surg.2005.06.017

  43. 43.

    Lyons C, Goldfarb D, Jones SL, Badhiwala N, Miles B, Link R, Dunkin B (2013) Which skills really matter? Proving face, content, and construct validity for a commercial robotic simulator. Surg Endosc 27(6):2020–2030. https://doi.org/10.1007/S00464-012-2704-7

Download references


The research group sincerely thank all doctors from the Centre for Robotic Surgery at Copenhagen University Hospital, Herlev-Gentofte and from Robotic Surgery Section at Department of Gynaecology, Copenhagen University Hospital, Rigshospitalet-Glostrup for participation.

Author information

Correspondence to Lisette Hvid Hovgaard.

Ethics declarations


Lisette H. Hovgaard and Drs. Steven A. W. Andersen, Lars Konge, Torur Dalsgaard and Christian R. Larsen have no conflicts of interest or financial ties to disclose.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 109 KB)

Supplementary material 2 (DOCX 93 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hovgaard, L.H., Andersen, S.A.W., Konge, L. et al. Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure. Surg Endosc 32, 4200–4208 (2018). https://doi.org/10.1007/s00464-018-6165-5

Download citation


  • Robotic surgery
  • Virtual reality simulation
  • Gynaecology
  • Assessment
  • Proficiency-based training