Skip to main content

Advertisement

SpringerLink
Log in

Objective assessment of robotic surgical skills: review of literature and future directions

  • Review Article
  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

Evaluation of robotic surgical skill has become increasingly important as robotic approaches to common surgeries become more widely utilized. However, evaluation of these currently lacks standardization. In this paper, we aimed to review the literature on robotic surgical skill evaluation.

Methods

A review of literature on robotic surgical skill evaluation was performed and representative literature presented over the past ten years.

Results

The study of reliability and validity in robotic surgical evaluation shows two main assessment categories: manual and automatic. Manual assessments have been shown to be valid but typically are time consuming and costly. Automatic evaluation and simulation are similarly valid and simpler to implement. Initial reports on evaluation of skill using artificial intelligence platforms show validity. Few data on evaluation methods of surgical skill connect directly to patient outcomes.

Conclusion

As evaluation in surgery begins to incorporate robotic skills, a simultaneous shift from manual to automatic evaluation may occur given the ease of implementation of these technologies. Robotic platforms offer the unique benefit of providing more objective data streams including kinematic data which allows for precise instrument tracking in the operative field. Such data streams will likely incrementally be implemented in performance evaluations. Similarly, with advances in artificial intelligence, machine evaluation of human technical skill will likely form the next wave of surgical evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Jung M, Morel P, Buehler L, Buchs NC, Hagen ME (2015) Robotic general surgery: current practice, evidence, and perspective. Langenbeck’s Arch Surg 400:283–292. https://doi.org/10.1007/s00423-015-1278-y.Feb18

    Article  CAS  Google Scholar 

  2. Szasz P, Louridas M, Harris KA, Aggarwal R, Grantcharov TP (2015) Assessing technical competence in surgical trainees: A systematic review. Ann Surg 261:1046–1055. https://doi.org/10.1097/SLA.0000000000000866.June

    Article  PubMed  Google Scholar 

  3. Thomas MP (2013) The role of simulation in the development of technical competence during surgical training: a literature review. Int J Med Educ 4:48–58. https://doi.org/10.5116/ijme.513b.2df7

    Article  PubMed Central  Google Scholar 

  4. Chen J, Cheng N, Cacciamani G, Oh P, Lin-Brande M, Remulla D, Gill IS, Hung AJ (2019) Objective assessment of robotic surgical technical skill: a systematic review. J Urol 201:461–469. https://doi.org/10.1016/j.juro.2018.06.078.Mar

    Article  PubMed  Google Scholar 

  5. Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: Validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187:247–252. https://doi.org/10.1016/j.juro.2011.09.032.Jan

    Article  PubMed  Google Scholar 

  6. Aghazadeh MA, Jayaratna IS, Hung AJ, Pan MM, Desai MM, Gill IS, Goh AC (2015) External validation of global evaluative assessment of robotic skills (GEARS). Surg Endosc 29:3261–3266. https://doi.org/10.1007/s00464-015-4070-8.Jan15

    Article  PubMed  Google Scholar 

  7. Sánchez R, Rodríguez O, Rosciano J et al (2016) Robotic surgery training: construct validity of Global Evaluative Assessment of Robotic Skills (GEARS. J Robot Surg 10:227–231. https://doi.org/10.1007/s11701-016-0572-1.Apr2

    Article  PubMed  Google Scholar 

  8. Goldenberg MG, Nabhani J, Wallis CJD, Chopra S, Hung AJ, Schuckman A, Djaladat H, Daneshmand S, Desai MM, Aron M, Gill IS, Satkunasivam R (2017) Feasibility of expert and crowd-sourced review of intraoperative video for quality improvement of intracorporeal urinary diversion during robotic radical cystectomy. Can Urol Assoc J 11:331–336. https://doi.org/10.5489/cuaj.4442.Oct

    Article  PubMed  PubMed Central  Google Scholar 

  9. Reznick R, Regehr G, Macrae H, Martin J, Mcculloch W (1997) Testing Technical Skill via an Innovative “Bench Station” Examination. Am J Surg 173:226–230. https://doi.org/10.1016/s0002-9610(97)89597-9.Mar

    Article  CAS  PubMed  Google Scholar 

  10. Siddiqui NY, Galloway ML, Geller EJ, Green IC, Hur HC, Langston K, Pitter MC, Tarr ME, Martina MA (2014) Validity and reliability of the robotic objective structured assessment of technical skills. Obstet Gynecol 123:1193–1199. https://doi.org/10.1097/AOG.0000000000000288.June

    Article  PubMed  PubMed Central  Google Scholar 

  11. Siddiqui NY, Tarr ME, Geller EJ, Advincula AP, Galloway ML, Green IC, Hur HC, Pitter MC, Burke EE, Martina MA (2016) Establishing benchmarks for minimum competence with dry lab robotic surgery drills. J Minim Invasive Gynecol 23:633–638. https://doi.org/10.1016/j.jmig.2016.03.014.Mar16

    Article  PubMed  Google Scholar 

  12. Liu M, Purohit S, Mazanetz J, Allen W, Kreaden US, Curet M (2017) Assessment of Robotic Console Skills (ARCS): construct validity of a novel global rating scale for technical skills in robotically assisted surgery. Surg Endosc 32:526–535. https://doi.org/10.1007/s00464-017-5694-7.July1

    Article  PubMed  Google Scholar 

  13. Frederick PJ, Szender JB, Hussein AA, Kesterson JP, Shelton JA, Anderson TL, Barnabei VM, Guru K (2016) Surgical competency for robot-assisted hysterectomy: development and validation of a robotic hysterectomy assessment score (RHAS. J Minim Invasive Gynecol 24:55–61. https://doi.org/10.1016/j.jmig.2016.10.004.Oct22

    Article  PubMed  Google Scholar 

  14. Prostatectomy Assessment and Competency Evaluation (2016) Hussein AA, Ghani KR, Peabody J, Sarle R, Abaza R, Eun D, Hu J, FUmo M, Lane B, Montgomery JS, Hinata N, Rooney D, Comstock B, Chan HK, Mane SS, Mohler JL, Wilding G, Miller D, Guru KA. (2017) Development and Validation of an Objective Scoring Tool for Robot-Assisted Radical Prostatectomy. J Urol 197:1237–1244. https://doi.org/10.1016/j.juro.2016.11.100,Nov29

    Article  Google Scholar 

  15. Raza SJ, Field E, Jay C, Eun D, Fumo M, Hu JC, Lee D, Mehboob Z, Nyquist J, Peabody JO, Sarle R, STricker H, Yang Z, Wilding G, Mohler JL, Guru K. (2015) Surgical competency for urethrovesical anastomosis during robot-assisted radical prostatectomy: Development and validation of the robotic anastomosis competency evaluation. J Urol 85(27–32):2015. https://doi.org/10.1016/j.urology.2014.09.017.Jan

    Article  Google Scholar 

  16. Hussein AA, Sexton KJ, May PR, Meng MV, Hosseini A, Eun DD, Daneshmand S, Bochner BH, Peabody JO, Abaza R, Skinner EC, Hautmann RE, Guru KA (2018) Development and validation of surgical training tool: cystectomy assessment and surgical evaluation (CASE) for robot-assisted radical cystectomy for men. Surg Endosc 32:4458–4464. https://doi.org/10.1007/s00464-018-6191-3,Apr13

    Article  PubMed  PubMed Central  Google Scholar 

  17. Guni A, Raison N, Challacombe B, Khan S, Dasgupta P, Ahmed K (2018) Development of a technical checklist for the assessment of suturing in robotic surgery. Surg Endosc 32:4402–4407. https://doi.org/10.1007/s00464-018-6407-6,Sep7

    Article  PubMed  Google Scholar 

  18. White LW, Kowalewski TM, Dockter RL, Comstock B, Hannaford B, Lendvay TS (2015) Crowd-sourced assessment of technical skill: A valid method for discriminating basic robotic surgery skills. J Endourol 29:1295–1301. https://doi.org/10.1089/end.2015.0191,Aug24

    Article  PubMed  Google Scholar 

  19. Chen C, White L, Kowalewski T, Aggarwal R, Lintott C, Comstock B, Kuksenok K, Aragon C, Holst D, Lendvay T (2014) Crowd-sourced assessment of technical skills: A novel method to evaluate surgical performance. J Surg Res 187(65–71):2013. https://doi.org/10.1016/j.jss.2013.09.024,Oct10

    Article  Google Scholar 

  20. Holst D, Kowalewski TM, White LW, Brand TC, Harper JD, Sorenson MD, Truong M, Simpson K, Tanaka A, Smith R, Lendvay TS (2015) (2015) Crowd-sourced assessment of technical skills: differentiating animate surgical skill through the wisdom of crowds. J Endourol 29:1183–1188. https://doi.org/10.1089/end.2015.0104,May26

    Article  PubMed  Google Scholar 

  21. Polin MR, Siddiqui NY, Comstock BA, Brown C, Heshan H, Lendvay TS, Martina MA (2015) Crowdsourcing: a valid alternative to expert evaluation of robotic surgery skills. Am J Ob and Gyn 215(644):e1–e7. https://doi.org/10.1016/j.ajog.2016.06.033,Oct15

    Article  Google Scholar 

  22. Holst D, Kowalewski TM, White LW, Brand TC, Harper JD, Soreesnson MD, Kirsch S, Lendvay TS (2015) Crowd-sourced assessment of technical skills: an adjunct to urology resident surgical simulation training. J Endourol 29:604–609. https://doi.org/10.1089/end.2014.0616,Jan7

    Article  PubMed  Google Scholar 

  23. Powers MK, Boonjindasup A, Pinsky M, Dorsey P, Maddox M, Su LM, Gettman M, Sandaram CP, Castle EP, Lee JY, Lee BR (2015) Crowdsourcing assessment of surgeon dissection of renal artery and vein during robotic partial nephrectomy: a novel approach for quantitative assessment of surgical performance. J Endourol 30:447–452. https://doi.org/10.1089/end.2015.0665

    Article  PubMed  Google Scholar 

  24. Oh PJ, Chen J, Hatcher D, Djaladat H, Hung AJ (2018) Crowdsourced versus expert evaluations of the vesico-urethral anastomosis in the robotic radical prostatectomy: is one superior at discriminating differences in automated performance metrics? J Robot Surg 12:705–711. https://doi.org/10.1007/s11701-018-0814-5

    Article  PubMed  Google Scholar 

  25. Ghani KR, Miller DC, Linsell S, Brachulis A, Lane B, Sarle R, Dalela D, Menon M, Comstock B, Lendvay TS, Montie J, Peabody JO (2016) Measuring to improve: Peer and crowd-sourced assessments of technical skill with robot-assisted radical prostatectomy. Eur Urol 69:547–550. https://doi.org/10.1016/j.eururo.2015.11.028

    Article  PubMed  Google Scholar 

  26. Kumar R, Jog A, Malpani A, Vagvolgyi B, Yuh D, Nguyen H, Hager G, Chen CCG (2011) Assessing system operation skills in robotic surgery trainees. Int J Med Robot Comput Assist Surg 8:118–124. https://doi.org/10.1002/rcs.449

    Article  Google Scholar 

  27. Hung AJ, Chen J, Jarc A, Hatcher D, Djaladat H, Gill IS (2017) Development and validation of objective performance metrics for robot-assisted radical prostatectomy: a pilot study. J Urol 199:296–304. https://doi.org/10.1016/j.juro.2017.07.081

    Article  PubMed  PubMed Central  Google Scholar 

  28. Chen A, Ghodoussipour S, Titus MB, Nguyen JH, Chen J, Ma R, Hung AJ (2019) Comparison of clinical outcomes and automated performance metrics in robot-assisted radical prostatectomy with and without trainee involvement. World J Urol 38:1615–1621. https://doi.org/10.1007/s00345-019-03010-3

    Article  PubMed  PubMed Central  Google Scholar 

  29. Lee JY, Mucksavage P, Kerbl DC, Huynh VB, Etafy M, McDougall EM (2012) Validation study of a virtual reality robotic simulator role as an assessment tool? J Urol 187:998–1002. https://doi.org/10.1016/j.juro.2011.10.160

    Article  PubMed  Google Scholar 

  30. Perrenot C, Perez M, Tran N, Jehl JP, Felblinger J, Bresler L, Hubert J (2012) The virtual reality simulator dV-Trainer(®) is a valid assessment tool for robotic surgical skills. Surg Endosc 26:2587–2593. https://doi.org/10.1007/s00464-012-2237-0

    Article  PubMed  Google Scholar 

  31. Liu M, Curet M (2015) A review of training research and virtual reality simulators for the da vinci surgical system. Teach Learn Med 27:12–26. https://doi.org/10.1080/10401334.2014.979181

    Article  PubMed  Google Scholar 

  32. Liss MA, Kane CJ, Chen T, Baumgartner J, Derweesh IH (2015) Virtual reality suturing task as an objective test for robotic experience assessment. BMC Urol 15:63. https://doi.org/10.1186/s12894-015-0051-4

    Article  PubMed  PubMed Central  Google Scholar 

  33. Havemann MC, Dalsgaard T, Sørensen JL, Rossaak K, Brisling S, Mosgaard BJ, Hogdall C, Bjerrum F (2018) Examining validity evidence for a simulation-based assessment tool for basic robotic surgical skills. J Robot Surg 13:99–106. https://doi.org/10.1007/s11701-018-0811-8

    Article  PubMed  Google Scholar 

  34. Dubin AK, Julian D, Tanaka A, Mattingly P, Smith R (2018) A model for predicting the GEARS score from virtual reality surgical simulator metrics. Surg Endosc 32:3576–3581. https://doi.org/10.1007/s00464-018-6082-7

    Article  PubMed  Google Scholar 

  35. Chandra V, Nehra D, Parent R, Woo R, Reyes R, Hernandez-Boussard T, Dutta S (2009) A comparison of laparoscopic and robotic assisted suturing performance by experts and novices. Surgery 147:830–839. https://doi.org/10.1016/j.surg.2009.11.002

    Article  PubMed  Google Scholar 

  36. Chowriappa AJ, Shi Y, Raza SJ, Ahmed K, Stegemann A, Wilding G, Kaouk J, Peabody JO, Menon M, Hassett JM, Kesavadas T, Guru KA (2013) Development and validation of a composite scoring system for robot-assisted surgical training the Robotic Skills Assessment Score. J Surg Res 185:561–569. https://doi.org/10.1016/j.jss.2013.06.054

    Article  PubMed  Google Scholar 

  37. Witthaus MW, Farooq S, Melnyk R, CampbellT SP, Mathrews E, Ezzat B, Ertefaie A, Frye TP, Wu G, Rashid H, Joseph JV, Ghazi A (2020) Incorporation and validation of clinically relevant performance metrics of simulation (CRPMS) into a novel full-immersion simulation platform for nerve-sparing robot-assisted radical prostatectomy (NS-RARP) utilizing three-dimensional printing and hydrogel casting technology. BJU Int 125(322–332):2019. https://doi.org/10.1111/bju.14940

    Article  Google Scholar 

  38. Baghdadi A, Hussein AA, Ahmed Y, Cavuoto LA, Guru KA (2018) A computer vision technique for automated assessment of surgical performance using surgeons’ console-feed videos. Int J Comput Assist Radiol Surg 14:697–707. https://doi.org/10.1007/s11548-018-1881-9

    Article  PubMed  Google Scholar 

  39. Hashimoto DA, Rosman G, Witkowski ER, Stafford C, Navarette-Welton AJ, Rattner DW, Lillemoe KD, Rus DL, Meireles OR (2019) Computer vision analysis of intraoperative video: automated recognition of operative steps of laparoscopic sleeve gastrectomy. Ann Surg 270:414–421. https://doi.org/10.1097/sla.0000000000003460

    Article  PubMed  Google Scholar 

  40. Mascagni P, Vardazaryan A, Alapatt D, Urade T, Emre T, Fiorillo C, Pessaux P, Mutter D, Marescaux J, Costamagna G, Dallemagne B, Padoy N (2020) Artificial Intelligence for Surgical Safety: Automatic Assessment of the Critical View of Safety in Laparoscopic Cholecystectomy using Deep Learning. Ann Surg. https://doi.org/10.1097/SLA.0000000000004351

    Article  PubMed  Google Scholar 

  41. Vassiliou MC, Dunkin BJ, Marks JM, Fried GM (2010) FLS and FES: Comprehensive models of training and assessment. Surg Clin North Am 90:535–558. https://doi.org/10.1016/j.suc.2010.02.012

    Article  PubMed  Google Scholar 

  42. Smith R, Patel V, Satava R (2013) Fundamentals of robotic surgery: A course of basic robotic surgery skills based upon a 14-society consensus template of outcomes measures and curriculum development. Int J Med Robot Comput Assist Surg 10:379–384. https://doi.org/10.1002/rcs.1559

    Article  Google Scholar 

  43. Zendejas B, Ruparel RK, Cook DA (2015) Validity evidence for the Fundamentals of Laparoscopic Surgery (FLS) program as an assessment tool: a systematic review. Surg Endosc 30:512–520. https://doi.org/10.1007/s00464-015-4233-7

    Article  PubMed  Google Scholar 

  44. Addison P, Yoo A, Duarte-Ramos J, Addy J, Dechario S, Husk G, Jarrett M, Teixera J, Antonacci A, Filicori F (2020) Correlation between operative time and crowd-sourced skills assessment for robotic bariatric surgery. Surg Endosc. https://doi.org/10.1007/s00464-020-08019-z

    Article  PubMed  Google Scholar 

  45. Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee BNJO (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369(1434–1442):2013. https://doi.org/10.1056/NEJMsa1300625

    Article  CAS  Google Scholar 

  46. Lee D, Yu HW, Kwon H, Kong HJ, Lee KE, Kim HC (2020) Evaluation of surgical skills during robotic surgery by deep learning-based multiple surgical instrument tracking in training and actual operations. J Clin Med 9:1964. https://doi.org/10.3390/jcm9061964

    Article  PubMed Central  Google Scholar 

  47. Lavanchy JL, Zindel J, Kirtac K, Twick I, Hosgor E, Candinas D, Beldi G (2021) Automation of surgical skill assessment using a three-stage machine learning algorithm. Nature Sci Reports 11:5197. https://doi.org/10.1038/s41598-021-84295-6

    Article  CAS  Google Scholar 

  48. Funke I, Mees ST, Weltz J, Speldel S (2019) Video-based surgical skill assessment using 3D convolutional neural networks. Int J Comp Assisted Radiol and Surg 14:1217–1225. https://doi.org/10.1007/s11548-019-01995-1

    Article  Google Scholar 

  49. Guzman-Garcia C, Gomez-Tome M, Sanchez-Gonzalez P, Oropesa I, Gomez EJ (2021) Speech-based surgical phase recognition for non-intrusive surgical skills’ assessment in educational contexts. Sensors (Basel) 21(1330):2021. https://doi.org/10.3390/s21041330

    Article  Google Scholar 

  50. Jung JJ, Juni P, Lebovic G, Grantcharov T (2020) First-year analysis of the operating room black box study. Ann Surg 27:122–127. https://doi.org/10.1097/SLA.0000000000002863

    Article  Google Scholar 

  51. Chen AB, Liang S, Nguyen JH, Liu Y, Hung AJ (2020) Machine learning analyses of automated performance metrics during granular sub-stitch phases predict surgeon experience. J Surg 169:1245–1249. https://doi.org/10.1016/j.surg.2020.09.020

    Article  Google Scholar 

  52. Taylor RH, Funda J, Eldridge B, Gomory S, Gruben K, LaRose D, Talamini M, Kavoussi L, Anderson J (1995) A telerobotic assistant for laparoscopic surgery. IEEE Eng Med Biol 14:279–288

    Article  Google Scholar 

  53. Talamini MA (2001) Surgery of the 21st Century. Ann Surg 234:8–9. https://doi.org/10.1097/00000658-200107000-00002

    Article  CAS  PubMed  Google Scholar 

  54. Talamini MA (2003) Overview - Current clinical and preclinical use of robotics for surgery. J Gastrointest Surg 7:479–480. https://doi.org/10.1016/S1091-255X(03)00031-3

    Article  PubMed  Google Scholar 

  55. Talamini MA, Hanly EJ (2005) Technology in the operating suite. JAMA 293:863–866. https://doi.org/10.1001/jama.293.7.863

    Article  CAS  PubMed  Google Scholar 

  56. Talamini MA, Horgan S, Melvin S, Chapman W (2003) A Prospective analysis of 211 robotic assisted surgical procedures. Surg Endosc 17:1521–1524. https://doi.org/10.1007/s00464-002-8853-3

    Article  CAS  PubMed  Google Scholar 

  57. MASTERY Trial. NCT04647188. Expected July 2022

  58. Waters PS, Flynn J, Larach JT, Fernando D, Peacock O, Foster, JD, Flood M, McCormick JJ, Warrier SK, Heriot AG (2021) Fellowship training in robotic colorectal surgery within the current hospital setting: an achievable goal? ANZ J Surg 91:2337–2344. https://doi.org/10.1111/ans.16677

Download references

Funding

IPAL is supported by 2020 SAGES Robotic Surgery Grant.

Author information

Authors and Affiliations

  1. Intraoperative Performance Analytics Laboratory (IPAL), Department of General Surgery, Northwell Health, Lenox Hill Hospital, 186 E. 76th Street, 1st Floor, New York, NY, 10021, USA

    Saratu Kutana, Daniel P. Bitner, Poppy Addison, Paul J. Chung & Filippo Filicori

  2. Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA

    Paul J. Chung, Mark A. Talamini & Filippo Filicori

Authors
  1. Saratu Kutana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel P. Bitner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Poppy Addison
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Paul J. Chung
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mark A. Talamini
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Filippo Filicori
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel P. Bitner.

Ethics declarations

Disclosures

Dr. Filippo Filicori has consulting affiliations with Digital Surgery, Boston Scientific, and Cambridge Medical Robotics. Dr. Mark Talamini, Dr. Paul Chung, Dr. Daniel Bitner, Dr. Poppy Addison, and Ms. Saratu Kutana have no conflicts of interest to disclose. Funding is from SAGES 2020 Robotic Surgery Grant [no specified Grant Number].

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kutana, S., Bitner, D.P., Addison, P. et al. Objective assessment of robotic surgical skills: review of literature and future directions. Surg Endosc 36, 3698–3707 (2022). https://doi.org/10.1007/s00464-022-09134-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-022-09134-9

Keywords

Search

Navigation