Skip to main content

Advertisement

SpringerLink
Log in

Review of automated performance metrics to assess surgical technical skills in robot-assisted laparoscopy

  • Review Article
  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Introduction

Robot-assisted laparoscopy is a safe surgical approach with several studies suggesting correlations between complication rates and the surgeon’s technical skills. Surgical skills are usually assessed by questionnaires completed by an expert observer. With the advent of surgical robots, automated surgical performance metrics (APMs)—objective measures related to instrument movements—can be computed. The aim of this systematic review was thus to assess APMs use in robot-assisted laparoscopic procedures. The primary outcome was the assessment of surgical skills by APMs and the secondary outcomes were the association between APM and surgeon parameters and the prediction of clinical outcomes.

Methods

A systematic review following the PRISMA guidelines was conducted. PubMed and Scopus electronic databases were screened with the query “robot-assisted surgery OR robotic surgery AND performance metrics” between January 2010 and January 2021. The quality of the studies was assessed by the medical education research study quality instrument. The study settings, metrics, and applications were analysed.

Results

The initial search yielded 341 citations of which 16 studies were finally included. The study settings were either simulated virtual reality (VR) (4 studies) or real clinical environment (12 studies). Data to compute APMs were kinematics (motion tracking), and system and specific events data (actions from the robot console). APMs were used to differentiate expertise levels, and thus validate VR modules, predict outcomes, and integrate datasets for automatic recognition models. APMs were correlated with clinical outcomes for some studies.

Conclusions

APMs constitute an objective approach for assessing technical skills. Evidence of associations between APMs and clinical outcomes remain to be confirmed by further studies, particularly, for non-urological procedures. Concurrent validation is also required.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Ozben V et al (2019) The da Vinci Xi system for robotic total/subtotal colectomy vs conventional laparoscopy: short-term outcomes. Tech Coloproctol 23:861–868

    CAS  PubMed  Google Scholar 

  2. Gallotta V et al (2018) Robotic versus laparoscopic radical hysterectomy in early cervical cancer: A case matched control study. Eur J Surg Oncol 44:754–759

    PubMed  Google Scholar 

  3. Stulberg JJ et al (2020) Association Between Surgeon Technical Skills and Patient Outcomes. JAMA Surg. https://doi.org/10.1001/jamasurg.2020.3007

    Article  PubMed  PubMed Central  Google Scholar 

  4. Larcher A et al (2019) The learning curve for robot-assisted partial nephrectomy: impact of surgical experience on perioperative outcomes. Eur Urol 75:253–256

    PubMed  Google Scholar 

  5. Vassiliou MC et al (2005) A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 190:107–113

    PubMed  Google Scholar 

  6. Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187:247–252

    PubMed  Google Scholar 

  7. Hung AJ et al (2018) Development and validation of objective performance metrics for robot-assisted radical prostatectomy: a pilot study. J Urol 199:296–304

    PubMed  Google Scholar 

  8. Vanlander AE et al (2020) Orsi consensus meeting on european robotic training (OCERT): results from the first multispecialty consensus meeting on training in robot-assisted surgery. Eur Urol 78:713–716

    PubMed  Google Scholar 

  9. Chowriappa AJ et al (2013) Development and validation of a composite scoring system for robot-assisted surgical training–the Robotic Skills Assessment Score. J Surg Res 185:561–569

    PubMed  Google Scholar 

  10. Scott SI, Dalsgaard T, Jepsen JV, von Buchwald C, Andersen SAW (2020) Design and validation of a cross-specialty simulation-based training course in basic robotic surgical skills. Int J Med Robot 16:1–10

    PubMed  Google Scholar 

  11. Jarc A, Curet MJ (2017) Viewpoint matters: objective performance metrics for surgeon endoscope control during robot-assisted surgery. Surg Endosc 31:1192–1202

    PubMed  Google Scholar 

  12. Liberati A et al (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol 62:e1-34

    PubMed  Google Scholar 

  13. Kaufman, K. & An, K. Chapter 6 - Biomechanics. in Kelley and Firestein’s Textbook of Rheumatology (Tenth Edition) (eds. Firestein, G. S., Budd, R. C., Gabriel, S. E., McInnes, I. B. & O’Dell, J. R.) 78–89 (Elsevier, 2017). doi: https://doi.org/10.1016/B978-0-323-31696-5.00006-1.

  14. Applicability of the Newcastle-Ottawa Scale (NOS) for rating quality of cohort studies: using studies regarding the predictors of returning to work after traumatic limb injuries for example | Colloquium Abstracts. /2010-keystone/applicability-newcastle-ottawa-scale-nos-rating-quality-cohort-studies-using-studies.

  15. Hvolbek AP, Nilsson PM, Sanguedolce F, Lund L (2019) A prospective study of the effect of video games on robotic surgery skills using the high-fidelity virtual reality RobotiX simulator. Adv Med Educ Pract 10:627–634

    PubMed  PubMed Central  Google Scholar 

  16. Chen J et al (2019) Effect of surgeon experience and bony pelvic dimensions on surgical performance and patient outcomes in robot-assisted radical prostatectomy. BJU Int 124:828–835

    PubMed  Google Scholar 

  17. Hung AJ et al (2019) Experts vs super-experts: differences in automated performance metrics and clinical outcomes for robot-assisted radical prostatectomy. BJU Int 123:861–868

    PubMed  Google Scholar 

  18. Hovgaard LH, Andersen SAW, Konge L, Dalsgaard T, Larsen CR (2018) Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure. Surg Endosc 32:4200–4208

    PubMed  Google Scholar 

  19. Harrison P et al (2018) The validation of a novel robot-assisted radical prostatectomy virtual reality module. J Surg Educ 75:758–766

    PubMed  Google Scholar 

  20. Ebbing J et al (2020) Development and validation of non-guided bladder-neck and neurovascular-bundle dissection modules of the RobotiX-Mentor® full-procedure robotic-assisted radical prostatectomy virtual reality simulation. Int J Med Robot. https://doi.org/10.1002/rcs.2195

    Article  PubMed  PubMed Central  Google Scholar 

  21. Ghodoussipour S et al (2020) an objective assessment of performance during robotic partial nephrectomy: validation and correlation of automated performance metrics with intraoperative outcomes. J Urol. https://doi.org/10.1097/JU.0000000000001557

    Article  PubMed  Google Scholar 

  22. Chen J et al (2018) Use of automated performance metrics to measure surgeon performance during robotic vesicourethral anastomosis and methodical development of a training tutorial. J Urol 200:895–902

    PubMed  Google Scholar 

  23. Brown KC, Bhattacharyya KD, Kulason S, Zia A, Jarc A (2020) How to bring surgery to the next level: interpretable skills assessment in robotic-assisted surgery. Visc Med 36:463–470

    PubMed  PubMed Central  Google Scholar 

  24. Chen A et al (2019) Comparison of clinical outcomes and automated performance metrics in robot-assisted radical prostatectomy with and without trainee involvement. World J Urol. https://doi.org/10.1007/s00345-019-03010-3

    Article  PubMed  PubMed Central  Google Scholar 

  25. Ghaderi I et al (2017) Surgical skills curricula in American College of surgeons accredited education institutes: an international survey. Am J Surg 213:678–686

    PubMed  Google Scholar 

  26. Hung AJ et al (2012) Concurrent and predictive validation of a novel robotic surgery simulator: a prospective, randomized study. J Urol 187:630–637

    PubMed  Google Scholar 

  27. Kutikov A, Uzzo RG (2009) The R.E.N.A.L. nephrometry score: a comprehensive standardized system for quantitating renal tumor size, location and depth. J Urol 182:844–853

    PubMed  Google Scholar 

  28. Chen A et al (2020) Comparison of clinical outcomes and automated performance metrics in robot-assisted radical prostatectomy with and without trainee involvement. World J Urol 38:1615–1621

    PubMed  Google Scholar 

  29. Hung AJ et al (2019) A deep-learning model using automated performance metrics and clinical features to predict urinary continence recovery after robot-assisted radical prostatectomy. BJU Int 124:487–495

    PubMed  PubMed Central  Google Scholar 

  30. Hung AJ et al (2018) Utilizing Machine Learning and Automated Performance Metrics to Evaluate Robot-Assisted Radical Prostatectomy Performance and Predict Outcomes. J Endourol 32:438–444

    PubMed  Google Scholar 

  31. Chen AB, Liang S, Nguyen JH, Liu Y, Hung AJ (2020) Machine learning analyses of automated performance metrics during granular sub-stitch phases predict surgeon experience. Surgery. https://doi.org/10.1016/j.surg.2020.09.020

    Article  PubMed  PubMed Central  Google Scholar 

  32. Katić D et al (2015) LapOntoSPM: an ontology for laparoscopic surgeries and its application to surgical phase recognition. Int J Comput Assist Radiol Surg 10:1427–1434

    PubMed  Google Scholar 

  33. Novel evaluation of surgical activity recognition models using task-based efficiency metrics - PubMed. https://pubmed.ncbi.nlm.nih.gov/31267333/.

  34. Khan N, Abboudi H, Khan MS, Dasgupta P, Ahmed K (2014) Measuring the surgical ‘learning curve’: methods, variables and competency. BJU Int 113:504–508

    PubMed  Google Scholar 

  35. Huaulmé, A. et al. (2021) MIcro-surgical anastomose workflow recognition challenge report. 2103: 13111

  36. Zia, A., Hung, A., Essa, I. & Jarc, A. Surgical Activity Recognition in Robot-Assisted Radical Prostatectomy Using Deep Learning: 21st International Conference, Granada, Spain, September 16–20, 2018, Proceedings, Part IV. in 273–280 (2018). doi: https://doi.org/10.1007/978-3-030-00937-3_32.

  37. Meeuwsen FC, van Luyn F, Blikkendaal MD, Jansen FW, van den Dobbelsteen JJ (2019) Surgical phase modelling in minimal invasive surgery. Surg Endosc 33:1426–1432

    CAS  PubMed  Google Scholar 

  38. Sarikaya, D. & Jannin, P. (2020) Towards generalizable surgical activity recognition using spatial temporal graph convolutional networks. [cs]

  39. Kitaguchi D et al (2020) Real-time automatic surgical phase recognition in laparoscopic sigmoidectomy using the convolutional neural network-based deep learning approach. Surg Endosc 34:4924–4931

    PubMed  Google Scholar 

  40. Huaulmé A et al (2019) Automatic annotation of surgical activities using virtual reality environments. Int J Comput Assist Radiol Surg 14:1663–1671

    PubMed  Google Scholar 

  41. Mason JD, Ansell J, Warren N, Torkington J (2013) Is motion analysis a valid tool for assessing laparoscopic skill? Surg Endosc 27:1468–1477

    PubMed  Google Scholar 

  42. Beulens AJW et al (2020) Analysis of the video motion tracking system ‘Kinovea’ to assess surgical movements during robot-assisted radical prostatectomy. Int J Med Robot 16:e2090

    PubMed  Google Scholar 

  43. Alexander HC et al (2018) Reporting of complications after laparoscopic cholecystectomy: a systematic review. HPB (Oxford) 20:786–794

    Google Scholar 

  44. Adelman MR, Bardsley TR, Sharp HT (2014) Urinary tract injuries in laparoscopic hysterectomy: a systematic review. J Minim Invasive Gynecol 21:558–566

    PubMed  Google Scholar 

  45. Despinoy F et al (2016) Unsupervised trajectory segmentation for surgical gesture recognition in robotic training. IEEE Trans Biomed Eng 63:1280–1291

    PubMed  Google Scholar 

  46. Einarsson JI, Suzuki Y (2009) Total laparoscopic hysterectomy: 10 steps toward a successful procedure. Rev Obstet Gynecol 2:57–64

    PubMed  PubMed Central  Google Scholar 

  47. Working group of ESGE (2019) Surgical steps of total laparoscopic hysterectomy: Part 1: Benign disease by the European Society for Gynaecological Endoscopy (ESGE)1. Facts Views Vis Obgyn 11:103–110

    Google Scholar 

  48. Bratu O et al (2017) Erectile dysfunction post-radical prostatectomy - a challenge for both patient and physician. J Med Life 10:13–18

    CAS  PubMed  PubMed Central  Google Scholar 

  49. Loughlin KR, Prasad MM (2010) Post-prostatectomy urinary incontinence: a confluence of 3 factors. J Urol 183:871–877

    PubMed  Google Scholar 

  50. Costello AJ (2020) Considering the role of radical prostatectomy in 21st century prostate cancer care. Nat Rev Urol 17:177–188

    PubMed  Google Scholar 

  51. Levinson AW, Su L-M (2007) Laparoscopic radical prostatectomy: current techniques. Curr Opin Urol 17:98–103

    PubMed  Google Scholar 

  52. Sheetz KH, Claflin J, Dimick JB (2020) Trends in the adoption of robotic surgery for common surgical procedures. JAMA Netw Open 3:e1918911

    PubMed  PubMed Central  Google Scholar 

  53. McFerrin C et al (2019) Charlson comorbidity score is associated with readmission to the index operative hospital after radical cystectomy and correlates with 90-day mortality risk. Int Urol Nephrol 51:1755–1762

    CAS  PubMed  Google Scholar 

  54. Pellino G et al (2018) Predictors of complications and mortality following left colectomy with primary stapled anastomosis for cancer: results of a multicentric study with 1111 patients. Colorectal Dis 20:986–995

    CAS  PubMed  Google Scholar 

  55. Adogwa O et al (2019) Extended length of stay after lumbar spine surgery: sick patients, postoperative complications, or practice style differences among hospitals and physicians? World Neurosurg 123:e734–e739

    PubMed  Google Scholar 

  56. Jung JJ, Jüni P, Lebovic G, Grantcharov T (2020) First-year analysis of the operating room black box study. Ann Surg 271:122–127

    PubMed  Google Scholar 

  57. Maier-Hein L et al (2017) Surgical data science for next-generation interventions. Nat Biomed Eng 1:691–696

    PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Gynaecology Obstetrics, Rennes University Hospital, Univ Rennes, INSERM, LTSI - UMR 1099, F35000, Rennes, France

    Sonia Guerin, Vincent Lavoue & Krystel Nyangoh Timoh

  2. University Rennes, INSERM, LTSI - UMR 1099, F35000, Rennes, France

    Sonia Guerin, Arnaud Huaulmé, Pierre Jannin & Krystel Nyangoh Timoh

Authors
  1. Sonia Guerin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arnaud Huaulmé
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vincent Lavoue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pierre Jannin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Krystel Nyangoh Timoh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sonia Guerin.

Ethics declarations

Disclosures

Sonia Guérin, Dr Arnaud Huaulmé, Pierre Jannin, and Dr Krystel Nyangoh Timoh have no conflicts of interest or financial ties to disclose. Pr Vincent Lavoué is proctor for interest in Intuitive surgery.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guerin, S., Huaulmé, A., Lavoue, V. et al. Review of automated performance metrics to assess surgical technical skills in robot-assisted laparoscopy. Surg Endosc 36, 853–870 (2022). https://doi.org/10.1007/s00464-021-08792-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-021-08792-5

Keywords

Search

Navigation