Skip to main content
SpringerLink
Log in

An objective approach to evaluate novice robotic surgeons using a combination of kinematics and stepwise cumulative sum (CUSUM) analyses

  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

Current evaluation methods for robotic-assisted surgery (ARCS or GEARS) are limited to 5-point Likert scales which are inherently time-consuming and require a degree of subjective scoring. In this study, we demonstrate a method to break down complex robotic surgical procedures using a combination of an objective cumulative sum (CUSUM) analysis and kinematics data obtained from the da Vinci® Surgical System to evaluate the performance of novice robotic surgeons.

Methods

Two HPB fellows performed 40 robotic-assisted hepaticojejunostomy reconstructions to model a portion of a Whipple procedure. Kinematics data from the da Vinci® system was recorded using the dV Logger® while CUSUM analyses were performed for each procedural step. Each kinematic variable was modeled using machine learning to reflect the fellows’ learning curves for each task. Statistically significant kinematics variables were then combined into a single formula to create the operative robotic index (ORI).

Results

The inflection points of our overall CUSUM analysis showed improvement in technical performance beginning at trial 16. The derived ORI model showed a strong fit to our observed kinematics data (R2 = 0.796) with an ability to distinguish between novice and intermediate robotic performance with 89.3% overall accuracy.

Conclusions

In this study, we demonstrate a novel approach to objectively break down novice performance on the da Vinci® Surgical System. We identified kinematics variables associated with improved overall technical performance to create an objective ORI. This approach to robotic operative evaluation demonstrates a valuable method to break down complex surgical procedures in an objective, stepwise fashion. Continued research into objective methods of evaluation for robotic surgery will be invaluable for future training and clinical implementation of the robotic platform.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Saini V, Brownlee S, Elshaug AG, Glasziou P, Heath I (2017) Addressing overuse and underuse around the world. Lancet 390(10090):105–107. https://doi.org/10.1016/s0140-6736(16)32573-9

    Article  PubMed  Google Scholar 

  2. Vonlanthen R et al (2011) The impact of complications on costs of major surgical procedures: a cost analysis of 1200 patients. Ann Surg 254(6):907–913. https://doi.org/10.1097/SLA.0b013e31821d4a43

    Article  PubMed  Google Scholar 

  3. Khuri SF, Henderson WG, DePalma RG, Mosca C, Healey NA, Kumbhani DJ (2005) Determinants of long-term survival after major surgery and the adverse effect of postoperative complications. Ann Surg 242(3):326–341

    Article  Google Scholar 

  4. Birkmeyer JD et al (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369(15):1434–1442. https://doi.org/10.1056/NEJMsa1300625

    Article  CAS  PubMed  Google Scholar 

  5. Hogg ME et al (2016) Grading of surgeon technical performance predicts postoperative pancreatic fistula for pancreaticoduodenectomy independent of patient-related variables. Ann Surg 264(3):482–491. https://doi.org/10.1097/sla.0000000000001862

    Article  PubMed  Google Scholar 

  6. Liu M, Purohit S, Mazanetz J, Allen W, Kreaden US, Curet M (2017) Assessment of robotic console skills (ARCS): construct validity of a novel global rating scale for technical skills in robotically assisted surgery. Surg Endosc. https://doi.org/10.1007/s00464-017-5694-7

    Article  PubMed  PubMed Central  Google Scholar 

  7. Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187(1):247–252. https://doi.org/10.1016/j.juro.2011.09.032

    Article  PubMed  Google Scholar 

  8. Deal SB et al (2017) Evaluation of crowd-sourced assessment of the critical view of safety in laparoscopic cholecystectomy. Surg Endosc 31(12):5094–5100. https://doi.org/10.1007/s00464-017-5574-1

    Article  PubMed  Google Scholar 

  9. Gao Y, et al (2014) JHU-ISI gesture and skill assessment working set (JIGSAWS): a surgical activity dataset for human motion modeling. In: MICCAI Workshop: M2CAI, 2014, vol 3

  10. Jog A, Itkowitz B, Liu M, DiMaio S, Hager G (2011) Towards integrating task information in skills assessment for dexterous tasks in surgery and simulation. In: 2011 IEEE International Conference on Robotics and Automation (pp. 5273–5278)

  11. Wohl H (1977) The cusum plot: its utility in the analysis of clinical data. N Engl J Med 296(18):1044–1045. https://doi.org/10.1056/nejm197705052961806

    Article  CAS  PubMed  Google Scholar 

  12. Chaput de Saintonge DM, Vere DW (1974) Why don't doctors use cusums? Lancet 1(7848):120–121

    Article  CAS  Google Scholar 

  13. Cavill I (1971) Quality control in routine haemoglobinometry. J Clin Pathol 24(8):701–704

    Article  CAS  Google Scholar 

  14. Bosker R, Groen H, Hoff C, Totte E, Ploeg R, Pierie JP (2013) Early learning effect of residents for laparoscopic sigmoid resection. J Surg Educ 70(2):200–205. https://doi.org/10.1016/j.jsurg.2012.10.004

    Article  PubMed  Google Scholar 

  15. Mackenzie H et al (2013) Clinical and educational proficiency gain of supervised laparoscopic colorectal surgical trainees. Surg Endosc 27(8):2704–2711. https://doi.org/10.1007/s00464-013-2806-x

    Article  PubMed  Google Scholar 

  16. Guend H et al (2017) Developing a robotic colorectal cancer surgery program: understanding institutional and individual learning curves. Surg Endosc 31(7):2820–2828. https://doi.org/10.1007/s00464-016-5292-0

    Article  PubMed  Google Scholar 

  17. Zhang L et al (2013) Characterizing the learning curve of the VBLaST-PT((c)) (Virtual Basic Laparoscopic Skill Trainer). Surg Endosc 27(10):3603–3615. https://doi.org/10.1007/s00464-013-2932-5

    Article  PubMed  PubMed Central  Google Scholar 

  18. De Gori M, Adamczewski B, Jenny JY (2017) Value of the cumulative sum test for the assessment of a learning curve: application to the introduction of patient-specific instrumentation for total knee arthroplasty in an academic department. Knee 24(3):615–621. https://doi.org/10.1016/j.knee.2017.03.007

    Article  PubMed  Google Scholar 

  19. Tam V et al (2017) Robotic pancreatoduodenectomy biotissue curriculum has validity and improves technical performance for surgical oncology fellows. J Surg Educ. https://doi.org/10.1016/j.jsurg.2017.05.016

    Article  PubMed  Google Scholar 

  20. Yap CH, Colson ME, Watters DA (2007) Cumulative sum techniques for surgeons: a brief review. ANZ J Surg 77(7):583–586. https://doi.org/10.1111/j.1445-2197.2007.04155.x

    Article  PubMed  Google Scholar 

  21. Kestin IG (1995) A statistical approach to measuring the competence of anaesthetic trainees at practical procedures. Br J Anaesth 75(6):805–809

    Article  CAS  Google Scholar 

  22. Hung AJ, Chen J, Gill IS (2018) Automated performance metrics and machine learning algorithms to measure surgeon performance and anticipate clinical outcomes in robotic surgery. JAMA Surg. https://doi.org/10.1001/jamasurg.2018.1512

    Article  PubMed  PubMed Central  Google Scholar 

  23. Hanzly MI et al (2015) Simulation-based training in robot-assisted surgery: current evidence of value and potential trends for the future. Curr Urol Rep 16(6):41. https://doi.org/10.1007/s11934-015-0508-8

    Article  PubMed  Google Scholar 

  24. Hogg ME et al (2017) Mastery-based virtual reality robotic simulation curriculum: the first step toward operative robotic proficiency. J Surg Educ 74(3):477–485. https://doi.org/10.1016/j.jsurg.2016.10.015

    Article  PubMed  Google Scholar 

  25. Newcomb LK et al (2017) Correlation of virtual reality simulation and dry lab robotic technical skills. J Minim Invasive Gynecol. https://doi.org/10.1016/j.jmig.2017.11.006

    Article  PubMed  Google Scholar 

  26. Tam V, Zeh HJ 3rd, Hogg ME (2017) Incorporating metrics of surgical proficiency into credentialing and privileging pathways. JAMA Surg 152(5):494–495. https://doi.org/10.1001/jamasurg.2017.0025

    Article  PubMed  Google Scholar 

Download references

Funding

Hardware (dV Logger®) and kinematics data supplied by Intuitive Surgical®. Subsequent ongoing related studies funded by Intuitive Surgical®.

Author information

Authors and Affiliations

  1. Department of Surgery, Carolinas Medical Center, 1000 Blythe Blvd, MEB Suite 601, Charlotte, NC, 28203, USA

    William B. Lyman

  2. Division of HPB Surgery, Department of Surgery, Carolinas Medical Center, Charlotte, NC, USA

    Michael J. Passeri, Keith Murphy, David A. Iannitti, John B. Martinie, Erin H. Baker & Dionisios Vrochides

  3. St. Vincent’s Medical Center, Bridgeport, CT, USA

    Imran A. Siddiqui

  4. Washington University School of Medicine, St. Louis, MO, USA

    Adeel S. Khan

Authors
  1. William B. Lyman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael J. Passeri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Keith Murphy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Imran A. Siddiqui
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Adeel S. Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. David A. Iannitti
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. John B. Martinie
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Erin H. Baker
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Dionisios Vrochides
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to William B. Lyman.

Ethics declarations

Disclosures

The “Black Box” or dV Logger® and kinematics data from the dV Logger® was supplied by Intuitive Surgical®. Dr. Lyman reports nonfinancial support from Intuitive Surgical, during the conduct of the study; grants from Intuitive Surgical, outside the submitted work. Dr. Khan reports personal fees from Intuitive Surgical, outside the submitted work; Dr. Martinie reports grants, personal fees and nonfinancial support from Intuitive Surgical, outside the submitted work; Dr. Vrochides reports nonfinancial support from Intuitive Surgical, during the conduct of the study; grants and nonfinancial support from Intuitive Surgical, outside the submitted work. Dr. Passeri, Mr. Murphy, Dr. Siddiqui, Dr. Iannitti, Dr. Baker have nothing to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (AVI 135451 kb)

Supplementary file2 (DOCX 13 kb)

Supplementary file3 (DOCX 294 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lyman, W.B., Passeri, M.J., Murphy, K. et al. An objective approach to evaluate novice robotic surgeons using a combination of kinematics and stepwise cumulative sum (CUSUM) analyses. Surg Endosc 35, 2765–2772 (2021). https://doi.org/10.1007/s00464-020-07708-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-020-07708-z

Keywords

Search

Navigation