Skip to main content

Advertisement

SpringerLink
Log in

Objective assessment of robotic surgical skill using instrument contact vibrations

  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

Surgical skill evaluation ordinarily requires tedious video review and survey completion, while new automatic approaches focus on evaluating the quality of the surgeon’s movements in free space. Robotic surgical instrument vibrations are simple to measure and physically correspond to how roughly instruments are handled, but they have yet to be studied as a measure of technical surgical skill.

Methods

Thirteen surgeons used a robotic surgery system (da Vinci S by Intuitive Surgical) to perform four trials each of peg transfer (PT), needle pass (NP), and intracorporeal suturing (IS). Completion time, instrument vibrations, and applied forces were measured for each trial; root mean square (RMS) and total sum of squares (TSS) were calculated from both the vibration and force recordings. Four experienced surgeons blindly assessed the task videos using a Global Rating Scale (GRS), and skill metrics were compared between the eight novices and five experienced participants. Stepwise regression was performed to predict GRS score from objective skill metrics. The concurrent validity of each metric was evaluated using receiver operating characteristic (ROC) analysis.

Results

The GRS demonstrated excellent internal consistency (Cronbach’s α = 0.91) and strong inter-rater reliability (ICC = 0.84). Compared to novices, experienced surgeons earned higher GRS scores and performed tasks with lower vibration magnitudes, lower forces, and shorter completion times in 15 of 18 task–metric combinations (p values ranging from 0.042 to <0.001). ROC analysis demonstrated that including vibration and force magnitudes along with completion time in skill prediction models improves the objective classification of subjects as novice or experienced for all tasks studied (PT: 90 % sensitivity, 75 % specificity; NP: 85 % sensitivity, 84 % specificity; suturing: 100 % sensitivity, 100 % specificity).

Conclusions

RMS and TSS instrument vibrations are novel construct-valid measures of robotic surgical skill that enable the development of objective skill assessment models comparable to observer-based ratings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Reznick RK (1993) Teaching and testing technical skills. Am J Surg 165(3):358–361

    Article  CAS  PubMed  Google Scholar 

  2. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C et al (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84(2):273–278

    Article  CAS  PubMed  Google Scholar 

  3. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W (1997) Testing technical skill via an innovative “bench station” examination. Am J Surg 173(3):226–230

    Article  CAS  PubMed  Google Scholar 

  4. Figert PL, Park AE, Witzke DB, Schwartz RW (2001) Transfer of training in acquiring laparoscopic skills. J Am Coll Surg 193(5):533–537

    Article  CAS  PubMed  Google Scholar 

  5. Amodeo A, Linares Quevedo A, Joseph JV, Belgrano E, Patel HR (2009) Robotic laparoscopic surgery: cost and training. Minerva Urol Nefrol 61(2):121–128

    CAS  PubMed  Google Scholar 

  6. van Hove PD, Tuijthof GJ, Verdaasdonk EG, Stassen LP, Dankelman J (2010) Objective assessment of technical surgical skills. Br J Surg 97(7):972–987. doi:10.1002/bjs.7115

    Article  PubMed  Google Scholar 

  7. Bark K, McMahan W, Remington A, Gewirtz J, Wedmid A, Lee DI et al (2013) In vivo validation of a system for haptic feedback of tool vibrations in robotic surgery. Surg Endosc 27(2):656–664. doi:10.1007/s00464-012-2452-8

    Article  PubMed  Google Scholar 

  8. McMahan W, Gewirtz J, Standish D, Martin P, Kunkel JA, Lilavois M et al (2011) Tool contact acceleration feedback for telerobotic surgery. Haptics IEEE Trans 4(3):210–220

    Article  CAS  Google Scholar 

  9. Vassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondre K, Stanbridge D et al (2005) A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 190(1):107–113. doi:10.1016/j.amjsurg.2005.04.004

    Article  PubMed  Google Scholar 

  10. Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187(1):247–252. doi:10.1016/j.juro.2011.09.032

    Article  PubMed  Google Scholar 

  11. Peters J, Fried GM, Swanstrom LL, Soper NJ, Sillin LF, Schirmer B et al (2004) Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery. Surgery 135(1):21–27

    Article  PubMed  Google Scholar 

  12. Gomez ED, Bark K, Rivera C, McMahan W, Remington A, Lee DI et al (2012) Construct validity of instrument vibrations as a measure of robotic surgical skill. J Am Coll Surg 215(3):S119–S120

    Article  Google Scholar 

  13. LaMotte RH, Mountcastle VB (1975) Capacities of humans and monkeys to discriminate vibratory stimuli of different frequency and amplitude: a correlation between neural events and psychological measurements. J Neurophysiol 38(3):539–559

    CAS  PubMed  Google Scholar 

  14. Darzi A, Mackay S (2001) Assessment of surgical competence. Qual Health Care 10(Suppl 2):ii64–ii69

    PubMed  PubMed Central  Google Scholar 

  15. Rosen J, MacFarlane M, Richards C, Hannaford B, Sinanan M (1999) Surgeon-tool force/torque signatures—evaluation of surgical skills in minimally invasive surgery. Stud Health Technol Inform 62:290–296

    CAS  PubMed  Google Scholar 

  16. Rosen J, Hannaford B, Richards CG, Sinanan MN (2001) Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. IEEE Trans Biomed Eng 48(5):579–591. doi:10.1109/10.918597

    Article  CAS  PubMed  Google Scholar 

  17. Singapogu RB, Smith DE, Long LO, Burg TC, Pagano CC, Burg KJ (2012) Objective differentiation of force-based laparoscopic skills using a novel haptic simulator. J Surg Educ 69(6):766–773. doi:10.1016/j.jsurg.2012.07.008

    Article  PubMed  Google Scholar 

  18. Bark K, McMahan W, Remington A, Gewirtz J, Wedmid A, Lee DI, Kuchenbecker KJ (2013) In vivo validation of a system for haptic feedback of tool vibrations in robotic surgery. Surg Endosc 27(2):656–664

    Article  PubMed  Google Scholar 

  19. Aggarwal R, Crochet P, Dias A, Misra A, Ziprin P, Darzi A (2009) Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg 96(9):1086–1093

    Article  CAS  PubMed  Google Scholar 

  20. Reiley CE, Hager GD (2009) Task versus subtask surgical skill evaluation of robotic minimally invasive surgery. Med Image Comput Comput Assist Interv 12(Pt 1):435–442

    PubMed  Google Scholar 

  21. Dosis A, Bello F, Gillies D, Undre S, Aggarwal R, Darzi A (2005) Laparoscopic task recognition using hidden Markov models. Stud Health Technol Inform 111:115–122

    PubMed  Google Scholar 

  22. Gujarati D (2003) Multicollinearity: What happens if the regressors are correlated? Basic econometrics, 4th edn. McGraw-Hill, Newyork, p 363

    Google Scholar 

  23. Smith S, Torkington J, Brown T, Taffinder N, Darzi A (2002) Motion analysis. Surg Endosc 16(4):640–645

    Article  CAS  PubMed  Google Scholar 

  24. Gomez ED, McMahan W, Hashimoto DA, Brzezinski A, Bark K, Dumon KR, Lee DI, Williams NN, Kuchenbecker KJ (2013) Haptic feedback of instrument vibrations improves performance during robotic surgical simulation: a randomized controlled trial. J Am Coll Surg 217(3):S65–S66

    Article  Google Scholar 

  25. McMahan W, Gomez E. D, Chen L, Bark K, Nappo J. C, Koch E. I et al (2013) A practical system for recording instrument interactions during live robotic surgery. J Robot Surg 7(4):351–358

    Article  Google Scholar 

  26. Gomez ED, McMahan W, Koch E, Hashimoto DA, Darzi A, Murayama KM, Dumon KR, Williams NN, Kuchenbecker KJ (2013) Instrument contact vibrations are a construct-valid measure of technical skill in fundamentals of laparoscopic surgery training tasks. In: Association for Surgical Education Annual Meeting, Orlando, FL

Download references

Acknowledgments

The authors would like to acknowledge Charlotte Rivera, Daniel Hashimoto, and Dr. Andrew Cucchiara for contributing to data analysis. The authors would like to acknowledge Dr. Noel N. Williams, Dr. Kristoffel R. Dumon, Dr. Kenric M. Murayama, and Dr. David I. Lee for providing skill assessments of study videos. The authors would like to thank Jamie Gewirtz, Dorsey Standish, Paul Martin, Jacquelyn Kunkel, Magalie Lilavois, Dr. Alexei Wedmid, and Dr. David I. Lee for their contributions to data collection and analysis.

Author information

Authors and Affiliations

  1. Department of Otorhinolaryngology – Head and Neck Surgery, Hospital of the University of Pennsylvania, Philadelphia, PA, USA

    Ernest D. Gomez

  2. Department of Surgery, McGill University Health Centre, Philadelphia, PA, USA

    Rajesh Aggarwal

  3. Arnold and Blema Steinberg Medical Simulation Centre, Philadelphia, PA, USA

    Rajesh Aggarwal

  4. Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania, 224 Towne Building, 220 South 33rd Street, Philadelphia, PA, 19104, USA

    Ernest D. Gomez, William McMahan, Karlin Bark & Katherine J. Kuchenbecker

Authors
  1. Ernest D. Gomez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rajesh Aggarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. William McMahan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Karlin Bark
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Katherine J. Kuchenbecker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Ernest D. Gomez or Katherine J. Kuchenbecker.

Ethics declarations

Funding

This work was supported by the Pennsylvania Department of Health via Health Research Formula Funds, by the National Science Foundation via Grant #IIS-0845670, by the National Center for Research Resources and the National Center for Advancing Translational Sciences of the National Institutes of Health via Grant UL1TR000003, by a Translational Research Award from the Coulter Foundation, and by the University of Pennsylvania.

Disclosures

Ernest D. Gomez co-invented the use of the vibration metrics to assess surgical skill. This invention is described in pending patent applications that have not been licensed to any company. Rajesh Aggarwal is a consultant for Applied Medical. William McMahan co-invented the vibration feedback technology evaluated in this paper and the use of the vibration metrics to assess surgical skill. Both inventions are described in pending patent applications that have not been licensed to any company. Karlin Bark co-invented the use of the vibration metrics to assess surgical skill. Professor Katherine J. Kuchenbecker co-invented the vibration feedback technology evaluated in this paper and the use of the vibration metrics to assess surgical skill. Dr. Gomez, Dr. Aggarwal, Dr. McMahan, Dr. Bark and Professor Kuchenbecker have no conflicts of interest or financial ties to disclose. Subsets of the data in this work were presented at the Hamlyn Symposium on Medical Robotics, London, England, June 2012, and at the American College of Surgeons 98th Clinical Congress, Chicago, IL, October 2012.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (MP4 870 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gomez, E.D., Aggarwal, R., McMahan, W. et al. Objective assessment of robotic surgical skill using instrument contact vibrations. Surg Endosc 30, 1419–1431 (2016). https://doi.org/10.1007/s00464-015-4346-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-015-4346-z

Keywords

Search

Navigation