Skip to main content
SpringerLink
Log in

Using objective robotic automated performance metrics and task-evoked pupillary response to distinguish surgeon expertise

  • Topic Paper
  • Published:
World Journal of Urology Aims and scope Submit manuscript

Abstract

Purpose

In this study, we investigate the ability of automated performance metrics (APMs) and task-evoked pupillary response (TEPR), as objective measures of surgeon performance, to distinguish varying levels of surgeon expertise during generic robotic surgical tasks. Additionally, we evaluate the association between APMs and TEPR.

Methods

Participants completed ten tasks on a da Vinci Xi Surgical System (Intuitive Surgical, Inc.), each representing a surgical skill type: EndoWrist® manipulation, needle targeting, suturing/knot tying, and excision/dissection. Automated performance metrics (instrument motion tracking, EndoWrist® articulation, and system events data) and TEPR were recorded by a systems data recorder (Intuitive Surgical, Inc.) and Tobii Pro Glasses 2 (Tobii Technologies, Inc.), respectively. The Kruskal–Wallis test determined significant differences between groups of varying expertise. Spearman’s rank correlation coefficient measured associations between APMs and TEPR.

Results

Twenty-six participants were stratified by robotic surgical experience: novice (no prior experience; n = 9), intermediate (< 100 cases; n = 9), and experts (≥ 100 cases; n = 8). Several APMs differentiated surgeon experience including task duration (p < 0.01), time active of instruments (p < 0.03), linear velocity of instruments (p < 0.04), and angular velocity of dominant instrument (p < 0.04). Task-evoked pupillary response distinguished surgeon expertise for three out of four task types (p < 0.04). Correlation trends between APMs and TEPR revealed that expert surgeons move more slowly with high cognitive workload (ρ < − 0.60, p < 0.05), while novices move faster under the same cognitive experiences (ρ > 0.66, p < 0.05).

Conclusions

Automated performance metrics and TEPR can distinguish surgeon expertise levels during robotic surgical tasks. Furthermore, under high cognitive workload, there can be a divergence in robotic movement profiles between expertise levels.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Intuitive Surgical, Inc.: Annual Report 2018. http://www.annualreports.com/Company/intuitive-surgical-inc. Accessed 18 May 2019

  2. Lerner MA, Ayalew M, Peine WJ, Sundaram CP (2010) Does training on a virtual reality robotic simulator improve performance on the da Vinci surgical system? J Endourol 24(3):467–472. https://doi.org/10.1089/end.2009.0190

    Article  PubMed  Google Scholar 

  3. Richstone L, Schwartz MJ, Seideman C, Cadeddu J, Marshall S, Kavoussi LR (2010) Eye metrics as an objective assessment of surgical skill. Ann Surg 252(1):177–182. https://doi.org/10.1097/SLA.0b013e3181e464fb

    Article  PubMed  Google Scholar 

  4. Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJ, Michigan Bariatric Surgery C (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369(15):1434–1442. https://doi.org/10.1056/NEJMsa1300625

    Article  CAS  PubMed  Google Scholar 

  5. Chen J, Cheng N, Cacciamani G, Oh P, Lin-Brande M, Remulla D, Gill IS, Hung AJ (2019) Objective assessment of robotic surgical technical skill: a systematic review. J Urol 201(3):461–469. https://doi.org/10.1016/j.juro.2018.06.078

    Article  PubMed  Google Scholar 

  6. Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187(1):247–252. https://doi.org/10.1016/j.juro.2011.09.032

    Article  PubMed  Google Scholar 

  7. Hung AJ, Chen J, Jarc A, Hatcher D, Djaladat H, Gill IS (2018) Development and validation of objective performance metrics for robot-assisted radical prostatectomy: a pilot study. J Urol 199(1):296–304. https://doi.org/10.1016/j.juro.2017.07.081

    Article  PubMed  Google Scholar 

  8. Ghani KR, Miller DC, Linsell S, Brachulis A, Lane B, Sarle R, Dalela D, Menon M, Comstock B, Lendvay TS, Montie J, Peabody JO, Michigan Urological Surgery Improvement C (2016) Measuring to improve: peer and crowd-sourced assessments of technical skill with robot-assisted radical prostatectomy. Eur Urol 69(4):547–550. https://doi.org/10.1016/j.eururo.2015.11.028

    Article  PubMed  Google Scholar 

  9. Hung AJ, Oh PJ, Chen J, Ghodoussipour S, Lane C, Jarc A, Gill IS (2019) Experts vs super-experts: differences in automated performance metrics and clinical outcomes for robot-assisted radical prostatectomy. BJU Int 123(5):861–868. https://doi.org/10.1111/bju.14599

    Article  PubMed  Google Scholar 

  10. Hung AJ, Chen J, Ghodoussipour S, Oh PJ, Liu Z, Nguyen J, Purushotham S, Gill IS, Liu Y (2019) A deep-learning model using automated performance metrics and clinical features to predict urinary continence recovery after robot-assisted radical prostatectomy. BJU Int. https://doi.org/10.1111/bju.14735

    Article  PubMed  PubMed Central  Google Scholar 

  11. Dias RD, Ngo-Howard MC, Boskovski MT, Zenati MA, Yule SJ (2018) Systematic review of measurement tools to assess surgeons’ intraoperative cognitive workload. Br J Surg 105(5):491–501. https://doi.org/10.1002/bjs.10795

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Ruiz-Rabelo JF, Navarro-Rodriguez E, Di-Stasi LL, Diaz-Jimenez N, Cabrera-Bermon J, Diaz-Iglesias C, Gomez-Alvarez M, Briceno-Delgado J (2015) Validation of the NASA-TLX score in ongoing assessment of mental workload during a laparoscopic learning curve in bariatric surgery. Obes Surg 25(12):2451–2456. https://doi.org/10.1007/s11695-015-1922-1

    Article  PubMed  Google Scholar 

  13. Szulewski A, Roth N, Howes D (2015) The use of task-evoked pupillary response as an objective measure of cognitive load in novices and trained physicians: a new tool for the assessment of expertise. Acad Med 90(7):981–987. https://doi.org/10.1097/ACM.0000000000000677

    Article  PubMed  Google Scholar 

  14. Abboudi H, Khan MS, Guru KA, Froghi S, de Win G, Van Poppel H, Dasgupta P, Ahmed K (2014) Learning curves for urological procedures: a systematic review. BJU Int 114(4):617–629. https://doi.org/10.1111/bju.12315

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Research reported in this publication was supported in part by the National Institute Of Biomedical Imaging And Bioengineering of the National Institutes of Health under Award Number K23EB026493 and an Intuitive Surgical Clinical Research Grant. Anthony Jarc and Liheng Guo (Intuitive Surgical, Inc.) assisted with automated performance metric processing.

Author information

Authors and Affiliations

  1. Center for Robotic Simulation and Education, Catherine and Joseph Aresty Department of Urology, University of Southern California Institute of Urology, 1441 Eastlake Avenue, Suite 7416, Los Angeles, CA, 90033, USA

    Jessica H. Nguyen, Jian Chen, Saum Ghodoussipour, Andrew Chen, Inderbir S. Gill & Andrew J. Hung

  2. EyeTracking, Inc., 512 Via De La Valle, Suite 200, Solana Beach, CA, 92075, USA

    Sandra P. Marshall

Authors
  1. Jessica H. Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jian Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sandra P. Marshall
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Saum Ghodoussipour
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andrew Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Inderbir S. Gill
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Andrew J. Hung
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JHN: data collection/management, data analysis, manuscript writing/editing. JC: data collection/management, data analysis, manuscript writing/editing. SPM: data analysis. SG: manuscript writing/editing. AC: manuscript writing/editing. ISG: project development. AJH: project development, manuscript writing/editing.

Corresponding author

Correspondence to Andrew J. Hung.

Ethics declarations

Conflict of interest

The study was supported in part by an Intuitive Surgical, Inc. clinical grant. Intuitive Surgical, Inc. provided the systems events data recorder.

Research involving human participants and/or animals

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 30 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nguyen, J.H., Chen, J., Marshall, S.P. et al. Using objective robotic automated performance metrics and task-evoked pupillary response to distinguish surgeon expertise. World J Urol 38, 1599–1605 (2020). https://doi.org/10.1007/s00345-019-02881-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00345-019-02881-w

Keywords

Search

Navigation