Using objective robotic automated performance metrics and task-evoked pupillary response to distinguish surgeon expertise



In this study, we investigate the ability of automated performance metrics (APMs) and task-evoked pupillary response (TEPR), as objective measures of surgeon performance, to distinguish varying levels of surgeon expertise during generic robotic surgical tasks. Additionally, we evaluate the association between APMs and TEPR.


Participants completed ten tasks on a da Vinci Xi Surgical System (Intuitive Surgical, Inc.), each representing a surgical skill type: EndoWrist® manipulation, needle targeting, suturing/knot tying, and excision/dissection. Automated performance metrics (instrument motion tracking, EndoWrist® articulation, and system events data) and TEPR were recorded by a systems data recorder (Intuitive Surgical, Inc.) and Tobii Pro Glasses 2 (Tobii Technologies, Inc.), respectively. The Kruskal–Wallis test determined significant differences between groups of varying expertise. Spearman’s rank correlation coefficient measured associations between APMs and TEPR.


Twenty-six participants were stratified by robotic surgical experience: novice (no prior experience; n = 9), intermediate (< 100 cases; n = 9), and experts (≥ 100 cases; n = 8). Several APMs differentiated surgeon experience including task duration (p < 0.01), time active of instruments (p < 0.03), linear velocity of instruments (p < 0.04), and angular velocity of dominant instrument (p < 0.04). Task-evoked pupillary response distinguished surgeon expertise for three out of four task types (p < 0.04). Correlation trends between APMs and TEPR revealed that expert surgeons move more slowly with high cognitive workload (ρ < − 0.60, p < 0.05), while novices move faster under the same cognitive experiences (ρ > 0.66, p < 0.05).


Automated performance metrics and TEPR can distinguish surgeon expertise levels during robotic surgical tasks. Furthermore, under high cognitive workload, there can be a divergence in robotic movement profiles between expertise levels.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2


  1. 1.

    Intuitive Surgical, Inc.: Annual Report 2018. Accessed 18 May 2019

  2. 2.

    Lerner MA, Ayalew M, Peine WJ, Sundaram CP (2010) Does training on a virtual reality robotic simulator improve performance on the da Vinci surgical system? J Endourol 24(3):467–472.

    Article  PubMed  Google Scholar 

  3. 3.

    Richstone L, Schwartz MJ, Seideman C, Cadeddu J, Marshall S, Kavoussi LR (2010) Eye metrics as an objective assessment of surgical skill. Ann Surg 252(1):177–182.

    Article  PubMed  Google Scholar 

  4. 4.

    Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJ, Michigan Bariatric Surgery C (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369(15):1434–1442.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  5. 5.

    Chen J, Cheng N, Cacciamani G, Oh P, Lin-Brande M, Remulla D, Gill IS, Hung AJ (2019) Objective assessment of robotic surgical technical skill: a systematic review. J Urol 201(3):461–469.

    Article  PubMed  PubMed Central  Google Scholar 

  6. 6.

    Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187(1):247–252.

    Article  PubMed  Google Scholar 

  7. 7.

    Hung AJ, Chen J, Jarc A, Hatcher D, Djaladat H, Gill IS (2018) Development and validation of objective performance metrics for robot-assisted radical prostatectomy: a pilot study. J Urol 199(1):296–304.

    Article  Google Scholar 

  8. 8.

    Ghani KR, Miller DC, Linsell S, Brachulis A, Lane B, Sarle R, Dalela D, Menon M, Comstock B, Lendvay TS, Montie J, Peabody JO, Michigan Urological Surgery Improvement C (2016) Measuring to improve: peer and crowd-sourced assessments of technical skill with robot-assisted radical prostatectomy. Eur Urol 69(4):547–550.

    Article  PubMed  PubMed Central  Google Scholar 

  9. 9.

    Hung AJ, Oh PJ, Chen J, Ghodoussipour S, Lane C, Jarc A, Gill IS (2019) Experts vs super-experts: differences in automated performance metrics and clinical outcomes for robot-assisted radical prostatectomy. BJU Int 123(5):861–868.

    Article  Google Scholar 

  10. 10.

    Hung AJ, Chen J, Ghodoussipour S, Oh PJ, Liu Z, Nguyen J, Purushotham S, Gill IS, Liu Y (2019) A deep-learning model using automated performance metrics and clinical features to predict urinary continence recovery after robot-assisted radical prostatectomy. BJU Int.

    Article  Google Scholar 

  11. 11.

    Dias RD, Ngo-Howard MC, Boskovski MT, Zenati MA, Yule SJ (2018) Systematic review of measurement tools to assess surgeons’ intraoperative cognitive workload. Br J Surg 105(5):491–501.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Ruiz-Rabelo JF, Navarro-Rodriguez E, Di-Stasi LL, Diaz-Jimenez N, Cabrera-Bermon J, Diaz-Iglesias C, Gomez-Alvarez M, Briceno-Delgado J (2015) Validation of the NASA-TLX score in ongoing assessment of mental workload during a laparoscopic learning curve in bariatric surgery. Obes Surg 25(12):2451–2456.

    Article  PubMed  Google Scholar 

  13. 13.

    Szulewski A, Roth N, Howes D (2015) The use of task-evoked pupillary response as an objective measure of cognitive load in novices and trained physicians: a new tool for the assessment of expertise. Acad Med 90(7):981–987.

    Article  PubMed  Google Scholar 

  14. 14.

    Abboudi H, Khan MS, Guru KA, Froghi S, de Win G, Van Poppel H, Dasgupta P, Ahmed K (2014) Learning curves for urological procedures: a systematic review. BJU Int 114(4):617–629.

    Article  PubMed  Google Scholar 

Download references


Research reported in this publication was supported in part by the National Institute Of Biomedical Imaging And Bioengineering of the National Institutes of Health under Award Number K23EB026493 and an Intuitive Surgical Clinical Research Grant. Anthony Jarc and Liheng Guo (Intuitive Surgical, Inc.) assisted with automated performance metric processing.

Author information




JHN: data collection/management, data analysis, manuscript writing/editing. JC: data collection/management, data analysis, manuscript writing/editing. SPM: data analysis. SG: manuscript writing/editing. AC: manuscript writing/editing. ISG: project development. AJH: project development, manuscript writing/editing.

Corresponding author

Correspondence to Andrew J. Hung.

Ethics declarations

Conflict of interest

The study was supported in part by an Intuitive Surgical, Inc. clinical grant. Intuitive Surgical, Inc. provided the systems events data recorder.

Research involving human participants and/or animals

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 30 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Nguyen, J.H., Chen, J., Marshall, S.P. et al. Using objective robotic automated performance metrics and task-evoked pupillary response to distinguish surgeon expertise. World J Urol 38, 1599–1605 (2020).

Download citation


  • Robotic surgical training
  • Surgeon assessment
  • Automated performance metrics
  • Task-evoked pupillary response