Explaining a model predicting quality of surgical practice: a first presentation to and review by clinical experts



Surgical Data Science (SDS) is an emerging research domain offering data-driven answers to challenges encountered by clinicians during training and practice. We previously developed a framework to assess quality of practice based on two aspects: exposure of the surgical scene (ESS) and the surgeon’s profile of practice (SPP). Here, we wished to investigate the clinical relevance of the parameters learned by this model by (1) interpreting these parameters and identifying associated representative video samples and (2) presenting this information to surgeons in the form of a video-enhanced questionnaire. To our knowledge, this is the first approach in the field of SDS for laparoscopy linking the choices made by a machine learning model predicting surgical quality to clinical expertise.


Spatial features and quality of practice scores extracted from labeled and segmented frames in 30 laparoscopic videos were used to predict the ESS and the SPP. The relationships between the inputs and outputs of the model were then analyzed and translated into meaningful sentences (statements, e.g., “To optimize the ESS, it is very important to correctly handle the spleen”). Representative video clips illustrating these statements were semi-automatically identified. Eleven statements and video clips were used in a survey presented to six experienced digestive surgeons to gather their opinions on the algorithmic analyses.


All but one of the surgeons agreed with the proposed questionnaire overall. On average, surgeons agreed with 7/11 statements.


This proof-of-concept study provides preliminary validation of our model which has a high potential for use to analyze and understand surgical practices.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6


  1. 1.

    Derathé A, Reche F, Moreau-Gaudry A, Jannin P, Gibaud B, Voros S (2019) Predicting the quality of surgical exposure using spatial and procedural features from laparoscopic videos. Int J Comput Assist Radiol Surg. https://doi.org/10.1007/s11548-019-02072-3

    Article  PubMed  Google Scholar 

  2. 2.

    El Ahmadieh TY, Harrop J, Batjer HH, Resnick DK, Bendok BR (2014) Blinded peer assessment of surgical skill is feasible and can predict complication rates: a step toward measuring surgical quality. Neurosurgery 74(6):N12–14. https://doi.org/10.1227/01.neu.0000450232.06740.ef

    Article  PubMed  Google Scholar 

  3. 3.

    Foster JD, Miskovic D, Allison AS, Conti JA, Ockrim J, Cooper EJ, Hanna GB, Francis NK (2016) Application of objective clinical human reliability analysis (OCHRA) in assessment of technical performance in laparoscopic rectal cancer surgery. Tech Coloproctol 20(6):361–367. https://doi.org/10.1007/s10151-016-1444-4

    CAS  Article  PubMed  Google Scholar 

  4. 4.

    Gagner M, Hutchinson C, Rosenthal R (2016) Fifth international consensus conference: current status of sleeve gastrectomy. Surg Obes Relat Dis 12(4):750–756. https://doi.org/10.1016/j.soard.2016.01.022

    Article  PubMed  Google Scholar 

  5. 5.

    Gordon L, Grantcharov T, Rudzicz F (2019) Explainable artificial intelligence for safe intraoperative decision support. JAMA Surg 154(11):1064–1065. https://doi.org/10.1001/jamasurg.2019.2821

    Article  PubMed  Google Scholar 

  6. 6.

    Huaulmé A, Jannin P, Reche F, Faucheron JL, Moreau-Gaudry A, Voros S (2020) Offline identification of surgical deviations in laparoscopic rectopexy. Artif Intell Med 104:101837. https://doi.org/10.1016/j.artmed.2020.101837

    Article  PubMed  Google Scholar 

  7. 7.

    Lundberg SM, Nair B, Vavilala MS, Horibe M, Eisses MJ, Adams T, Liston DE, Low DKW, Newman SF, Kim J, Lee SI (2018) Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nat Biomed Eng 2(10):749–760. https://doi.org/10.1038/s41551-018-0304-0

    Article  PubMed  PubMed Central  Google Scholar 

  8. 8.

    Maier-Hein L, Vedula SS, Speidel S, Navab N, Kikinis R, Park A, Eisenmann M, Feussner H, Forestier G, Giannarou S, Hashizume M, Katic D, Kenngott H, Kranzfelder M, Malpani A, März K, Neumuth T, Padoy N, Pugh C, Schoch N, Stoyanov D, Taylor R, Wagner M, Hager GD, Jannin P (2017) Surgical data science for next-generation interventions. Nat Biomed Eng 1(9):691–696. https://doi.org/10.1038/s41551-017-0132-7

    Article  PubMed  Google Scholar 

  9. 9.

    Malpani A, Vedula SS, Chen CCG, Hager GD (2015) A study of crowdsourced segment-level surgical skill assessment using pairwise rankings. Int J Comput Assist Radiol Surg 10(9):1435–1447. https://doi.org/10.1007/s11548-015-1238-6

    Article  PubMed  Google Scholar 

  10. 10.

    Mangano A, Kim HY, Wu CW, Rausei S, Hui S, Xiaoli L, Chiang FY, Roukos DH, Lianos GD, Volpi E, Dionigi G (2016) Continuous intraoperative neuromonitoring in thyroid surgery: safety analysis of 400 consecutive electrode probe placements with standardized procedures. Head Neck 38(S1):E1568–E1574. https://doi.org/10.1002/hed.24280

    Article  PubMed  Google Scholar 

  11. 11.

    Mascagni P, Vardazaryan A, Alapatt D, Urade T, Emre T, Fiorillo C, Pessaux P, Mutter D, Marescaux J, Costamagna G, Dallemagne B, Padoy N (2020) Artificial intelligence for surgical safety: automatic assessment of the critical view of safety in laparoscopic cholecystectomy using deep learning. Ann Surg. https://doi.org/10.1097/SLA.0000000000004351

    Article  PubMed  Google Scholar 

  12. 12.

    Pernek I, Ferscha A (2017) A survey of context recognition in surgery. Med Biol Eng Comput 55(10):1719–1734. https://doi.org/10.1007/s11517-017-1670-6

    Article  PubMed  Google Scholar 

  13. 13.

    Radivojac P, Obradovic Z, Dunker AK, Vucetic S (2004) Feature selection filters based on the permutation test. In: Boulicaut JF, Esposito F, Giannotti F, Pedreschi D (eds) Machine learning: ECML 2004. Lecture notes in computer science. Springer, Berlin, pp 334–346. https://doi.org/10.1007/978-3-540-30115-8-32

    Chapter  Google Scholar 

  14. 14.

    Rätsch G, Sonnenburg S, Schäer C (2006) Learning interpretable SVMs for biological sequence classification. BMC Bioinform 7(1):S9. https://doi.org/10.1186/1471-2105-7-S1-S9

    CAS  Article  Google Scholar 

  15. 15.

    The COlon cancer Laparoscopic or Open Resection Study Group (2005) Laparoscopic surgery versus open surgery for colon cancer: short-term outcomes of a randomised trial. Lancet Oncol 6(7):477–484. https://doi.org/10.1016/S1470-2045(05)70221-7

    Article  PubMed  Google Scholar 

  16. 16.

    Thiele RH, Rea KM, Turrentine FE, Friel CM, Hassinger TE, Goudreau BJ, Umapathi BA, Kron IL, Sawyer RG, Hedrick TL, McMurry TL (2015) Standardization of care: impact of an enhanced recovery protocol on length of stay, complications, and direct costs after colorectal surgery. J Am Coll Surg 220(4):430–443. https://doi.org/10.1016/j.jamcollsurg.2014.12.042

    Article  PubMed  Google Scholar 

  17. 17.

    Zhang W, Wang L, Wang Z (2019) Interpretable machine learning study of the many-body localization transition in disordered quantum Ising spin chains. Phys Rev B 99(5):054208. https://doi.org/10.1103/PhysRevB.99.054208

    CAS  Article  Google Scholar 

Download references


This work was supported by funding from the French government managed by the ANR as part of the Investissements d’Avenir Programme (Labex CAMI) under reference ANR-11-LABX-0004. This work was partially supported by MIAI @ Grenoble Alpes, (ANR-19-P3IA-0003). The authors thank the IRT b\(<>\)com for providing the “Surgery Workflow Toolbox [annotate],” software used in this study.

Author information



Corresponding author

Correspondence to Sandrine Voros.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest to declare.

Ethical approval

All procedures performed in the study presented involving human participants were carried out in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Declaration of Helsinki and its later amendments, or comparable ethical standards.

Informed consent

The study protocol was approved by the institutional ethics committee on May 24, 2018 (CECIC Rhône-Alpes-Auvergne, Clermont-Ferrand, IRB 5891).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Derathé, A., Reche, F., Jannin, P. et al. Explaining a model predicting quality of surgical practice: a first presentation to and review by clinical experts. Int J CARS (2021). https://doi.org/10.1007/s11548-021-02422-0

Download citation


  • Surgical skills
  • Video-based assessment
  • Explainable artificial intelligence