Abstract
Purpose
Classic methods of surgery skills evaluation tend to classify the surgeon performance in multi-categorical discrete classes. If this classification scheme has proven to be effective, it does not provide in-between evaluation levels. If these intermediate scoring levels were available, they would provide more accurate evaluation of the surgeon trainee.
Methods
We propose a novel approach to assess surgery skills on a continuous scale ranging from 1 to 5. We show that the proposed approach is flexible enough to be used either for scores of global performance or several sub-scores based on a surgical criteria set called Objective Structured Assessment of Technical Skills (OSATS). We established a combined CNN+BiLSTM architecture to take advantage of both temporal and spatial features of kinematic data. Our experimental validation relies on real-world data obtained from JIGSAWS database. The surgeons are evaluated on three tasks: Knot-Tying, Needle-Passing and Suturing. The proposed framework of neural networks takes as inputs a sequence of 76 kinematic variables and produces an output float score ranging from 1 to 5, reflecting the quality of the performed surgical task.
Results
Our proposed model achieves high-quality OSATS scores predictions with means of Spearman correlation coefficients between the predicted outputs and the ground-truth outputs of 0.82, 0.60 and 0.65 for Knot-Tying, Needle-Passing and Suturing, respectively. To our knowledge, we are the first to achieve this regression performance using the OSATS criteria and the JIGSAWS kinematic data.
Conclusion
An effective deep learning tool was created for the purpose of surgical skills assessment. It was shown that our method could be a promising surgical skills evaluation tool for surgical training programs.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Tevis SE, Kennedy GD (2013) Postoperative complications and implications on patient-centered outcomes. J Surg Res 181(1):106–113
Semel ME, Lipsitz SR, Funk LM (2012) Rates and patterns of death after surgery in the united states, 1996 and 2006. Surgery 151(2):171–182
Sweeney JF (2013) Postoperative complications and hospital readmissions in surgical patients: an important association. Ann Surg 258(1):19
Lawson EH, Hall BL, Louie R (2013) Association between occurrence of a postoperative complication and readmission: implications for quality improvement and cost savings. Ann Surg 258(1):10–18
Polavarapu HV, Kulaylat AN, Sun S (2013) 100 years of surgical education: the past, present, and future. Bull Am Coll Surg 98(7):22–27
Kassahun Y, Yu B, Tibebu AT (2016) Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions. Int J Comput Assist Radiol Surg 11(4):553–568
Satava RM (1993) Virtual reality surgical simulator. Surg Endosc 7(3):203–205
Sun LW, Van Meer F, Bailly Y (2007) Design and development of a da vinci surgical system simulator. In: 2007 International conference on mechatronics and automation, IEEE, pp 1050–1055
DiMaio S, Hanuschik M, Kreaden U (2011) The da Vinci surgical system. Springer, Boston, pp 199–217
Levin M, McKechnie T, Khalid S (2019) Automated methods of technical skill assessment in surgery: a systematic review. J Surg Educ 76(6):1629–1639
Ahmidi N, Hager GD, Ishii L (2010) Surgical task and skill classification from eye tracking and tool motion in minimally invasive surgery. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 295–302
Asif H, McInnis C, Dang F (2021) Objective structured assessment of technical skill (osats) in the surgical skills and technology elective program (sstep): Comparison of peer and expert raters. Am J Surg. https://doi.org/10.1016/j.amjsurg.2021.03.064
Martin J, Regehr G, Reznick R (1997) Objective structured assessment of technical skill (osats) for surgical residents. J Br Surg 84(2):273–278
Lubke GH, Muthén BO (2004) Applying multigroup confirmatory factor models for continuous outcomes to likert scale data complicates meaningful group comparisons. Struct Equ Model 11(4):514–534
Sullivan GM, Artino AR Jr (2013) Analyzing and interpreting data from likert-type scales. J Grad Med Educ 5(4):541–542
Jamieson S (2004) Likert scales: how to (ab) use them? Med Educ 38(12):1217–1218
Carifio J, Perla R (2008) Resolving the 50-year debate around using and misusing likert scales
Benmansour M, Handouzi W, Malti A (2018) Task-specific surgical skill assessment with neural networks. In: International conference on advanced intelligent systems for sustainable development, Springer, pp 159–167
Benmansour M, Malti A (2019) Skills evaluation of specific surgical tasks using long short term memory networks. In: International conference on advanced intelligent systems for sustainable development, Springer, pp 331–339
Benmansour M, Malti A (2018) Simple and efficient recurrent neural network to evaluate classified surgery tasks. In: 5th International conference on automation, control engineering and computer science-ACECS
Khalid S, Goldenberg M, Grantcharov T (2020) Evaluation of deep learning models for identifying surgical actions and measuring performance. JAMA Netw open 3(3):e201664
Kowalewski KF, Garrow C, Schmidt M (2019) Sensor-based machine learning for workflow detection and as key to detect expert level in laparoscopic suturing and knot-tying. Surg Endosc. https://doi.org/10.1007/s00464-019-06667-4
Liu D, Jiang T, Wang Y, (2019b) Surgical skill assessment on in-vivo clinical data via the clearness of operating field. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 476–484
Hung A, Chen J, Che Z (2018) Utilizing machine learning and automated performance metrics to evaluate robot-assisted radical prostatectomy performance and predict outcomes. J Endourol. https://doi.org/10.1089/end.2018.0035
Ismail Fawaz H, Forestier G, Weber J (2019) Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks. Int J Comput Assist Radiol Surg 14(9):1611–1617
Pascanu R, Mikolov T, Bengio Y (2012) Understanding the exploding gradient problem. CoRR abs/1211.5063. arXiv:1211.5063
Gao Y, Vedula SS, Reiley CE, (2014) Jhu-isi gesture and skill assessment working set (jigsaws): a surgical activity dataset for human motion modeling. In: MICCAI workshop: M2cai
Wang Z, Majewicz Fey A (2018) Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery. Int J Comput Assist Radiol Surg 13(12):1959–1970
Zia A, Essa I (2017) Automated surgical skill assessment in rmis training. Int J Comput Assist Radiol Surg. https://doi.org/10.1007/s11548-018-1735-5
Chollet F (2015) Keras. https://keras.io, https://github.com/fchollet/keras
Acknowledgements
This research did not receive any specific Grant from funding agencies in the public, commercial, or not-for-profit sectors.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
For this type of study, formal consent is not required.
Informed consent
This article does not contain patient data.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Benmansour, M., Malti, A. & Jannin, P. Deep neural network architecture for automated soft surgical skills evaluation using objective structured assessment of technical skills criteria. Int J CARS 18, 929–937 (2023). https://doi.org/10.1007/s11548-022-02827-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-022-02827-5