Abstract
Background
Operating room planning is a complex task as pre-operative estimations of procedure duration have a limited accuracy. This is due to large variations in the course of procedures. Therefore, information about the progress of procedures is essential to adapt the daily operating room schedule accordingly. This information should ideally be objective, automatically retrievable and in real-time. Recordings made during endoscopic surgeries are a potential source of progress information. A trained observer is able to recognize the ongoing surgical phase from watching these videos. The introduction of deep learning techniques brought up opportunities to automatically retrieve information from surgical videos. The aim of this study was to apply state-of-the art deep learning techniques on a new set of endoscopic videos to automatically recognize the progress of a procedure, and to assess the feasibility of the approach in terms of performance, scalability and practical considerations.
Methods
A dataset of 33 laparoscopic cholecystectomies (LC) and 35 total laparoscopic hysterectomies (TLH) was used. The surgical tools that were used and the ongoing surgical phases were annotated in the recordings. Neural networks were trained on a subset of annotated videos. The automatic recognition of surgical tools and phases was then assessed on another subset. The scalability of the networks was tested and practical considerations were kept up.
Results
The performance of the surgical tools and phase recognition reached an average precision and recall between 0.77 and 0.89. The scalability tests showed diverging results. Legal considerations had to be taken into account and a considerable amount of time was needed to annotate the datasets.
Conclusion
This study shows the potential of deep learning to automatically recognize information contained in surgical videos. This study also provides insights in the applicability of such a technique to support operating room planning.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Eijkemans MJC, Van Houdenhoven M, Nguyen T et al (2010) Predicting the unpredictable: a new prediction model for operating room times using individual characteristics and the surgeon’s estimate. Anesthesiology. https://doi.org/10.1097/ALN.0b013e3181c294c2
Dexter F, Ph D, Epstein RH et al (2017) Making management decisions on the day of surgery based on operating room efficiency and patient waiting. J Am Soc Anesthesiol 101:1444–1453
Edelman ER, Van KSMJ, Hamaekers AEW et al (2017) Improving the prediction of total surgical procedure time using linear regression modeling. Front Med 4:1–5. https://doi.org/10.3389/fmed.2017.00085
van Eijk RPA, Van V-B, Kazemier G, Eijkemans MJC (2016) Effect of individual surgeons and anesthesiologists on operating room time. Anesth Anal. https://doi.org/10.1213/ANE.0000000000001430
Gupta N, Ranjan G, Arora MP et al (2013) Validation of a scoring system to predict difficult laparoscopic cholecystectomy. Int J Surg 11:1002–1006. https://doi.org/10.1016/j.ijsu.2013.05.037
Wiegmann DA, ElBardissi AW, Dearani JA et al (2007) Disruptions in surgical flow and their relationship to surgical errors: an exploratory investigation. Surgery 142:658–665. https://doi.org/10.1016/j.surg.2007.07.034
Arora S, Hull L, Sevdalis N et al (2010) Factors compromising safety in surgery: stressful events in the operating room. Am J Surg 199:60–65. https://doi.org/10.1016/j.amjsurg.2009.07.036
Blum T, Padoy N, Feußner H, Navab N (2008) Modeling and online recognition of surgical phases using hidden Markov models. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) LNCS 5242:627–635. https://doi.org/10.1007/978-3-540-85990-1-75
Guédon ACP, Paalvast M, Meeuwsen FC et al (2016) ‘It is Time to Prepare the Next patient’ Real-Time Prediction of Procedure Duration in Laparoscopic Cholecystectomies. J Med Syst. https://doi.org/10.1007/s10916-016-0631-1
Meeuwsen FC, van Luyn F, Blikkendaal MD et al (2019) Surgical phase modelling in minimal invasive surgery. Surg Endosc. https://doi.org/10.1007/s00464-018-6417-4
Padoy N, Blum T, Ahmadi SA et al (2012) Statistical modeling and recognition of surgical workflow. Med Image Anal 16:632–641. https://doi.org/10.1016/j.media.2010.10.001
Blum T, Feußner H, Navab N (2010) Modeling and segmentation of surgical workflow from laparoscopic video. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) LNCS 6363:400–407. https://doi.org/10.1007/978-3-642-15711-0_50
Bouarfa L, Jonker PP, Dankelman J (2011) Discovery of high-level tasks in the operating room. J Biomed Inform 44:455–462. https://doi.org/10.1016/j.jbi.2010.01.004
Lalys F, Riffaud L, Morandi X, Jannin P (2011) Surgical phases detection from microscope videos by combining SVM and HMM. In: Menze B, Langs G, Tu Z, Criminisi A (eds) Medical computer vision. Recognition techniques and applications in medical imaging. Lecture notes in computer science. Springer, Berlin
Volkov M, Hashimoto DA, Rosman G et al (2017) Machine learning and coresets for automated real-time video segmentation of laparoscopic and robot-assisted surgery. Proc IEEE Int Conf Robot Autom. https://doi.org/10.1109/ICRA.2017.7989093
Lalys F, Riffaud L, Bouget D, Jannin P (2012) A framework for the recognition of high-level surgical tasks from video images for cataract surgeries. IEEE Trans Biomed Eng 59:966–976. https://doi.org/10.1109/TBME.2011.2181168
Aksamentov I, Twinanda AP, Mutter D et al (2017) Deep neural networks predict remaining surgery duration from cholecystectomy videos. In: Descoteaux M, Maier-Hein L, Franz A, Jannin P, Collins D, Duchesne S (eds) Medical image computing and computer-assisted intervention—MICCAI 2017. Lecture notes in computer science. Springer, Cham
Twinanda AP, Shehata S, Mutter D et al (2017) EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36:86–97. https://doi.org/10.1109/TMI.2016.2593957
Twinanda AP, Yengera G, Mutter D et al (2018) RSDNet: learning to predict remaining surgery duration from laparoscopic videos without manual annotations. IEEE Trans Med Imaging. https://doi.org/10.1109/TMI.2018.2878055
Yu F, Croso SG et al (2019) Assessment of automated identification of phases in videos of cataract surgery using machine learning and deep learning techniques. JAMA Netw Open. https://doi.org/10.1001/jamanetworkopen.2019.1860
Blikkendaal MD, Driessen SRC, Rodrigues SP et al (2017) Surgical flow disturbances in dedicated minimally invasive surgery suites: an observational study to assess its supposed superiority over conventional suites. Surg Endosc. https://doi.org/10.1007/s00464-016-4971-1
Szegedy C, Vanhoucke V, Ioffe S et al (2016) Rethinking the inception architecture for computer vision. IEEE Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2016.308
Nwoye CI, Mutter D, Marescaux J, Padoy N (2019) Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos. Int J Comput Assist Radiol Surg. https://doi.org/10.1007/s11548-019-01958-6
Chen W, Feng J, Lu J, Zhou J (2018) Endo3D: online workflow analysis for endoscopic surgeries based on 3D CNN and LSTM. In: Stoyanov D et al (eds) OR 2.0 context-aware operating theaters, computer assisted robotic endoscopy clinical image-based procedures and skin image analysis. Lecture notes in computer science. Springer, Cham
Van Dalen ASHM, Legemaate J, Schlack WS et al (2019) Legal perspectives on black box recording devices in the operating environment. Br J Surg 106(11):1433–2144. https://doi.org/10.1002/bjs.11198
Gordon L, Grantcharov T, Rudzicz F (2019) Explainable artificial intelligence for safe intraoperative decision support. JAMA Surg 154(11):1064–1065. https://doi.org/10.1001/jamasurg.2019.2821
Funding
No funding.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Disclosures
Annetje C. P. Guédon, Senna E. P. Meij, Karim N. M. M. H. Osman, Helena A. Kloosterman, Karlijn J. van Stralen, Matthijs C. M. Grimbergen, Quirijn A. J. Eijsbouts, John J. van den Dobbelsteen, Andru P. Twinanda have no conflict of interest or financial ties to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Guédon, A.C.P., Meij, S.E.P., Osman, K.N.M.M.H. et al. Deep learning for surgical phase recognition using endoscopic videos. Surg Endosc 35, 6150–6157 (2021). https://doi.org/10.1007/s00464-020-08110-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00464-020-08110-5