Skip to main content
SpringerLink
Log in

Towards reliable hepatocytic anatomy segmentation in laparoscopic cholecystectomy using U-Net with Auto-Encoder

  • Dynamic Manuscript
  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

Most bile duct (BDI) injuries during laparoscopic cholecystectomy (LC) occur due to visual misperception leading to the misinterpretation of anatomy. Deep learning (DL) models for surgical video analysis could, therefore, support visual tasks such as identifying critical view of safety (CVS). This study aims to develop a prediction model of CVS during LC. This aim is accomplished using a deep neural network integrated with a segmentation model that is capable of highlighting hepatocytic anatomy.

Methods

Still images from LC videos were annotated with four hepatocystic landmarks of anatomy segmentation. A deep autoencoder neural network with U-Net to investigate accurate medical image segmentation was trained and tested using fivefold cross-validation. Accuracy, Loss, Intersection over Union (IoU), Precision, Recall, and Hausdorff Distance were computed to evaluate the model performance versus the annotated ground truth.

Results

A total of 1550 images from 200 LC videos were annotated. Mean IoU for segmentation was 74.65%. The proposed approach performed well for automatic hepatocytic landmarks identification with 92% accuracy and 93.9% precision and can segment challenging cases.

Conclusion

DL, can potentially provide an intraoperative model for surgical video analysis and can be trained to guide surgeons toward reliable hepatocytic anatomy segmentation and produce selective video documentation of this safety step of LC.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Abbreviations

CVS:

Critical view of safety

LC:

Laparoscopic cholecystectomy

BDI:

Bile duct injury

DL:

Deep learning

AI:

Artificial Intelligence

CNN:

Convolutional neural network

References

  1. Shaffer EA (2005) Epidemiology and risk factors for gallstone disease: has the paradigm changed in the 21st century? Curr Gastroenterol Rep 7(2):132–140. https://doi.org/10.1007/s11894-005-0051-8

    Article  PubMed  Google Scholar 

  2. Buddingh KT, Weersma RK, Savenije RA, Van Dam GM, Nieuwenhuijs VB (2011) Lower rate of major bile duct injury and increased intraoperative management of common bile duct stones after implementation of routine intraoperative cholangiography. J Am Coll Surg 213(2):267–274. https://doi.org/10.1016/j.jamcollsurg.2011.03.004

    Article  PubMed  Google Scholar 

  3. Törnqvist B, Strömberg C, Persson G, Nilsson M (2012) Effect of intended intraoperative cholangiography and early detection of bile duct injury on survival after cholecystectomy: population based cohort study. BMJ (Clin Res Ed) 345:e6457. https://doi.org/10.1136/bmj.e6457

    Article  Google Scholar 

  4. Berci G, Hunter J, Morgenstern L, Arregui M, Mrunt M, Carroll B, Edye M, Fermelia D, Ferzli G, Greene F, Petelin J, Phillips E, Ponsky J, Sax H, Schwaitzberg S, Soper N, Swanstrom L, Rraverso W (2013) Laparoscopic cholecystectomy: first, do no harm; second, take care of bile duct stones. Surg Endosc 27(4):1051–1054. https://doi.org/10.1007/s00464-012-2767-5

    Article  PubMed  Google Scholar 

  5. Strasberg SM, Hertl M, Soper NJ (1995) An analysis of the problem of biliary injury during laparoscopic cholecystectomy. J Am Coll Surg 180(1):101–125

    CAS  PubMed  Google Scholar 

  6. Way LW, Stewart L, Gantert W, Liu K, Lee CM, Whang K, Hunter JG (2003) Causes and prevention of laparoscopic bile duct injuries: analysis of 252 cases from a human factors and cognitive psychology perspective. Ann Surg 237(4):460–469. https://doi.org/10.1097/01.sla.0000060680.92690.e9

    Article  PubMed  PubMed Central  Google Scholar 

  7. Mascagni P, Fiorillo C, Urade T, Emre T, Yu T, Wakabayashi T, Felli E, Perretta S, Swanstrom L, Mutter D, Marescaux J, Pessaux P, Costamagna G, Padoy N, Dallemagne B (2020) Formalizing video documentation of the critical view of safety in laparoscopic cholecystectomy: a step towards artificial intelligence assistance to improve surgical safety. Surg Endosc 34(6):2709–2714. https://doi.org/10.1007/s00464-019-07149-3

    Article  PubMed  Google Scholar 

  8. Nijssen MA, Schreinemakers JM, Meyer Z, Van Der Schelling GP, Crolla RM, Rijken AM (2015) Complications after laparoscopic cholecystectomy: a video evaluation study of whether the critical view of safety was reached. World J Surg 39(7):1798–1803. https://doi.org/10.1007/s00268-015-2993-9

    Article  CAS  PubMed  Google Scholar 

  9. Al-Masni MA, Kim DH (2021) CMM-Net: contextual multi-scale multi-level network for efficient biomedical image segmentation. Sci Rep 11(1):10191. https://doi.org/10.1038/s41598-021-89686-3

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Ronneberger O, Fischer P, Prox T (2015) U-Net: convolutional networks for biomedical image segmentation. International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 234–241

    Google Scholar 

  11. http://camma.u-strasbg.fr/datasets. Accessed 12 Jan 2023

  12. https://www.laparoscopyhospital.com/online_laparoscopic_videos.html. Accessed 3 Jan 2023

  13. https://www.youtube.com/c/sagesvideo/featured. Accessed 3 Jan 2023

  14. https://med.wmich.edu/node/66. Accessed 3 Jan 2023

  15. Mascagni P, Alapatt D, Garcia A, Okamoto N, Vardazaryan A, Costamagna G et al (2021) Surgical data science for safe cholecystectomy: a protocol for segmentation of hepatocystic anatomy and assessment of the critical view of safety. arXiv:2106.10916.

  16. Thomas L, Schaefer F, Gehrig J (2021) Fiji plugins for qualitative image annotations: routine analysis and application to image classification. F1000Research 9:1248

    Google Scholar 

  17. Perez L, Wang J (2017) The effectiveness of data augmentation in image classification using deep learning. arXiv:1712.04621v1

  18. https://keras.io/api/data_loading/image/. Keras Image Preprocessing. March 2015. Accessed 3 Nov 2021

  19. Goodfellow I, Bengio Y, Courville A, Bengio Y (2016) Deep learning. MIT Press, Cambridge

    Google Scholar 

  20. Ward TM, Fer DM, Ban Y, Rosman G, Meireles OR, Hashimoto DA (2021) Challenges in surgical video annotation. Comput Assist Surg 26(1):58–68

    Article  Google Scholar 

  21. Jing L, Tian Y (2020) Self-supervised visual feature learning with deep neural networks: a survey. IEEE Trans Pattern Anal Mach Intell 43(11):4037–4058

    Article  Google Scholar 

  22. Ferreira MF, Camacho R, Teixeira LF (2020) Using autoencoders as a weight initialization method on deep neural networks for disease detection. BMC Med Inform Decis Mak 20(Suppl 5):141. https://doi.org/10.1186/s12911-020-01150-w

    Article  PubMed  PubMed Central  Google Scholar 

  23. Badr W (2019) Auto-encoder: what is it? and what is it used for? (part 1). https://towardsdatascience.com/Auto-Encoder-what-is-it-and-what-is-it-used-for-part-1-3e5c6f017726. Accessed 3 Jan 2023

  24. https://colab.research.google.com/signup\. Access 3 Jan 2023

  25. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow IJ, Harp A, Irving G, Isard M, Jia Y, Józefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray DG, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker PA, Vanhoucke V, Vasudevan V, Viégas FB, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2016) TensorFlow: Large-scale machine learning on heterogeneous distributed systems. arXiv:1603.04467.

  26. Twinanda AP, Shehata S, Mutter D, Marescaux J, De Mathelin M, Padoy N (2017) Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36(1):86–97. https://doi.org/10.1109/tmi.2016.2593957

    Article  PubMed  Google Scholar 

  27. Liu X, Song L, Liu S, Zhang Y (2021) A review of deep-learning-based medical image segmentation methods. Sustainability 13(3):1224

    Article  Google Scholar 

  28. https://image-net.org/update-mar-11-2021.php. Accessed 3 Jan 2023

  29. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 9992–10002.

  30. Mascagni P, Vardazaryan A, Alapatt D, Urade T, Emre T, Fiorillo C et al (2022) Artificial intelligence for surgical safety: automatic assessment of the critical view of safety in laparoscopic cholecystectomy using deep learning. Ann Surg 275(5):955–961

    Article  PubMed  Google Scholar 

  31. Madani A, Namazi B, Altieri MS, Hashimoto DA, Rivera AM, Pucher PH, Navarrete-Welton A, Sankaranarayanan G, Brunt LM, Okrainec A, Alseidi A (2022) Artificial intelligence for intraoperative guidance: using semantic segmentation to identify surgical anatomy during laparoscopic cholecystectomy. Ann Surg 276(2):363–369. https://doi.org/10.1097/sla.0000000000004594

    Article  PubMed  Google Scholar 

  32. Namazi B, Sankaranarayanan G, Devarajan V, Fleshman J (2017) A deep learning system for automatically identifying critical view of safety in laparoscopic cholecystectomy videos for assessment. In: Sages 2017 annual meeting. Sages, Houston

  33. Tokuyasu T, Iwashita Y, Matsunobu Y, Kamiyama T, Ishikake M, Sakaguchi S, Ebe K, Tada K, Endo Y, Etoh T, Nakashima M, Inomata M (2021) Development of an artificial intelligence system using deep learning to indicate anatomical landmarks during laparoscopic cholecystectomy. Surg Endosc 35(4):1651–1658. https://doi.org/10.1007/s00464-020-07548-x

    Article  PubMed  Google Scholar 

  34. https://thinklikeasurgeon.ca/. Accessed 3 Jan 2023.

  35. Breheret A (2017) Pixel annotation tool. https://github.com/abreheret/pixelannotationtool. Accessed 3 Jan 2023.

Download references

Acknowledgements

The authors want to thank the Department of Surgery at Western Michigan University Homer Stryker M.D. School of Medicine for their support and consultation to create the database and this work.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Western Michigan University, Kalamazoo, MI, USA

    Koloud N. Alkhamaiseh, Janos L. Grantner & Ikhlas Abdel-Qader

  2. Western Michigan University Homer Stryker MD School of Medicine, Kalamazoo, MI, USA

    Saad Shebrain

Authors
  1. Koloud N. Alkhamaiseh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Janos L. Grantner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Saad Shebrain
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ikhlas Abdel-Qader
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Study conception and design: KNA. Acquisition of data: KNA, SS. Analysis and interpretation of data: KNA, SS, JLG, IA-Q. Drafting of manuscript: KNA, SS, JLG, IA-Q. Critical revision: SS, JLG, IA-Q.

Corresponding author

Correspondence to Koloud N. Alkhamaiseh.

Ethics declarations

Disclosures

Koloud N. Alkhamaiseh, Janos L. Grantner, Saad Shebrain, and Ikhlas Abdel-Qader have no conflicts of interest or financial ties to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alkhamaiseh, K.N., Grantner, J.L., Shebrain, S. et al. Towards reliable hepatocytic anatomy segmentation in laparoscopic cholecystectomy using U-Net with Auto-Encoder. Surg Endosc 37, 7358–7369 (2023). https://doi.org/10.1007/s00464-023-10306-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-023-10306-4

Keywords

Search

Navigation