Abstract
Purpose
A clear surgical field of view is a prerequisite for successful laparoscopic surgery. Surgical smoke, image blur, and lens fogging can affect the clarity of laparoscopic imaging. We aimed to develop a real-time assistance system (namely LVQIS) for removing these interfering factors during laparoscopic surgery, thereby improving laparoscopic video quality.
Methods
LVQIS was developed with generative adversarial networks (GAN) and transfer learning, which included two classification models (ResNet-50), a motion blur removal model (MPRNet), and a smoke/fog removal model (GAN). 136 laparoscopic surgery videos were retrospectively collected in a tripartite dataset for training and validation. A synthetic dataset was simulated using the image enhancement library Albumentations and the 3D rendering software Blender. The objective evaluation results were through PSNR, SSIM and FID, and the subjective evaluation includes the operation pause time and the degree of anxiety of surgeons.
Results
The synthesized dataset contained 19,245 clear images, 19,245 motion blur images, and 19,245 smoke/fog images. The ResNet-50 CNN model identified whether a single laparoscopic image had motion blur and smoke/fog with an accuracy of over 0.99. The PSNR, SSIM and FID of the de-smoke model were 29.67, 0.9551 and 74.72, respectively, and the PSNR, SSIM and FID of the de-blurring model were 26.78, 0.9020 and 80.10, respectively, which were better than other advanced de-blurring and de-smoke/fog models. In a comparative study of 100 laparoscopic surgeries, the use of LVQIS significantly reduced the operation pause time (P < 0.001) and the anxiety of surgeons (P = 0.004).
Conclusions
In this study, LVQIS is an efficient and robust system that can improve the quality of laparoscopic video, reduce surgical pause time and the anxiety of surgeons, and has the potential for real-time application in real clinical settings.
Similar content being viewed by others
Data availability
Code and data to be released with this paper.
Change history
05 November 2022
Small error in the Discussion section, penultimate paragraph: the word has been from changed “Interfering” to “interfering”
Abbreviations
- DL:
-
Deep learning
- GAN:
-
Generative adversarial networks
- CNN:
-
Convolutional neural network
- SSIM:
-
Structural similarity index measure
- PSNR:
-
Peak signal-to-noise ratio
- FID:
-
Frechet inception distance
- VRT:
-
Video restoration transformer
- AOD-NET:
-
All-in-one dehazing network
- FFA-NET:
-
Feature fusion attention network
- EDN-GTM:
-
Encoder-decoder network for transfer graphs
- DW-GAN:
-
Discrete wavelet transform GAN
References
Johnson A (1997) Laparoscopic surgery. Lancet 349:631–635
Trilling B, Mancini A, Fiard G, Barraud PA, Decrouez M, Vijayan S, Tummers M, Faucheron JL, Silvent S, Schwartz C, Voros S (2021) Improving vision for surgeons during laparoscopy: the enhanced laparoscopic vision system (ELViS). SURG ENDOSC 35:2403–2415
Lawrentschuk N, Fleshner NE, Bolton DM (2010) Laparoscopic lens fogging: a review of etiology and methods to maintain a clear visual field. J Endourol 24:905–913
Stoyanov D (2012) Surgical vision. Ann Biomed Eng 40:332–345
Ha HI, Choi MC, Jung SG, Joo WD, Lee C, Song SH, Park H (2019) Chemicals in surgical smoke and the efficiency of built-in-filter ports. JSLS 23:e201900037
Choi SH, Kwon TG, Chung SK, Kim TH (2014) Surgical smoke may be a biohazard to surgeons performing laparoscopic surgery. Surg Endosc 28:2374–2380
Beckmeier L, Klapdor R, Soergel P, Kundu S, Hillemanns P, Hertel H (2014) Evaluation of active camera control systems in gynecological surgery: construction, handling, comfort, surgeries and results. Arch Gynecol Obstet 289:341–348
Song T, Lee DH (2020) A randomized Comparison of laparoscopic LEns defogging using Anti-fog solution, waRm saline, and chlorhexidine solution (CLEAR). Surg Endosc 34:940–945
Baid A, Kotwal A, Bhalodia R, Merchant SN, Awate SP (2017), IEEE: joint desmoking, specularity removal, and denoising of laparoscopy images via graphical models and bayesian inference. In: 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017), IEEE 14th international symposium on biomedical imaging (ISBI) - From Nano to Macro, pp 732–736
Kotwal A, Bhalodia R, Awate SP (2016) IEEE: joint desmoking and denoising of laparoscopy images. In: 2016 IEEE 13th international symposium on biomedical imaging (ISBI), 13th IEEE International Symposium on Biomedical Imaging (ISBI), pp 1050–1054
Luo XB, McLeod AJ, Pautler SE, Schlachta CM, Peters TM (2017) Vision-based surgical field defogging. IEEE T Med Imaging 36:2021–2030
Shin J, Kim M, Paik J, Lee S (2020) Radiance-reflectance combined optimization and structure-guided $\ell _0$-norm for single image dehazing. IEEE T Multimedia 22:30–44
Wang C, Cheikh FA, Kaaniche M, Elle OJ (2018) A smoke removal method for laparoscopic images. arXiv preprint arXiv:1803.08410
Wang C, Alaya CF, Kaaniche M, Beghdadi A, Elle OJ (2018) Variational based smoke removal in laparoscopic images. Biomed Eng Online 17:139
Gu L, Liu PL, Jiang CH, Luo M, Xu Q (2015) Virtual digital defogging technology improves laparoscopic imaging quality. Surg Innov 22:171–176
Zhu QS, Mai JM, Shao L (2015) A fast single image haze removal algorithm using color attenuation prior. IEEE T Image Process 24:3522–3533
Tchaka K, Pawar VM, Stoyanov D (2017) Chromaticity Based Smoke Removal in Endoscopic Images. In: Medical Imaging 2017: Image Processing. Styner MA and Angelini ED (eds.), Conference on Medical Imaging - Image Processing
Bolkar S, Wang CC, Cheikh FA, Yildirim S (2018) IEEE: deep smoke removal from minimally invasive surgery videos. In: 2018 25th IEEE international conference on image processing (ICIP), 25th IEEE International Conference on Image Processing (ICIP), pp 3403–3407
Vishal V, Sharma N, Singh M (2019) Guided Unsupervised Desmoking of Laparoscopic Images Using Cycle-Desmoke. In: OR 2.0 context-aware operating theaters and machine learning in clinical neuroimaginG. Zhou L, Sarikaya D and Kia SM, et al. (eds.), 2nd International Workshop on Context-Aware Surgical Theaters (OR)/2nd International Workshop on Machine Learning in Clinical Neuroimaging (MLCN), pp 21–28
Vishal V, Lochan K, Venkatesh V, Singh M, Sharma N (2020) Unsupervised Desmoking of Laparoscopy Images Using Multi-scale DesmokeNet. Springer, Cham
Wang CC, Mohammed AK, Cheikh FA, Beghdadi A, Elle OJ (2019) Multiscale deep desmoking for laparoscopic surgery. In: Medical Imaging 2019: Image Processing. Angelini ED and Landman BA (eds.), Conference on Medical Imaging: Image Processing
Pan Y, Bano S, Vasconcelos F, Park H, Jeong TT, Stoyanov D (2022) DeSmoke-LAP: improved unpaired image-to-image translation for desmoking in laparoscopic surgery. Int J Comput Assist Radiol Surg 17:885–893
Mascagni P, Vardazaryan A, Alapatt D, Urade T, Emre T, Fiorillo C, Pessaux P, Mutter D, Marescaux J, Costamagna G, Dallemagne B, Padoy N (2022) Artificial intelligence for surgical safety: automatic assessment of the critical view of safety in laparoscopic cholecystectomy using deep learning. Ann Surg 275:955–961
Zheng Q, Yang R, Yang S, Ni X, Li Y, Jiang Z, Wang X, Wang L, Chen Z, Liu X (2022) Development and validation of a deep-learning based assistance system for enhancing laparoscopic control level. Int J Med Robot Comput Assist Surg : MRCAS. https://doi.org/10.1002/rcs.2449
Zhang J, Gao X (2020) Object extraction via deep learning-based marker-free tracking framework of surgical instruments for laparoscope-holder robots. Int J Comput Assist Radiol Surg 15:1335–1345
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Advances in neural information processing systems, vol 27. Curran, Red Hook, NY
Lu T, Chen T, Gao F, Sun B, Ntziachristos V, Li J (2021) LV-GAN: a deep learning approach for limited-view optoacoustic imaging based on hybrid datasets. J BIOPHOTONICS 14:e202000325
Ma Y, Liu J, Liu Y, Fu H, Hu Y, Cheng J, Qi H, Wu Y, Zhang J, Zhao Y (2021) Structure and illumination constrained gan for medical image enhancement. IEEE Trans Med Imaging 40:3955–3967
Liu T, de Haan K, Rivenson Y, Wei Z, Zeng X, Zhang Y, Ozcan A (2019) Deep learning-based super-resolution in coherent imaging systems. Sci Rep 9:3926
He KM, Zhang XY, Ren SQ, Sun J (2016) IEEE: Deep Residual Learning for Image Recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 770–778, 2016
Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang MH, Shao L (2021) IEEE CS: Multi-Stage Progressive Image Restoration. In: 2021 IEEE/CVF conference on computer vision and pattern recognition, CVPR 2021, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 14816–14826, 2021
Salazar-Colores S, Jiménez HM, Ortiz-Echeverri CJ, Flores G (2020) Desmoking laparoscopy surgery images using an image-to-image translation guided by an embedded dark channel. IEEE ACCESS 8:208898–208909
Twinanda AP, Shehata S, Mutter D, Marescaux J, Padoy N (2016) EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos. IEEE T MED IMAGING 36:86–97
Stauder R, Ostler D, Kranzfelder M, Koller S, Feußner H, Navab N (2016) The TUM LapChole dataset for the M2CAI 2016 workflow challenge. arXiv preprint arXiv:1610.09278.
Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13:600–612
Hensel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S 2017 GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In: advances in neural information processing systems 30 (NIPS 2017). Guyon I, Luxburg UV and Bengio S, et al. (eds.), 31st Annual Conference on Neural Information Processing Systems (NIPS)
Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Kupyn O, Budzan V, Mykhailych M, Mishkin D, Matas J (2018) IEEE: DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR), 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 8183–8192
Kupyn O, Martyniuk T, Wu JR, Wang ZY (2019) IEEE: DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better. In: 2019 IEEE/CVF international conference on computer vision (ICCV 2019), IEEE/CVF International Conference on Computer Vision (ICCV), pp 8877–8886
Liang J, Cao J, Fan Y, Zhang K, Ranjan R, Li Y, Timofte R, Van Gool L (2022) Vrt: A video restoration transformer. arXiv preprint arXiv:2201.12288.
Li BY, Peng XL, Wang ZY, Xu JZ, Feng D (2017) IEEE: AOD-Net: All-in-One Dehazing Network. In: 2017 IEEE international conference on computer vision (ICCV), 16th IEEE International Conference on Computer Vision (ICCV), pp 4780–4788
Qin X, Wang ZL, Bai YC, Xie XD, Jia HZ (2020) Assoc AAI: FFA-Net: Feature Fusion Attention Network for Single Image Dehazing. In: thirty-fourth AAAI conference on artificial intelligence, the thirty-second innovative applications of artificial intelligence conference and the tenth AAAI symposium on educational advances in artificial intelligence, 34th AAAI Conference on Artificial Intelligence/32nd Innovative Applications of Artificial Intelligence Conference/10th AAAI Symposium on Educational Advances in Artificial Intelligence, pp 11908–11915
Tran L, Moon S, Park D (2022) A novel encoder-decoder network with guided transmission map for single image dehazing. arXiv preprint arXiv:2202.04757.
Fu MH, Liu H, Yu YK, Chen J, Wang KY (2021) IEEE CS: DW-GAN: A Discrete Wavelet Transform GAN for NonHomogeneous Dehazing. In: 2021 IEEE/CVF conference on computer vision and pattern recogition workshops (CVPRW 2021), IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 203–212
Chen L, Tang W, John NW, Wan TR, Zhang JJ (2020) De-smokeGCN: generative cooperative networks for joint surgical smoke detection and removal. IEEE Trans Med Imaging 39:1615–1625
Gu L, Liu PL, Zhou H, Xu Q (2018) A pilot study for a better visibility in the 3D laparoscopic right colectomy surgery. World J Surg 42:1872–1876
Chen L, Tang W, John NW, Wan TR, Zhang JJ (2018) SLAM-based dense surface reconstruction in monocular minimally invasive surgery and its application to augmented reality. Comput Meth Prog Bio 158:135–146
Acknowledgements
We acknowledge the grant from the Project of Hubei Province Key Research and Development Project of China (Grant No. 2020BCB051), the Ministry of Education Industry-University Cooperation and Collaborative Education Project of China (Grant No. 202102012021), and the National Medical Education Development Center Medical Simulation Education Research Project of China (Grant No. 2021MNYB11) for supporting the work.
Author information
Authors and Affiliations
Contributions
XHL reports being Director and the sponsor of this study. QYZ and RY performed the research, analysed the data, and wrote the paper. XMN, SY, ZYJ, LW and ZYC contributed essential tools.
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that they have no conflicts of interest with regard to the work presented in this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Visual comparison of laparoscopic video clip with and without LVQIS: https://youtu.be/-0pR_3JmaPw.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zheng, Q., Yang, R., Ni, X. et al. Development and validation of a deep learning-based laparoscopic system for improving video quality. Int J CARS 18, 257–268 (2023). https://doi.org/10.1007/s11548-022-02777-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-022-02777-y