Image-Based Smoke Detection in Laparoscopic Videos

  • Andreas Leibetseder
  • Manfred Jürgen Primus
  • Stefan Petscharnig
  • Klaus Schoeffmann
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10550)

Abstract

The development and improper removal of smoke during minimally invasive surgery (MIS) can considerably impede a patient’s treatment, while additionally entailing serious deleterious health effects. Hence, state-of-the-art surgical procedures employ smoke evacuation systems, which often still are activated manually by the medical staff or less commonly operate automatically utilizing industrial, highly-specialized and operating room (OR) approved sensors. As an alternate approach, video analysis can be used to take on said detection process – a topic not yet much researched in aforementioned context. In order to advance in this sector, we propose utilizing an image-based smoke classification task on a pre-trained convolutional neural network (CNN). We provide a custom data set of over 30 000 laparoscopic smoke/non-smoke images, part of which served as training data for GoogLeNet-based [41] CNN models. To be able to compare our research for evaluation, we separately developed a non-CNN classifier based on observing the saturation channel of a sample picture in the HSV color space. While the deep learning approaches yield excellent results with Receiver Operating Characteristic (ROC) curves enclosing areas of over 0.98, the computationally much less costly analysis of an image’s saturation histogram under certain circumstances can, surprisingly, as well be a good indicator for smoke with areas under the curves (AUCs) of around 0.92–0.97.

Keywords

Smoke detection Endoscopy Image processing Deep learning 

References

  1. 1.
    Linux mint 17.3 “rosa” - cinnamon (64-bit) (2006). https://linuxmint.com/edition.php?id=204. Accessed 28 Mar 2017
  2. 2.
    Lightning memory-mapped database (2016). https://symas.com/offerings/lightning-memory-mapped-database. Accessed 28 Mar 2017
  3. 3.
    OpenCV library (2017). http://opencv.org/
  4. 4.
    Python programming language (2017). https://www.python.org/
  5. 5.
    Al Sahaf, O.S., Vega-Carrascal, I., Cunningham, F.O., McGrath, J.P., Bloomfield, F.J.: Chemical composition of smoke produced by high-frequency electrosurgery. Irish J. Med. Sci. 176(3), 229–232 (2007)CrossRefGoogle Scholar
  6. 6.
    Anthimopoulos, M., Christodoulidis, S., Ebner, L., Christe, A., Mougiakakou, S.: Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans. Med. Imaging 35(5), 1207–1216 (2016). http://ieeexplore.ieee.org CrossRefGoogle Scholar
  7. 7.
    Ball, K.: Controlling surgical smoke: A team approach. Information Booklet (2004). http://www.megadyne.com/pdf/Kay-Ball-Smoke-Booklet.pdf
  8. 8.
    Calderara, S., Piccinini, P., Cucchiara, R.: Vision based smoke detection system using image energy and color information. Mach. Vis. Appl. 22(4), 705–719 (2011). http://link.springer.com/10.1007/s00138-010-0272-1 CrossRefGoogle Scholar
  9. 9.
    Chen-Rui Chou, M.C.L.: System and Method for Smoke Detection During Anatomical Surgery (2016). https://www.google.com/patents/US20160239967
  10. 10.
    Choi, S.H., Kwon, T.G., Chung, S.K., Kim, T.H.: Surgical smoke may be a biohazard to surgeons performing laparoscopic surgery. Surg. Endosc. Interv. Tech. 28(8), 2374–2380 (2014)CrossRefGoogle Scholar
  11. 11.
    Chunyu, Y., Jun, F., Jinjun, W., Yongming, Z.: Video fire smoke detection using motion and color features. Fire Technol. 46(3), 651–663 (2010). http://link.springer.com/10.1007/s10694-009-0110-z CrossRefGoogle Scholar
  12. 12.
    Cosmescu, I.: Automatic smoke evacuator system for a surgical laser apparatus and method therefor (1991). https://www.google.com/patents/US5199944
  13. 13.
    Cosmescu, I.: Automatic smoke evacuator and insufflation system for surgical procedures (2006). https://www.google.com/patents/US20070249990
  14. 14.
    Dobrogowski, M., Wesołowski, W., Kucharska, M., Sapota, A., Pomorski, L.: Chemical composition of surgical smoke formed in the abdominal cavity during laparoscopic cholecystectomy—assessment of the risk to the patient. Int. J. Occup. Med. Environ. Health 27(2), 314–325 (2014). http://ijomeh.eu/Chemical-composition-of-surgical-smoke-formed-in-the-abdominal-cavity-during-laparoscopic-cholecystectomy-assessment-of-the-risk-to-the-patient,2054,0,2.html CrossRefGoogle Scholar
  15. 15.
    Ferrari, R.J., Zhang, H., Kube, C.R.: Real-time detection of steam in video images. Pattern Recogn. 40(3), 1148–1159 (2007)CrossRefMATHGoogle Scholar
  16. 16.
    Gubbi, J., Marusic, S., Palaniswami, M.: Smoke detection in video using wavelets and support vector machines. Fire Saf. J. 44(8), 1110–1115 (2009)CrossRefGoogle Scholar
  17. 17.
    Häfner, M., Gangl, A., Liedlgruber, M., Uhl, A., Vécsei, A., Wrba, F.: Combining Gaussian Markov random fields with the discretewavelet transform for endoscopic image classification. In: Proceedings of the DSP 2009: 16th International Conference on Digital Signal Processing (2009)Google Scholar
  18. 18.
    Hafner, M., Gangl, A., Liedlgruber, M., Uhl, A., Vecsei, A., Wrba, F.: Endoscopic image classification using edge-based features. In: 2010 20th International Conference on Pattern Recognition, pp. 2724–2727. IEEE, August 2010. http://ieeexplore.ieee.org/document/5597011/
  19. 19.
    Häfner, M., Liedlgruber, M., Uhl, A., Vécsei, A., Wrba, F.: Color treatment in endoscopic image classification using multi-scale local color vector patterns. Med. Image Anal. 16(1), 75–86 (2012). http://www.sciencedirect.com/science/article/pii/S1361841511000569 CrossRefGoogle Scholar
  20. 20.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition, December 2015. http://arxiv.org/abs/1512.03385
  21. 21.
    Hensman, C., Baty, D., Willis, R., Cuschieri, A.: Chemical composition of smoke produced by high-frequency electrosurgery in a closed gaseous environment. Surg. Endosc. 12, 1017 (1998). http://www.springerlink.com/index/3PDVCC89D248BJT0.pdf CrossRefGoogle Scholar
  22. 22.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on Multimedia, pp. 675–678. ACM (2014)Google Scholar
  23. 23.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  24. 24.
    Kolesov, I., Karasev, P., Tannenbaum, A., Haber, E.: Fire and smoke detection in video with optimal mass transport based optical flow and neural networks. In: 2010 IEEE International Conference on Image Processing, pp. 761–764. IEEE, September 2010. http://ieeexplore.ieee.org/document/5652119/
  25. 25.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet Classification with Deep Convolutional Neural Networks, pp. 1097–1105. Curran Associates Inc., Nevada (2012). http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf Google Scholar
  26. 26.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105. Curran Associates, Inc., Nevada (2012). http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
  27. 27.
    Kudo, S., Hirota, S., Nakajima, T., Hosobe, S., Kusaka, H., Kobayashi, T., Himori, M., Yagyuu, A.: Colorectal tumours and pit pattern. J. Clin. Pathol. 47(10), 880–885 (1994). http://www.ncbi.nlm.nih.gov/pubmed/7962600, http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC502170 CrossRefGoogle Scholar
  28. 28.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  29. 29.
    Lee, C.Y., Lin, C.T., Hong, C.T., Su, M.T.: Smoke detection using spatial and temporal analyses. Int. J. Innov. Comput. Inf. Control 8(7A), 4749–4770 (2012)Google Scholar
  30. 30.
    Li, Q., Cai, W., Wang, X., Zhou, Y., Feng, D.D., Chen, M.: Medical image classification with convolutional neural network. In: 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV), pp. 844–848. IEEE, December 2014. http://ieeexplore.ieee.org/document/7064414/
  31. 31.
    Liedlgruber, M., Uhl, A.: Endoscopic image processing - an overview. In: 2009 Proceedings of 6th International Symposium on Image and Signal Processing and Analysis, pp. 707–712. IEEE, September 2009. http://ieeexplore.ieee.org/document/5297635/
  32. 32.
    Buffalo Filter LLC: Surgical Smoke: Education and Training (2017). http://www.buffalofilter.com/files/7914/1443/3525/Website_Training__Education_Section_10_27_2014.pdf
  33. 33.
    Loukas, C., Georgiou, E.: Smoke detection in endoscopic surgery videos: a first step towards retrieval of semantic events: smoke detection in endoscopic surgery videos. Int. J. Med. Robot. Comput. Assist. Surg. 11(1), 80–94 (2015). http://doi.wiley.com/10.1002/rcs.1578 CrossRefGoogle Scholar
  34. 34.
    Mattes, D., Silajdzic, E., Mayer, M., Horn, M., Scheidbach, D., Wackernagel, W., Langmann, G., Wedrich, A.: Surgical smoke management for minimally invasive (micro)endoscopy: an experimental study. Surg. Endosc. Interv. Tech. 24(10), 2492–2501 (2010)CrossRefGoogle Scholar
  35. 35.
    Menes, T., Spivak, H.: Laparoscopy: searching for the proper insufflation gas. Surg. Endosc. 14(11), 1050–1056 (2000). http://www.ncbi.nlm.nih.gov/pubmed/11116418 CrossRefGoogle Scholar
  36. 36.
    Ojo, J., Oladosu, J.: Video-based smoke detection algorithms: a chronological survey. Comput. Eng. Intell. Syst. 5(7), 38–50 (2014)Google Scholar
  37. 37.
    Ott, D.: Smoke production and smoke reduction in endoscopic surgery: preliminary report. Endosc. Surg. Allied Technol. 1(4), 230–232 (1993). http://www.ncbi.nlm.nih.gov/pubmed/8050026 Google Scholar
  38. 38.
    Park, S.Y., Sargent, D.: Colonoscopic polyp detection using convolutional neural networks. In: International Society for Optics and Photonics, p. 978528, March 2016. http://proceedings.spiedigitallibrary.org/proceeding.aspx?doi=10.1117/12.2217148
  39. 39.
    Petscharnig, S., Schöffmann, K.: Deep learning for shot classification in gynecologic surgery videos. In: Amsaleg, L., Guðmundsson, G.Þ., Gurrin, C., Jónsson, B.Þ., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10132, pp. 702–713. Springer, Cham (2017). doi:10.1007/978-3-319-51811-4_57 CrossRefGoogle Scholar
  40. 40.
    Sahiner, B., Chan, H.-P., Petrick, N., Wei, D., Helvie, M., Adler, D., Goodsitt, M.: Classification of mass and normal breast tissue: a convolution neural network classifier with spatial domain and texture images. IEEE Trans. Med. Imaging 15(5), 598–610 (1996). http://ieeexplore.ieee.org/document/538937/ CrossRefGoogle Scholar
  41. 41.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)Google Scholar
  42. 42.
    Takahashi, H., Yamasaki, M., Hirota, M., Miyazaki, Y., Moon, J.H., Souma, Y., Mori, M., Doki, Y., Nakajima, K.: Automatic smoke evacuation in laparoscopic surgery: a simplified method for objective evaluation. Surg. Endosc. 27(8), 2980–2987 (2013). http://link.springer.com/10.1007/s00464-013-2821-y CrossRefGoogle Scholar
  43. 43.
    Thiébaud, H.P., Knize, M.G., Kuzmicky, P.A., Hsieh, D.P., Felton, J.S.: Airborne mutagens produced by frying beef, pork and a soy-based food. Food Chem. Toxicol. 33(10), 821–828 (1995)CrossRefGoogle Scholar
  44. 44.
    Tian, H., Li, W., Wang, L., Ogunbona, P.: A novel video-based smoke detection method using image separation. In: Proceedings - IEEE International Conference on Multimedia and Expo, pp. 532–537 (2012)Google Scholar
  45. 45.
    Toreyin, B.U., Dedeoglu, Y., Cetin, A.E.: Contour Based Smoke Detection in Video Using Wavelets, pp. 1–5. IEEE (2006)Google Scholar
  46. 46.
    Tsui, C., Klein, R., Garabrant, M.: Minimally invasive surgery: national trends in adoption and future directions for hospital strategy. Surg. Endosc. 27(7), 2253–2257 (2013). http://link.springer.com/10.1007/s00464-013-2973-9 CrossRefGoogle Scholar
  47. 47.
    Wu, S., Yuan, F., Yang, Y., Fang, Z., Fang, Y.: Real-time image smoke detection using staircase searching-based dual threshold AdaBoost and dynamic analysis. IET Image Process. 9(10), 849–856 (2015). http://digital-library.theiet.org/content/journals/10.1049/iet-ipr.2014.1032 CrossRefGoogle Scholar
  48. 48.
    Yan, Z., Zhan, Y., Peng, Z., Liao, S., Shinagawa, Y., Zhang, S., Metaxas, D.N., Zhou, X.S.: Multi-Instance deep learning: discover discriminative local anatomies for bodypart recognition. IEEE Trans. Med. Imaging 35(5), 1332–1343 (2016). http://ieeexplore.ieee.org/document/7398101/ CrossRefGoogle Scholar
  49. 49.
    Yuan, F.: Video-based smoke detection with histogram sequence of LBP and LBPV pyramids. Fire Saf. J. 46(3), 132–139 (2011)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Andreas Leibetseder
    • 1
  • Manfred Jürgen Primus
    • 1
  • Stefan Petscharnig
    • 1
  • Klaus Schoeffmann
    • 1
  1. 1.Institute of Information TechnologyAlpen-Adria UniversityKlagenfurtAustria

Personalised recommendations