Skip to main content
SpringerLink
Log in

Airway label prediction in video bronchoscopy: capturing temporal dependencies utilizing anatomical knowledge

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Navigation guidance is a key requirement for a multitude of lung interventions using video bronchoscopy. State-of-the-art solutions focus on lung biopsies using electromagnetic tracking and intraoperative image registration w.r.t. preoperative CT scans for guidance. The requirement of patient-specific CT scans hampers the utilization of navigation guidance for other applications such as intensive care units.

Methods

This paper addresses bronchoscope tracking by solely incorporating video data. In contrast to state-of-the-art approaches, we entirely omit the use of electromagnetic tracking and patient-specific CT scans to avoid changes in clinical workflows and additional hardware requirements in intensive care units. Guidance is enabled by means of topological bronchoscope localization w.r.t. a generic airway model. Particularly, we take maximally advantage of anatomical constraints of airway trees being sequentially traversed. This is realized by incorporating sequences of CNN-based airway likelihoods into a hidden Markov model.

Results

Our approach is evaluated based on multiple experiments inside a lung phantom model. With the consideration of temporal context and use of anatomical knowledge for regularization, we are able to improve the accuracy up to to 0.98 compared to 0.81 (weighted F1: 0.98 compared to 0.81) for a classification based on individual frames.

Conclusion

We combine CNN-based single image classification of airway segments with anatomical constraints and temporal HMM-based inference for the first time. Our approach shows first promising results in vision-based guidance for bronchoscopy interventions in the absence of electromagnetic tracking and patient-specific CT scans.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Yoo JY, Kang SY, Park JS, Cho YJ, Park SY, Yoon HI, Park SJ, Jeong HG, Kim T (2021) Deep learning for anatomical interpretation of video bronchoscopy images. Sci Rep 11(1):23765. https://doi.org/10.1038/s41598-021-03219-6

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Eberhardt R, Kahn N, Gompelmann D, Schumann M, Heussel CP, Herth FJF (2010) LungPoint-A new approach to peripheral lesions. J Thorac Oncol 5(10):1559–1563. https://doi.org/10.1097/JTO.0b013e3181e8b308

    Article  PubMed  Google Scholar 

  3. Smith BM, Traboulsi H, Austin JHM, Manichaikul A, Hoffman EA, Bleecker ER, Cardoso WV, Cooper C, Couper DJ, Dashnaw SM, Guo J, Han MK, Hansel NN, Hughes EW, Jacobs DR, Kanner RE, Kaufman JD, Kleerup E, Lin CL, Liu K, Lo Cascio CM, Martinez FJ, Nguyen JN, Prince MR, Rennard S, Rich SS, Simon L, Sun Y, Watson KE, Woodruff PG, Baglole CJ, Barr RG, MESA Lung and SPIROMICS investigators (2018) Human airway branch variation and chronic obstructive pulmonary disease. PNAS 115(5):E974–E981. https://doi.org/10.1073/pnas.1715564115

  4. Falta F, Hansen L, Himstedt M, Heinrich MP (2022) Learning an Airway Atlas from lung CT using semantic inter-patient deformable registration. In: Proceeding of BVM. Springer, pp 75–80. https://doi.org/10.1007/978-3-658-36932-3_15

  5. Sganga J (2019) Autonomous navigation of a flexible surgical robot in the lungs. PhD thesis Stanford University

  6. Mori K, Hasegawa J, Suenaga Y, Toriwaki J (2000) Automated anatomical labeling of the bronchial branch and its application to the virtual bronchoscopy system. IEEE Trans Med Imaging 19(2):103–114. https://doi.org/10.1109/42.836370

    Article  CAS  PubMed  Google Scholar 

  7. Mori K, Deguchi D, Sugiyama J, Suenaga Y, Toriwaki J, Maurer CR Jr, Takabatake H, Natori H (2002) Tracking of a bronchoscope using epipolar geometry analysis and intensity-based image registration of real and virtual endoscopic images. Med Image Anal 6(3):321–336. https://doi.org/10.1016/S1361-8415(02)00089-0

    Article  CAS  PubMed  Google Scholar 

  8. Nagao J, Mori K, Enjouji T, Deguchi D, Kitasaka T, Suenaga Y, Hasegawa Ji, Toriwaki Ji, Takabatake H, Natori H (2004) Fast and accurate bronchoscope tracking using image registration and motion prediction. In: MICCAI 2004. Springer, pp 551–558. https://doi.org/10.1007/978-3-540-30136-3_68

  9. Deligianni F, Chung A, Yang GZ (2004) Patient-specific bronchoscope simulation with pq-space-based 2D/3D registration. Comput Aided Surg 9(5):215–226. https://doi.org/10.3109/10929080500144927

    Article  PubMed  Google Scholar 

  10. Reichl T, Luo X, Menzel M, Hautmann H, Mori K, Navab N (2013) Hybrid electromagnetic and image-based tracking of endoscopes with guaranteed smooth output. IJCARS. 8:955–965. https://doi.org/10.1007/s11548-013-0835-5

    Article  Google Scholar 

  11. Reichl T, Luo X, Menzel M, Hautmann H, Mori K, Navab N (2011) Deformable registration of bronchoscopic video sequences to CT volumes with guaranteed smooth output. In: MICCAI 2011. Springer, pp 17–24. https://doi.org/10.1007/978-3-642-23623-5_3

  12. Deligianni F, Chung AJ, Yang GZ (2006) Nonrigid 2-D/3-D registration for patient specific bronchoscopy simulation with statistical shape modeling: Phantom validation. IEEE Trans Med Imaging 25(11):1462–1471. https://doi.org/10.1109/TMI.2006.883452

    Article  PubMed  Google Scholar 

  13. Sganga J, Eng D, Graetzel C, Camarillo D (2019) Offsetnet: deep learning for localization in the lung using rendered images. IEEE Int Conf Robot Autom IEEE 5046–5052. https://doi.org/10.1109/ICRA.2019.8793940

  14. Shen M, Gu Y, Liu N, Yang GZ (2019) Context-aware depth and pose estimation for bronchoscopic navigation. IEEE Robot Autom Lett. 4(2):732–739. https://doi.org/10.1109/LRA.2019.2893419

    Article  Google Scholar 

  15. Zhao C, Shen M, Sun L, Yang GZ (2019) Generative localization with uncertainty estimation through video-CT data for bronchoscopic biopsy. IEEE Robot Autom Lett. 5(1):258–265. https://doi.org/10.1109/LRA.2019.2955941

    Article  Google Scholar 

  16. Banach A, King F, Masaki F, Tsukada H, Hata N (2021) Visually navigated bronchoscopy using three cycle-consistent generative adversarial network for depth estimation. Med Image Anal 73:102164. https://doi.org/10.1016/j.media.2021.102164

    Article  PubMed  PubMed Central  Google Scholar 

  17. Wang C, Oda M, Hayashi Y, Villard B, Kitasaka T, Takabatake H, Mori M, Honma H, Natori H, Mori K (2020) A visual SLAM-based bronchoscope tracking scheme for bronchoscopic navigation. IJCARS. 15(10):1619–1630. https://doi.org/10.1007/s11548-020-02241-9

    Article  Google Scholar 

  18. Visentini-Scarzanella M, Sugiura T, Kaneko T, Koto S (2017) Deep monocular 3D reconstruction for assisted navigation in bronchoscopy. IJCARS. 12(7):1089–1099. https://doi.org/10.1007/S11548-017-1609-2

    Article  Google Scholar 

  19. Borrego-Carazo J, Sanchez C, Castells-Rufas D, Carrabina J, Gil D (2023) BronchoPose: an analysis of data and model configuration for vision-based bronchoscopy pose estimation. Comput Methods Prog Biomed 228:107241. https://doi.org/10.1016/j.cmpb.2022.107241

    Article  Google Scholar 

  20. Howard A, Sandler M, Chen B, Wang W, Chen LC, Tan M, Chu G, Vasudevan V, Zhu Y, Pang R, Adam H, Le Q (2019) Searching for MobileNetV3. In: Proceedings of the IEEE conference on computer vision and pattern Recognition, pp 1314–1324

  21. Keuth R, Heinrich M, Eichenlaub M, Himstedt M (2023) Weakly supervised airway orifice segmentation in video bronchoscopy. In: Med Imaging 2023: Image Process. vol. 12464. International Society for Optics and Photonics. SPIE. pp 124640A. https://doi.org/10.1117/12.2654229

  22. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern Recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  23. Prince SJD (2012) Computer vision: models, learning, and inference, 1st edn. Cambridge University Press

    Book  Google Scholar 

  24. Guo C, Pleiss G, Sun Y, Weinberger KQ (2017) On calibration of modern neural networks. In: Precup D, Teh YW (eds) Proceeding of ICML. vol 70. PMLR, pp 1321–1330. https://proceedings.mlr.press/v70/guo17a.html

  25. Aho AV, Hopcroft JE, Ullman JD (1974) The design and analysis of computer algorithms. Addison-Wesley series in computer science and information processing. Addison-Wesley Pub. Co, Reading

  26. Liu DC, Nocedal J (1989) On the limited memory BFGS method for large scale optimization. Math Program 45(1):503–528. https://doi.org/10.1007/BF01589116

Download references

Author information

Authors and Affiliations

  1. Medical Informatics, University of Lübeck, Ratzeburger Allee 160, 23562, Lübeck, Germany

    Ron Keuth, Mattias Heinrich & Marian Himstedt

  2. Clinic of Pneumology, University Hospital Freiburg, Germany, Breisacher Straße 153, 79110, Freiburg, Germany

    Martin Eichenlaub

Authors
  1. Ron Keuth
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mattias Heinrich
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Martin Eichenlaub
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marian Himstedt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ron Keuth.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Keuth, R., Heinrich, M., Eichenlaub, M. et al. Airway label prediction in video bronchoscopy: capturing temporal dependencies utilizing anatomical knowledge. Int J CARS 19, 713–721 (2024). https://doi.org/10.1007/s11548-023-03050-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-023-03050-6

Keywords

Search

Navigation