Skip to main content

Advertisement

SpringerLink
Log in

Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Optical imaging is evolving as a key technique for advanced sensing in the operating room. Recent research has shown that machine learning algorithms can be used to address the inverse problem of converting pixel-wise multispectral reflectance measurements to underlying tissue parameters, such as oxygenation. Assessment of the specific hardware used in conjunction with such algorithms, however, has not properly addressed the possibility that the problem may be ill-posed.

Methods

We present a novel approach to the assessment of optical imaging modalities, which is sensitive to the different types of uncertainties that may occur when inferring tissue parameters. Based on the concept of invertible neural networks, our framework goes beyond point estimates and maps each multispectral measurement to a full posterior probability distribution which is capable of representing ambiguity in the solution via multiple modes. Performance metrics for a hardware setup can then be computed from the characteristics of the posteriors.

Results

Application of the assessment framework to the specific use case of camera selection for physiological parameter estimation yields the following insights: (1) estimation of tissue oxygenation from multispectral images is a well-posed problem, while (2) blood volume fraction may not be recovered without ambiguity. (3) In general, ambiguity may be reduced by increasing the number of spectral bands in the camera.

Conclusion

Our method could help to optimize optical camera design in an application-specific manner.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Alerstam E, Lo WCY, Han TD, Rose J, Andersson-Engels S, Lilge L (2010) Next-generation acceleration and code optimization for light transport in turbid media using GPUs. Biomed Opt Express 1(2):658–675

    Article  PubMed  PubMed Central  Google Scholar 

  2. Ardizzone L, Kruse J, Rother C, Köthe U (2019) Analyzing inverse problems with invertible neural networks. In: International conference on learning representations. https://openreview.net/forum?id=rJed6j0cKX. Accessed 3 Mar 2019

  3. Clancy NT, Arya S, Stoyanov D, Singh M, Hanna GB, Elson DS (2015) Intraoperative measurement of bowel oxygen saturation using a multispectral imaging laparoscope. Biomed Opt Express 6(10):4179–4190

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Cox B, Laufer J, Beard P (2009) The challenges for quantitative photoacoustic imaging. In: Photons plus ultrasound: imaging and sensing 2009, vol 7177. International Society for Optics and Photonics, p 717713

  5. Dinh L, Sohl-Dickstein J, Bengio S (2016) Density estimation using real NVP. arXiv preprint arXiv:1605.08803

  6. Feindt M (2004) A neural Bayesian estimator for conditional probability densities. arXiv:physics/0402093

  7. Gal Y, Ghahramani Z (2016) Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: International conference on machine learning, pp 1050–1059

  8. Gretton A, Borgwardt KM, Rasch MJ, Schölkopf B, Smola A (2012) A kernel two-sample test. J Mach Learn Res 13(Mar):723–773

    Google Scholar 

  9. Gu X, Han Z, Yao L, Zhong Y, Shi Q, Fu Y, Liu C, Wang X, Xie T (2016) Image enhancement based on in vivo hyperspectral gastroscopic images: a case study. J Biomed Opt 21(10):101412. https://doi.org/10.1117/1.JBO.21.10.101412

    Article  PubMed  Google Scholar 

  10. Han Z, Zhang A, Wang X, Sun Z, Wang MD, Xie T (2016) In vivo use of hyperspectral imaging to develop a noncontact endoscopic diagnosis support system for malignant colorectal tumors. J Biomed Opt 21(1):016001

    Article  Google Scholar 

  11. Jacques SL (2013) Optical properties of biological tissues: a review. Phys Med Biol 58(11):R37

    Article  Google Scholar 

  12. Kaneko K, Yamaguchi H, Saito T, Yano T, Oono Y, Ikematsu H, Nomura S, Sato A, Kojima M, Esumi H, Ochiai A (2014) Hypoxia imaging endoscopy equipped with laser light source from preclinical live animal study to first-in-human subject research. PLoS ONE 9(6):e99055

    Article  PubMed  PubMed Central  Google Scholar 

  13. Kirchner T, Gröhl J, Maier-Hein L (2018) Context encoding enables machine learning-based quantitative photoacoustics. J Biomed Opt 23(5):056008. https://doi.org/10.1117/1.JBO.23.5.056008

    Article  Google Scholar 

  14. Kohl S.A, Romera-Paredes B, Meyer C, De Fauw J, Ledsam JR, Maier-Hein KH, Eslami S, Rezende DJ, Ronneberger O (2018) A probabilistic U-Net for segmentation of ambiguous images. arXiv preprint arXiv:1806.05034

  15. Lakshminarayanan B, Pritzel A, Blundell C (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in neural information processing systems, vol 30. Curran Associates, Inc., Red Hook, pp 6402–6413

    Google Scholar 

  16. Leibig C, Allken V, Ayhan MS, Berens P, Wahl S (2017) Leveraging uncertainty information from deep neural networks for disease detection. Sci Rep 7(1):17816. https://doi.org/10.1038/s41598-017-17876-z

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Li Y, Gal Y (2017) Dropout inference in Bayesian neural networks with alpha-divergences. arXiv:1703.02914 [cs, stat]

  18. Maier-Hein L, Eisenmann M, Reinke A, Onogur S, Stankovic M, Scholz P, Arbel T, Bogunovic H, Bradley AP, Carass A, Feldmann C, Frangi AF, Full PM, van Ginneken B, Hanbury A, Honauer K, Kozubek M, Landman BA, März K, Maier O, Maier-Hein K, Menze BH, Müller H, Neher PF, Niessen W, Rajpoot N, Sharp GC, Sirinukunwattana K, Speidel S, Stock C, Stoyanov D, Taha AA, van der Sommen F, Wang CW, Weber MA, Zheng G, Jannin P, Kopp-Schneider A (2018) Why rankings of biomedical image analysis competitions should be interpreted with care. Nat Commun 9(1):5217

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Moccia S, Wirkert SJ, Kenngott H, Vemuri AS, Apitz M, Mayer B, De Momi E, Mattos LS, Maier-Hein L (2018) Uncertainty-aware organ classification for surgical data science applications in laparoscopy. IEEE Trans Biomed Eng 65(11):2649–2659

    Article  PubMed  Google Scholar 

  20. Niculescu-Mizil A, Caruana R (2005) Predicting good probabilities with supervised learning. In: Proceedings of the 22nd international conference on machine learning. ACM, New York, pp 625–632

  21. Nouri D, Lucas Y, Treuillet S (2014) Efficient tissue discrimination during surgical interventions using hyperspectral imaging. In: International conference on information processing in computer-assisted interventions. Springer, Berlin, pp 266–275. http://link.springer.com/chapter/10.1007/978-3-319-07521-1_28

  22. Nouri D, Lucas Y, Treuillet S (2016) Hyperspectral interventional imaging for enhanced tissue visualization and discrimination combining band selection methods. Int J Comput Assist Radiol Surg 11(12):2185–2197. https://doi.org/10.1007/s11548-016-1449-5

    Article  PubMed  Google Scholar 

  23. Smith L, Gal Y (2018) Understanding measures of uncertainty for adversarial example detection. arXiv:1803.08533 [cs, stat]

  24. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    Google Scholar 

  25. Waibel D, Gröhl J, Isensee F, Kirchner T, Maier-Hein K, Maier-Hein L (2018) Reconstruction of initial pressure from limited view photoacoustic images using deep learning. In: Photons plus ultrasound: imaging and sensing 2018, vol 10494. International Society for Optics and Photonics, p 104942S

  26. Wang L, Jacques SL, Zheng L (1995) MCML: Monte Carlo modeling of light transport in multi-layered tissues. Comput Methods Programs Biomed 47(2):131–146

    Article  CAS  PubMed  Google Scholar 

  27. Wirkert SJ, Clancy NT, Stoyanov D, Arya S, Hanna GB, Schlemmer HP, Sauer P, Elson DS, Maier-Hein L (2014) Endoscopic Sheffield Index for unsupervised in vivo spectral band selection. In: Luo X, Reichl T, Mirota D, Soper T (eds) Computer-assisted and robotic endoscopy, vol 8899. Springer, Cham, pp 110–120

    Google Scholar 

  28. Wirkert SJ, Isensee F, Vemuri AS, Maier-Hein K, Fei B, Maier-Hein L (2018) Domain and task specific multispectral band selection (conference presentation). In: Design and quality for biomedical technologies XI, p. nil. https://doi.org/10.1117/12.2287824

  29. Wirkert SJ, Kenngott H, Mayer B, Mietkowski P, Wagner M, Sauer P, Clancy NT, Elson DS, Maier-Hein L (2016) Robust near real-time estimation of physiological parameters from megapixel multispectral images with inverse Monte Carlo and random forest regression. Int J Comput Assist Radiol Surg 11(6):909–917

    Article  PubMed  PubMed Central  Google Scholar 

  30. Wirkert SJ, Vemuri AS, Kenngott HG, Moccia S, Götz M, Mayer BF, Maier-Hein KH, Elson DS, Maier-Hein L (2017) Physiological parameter estimation from multispectral images unleashed. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 134–141

  31. Zhu Y, Zabaras N (2018) Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification. J Comput Phys 366:415–447. https://doi.org/10.1016/j.jcp.2018.04.018

    Article  Google Scholar 

Download references

Acknowledgements

This study has received funding from the European Unions Horizon 2020 research and innovation program through the ERC starting grant COMBIOSCOPY under Grant Agreement No. ERC-2015-StG-37960.

Author information

Authors and Affiliations

  1. Computer Assisted Medical Interventions, Deutsches Krebsforschungszentrum, Im Neuenheimer Feld 223, 69120, Heidelberg, Germany

    Tim J. Adler, Anant Vemuri, Leonardo Ayala, Janek Gröhl, Thomas Kirchner, Sebastian Wirkert & Lena Maier-Hein

  2. Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany

    Tim J. Adler

  3. Visual Learning Lab, Heidelberg University, Heidelberg, Germany

    Lynton Ardizzone, Jakob Kruse, Carsten Rother & Ullrich Köthe

  4. Medical Faculty, Heidelberg University, Heidelberg, Germany

    Janek Gröhl

  5. Faculty of Physics and Astronomy, Heidelberg University, Heidelberg, Germany

    Thomas Kirchner

Authors
  1. Tim J. Adler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lynton Ardizzone
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anant Vemuri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Leonardo Ayala
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Janek Gröhl
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Thomas Kirchner
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Sebastian Wirkert
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jakob Kruse
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Carsten Rother
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Ullrich Köthe
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Lena Maier-Hein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tim J. Adler.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical standard

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. All applicable international, national and/or institutional guidelines for the care and use of animals were followed. All procedures performed in studies involving animals were in accordance with the ethical standards of the institution or practice at which the studies were conducted. For this type of study, formal consent is not required.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Adler, T.J., Ardizzone, L., Vemuri, A. et al. Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks. Int J CARS 14, 997–1007 (2019). https://doi.org/10.1007/s11548-019-01939-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-019-01939-9

Keywords

Search

Navigation