Uncertainty Estimation in Medical Image Denoising with Bayesian Deep Image Prior

Max-Heinrich Laves; Malte Tölle; Tobias Ortmaier

doi:10.1007/978-3-030-60365-6_9

International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging

International Workshop on Graphs in Biomedical Image Analysis

UNSURE 2020, GRAIL 2020: Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis pp 81-96 | Cite as

Uncertainty Estimation in Medical Image Denoising with Bayesian Deep Image Prior

Authors
Authors and affiliations

Max-Heinrich Laves
Malte Tölle
Tobias Ortmaier

Conference paper

First Online: 05 October 2020

3 Mentions
767 Downloads

Part of the Lecture Notes in Computer Science book series (LNCS, volume 12443)

Abstract

Uncertainty quantification in inverse medical imaging tasks with deep learning has received little attention. However, deep models trained on large data sets tend to hallucinate and create artifacts in the reconstructed output that are not anatomically present. We use a randomly initialized convolutional network as parameterization of the reconstructed image and perform gradient descent to match the observation, which is known as deep image prior. In this case, the reconstruction does not suffer from hallucinations as no prior training is performed. We extend this to a Bayesian approach with Monte Carlo dropout to quantify both aleatoric and epistemic uncertainty. The presented method is evaluated on the task of denoising different medical imaging modalities. The experimental results show that our approach yields well-calibrated uncertainty. That is, the predictive uncertainty correlates with the predictive error. This allows for reliable uncertainty estimates and can tackle the problem of hallucinations and artifacts in inverse medical imaging tasks.

Keywords

Variational inference Hallucination Deep learning

This is a preview of subscription content, to check access.

A Appendix

A.1 Additional Figures

(See Figs. 6, 7, 8 and 10)

Fig. 6.
Denoised images after convergence.

Fig. 7.
Denoised images with early-stopping applied.

Fig. 8.
MSE (top row) between denoised $ \hat{\textit{\textbf{x}}} $ image and noisy image $ \tilde{\textit{\textbf{x}}} $ and SSIM (bottom row) between denoised $ \hat{\textit{\textbf{x}}} $ image and ground truth $ \textit{\textbf{x}} $ vs. iteration. Only MCDIP does not overfit the noisy image and converges with highest similarity to the ground truth. Despite the claim of the authors, SGLD suffers from overfitting and creates the need for carefully applied early stopping [4]. Note: We compared both our own implementation of SGLD and the original code provided by the authors. The plots show means from 3 runs with different random initialization.

A.2 Additional Tables

(See Table 2, 3 and 4)

Table 2.

PSNR with early-stopping.

PSNR	DIP	SGLD	SGLD+NLL	MCDIP
OCT	$ 29.88 \pm 0.02 $	$ 29.89 \pm 0.05 $	$ 29.77 \pm 0.07 $	$ \mathbf {29.92} \pm 0.03 $
US	$ 29.74 \pm 0.05 $	$ \mathbf {29.78} \pm 0.02 $	$ 29.54 \pm 0.03 $	$ 29.7 \pm 0.07 $
X-ray	$ 30.91 \pm 0.05 $	$ 30.98 \pm 0.09 $	$ 30.74 \pm 0.03 $	$ \mathbf {31.22} \pm 0.1 $

Table 3.

SSIM after convergence.

SSIM	DIP	SGLD	SGLD+NLL	MCDIP
OCT	$ 0.582 \pm 0.0 $	$ 0.574 \pm 0.0 $	$ 0.66 \pm 0.0 $	$ \mathbf {0.872} \pm 0.0 $
US	$ 0.687 \pm 0.0 $	$ 0.703 \pm 0.0 $	$ 0.723 \pm 0.0 $	$ \mathbf {0.902} \pm 0.0 $
X-ray	$ 0.625 \pm 0.0 $	$ 0.631 \pm 0.0 $	$ 0.686 \pm 0.0 $	$ \mathbf {0.922} \pm 0.0 $

Fig. 9.
Calibration diagrams and uncertainty maps for SGLD+NLL and MCDIP after convergence (best viewed with digital zoom). (Left) The calibration diagrams show MSE vs. uncertainty and provide mean uncertainty (U) and UCE values. (Right) Uncertainty maps show per-pixel uncertainty. Due to overfitting, the MSE and uncertainty from SGLD+NLL concentrates around 0.0.

Fig. 10.
Calibration diagrams and uncertainty maps for SGLD+NLL after early stopping and MCDIP after convergence (best viewed with digital zoom). (Left) The calibration diagrams show MSE vs. uncertainty and provide mean uncertainty (U) and UCE values. (Right) Uncertainty maps show per-pixel uncertainty.

Table 4.

SSIM with early-stopping.

SSIM	DIP	SGLD	SGLD+NLL	MCDIP
OCT	$ 0.872 \pm 0.0 $	$ 0.872 \pm 0.0 $	$ 0.872 \pm 0.0 $	$ 0.872 \pm 0.0 $
US	$ 0.902 \pm 0.0 $	$ \mathbf {0.903} \pm 0.0 $	$ 0.899 \pm 0.0 $	$ \mathbf {0.903} \pm 0.0 $
X-ray	$ 0.915 \pm 0.0 $	$ 0.917 \pm 0.0 $	$ 0.912 \pm 0.0 $	$ \mathbf {0.923} \pm 0.0 $

A.3 SGLD with Step Size Decay

Additionall, we implement SGLD with step size decay as described by Welling et al. [26]. The step size $ \epsilon $ is used to scale the parameter update in the SGD step (i.e. the learning rate) and defines the variance of the noise that is injected into the gradients. Here, we reduce the step size at each step t exponentially with $ \epsilon _{t} = 0.999^{t} \epsilon _{0} $. To satisfy the step size property (Eq. (2) in [26]), we fix the step size once it decreases below 1e-8. We observe no overfitting of the noisy image with step size decay (see Fig. 11). However, the quality of the resulting denoised image is very sensitive to the decay scheme. Choosing a decrease that is too low (i.e. $ \epsilon _{t} = 0.9999^{t} \epsilon _{0} $) results in overfitting; a decrease that is too high (i.e. $ \epsilon _{t} = 0.99^{t} \epsilon _{0} $) results in convergence to a subpar reconstruction. This is equivalent to carefully applied early stopping and therefore nullifies the advantage of SGLD for denoising of medical images.

Fig. 11.
Comparison of SGLD and SGLD+LR (with step size decay). Carefully chosen step size decay impedes overfitting the noisy image. (Right) Reconstruction of SGLD+LR after convergence (no early stopping applied).

A.4 Downsampling

Here, we provide justification why downsampling of an image by averaging neighboring pixels reduces the noise level and can be used as an approximation to a ground truth noise-free image (by sacrificing image resolution).

Proposition 1

Downsampling of an image reduces the observation noise.

Proof

Let $ X = \mu _{x} + \varepsilon _{x} $ and $ Y = \mu _{y} + \varepsilon _{y} $ be two neighboring pixels affected by additive i.i.d. noise $ \varepsilon _{x} , \varepsilon _{y} \sim \mathcal {N}(0, \sigma ^{2}) $. The pixels are assumed to be uncorrelated to noise. Pixels in a local neighborhood are highly correlated and assumed to be of high similarity $ \mu _{x} \approx \mu _{y} = \mu $. Let $ Z = \tfrac{1}{2} \left( X + Y \right) $ be the average of two neighboring pixels (i.e. the result of downsampling). The expectation is given by

$$\begin{aligned} \mathbb {E}[Z]&= \frac{1}{2} \left( \mathbb {E}[X] + \mathbb {E}[Y] \right) \end{aligned}$$

(9)

$$\begin{aligned}&= \frac{1}{2} 2 \, \mathbb {E}[X] \end{aligned}$$

(10)

$$\begin{aligned}&= \mu \end{aligned}$$

(11)

and the variance is given by

$$\begin{aligned} \mathrm {Var}\left[ Z\right]&= \mathrm {Var}\left[ \frac{1}{2} \left( X + Y \right) \right] \end{aligned}$$

(12)

$$\begin{aligned}&= \frac{1}{2^{2}} \left( \mathrm {Var}\left[ X\right] + \mathrm {Var}\left[ Y\right] \right) \end{aligned}$$

(13)

$$\begin{aligned}&= \frac{1}{2^{2}} 2 \mathrm {Var}\left[ X\right] \end{aligned}$$

(14)

$$\begin{aligned}&= \frac{1}{2} \sigma ^{2} ~ . \end{aligned}$$

(15)

Thus, if the similarity of neighboring pixels is sufficiently high, downsampling reduces the variance of average pixel Z by a factor of 2. $\square $

Naturally, two neighboring pixels are not exactly equal. However, downsampling can also be viewed as superposing two signals, each with a highly correlated and an uncorrelated part. Without providing proof, the amplitude of the addition of two signals can be viewed as vector addition. In the uncorrelated case, the two signals are perpendicular to each other and in the correlated case, the angle between the two signals is acute. Thus, the correlated parts of the two signals have a higher impact on the resulting addition than the uncorrelated (noise) parts. In the ideal case, where the noise is uncorrelated and the signals are in parallel, the same noise reduction as above follows.

A.5 Link Between Poisson Distribution and Normal Distribution

We approximate the Poisson noise to simulate a low-dose X-ray image with a Normal distribution. It is well-known that the limiting distribution of $ \mathsf {Poisson}(\lambda ) $ is Normal as $ \lambda \rightarrow \infty $ [10]. For completeness, we list a common proof using the moment generating function of a standardized Poisson random variable:

Theorem 1

The Poisson($\lambda $) distribution can be approximated with a Normal distribution as $ \lambda \rightarrow \infty $.

Proof

Let $ X_{\lambda } \sim \mathsf {Poisson}(\lambda ), ~ \lambda \in \{ 1, 2, \ldots \} $. The probability mass function of $ X_{\lambda } $ is given by

$$\begin{aligned} f_{X_{\lambda }}(x) = \frac{\lambda ^{x}e^{-\lambda }}{x!} \quad x \in \{ 0, 1, 2, \ldots \} ~ . \end{aligned}$$

(16)

The moment generating function is given by [10]

$$\begin{aligned} M_{X_{\lambda }}(t) = \mathbb {E} [ e^{t X_{\lambda }} ] = e^{\lambda (e^{t}-1)} ~ . \end{aligned}$$

(17)

The standardized Poisson random variable

$$\begin{aligned} Z = \frac{X_{\lambda } - \lambda }{\sqrt{\lambda }} \end{aligned}$$

(18)

has the limiting moment generating function

$$\begin{aligned} \lim _{\lambda \rightarrow \infty } M_{Z} (t)&= \lim _{\lambda \rightarrow \infty } \mathbb {E} \left[ \exp {\left( t \cdot \frac{X_{\lambda } - \lambda }{\sqrt{\lambda }} \right) } \right] \end{aligned}$$

(19)

$$\begin{aligned}&= \lim _{\lambda \rightarrow \infty } \exp { \left( -t \sqrt{\lambda } \right) } \mathbb {E} \left[ \exp {\left( \frac{t X_{\lambda }}{\sqrt{\lambda }} \right) } \right] \end{aligned}$$

(20)

$$\begin{aligned}&= \lim _{\lambda \rightarrow \infty } \exp { \left( -t \sqrt{\lambda } \right) } \exp {\left( \lambda \left( e^{t/\sqrt{\lambda }} - 1 \right) \right) } \end{aligned}$$

(21)

$$\begin{aligned}&= \lim _{\lambda \rightarrow \infty } \exp { \left( -t \sqrt{\lambda } + \lambda \left( t \lambda ^{-1/2} + t^{2} \lambda ^{-1}/2 + t^{3} \lambda ^{-3/2}/6 + \ldots \right) \right) } \end{aligned}$$

(22)

$$\begin{aligned}&= \lim _{\lambda \rightarrow \infty } \exp { \left( t^{2} / 2 + t^{3}\lambda ^{-1/2}/6 + \ldots \right) } \end{aligned}$$

(23)

$$\begin{aligned}&= \exp {\left( t^{2} / 2 \right) } \end{aligned}$$

(24)

which is the moment generating function of a standard normal random variable. $\square $

References

1.
Agostinelli, F., Anderson, M.R., Lee, H.: Adaptive multi-column deep neural networks with application to robust image denoising. In: Advances in Neural Information Processing Systems, pp. 1493–1501 (2013)Google Scholar
2.
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Boston (2006). https://doi.org/10.1007/978-1-4615-7566-5CrossRefzbMATHGoogle Scholar
3.
Chang, S.G., Yu, B., Vetterli, M.: Adaptive wavelet thresholding for image denoising and compression. IEEE Trans. Image Process. 9(9), 1532–1546 (2000). https://doi.org/10.1109/83.862633MathSciNetCrossRefzbMATHGoogle Scholar
4.
Cheng, Z., Gadelha, M., Maji, S., Sheldon, D.: A Bayesian perspective on the deep image prior. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5443–5451 (2019)Google Scholar
5.
Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. Trans. Image Process. 16(8), 2080–2095 (2007). https://doi.org/10.1109/TIP.2007.901238MathSciNetCrossRefGoogle Scholar
6.
Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In: ICML, pp. 1050–1059 (2016)Google Scholar
7.
Gondara, L.: Medical image denoising using convolutional denoising autoencoders. In: International Conference on Data Mining Workshops, pp. 241–246 (2016). https://doi.org/10.1109/ICDMW.2016.0041
8.
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: ICML, pp. 1321–1330 (2017)Google Scholar
9.
van den Heuvel, T.L., de Bruijn, D., de Korte, C.L., Ginneken, B.v.: Automated measurement of fetal head circumference using 2D ultrasound images. PloS One 13(8), e0200412 (2018). https://doi.org/10.1371/journal.pone.0200412. US dataset source
10.
Hogg, R.V., McKean, J., Craig, A.T.: Introduction to Mathematical Statistics, 8th edn. Pearson, New York (2018)Google Scholar
11.
Jain, V., Seung, S.: Natural image denoising with convolutional networks. In: Advances in Neural Information Processing Systems, pp. 769–776 (2009)Google Scholar
12.
Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: NeurIPS, pp. 5574–5584 (2017)Google Scholar
13.
Kermany, D.S., et al.: Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172(5), 1122–1131 (2018). https://doi.org/10.1016/j.cell.2018.02.010CrossRefGoogle Scholar
14.
Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: ICLR (2014)Google Scholar
15.
Laves, M.H., Ihler, S., Fast, J.F., Kahrs, L.A., Ortmaier, T.: Well-calibrated regression uncertainty in medical imaging with deep learning. In: Medical Imaging with Deep Learning (2020)Google Scholar
16.
Laves, M.H., Ihler, S., Kahrs, L.A., Ortmaier, T.: Semantic denoising autoencoders for retinal optical coherence tomography. In: SPIE/OSA European Conference on Biomedical Optics, vol. 11078, pp. 86–89 (2019). https://doi.org/10.1117/12.2526936
17.
Lee, S., Lee, M.S., Kang, M.G.: Poisson-gaussian noise analysis and estimation for low-dose x-ray images in the NSCT domain. Sensors 18(4), 1019 (2018)CrossRefGoogle Scholar
18.
Lempitsky, V., Vedaldi, A., Ulyanov, D.: Deep Image Prior. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9446–9454 (2018). https://doi.org/10.1109/CVPR.2018.00984
19.
Levi, D., Gispan, L., Giladi, N., Fetaya, E.: Evaluating and calibrating uncertainty prediction in regression tasks. arXiv arXiv:1905.11659 (2019)
20.
Li, C., Chen, C., Carlson, D., Carin, L.: Preconditioned stochastic gradient Langevin dynamics for deep neural networks. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp. 1788–1794 (2016)Google Scholar
21.
Michailovich, O.V., Tannenbaum, A.: Despeckling of medical ultrasound images. Trans. Ultrason. Ferroelectr. Freq. Control 53(1), 64–78 (2006). https://doi.org/10.1109/TUFFC.2006.1588392CrossRefGoogle Scholar
22.
Rabbani, H., Nezafat, R., Gazor, S.: Wavelet-domain medical image denoising using bivariate Laplacian mixture model. Trans. Biomed. Eng. 56(12), 2826–2837 (2009). https://doi.org/10.1109/TBME.2009.2028876CrossRefGoogle Scholar
23.
Salinas, H.M., Fernandez, D.C.: Comparison of PDE-based nonlinear diffusion approaches for image enhancement and denoising in optical coherence tomography. IEEE Trans. Med. Imaging 26(6), 761–771 (2007). https://doi.org/10.1109/TMI.2006.887375CrossRefGoogle Scholar
24.
Sotiras, A., Davatzikos, C., Paragios, N.: Deformable medical image registration: a survey. IEEE Trans. Med. Imaging 32(7), 1153–1190 (2013). https://doi.org/10.1109/TMI.2013.2265603CrossRefGoogle Scholar
25.
Wang, N., Tao, D., Gao, X., Li, X., Li, J.: A comprehensive survey to face hallucination. Int. J. Comput. Vis. 106(1), 9–30 (2014). https://doi.org/10.1007/s11263-013-0645-9CrossRefGoogle Scholar
26.
Welling, M., Teh, Y.W.: Bayesian learning via stochastic gradient Langevin dynamics. In: ICML, pp. 681–688 (2011)Google Scholar
27.
Žabić, S., Wang, Q., Morton, T., Brown, K.M.: A low dose simulation tool for CT systems with energy integrating detectors. Med. Phys. 40(3), 031102 (2013). https://doi.org/10.1118/1.4789628CrossRefGoogle Scholar
28.
Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017). https://doi.org/10.1109/TIP.2017.2662206MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

Authors and Affiliations

Max-Heinrich Laves
- 1
Email author
Malte Tölle
- 1
Tobias Ortmaier
- 1

1.Leibniz Universität HannoverHanoverGermany

PSNR	DIP	SGLD	SGLD+NLL	MCDIP
OCT	\( 29.88 \pm 0.02 \)	\( 29.89 \pm 0.05 \)	\( 29.77 \pm 0.07 \)	\( \mathbf {29.92} \pm 0.03 \)
US	\( 29.74 \pm 0.05 \)	\( \mathbf {29.78} \pm 0.02 \)	\( 29.54 \pm 0.03 \)	\( 29.7 \pm 0.07 \)
X-ray	\( 30.91 \pm 0.05 \)	\( 30.98 \pm 0.09 \)	\( 30.74 \pm 0.03 \)	\( \mathbf {31.22} \pm 0.1 \)

SSIM	DIP	SGLD	SGLD+NLL	MCDIP
OCT	\( 0.582 \pm 0.0 \)	\( 0.574 \pm 0.0 \)	\( 0.66 \pm 0.0 \)	\( \mathbf {0.872} \pm 0.0 \)
US	\( 0.687 \pm 0.0 \)	\( 0.703 \pm 0.0 \)	\( 0.723 \pm 0.0 \)	\( \mathbf {0.902} \pm 0.0 \)
X-ray	\( 0.625 \pm 0.0 \)	\( 0.631 \pm 0.0 \)	\( 0.686 \pm 0.0 \)	\( \mathbf {0.922} \pm 0.0 \)

SSIM	DIP	SGLD	SGLD+NLL	MCDIP
OCT	\( 0.872 \pm 0.0 \)	\( 0.872 \pm 0.0 \)	\( 0.872 \pm 0.0 \)	\( 0.872 \pm 0.0 \)
US	\( 0.902 \pm 0.0 \)	\( \mathbf {0.903} \pm 0.0 \)	\( 0.899 \pm 0.0 \)	\( \mathbf {0.903} \pm 0.0 \)
X-ray	\( 0.915 \pm 0.0 \)	\( 0.917 \pm 0.0 \)	\( 0.912 \pm 0.0 \)	\( \mathbf {0.923} \pm 0.0 \)

Uncertainty Estimation in Medical Image Denoising with Bayesian Deep Image Prior

Abstract

Keywords

References

Copyright information

Authors and Affiliations

Personalised recommendations

Cite paper