Abstract
Predictive variability due to data ambiguities has typically been addressed via construction dedicated models with built-in probabilistic capabilities that are trained to predict uncertainty estimates as variables of interest. These approaches require distinct architectural components and training mechanisms, may include restrictive assumptions and exhibit overconfidence, i.e., high confidence in imprecise predictions. In this work, we propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions. It is architecture agnostic and can be applied to any feed-forward deterministic network without changes to the architecture nor training procedure. Experiments on regression tasks on imaging and non-imaging input data show the method’s ability to generate diverse and multi-modal predictive distributions and how estimated uncertainty correlates with prediction error.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
UK Biobank homepage. https://www.ukbiobank.ac.uk/about-biobank-uk. Accessed 24 May 2021
Baumgartner, C.F., et al.: PHiSeg: capturing uncertainty in medical image segmentation. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 119–127. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32245-8_14
Begoli, E., Bhattacharya, T., Kusnezov, D.: The need for uncertainty quantification in machine-assisted medical decision making. Nat. Mach. Intell. 1(1), 20–23 (2019). https://doi.org/10.1038/s42256-018-0004-1
Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics, Springer, New York (2006)
Brooks, S., Gelman, A., Jones, G., Meng, X.L.: Handbook of Markov Chain Monte Carlo. CRC Press (2011)
Chang, J., Fisher, J.W.I.: Efficient MCMC sampling with implicit shape representations. In: CVPR, pp. 2081–2088. IEEE Computer Society (2011). http://dblp.uni-trier.de/db/conf/cvpr/cvpr2011.html#ChangF11
Chen, S., Radke, R.J.: Markov chain Monte Carlo shape sampling using level sets. In: 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, pp. 296–303 (2009). https://doi.org/10.1109/ICCVW.2009.5457687
Draper, D.: Assessment and propagation of model uncertainty. J. R. Stat. Soci. Ser. B (Methodol.) 57(1), 45–97 (1995). http://www.jstor.org/stable/2346087
Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml
Erdil, E., Yildirim, S., Çetin, M., Tasdizen, T.: MCMC shape sampling for image segmentation with nonparametric shape priors. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 411–419 (2016). https://doi.org/10.1109/CVPR.2016.51
Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: Balcan, M.F., Weinberger, K.Q. (eds.) Proceedings of The 33rd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 48, pp. 1050–1059. PMLR, New York, 20–22 June 2016. http://proceedings.mlr.press/v48/gal16.html
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
Gelman, A., Carlin, J.B., Stern, H.S., Rubin, D.B.: Bayesian Data Analysis, 2nd edn. Chapman and Hall/CRC (2004)
Graf, F., Kriegel, H.-P., Schubert, M., Pölsterl, S., Cavallaro, A.: 2D image registration in CT images using radial image descriptors. In: Fichtinger, G., Martel, A., Peters, T. (eds.) MICCAI 2011. LNCS, vol. 6892, pp. 607–614. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23629-7_74
Hammersley, J.M., Handscomb, D.C.: Monte Carlo Methods. Springer, Dordrecht (1964). https://doi.org/10.1007/978-94-009-5819-7
Hastings, W.K.: Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57(1), 97–109 (1970). http://www.jstor.org/stable/2334940
Houhou, N., Thiran, J.P., Bresson, X.: Fast texture segmentation model based on the shape operator and active contour. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008). https://doi.org/10.1109/CVPR.2008.4587449
Izenman, A.J.: Modern Multivariate Statistical Techniques. Regression, Classification and Manifold Learning. Springer, New York (2008). https://doi.org/10.1007/978-0-387-78189-1
Karani, N., Erdil, E., Chaitanya, K., Konukoglu, E.: Test-time adaptable neural networks for robust medical image segmentation. Med. Image Anal. 68, 101907 (2021). https://doi.org/10.1016/j.media.2020.101907, https://www.sciencedirect.com/science/article/pii/S1361841520302711
Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: Guyon, I., von Luxburg, U., et al. (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 December 2017, Long Beach, CA, USA, pp. 5574–5584 (2017)
Kim, J., Fisher, J.W., III., Yezzi, A., Çetin, M., Willsky, A.S.: A nonparametric statistical method for image segmentation using information theory and curve evolution. IEEE Trans. Image Process. 14(10), 1486–1502 (2005)
Kindermann, J., Linden, A.: Inversion of neural networks by gradient descent. Parallel Comput. 14(3), 277–286 (1990). https://doi.org/10.1016/0167-8191(90)90081-J
Kohl, S., et al.: A probabilistic u-net for segmentation of ambiguous images. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 31. Curran Associates, Inc. (2018). https://proceedings.neurips.cc/paper/2018/file/473447ac58e1cd7e96172575f48dca3b-Paper.pdf
Levy, D., Sohl-dickstein, J., Hoffman, M.: Generalizing Hamiltonian Monte Carlo with neural networks. In: ICLR 2018 (2018). https://openreview.net/pdf?id=B1n8LexRZ
Michailovich, O., Rathi, Y., Tannenbaum, A.: Image segmentation using active contours driven by the Bhattacharyya gradient flow. IEEE Trans. Image Process. 16(11), 2787–2801 (2007). https://doi.org/10.1109/TIP.2007.908073
Milletari, F., Rothberg, A., Jia, J., Sofka, M.: Integrating statistical prior knowledge into convolutional neural networks. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 161–168. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66182-7_19
Neal, R.M.: Probabilistic inference using Markov chain Monte Carlo methods. Technical report, CRG-TR-93-1, Department of Computer Science, University of Toronto (1993)
Neal, R.M.: Bayesian Learning for Neural Networks. Lecture Notes in Statistics, vol. 118. Springer, New York (1996). https://doi.org/10.1007/978-1-4612-0745-0
Oktay, O., et al.: Anatomically constrained neural networks (ACNNs): application to cardiac image enhancement and segmentation. IEEE Trans. Med. Imaging 37(2), 384–395 (2018). https://doi.org/10.1109/TMI.2017.2743464
Qiu, X., Meyerson, E., Miikkulainen, R.: Quantifying point-prediction uncertainty in neural networks via residual estimation with an I/O kernel. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, 26–30 April 2020. OpenReview.net (2020). https://openreview.net/forum?id=rkxNh1Stvr
Rubin, D.B.: Bayesianly justifiable and relevant frequency calculations for the applied statistician. Ann. Stat. 12(4), 1151–1172 (1984). https://doi.org/10.1214/aos/1176346785
Saltelli, A., Tarantola, S., Campolongo, F., Ratto, M., et al.: Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models. Wiley, Chichester (2004)
Song, J., Zhao, S., Ermon, S.: A-NICE-MC: adversarial training for MCMC. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper/2017/file/2417dc8af8570f274e6775d4d60496da-Paper.pdf
Song, Y., Meng, C., Ermon, S.: MintNet: building invertible neural networks with masked convolutions. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019). https://proceedings.neurips.cc/paper/2019/file/70a32110fff0f26d301e58ebbca9cb9f-Paper.pdf
Tanno, R., et al.: Bayesian image quality transfer with CNNs: exploring uncertainty in dMRI super-resolution. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 611–619. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66182-7_70
Tanno, R., et al.: Uncertainty modelling in deep learning for safer neuroimage enhancement: demonstration in diffusion MRI. NeuroImage 225, 117366 (2021). https://doi.org/10.1016/j.neuroimage.2020.117366, https://www.sciencedirect.com/science/article/pii/S1053811920308521
Tóthová, K., et al.: Probabilistic 3D surface reconstruction from sparse MRI information. In: MICCAI 2020. LNCS, vol. 12261, pp. 813–823. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59710-8_79
Tóthová, K., et al.: Uncertainty quantification in CNN-based surface prediction using shape priors. In: Reuter, M., Wachinger, C., Lombaert, H., Paniagua, B., Lüthi, M., Egger, B. (eds.) ShapeMI 2018. LNCS, vol. 11167, pp. 300–310. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04747-4_28
Wang, H., Levi, D.M., Klein, S.A.: Intrinsic uncertainty and integration efficiency in bisection acuity. Vis. Res. 36(5), 717–739 (1996). https://doi.org/10.1016/0042-6989(95)00143-3
Acknowledgements
This research has been conducted using the UK Biobank Resource under Application Number 17806.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tóthová, K., Ladický, Ľ., Thul, D., Pollefeys, M., Konukoglu, E. (2022). Quantification of Predictive Uncertainty via Inference-Time Sampling. In: Sudre, C.H., et al. Uncertainty for Safe Utilization of Machine Learning in Medical Imaging. UNSURE 2022. Lecture Notes in Computer Science, vol 13563. Springer, Cham. https://doi.org/10.1007/978-3-031-16749-2_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-16749-2_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16748-5
Online ISBN: 978-3-031-16749-2
eBook Packages: Computer ScienceComputer Science (R0)
