A Bayesian Neural Net to Segment Images with Uncertainty Estimates and Good Calibration

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11492)


We propose a novel Bayesian decision theoretic deep-neural-network (DNN) framework for image segmentation, enabling us to define a principled measure of uncertainty associated with label probabilities. Our framework estimates uncertainty analytically at test time, unlike the state of the art that relies on approximate and expensive algorithms using sampling or multiple passes. Moreover, our framework leads to a novel Bayesian interpretation of the softmax layer. We propose a novel method to improve DNN calibration. Results on three large datasets show that our framework improves segmentation quality and calibration, and provides more realistic uncertainty estimates, over existing methods.


Image segmentation Deep neural network Bayesian decision theory Generative model Bayesian utility Uncertainty Calibration 


  1. 1.
    Awate, S.P., Whitaker, R.: Multiatlas segmentation as nonparametric regression. IEEE Trans. Med. Imag. 33(9), 1803–1817 (2014)CrossRefGoogle Scholar
  2. 2.
    Brosch, T., Yoo, Y., Tang, L.Y.W., Li, D.K.B., Traboulsee, A., Tam, R.: Deep convolutional encoder networks for multiple sclerosis lesion segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 3–11. Springer, Cham (2015). Scholar
  3. 3.
    Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., Pal, C.: The importance of skip connections in biomedical image segmentation. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 179–187. Springer, Cham (2016). Scholar
  4. 4.
    Fan, A.C., Fisher, J.W., Wells, W.M., Levitt, J.J., Willsky, A.S.: MCMC curve sampling for image segmentation. In: Ayache, N., Ourselin, S., Maeder, A. (eds.) MICCAI 2007. LNCS, vol. 4792, pp. 477–485. Springer, Heidelberg (2007). Scholar
  5. 5.
    Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: Advances in Neural Information Processing Systems (2016)Google Scholar
  6. 6.
    Gal, Y., Hron, J., Kendall, A.: Concrete dropout. In: Advances in Neural Information Processing Systems, pp. 3584–3593 (2017)Google Scholar
  7. 7.
    Garg, S., Awate, S.P.: Perfect MCMC sampling in Bayesian MRFs for uncertainty estimation in segmentation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 673–681. Springer, Cham (2018). Scholar
  8. 8.
    Guo, C., Pleiss, G., Sun, Y., Weinberger, K.: On calibration of modern neural networks. In: International Conference on Machine Learning, pp. 1321–1330 (2017)Google Scholar
  9. 9.
    Chen, H., Qi, X., Yu, L., Heng, P.A.: DCAN: deep contour-aware networks for accurate gland segmentation. In: IEEE Computer Vision and Pattern Recognition, pp. 2487–2496 (2016)Google Scholar
  10. 10.
    Havaei, M., et al.: Brain tumor segmentation with deep neural networks. Med. Imag. Anal. 35, 18–31 (2017)CrossRefGoogle Scholar
  11. 11.
    Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: Advances in Neural Information Processing Systems, pp. 5580–5590 (2017)Google Scholar
  12. 12.
    Kingma, D., Welling, M.: Auto-encoding variational Bayes. arXiv:1312.6114 (2013)
  13. 13.
    Le, M., Unkelbach, J., Ayache, N., Delingette, H.: Sampling image segmentations for uncertainty quantification. Med. Imag. Anal. 34, 42–51 (2016)CrossRefGoogle Scholar
  14. 14.
    Lin, T., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: International Conference on Computer Vision (2017)Google Scholar
  15. 15.
    Milletari, F., Navab, N., Ahmadi, S.: V-net: fully convolutional neural networks for volumetric medical image segmentation. In: International Conference on 3D Vision, pp. 565–571 (2016)Google Scholar
  16. 16.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  17. 17.
    Shah, M.P., Merchant, S.N., Awate, S.P.: MS-net: mixed-supervision fully-convolutional networks for full-resolution segmentation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 379–387. Springer, Cham (2018). Scholar
  18. 18.
    Tanno, R., et al.: Bayesian image quality transfer with CNNs: exploring uncertainty in dMRI super-resolution. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 611–619. Springer, Cham (2017). Scholar
  19. 19.
    Wang, G., Li, W., Ourselin, S., Vercauteren, T.: Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks. In: Crimi, A., Bakas, S., Kuijf, H., Menze, B., Reyes, M. (eds.) BrainLes 2017. LNCS, vol. 10670, pp. 178–190. Springer, Cham (2018). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Computer Science and EngineeringIndian Institute of Technology BombayMumbaiIndia

Personalised recommendations