Ensemble of Deep Convolutional Neural Networks with Monte Carlo Dropout Sampling for Automated Image Segmentation Quality Control and Robust Deep Learning Using Small Datasets

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12722)


Recent progress on deep learning (DL)-based medical image segmentation can enable fast extraction of clinical parameters for efficient clinical workflows. However, current DL methods can still fail and require manual visual inspection of outputs, which is time-consuming and diminishes the advantages of automation. For clinical applications, it is essential to develop DL approaches that can not only perform accurate segmentation, but also predict the segmentation quality and flag poor-quality results to avoid errors in diagnosis. To achieve robust performance, DL-based methods often require large datasets, which are not always readily available. It would be highly desirable to be able to train DL models using only small datasets, but this requires a quality prediction method to ensure reliability. We present a novel segmentation framework utilizing an ensemble of deep convolutional neural networks with Monte Carlo sampling. The proposed framework merges the advantages of both state-of-the-art deep ensembles and Bayesian approaches, to provide robust segmentation with inherent quality control. We successfully developed and tested this framework using just a small MRI dataset of 45 subjects. The framework obtained high mean Dice similarity coefficients (DSC) for segmentation of the endocardium (0.922) and the epicardium (0.942); importantly, segmentation DSC can be accurately predicted with low mean absolute errors (≤0.035), in the absence of the manual ground truth. Furthermore, binary classification of segmentation quality achieved a near-perfect accuracy of 99%. The proposed framework can enable fast and reliable medical image analysis with accurate quality control, and training of DL-based methods using even small datasets.


Automated quality assessment Segmentation Ensemble learning Monte Carlo sampling 


  1. 1.
    Roth, G.A., et al.: Global, regional, and national age-sex-specific mortality for 282 causes of death in 195 countries and territories, 1980–2017: a systematic analysis for the Global Burden of Disease Study 2017. The Lancet 392(10159), 1736–1788 (2018)CrossRefGoogle Scholar
  2. 2.
    Radau, P., et al.: Evaluation framework for algorithms segmenting short axis cardiac MRI. MIDAS J. 49 (2009)Google Scholar
  3. 3.
    Bernard, O., et al.: Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE Trans. Med. Imaging 37(11), 2514–2525 (2018)CrossRefGoogle Scholar
  4. 4.
    Petersen, S.E., et al.: Imaging in population science: cardiovascular magnetic resonance in 100,000 participants of UK Biobank - rationale, challenges and approaches. J. Cardiovasc. Magn. Reson. 15(1), 46 (2013)CrossRefGoogle Scholar
  5. 5.
    Constantinides, C., et al.: Semi-automated cardiac segmentation on cine magnetic resonance images using GVF-Snake deformable models. MIDAS J. 77 (2009)Google Scholar
  6. 6.
    Casta, C., et al.: Evaluation of the dynamic deformable elastic template model for the segmentation of the heart in MRI sequences. MIDAS J. (2009)Google Scholar
  7. 7.
    Huang, S., et al.: Segmentation of the left ventricle from cine MR images using a comprehensive approach. MIDAS J. (2009)Google Scholar
  8. 8.
    Lu, Y., et al.: Automatic image-driven segmentation of left ventricle in cardiac cine MRI. MIDAS J. (2009)Google Scholar
  9. 9.
    Jolly, M.: Fully automatic left ventricle segmentation in cardiac cine MR images using registration and minimum surfaces. MIDAS J. (2009)Google Scholar
  10. 10.
    O'Brien, S., Ghita, O., Whelan, P.F.: Segmenting the left ventricle in 3D using a coupled ASM and a learned non-rigid spatial model. MIDAS J. (2009)Google Scholar
  11. 11.
    Wijnhout, J., et al.: LV challenge LKEB contribution: fully automated myocardial contour detection. MIDAS J. (2009)Google Scholar
  12. 12.
    Chen, C., et al.: Deep learning for cardiac image segmentation: a review. Front. Cardiovasc. Med. 7 (2020)Google Scholar
  13. 13.
    Roy, A., Conjeti, S., Navab, N., Wachinger, C.: Inherent brain segmentation quality control from fully convnet monte carlo sampling. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I, pp. 664–672. Springer International Publishing, Cham (2018). Scholar
  14. 14.
    DeVries, T., Graham, T.: Leveraging Uncertainty Estimates for Predicting Segmentation Quality. arXiv pre-print server (2018)Google Scholar
  15. 15.
    Hann, E., et al.: Quality control-driven image segmentation towards reliable automatic image analysis in large-scale cardiovascular magnetic resonance aortic cine imaging. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 750–758. Springer, Cham (2019).
  16. 16.
    Lakshminarayanan, B., Pritzel, V., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv pre-print server (2017)Google Scholar
  17. 17.
    Hann, E., et al.: Deep neural network ensemble for on-the-fly quality control-driven segmentation of cardiac MRI T1 mapping. Med. Image Anal. 71, 102 (2021)CrossRefGoogle Scholar
  18. 18.
    Mehrtash, A., et al.: Confidence calibration and predictive uncertainty estimation for deep medical image segmentation. IEEE Trans. Med. Imaging 39(12), 3868–3878 (2020)CrossRefGoogle Scholar
  19. 19.
    Fort, S., Hu, H., Lakshminarayanan, B.: Deep ensembles: a loss landscape perspective. arXiv pre-print server (2020)Google Scholar
  20. 20.
    Pop, R., Fulop, P.: Deep ensemble bayesian active learning : addressing the mode collapse issue in monte carlo dropout via ensembles. arXiv pre-print server (2018)Google Scholar
  21. 21.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, Nassir, Hornegger, Joachim, Wells, William M., Frangi, Alejandro F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III, pp. 234–241. Springer International Publishing, Cham (2015). Scholar
  22. 22.
    Li, X., et al.: Estimating the ground truth from multiple individual segmentations incorporating prior pattern analysis with application to skin lesion segmentation. IEEE (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2021

Authors and Affiliations

  1. 1.Oxford Centre for Clinical Magnetic Resonance Research, Division of Cardiovascular Medicine, Radcliffe Department of MedicineUniversity of OxfordOxfordUK

Personalised recommendations