Abstract
Purpose
Estimating uncertainty in predictions made by neural networks is critically important for increasing the trust medical experts have in automatic data analysis results. In segmentation tasks, quantifying levels of confidence can provide meaningful additional information to aid clinical decision making. In recent work, we proposed an interpretable uncertainty measure to aid clinicians in assessing the reliability of developmental dysplasia of the hip metrics measured from 3D ultrasound screening scans, as well as that of the US scan itself. In this work, we propose a technique to quantify confidence in the associated segmentation process that incorporates voxel-wise uncertainty into the binary loss function used in the training regime, which encourages the network to concentrate its training effort on its least certain predictions.
Methods
We propose using a Bayesian-based technique to quantify 3D segmentation uncertainty by modifying the loss function within an encoder-decoder type voxel labeling deep network. By appending a voxel-wise uncertainty measure, our modified loss helps the network improve prediction uncertainty for voxels that are harder to train. We validate our approach by training a Bayesian 3D U-Net with the proposed modified loss function on a dataset comprising 92 clinical 3D US neonate scans and test on a separate hold-out dataset of 24 patients.
Results
Quantitatively, we show that the Dice score of ilium and acetabulum segmentation improves by 5% when trained with our proposed voxel-wise uncertainty loss compared to training with standard cross-entropy loss. Qualitatively, we further demonstrate how our modified loss function results in meaningful reduction of voxel-wise segmentation uncertainty estimates, with the network making more confident accurate predictions.
Conclusion
We proposed a Bayesian technique to encode voxel-wise segmentation uncertainty information into deep neural network optimization, and demonstrated how it can be leveraged into meaningful confidence measures to improve the model’s predictive performance.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Change history
20 April 2022
A Correction to this paper has been published: https://doi.org/10.1007/s11548-022-02594-3
References
Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 424–432
El-Hariri H, Mulpuri K, Hodgson A, Garbi R (2019) Comparative evaluation of hand-engineered and deep-learned features for neonatal hip bone segmentation in ultrasound. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp 12–20
Gal Y, Ghahramani Z (2016) Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: international conference on machine learning, pp 1050–1059
Iglesias JE, Sabuncu MR, Van Leemput K, Initiative ADN (2013) Improved inference in bayesian segmentation using monte carlo sampling: application to hippocampal subfield volumetry. Med Image Anal 17(7):766–778
Jungo A, Reyes M (2019) Assessing reliability and challenges of uncertainty estimations for medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp 48–56
Kannan A, Hodgson A, Mulpuri K, Garbi R (2020) Uncertainty estimation for assessment of 3d us scan adequacy and ddh metric reliability. Uncertainty for safe utilization of machine learning in medical imaging, and graphs in biomedical image analysis. Springer International Publishing, Cham, pp 97–105
Kendall A, Gal Y (2017) What uncertainties do we need in bayesian deep learning for computer vision? In: Advances in neural information processing systems, pp 5574–5584
Kervadec H, Dolz J, Tang M, Granger E, Boykov Y, Ayed IB (2019) Constrained-cnn losses for weakly supervised segmentation. Med Image Anal 54:88–99
Lakshminarayanan B, Pritzel A, Blundell C (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. In: Advances in neural information processing systems, pp 6402–6413
Lee HJ, Kim ST, Navab N, Ro YM (2020) Efficient ensemble model generation for uncertainty estimation with bayesian approximation in segmentation. arXiv preprint arXiv:2005.10754
Loder RT, Skopelja EN (2011) The epidemiology and demographics of hip dysplasia. ISRN Orthopedics 2011
Mehrtash A, Wells W, Tempany C, Abolmaesumi P, Kapur T (2020) Confidence calibration and predictive uncertainty estimation for deep medical image segmentation. IEEE Trans Med Imag 39:3868–3878
Nair T, Precup D, Arnold DL, Arbel T (2020) Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Med Image Anal 59:101557
Ozdemir O, Russell RL, Berlin AA (2019) A 3d probabilistic deep learning system for detection and diagnosis of lung cancer using low-dose ct scans. IEEE Trans Med Imag 39(5):1419–1429
Quader N, Hodgson A, Mulpuri K, Cooper A, Abugharbieh R (2016) Towards reliable automatic characterization of neonatal hip dysplasia from 3d ultrasound images. In: Ourselin S, Joskowicz L, Sabuncu MR, Unal G, Wells W (eds) Medical image computing and computer-Aasisted intervention - MICCAI 2016. Springer International Publishing, Cham, pp 602–609
Quader N, Hodgson AJ, Mulpuri K, Cooper A, Abugharbieh R (2017) A 3d femoral head coverage metric for enhanced reliability in diagnosing hip dysplasia. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp 100–107
Quader N, Schaeffer EK, Hodgson AJ, Abugharbieh R, Mulpuri K (2018) A systematic review and meta-analysis on the reproducibility of ultrasound-based metrics for assessing developmental dysplasia of the hip. J Pediatric Orthopaed 38(6):e305–e311
Quader N, Hodgson AJ, Mulpuri K, Cooper A, Garbi R (2020) 3-d ultrasound imaging reliability of measuring dysplasia metrics in infants. Ultrasound in Med Biol 25:58
Risholm P, Pieper S, Samset E, Wells WM (2010) Summarizing and visualizing uncertainty in non-rigid registration. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp 554–561
Shaw BA, Segal LS (2016) Evaluation and referral for developmental dysplasia of the hip in infants. Pediatrics 138(6):256
Taghanaki SA, Zheng Y, Zhou SK, Georgescu B, Sharma P, Xu D, Comaniciu D, Hamarneh G (2019) Combo loss: handling input and output imbalance in multi-organ segmentation. Computer Med Imag Gr 75:24–33
Tanno R, Worrall DE, Ghosh A, Kaden E, Sotiropoulos SN, Criminisi A, Alexander DC (2017) Bayesian image quality transfer with cnns: exploring uncertainty in dmri super-resolution. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp 611–619
Wenger D, Düppe H, Tiderius CJ (2013) Acetabular dysplasia at the age of 1 year in children with neonatal instability of the hip: a cohort study of 243 infants. Acta Orthopaedica 84(5):483–488
Acknowledgements
This work was funded by the Natural Sciences and Engineering Research Council of Canada. We also acknowledge support from the Institute of Computing, Information and Cognitive Systems.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflicts of interest.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Kannan, A., Hodgson, A., Mulpuri, K. et al. Leveraging voxel-wise segmentation uncertainty to improve reliability in assessment of paediatric dysplasia of the hip . Int J CARS 16, 1121–1129 (2021). https://doi.org/10.1007/s11548-021-02389-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-021-02389-y