Abstract
Purpose
Segmenting bone surfaces in ultrasound (US) is a fundamental step in US-based computer-assisted orthopaedic surgeries. Neural network-based segmentation techniques are a natural choice for this, given promising results in related tasks. However, to gain widespread use, we must be able to know how much to trust segmentation networks during clinical deployment when ground-truth data is unavailable.
Methods
We investigated alternative ways to measure the uncertainty of trained networks by implementing a baseline U-Net trained on a large dataset, together with three uncertainty estimation modifications: Monte Carlo dropout, test time augmentation, and ensemble learning. We measured the segmentation performance, calibration quality, and the ability to predict segmentation performance on test data. We further investigated the effect of data quality on these measures.
Results
Overall, we found that ensemble learning with binary cross-entropy (BCE) loss achieved the best segmentation performance (mean Dice: 0.75–0.78 and RMS distance: 0.62–0.86mm) and the lowest calibration errors (mean: 0.22–0.28%). In contrast to previous studies of area or volumetric segmentation, we found that the resulting uncertainty measures are not reliable proxies for surface segmentation performance.
Conclusion
Our experiments indicate that a significant performance and confidence calibration boost can be achieved with ensemble learning and BCE loss, as tested on 13,687 US images containing various anatomies and imaging parameters. However, these techniques do not allow us to reliably predict future segmentation performance. The results of this study can be used to improve the calibration and performance of US segmentation networks.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ungi T, Fichtinger G, Greer H, Sunderland K, Wu V, Baum ZMC, Schlenger C, Oetgen M, Cleary K, Aylward S (2020) Automatic spine ultrasound segmentation for scoliosis visualization and measurement. IEEE Trans Biomed Eng 67(11):3234–3241
Pandey PU, Quader N, Guy P, Garbi R, Hodgson AJ (2020) Ultrasound bone segmentation: a scoping review of techniques and validation practices. Ultrasound Med Biol 46(4):921–935
Mehrtash A, Wells WM, Tempany CM, Abolmaesumi P, Kapur T (2020) Confidence calibration and predictive uncertainty estimation for deep medical image segmentation. IEEE Trans Med Imag 39(12):3868–3878
Hüllermeier E, Waegeman W (2021) Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Mach Learn 110(3):457–506
Gal Y, Ghahramani Z (2016) Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: Proceedings of the 33rd international conference on machine learning ICML 2016
Wang G, Li W, Aertsen M, Deprest J, Ourselin S, Vercauteren T (2019) Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing 338:34–45
Lakshminarayanan B, Pritzel A, Blundell C (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. Adv Neural Inf Process Syst
Kannan A, Hodgson A, Mulpuri K, Garbi R (2021) Leveraging voxel-wise segmentation uncertainty to improve reliability in assessment of paediatric dysplasia of the hip. Int J Comput Assist Radiol Surg 16(7):1121–1129
Fedorov A, Beichel R, Kalpathy-Cramer J, Finet J, Fillion-Robin J-CC, Pujol S, Bauer C, Jennings D, Fennessy F, Sonka M, Buatti, J, Aylward S, Miller JV, Pieper S, Kikinis R (2012) 3D slicer as an image computing platform for the quantitative imaging network. Magn Reson Imag 30(9), 1323–1341
Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention—MICCAI 2015, pp 234–241. Springer, Cham
Isensee F, Jaeger PF, Kohl SAA, Petersen J, Maier-Hein KH (2021) nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods 18(2):203–211
Liu L, Jiang H, He P, Chen W, Liu X, Gao J, Han J (2019) On the variance of the adaptive learning rate and beyond
Osband I, Blundell C, Pritzel A, Roy BV Deep exploration via bootstrapped DQN
Guo C, Pleiss G, Sun Y, Weinberger KQ (2017) On calibration of modern neural networks. In: Proceedings of the 34th international conference on machine learning ICML 2017
Roy AG, Conjeti S, Navab N, Wachinger C (2019) Bayesian QuickNAT: model uncertainty in deep whole-brain segmentation for structure-wise quality control. Neuroimage 195:11–22
Haas J, Rabus B (2021) Uncertainty estimation for deep learning-based segmentation of roads in synthetic aperture radar imagery. Remote Sens. 13(8):1472
Sander J, de Vos BD, Wolterink JM, Išgum I (2019) Towards increased trustworthiness of deep learning segmentation methods on cardiac MRI. In: Medical imaging 2019 Image Processing, vol. 10949, p. 44. SPIE
Kervadec H, Bouchtiba J, Desrosiers C, Granger E, Dolz J, Ben Ayed I (2021) Boundary loss for highly unbalanced segmentation. Med Image Anal 67:285–296
Acknowledgements
The authors would like to thank the Centre for Hip Health and Mobility for providing facilities and technical support.
Funding
This study was partially funded by the Natural Sciences and Engineering Research Council of Canada, and the Orthopaedic Trauma Association.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional committee.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Pandey, P.U., Guy, P. & Hodgson, A.J. Can uncertainty estimation predict segmentation performance in ultrasound bone imaging?. Int J CARS 17, 825–832 (2022). https://doi.org/10.1007/s11548-022-02597-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-022-02597-0