Abstract
This paper investigates the numerical uncertainty of Convolutional Neural Networks (CNNs) inference for structural brain MRI analysis. It applies Random Rounding—a stochastic arithmetic technique— to CNN models employed in non-linear registration (SynthMorph) and whole-brain segmentation (FastSurfer), and compares the resulting numerical uncertainty to the one measured in a reference image-processing pipeline (FreeSurfer recon-all). Results obtained on 32 representative subjects show that CNN predictions are substantially more accurate numerically than traditional image-processing results (non-linear registration: 19 vs 13 significant bits on average; whole-brain segmentation: 0.99 vs 0.92 Sørensen-Dice score on average), which suggests a better reproducibility of CNN results across execution environments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Botvinik-Nezer, R., et al.: Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582(7810), 84–88 (2020)
Fanelli, D.: Is Science Really facing a reproducibility crisis, and do we need it to? Proc. Nat. Acad. Sci. 115(11), 2628–2631 (2018). https://www.pnas.org/doi/abs/10.1073/pnas.1708272114
Baker, M.: 1,500 scientists lift the lid on reproducibility. Nature 533(7604), 452–454 (2016)
Gronenschild, E.H., et al.: The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements. PLOS ONE 7(6), e38234 (2012)
Glatard, T., et al.: Reproducibility of neuroimaging analyses across operating systems. Front. Neuroinform. 9, 12 (2015)
Salari, A., Chatelain, Y., Kiar, G., Glatard, T.: Accurate simulation of operating system updates in neuroimaging using Monte-Carlo arithmetic. In: Sudre, C.H., et al. (eds.) UNSURE/PIPPI -2021. LNCS, vol. 12959, pp. 14–23. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87735-4_2
Kiar, G., et al.: Numerical uncertainty in analytical pipelines lead to impactful variability in brain networks. PLOS ONE 16(11), e0250755 (2021)
Hoffmann, M., Billot, B., Greve, D.N., Iglesias, J.E., Fischl, B., Dalca, A.V.: SynthMorph: learning contrast-invariant registration without acquired images. IEEE Trans. Med. Imaging 41(3), 543–558 (2021)
Iglesias, J.E.: A ready-to-use machine learning tool for symmetric multi-modality registration of brain MRI. Sci. Rep. 13(1), 6657 (2023)
Balakrishnan, G., Zhao, A., Sabuncu, M., Guttag, J., Dalca, A.V.: VoxelMorph: a learning framework for deformable medical image registration. IEEE TMI Trans. Med. Imaging 38, 1788–1800 (2019)
Roy, A.G., Conjeti, S., Navab, N., Wachinger, C., Initiative, A.D.N., et al.: QuickNAT: a fully convolutional network for quick and accurate segmentation of neuroanatomy. Neuroimage 186, 713–727 (2019)
Henschel, L., Conjeti, S., Estrada, S., Diers, K., Fischl, B., Reuter, M.: FastSurfer - a fast and accurate deep learning based neuroimaging pipeline. Neuroimage 219, 117012 (2020)
Jog, A., Hoopes, A., Greve, D.N., Van Leemput, K., Fischl, B.: PSACNN: pulse sequence adaptive fast whole brain segmentation. Neuroimage 199, 553–569 (2019)
Li, W., Wang, G., Fidon, L., Ourselin, S., Cardoso, M.J., Vercauteren, T.: On the compactness, efficiency, and representation of 3D convolutional networks: brain parcellation as a pretext task. In: Niethammer, M., et al. (eds.) IPMI 2017. LNCS, vol. 10265, pp. 348–360. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59050-9_28
Higham, N.J.: Accuracy and Stability of Numerical Algorithms. SIAM (2002)
Kloberdanz, E., Kloberdanz, K.G., Le, W.: DeepStability: a study of unstable numerical methods and their solutions in deep learning. In: Proceedings of the 44th International Conference on Software Engineering, pp. 586–597 (2022)
Chakraborty, A., Alam, M., Dey, V., Chattopadhyay, A., Mukhopadhyay, D.: A survey on adversarial attacks and defences. CAAI Trans. Intell. Technol. 6(1), 25–45 (2021)
Forsythe, G.E.: Reprint of a note on rounding-off errors. SIAM Rev. 1(1), 66 (1959)
Févotte, F., Lathuiliere, B.: VERROU: a CESTAC evaluation without recompilation. SCAN 2016, 47 (2016)
Fischl, B., et al.: Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron 33(3), 341–355 (2002)
Parker, D.S.: Monte Carlo Arithmetic: Exploiting Randomness in Floating-Point Arithmetic. University of California (Los Angeles). Computer Science Department (1997)
Verrou: floating-point error checker. https://github.com/edf-hpc/verrou. Accessed 28 Jun 2023
Nethercote, N., Seward, J.: Valgrind: a framework for heavyweight dynamic binary instrumentation. ACM SIGPLAN Not. 42(6), 89–100 (2007)
Denis, C., Castro, P.D.O., Petit, E.: Verificarlo: checking floating point accuracy through Monte Carlo arithmetic. In: 2016 IEEE 23nd Symposium on Computer Arithmetic (ARITH), Los Alamitos, CA, USA, July 2016, pp. 55–62. IEEE Computer Society (2016)
Sohier, D., Castro, P.D.O., Févotte, F., Lathuilière, B., Petit, E., Jamond, O.: Confidence intervals for stochastic arithmetic. ACM Trans. Math. Softw. (TOMS) 47(2), 1–33 (2021)
Significant digits package. https://github.com/verificarlo/significantdigits. Accessed 28 Jun 2023
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Unsupervised learning for image registration. https://github.com/voxelmorph/voxelmorph#SynthMorph. Accessed 28 Jun 2023
Fischl, B., et al.: Sequence-independent segmentation of magnetic resonance images. Neuroimage 23, S69–S84 (2004)
PyTorch implementation of FastSurferCNN. https://github.com/Deep-MI/FastSurfer. Accessed 28 Jun 2023
Zuo, X.-N., et al.: An open science resource for establishing reliability and reproducibility in functional connectomics. Sci. Data 1(1), 1–13 (2014)
Halchenko, Y., et al.: DataLad: distributed system for joint management of code, data, and their relationship. J. Open Source Softw. 6(63), 3262 (2021)
Corr DataLad dataset. http://datasets.datalad.org/?dir=/corr/RawDataBIDS. Accessed 28 Jun 2023
Verrou fix for SynthMorph. https://github.com/yohanchatelain/verrou/tree/synchroLib-v2.4.0. Accessed 28 Jun 2023
Scripts used in the experiments. https://github.com/InesGP/structural_mri_uncertainty/tree/main. Accessed 28 Jun 2023
Pepe, I.G., Chatelain, Y., Kiar, G., Glatard, T.: Numerical stability of DeepGOPlus inference. arXiv preprint arXiv:2212.06361 (2022)
Salari, A., Kiar, G., Lewis, L., Evans, A.C., Glatard, T.: File-based localization of numerical perturbations in data analysis pipelines. GigaScience 9(12), giaa106 (2020)
Acknowledgements
Computations were made on the Narval and Béluga supercomputers from École de Technologie Supérieure (ETS, Montréal), managed by Calcul Québec and The Digital Alliance of Canada. The operation of these supercomputers are funded by the Canada Foundation for Innovation (CFI), le Ministère de l’Économie, des Sciences et de l’Innovation du Québec (MESI) and le Fonds de recherche du Québec - Nature et technologies (FRQ-NT).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pepe, I.G., Sivakolunthu, V., Park, H.L., Chatelain, Y., Glatard, T. (2023). Numerical Uncertainty of Convolutional Neural Networks Inference for Structural Brain MRI Analysis. In: Sudre, C.H., Baumgartner, C.F., Dalca, A., Mehta, R., Qin, C., Wells, W.M. (eds) Uncertainty for Safe Utilization of Machine Learning in Medical Imaging. UNSURE 2023. Lecture Notes in Computer Science, vol 14291. Springer, Cham. https://doi.org/10.1007/978-3-031-44336-7_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-44336-7_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44335-0
Online ISBN: 978-3-031-44336-7
eBook Packages: Computer ScienceComputer Science (R0)
