Abstract
Generative modelling and synthetic data can be a surrogate for real medical imaging datasets, whose scarcity and difficulty to share can be a nuisance when delivering accurate deep learning models for healthcare applications. In recent years, there has been an increased interest in using these models for data augmentation and synthetic data sharing, using architectures such as generative adversarial networks (GANs) or diffusion models (DMs). Nonetheless, the application of synthetic data to tasks such as 3D magnetic resonance imaging (MRI) segmentation remains limited due to the lack of labels associated with the generated images. Moreover, many of the proposed generative MRI models lack the ability to generate arbitrary modalities due to the absence of explicit contrast conditioning. These limitations prevent the user from adjusting the contrast and content of the images and obtaining more generalisable data for training task-specific models. In this work, we propose brainSPADE3D, a 3D generative model for brain MRI and associated segmentations, where the user can condition on specific pathological phenotypes and contrasts. The proposed joint imaging-segmentation generative model is shown to generate high-fidelity synthetic images and associated segmentations, with the ability to combine pathologies. We demonstrate how the model can alleviate issues with segmentation model performance when unexpected pathologies are present in the data.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bakas, S., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. Data 4 (2017). https://doi.org/10.1038/SDATA.2017.117, https://pubmed.ncbi.nlm.nih.gov/28872634/
Bakas, S., et al.: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. Sandra Gonzlez-Vill 124 (2018). https://arxiv.org/abs/1811.02629v3
Barile, B., Marzullo, A., Stamile, C., Durand-Dubief, F., Sappey-Marinier, D.: Data augmentation using generative adversarial neural networks on brain structural connectivity in multiple sclerosis. Comput. Methods Programs Biomed. 206 (2021). https://doi.org/10.1016/J.CMPB.2021.106113, https://pubmed.ncbi.nlm.nih.gov/34004501/
Basaran, B.D., Matthews, P.M., Bai, W.: New lesion segmentation for multiple sclerosis brain images with imaging and lesion-aware augmentation. Front. Neurosci. 16 (2022). https://doi.org/10.3389/FNINS.2022.1007453, https://pubmed.ncbi.nlm.nih.gov/36340756/
Billot, B., Magdamo, C., Arnold, S.E., Das, S., Iglesias, J.E.: Robust machine learning segmentation for large-scale analysis of heterogeneous clinical brain MRI datasets. Proc. Natl. Acad. Sci. 120(9), e2216399120 (2022). https://doi.org/10.1073/PNAS.2216399120/SUPPL_FILE/PNAS.2216399120.SAPP.PDF, https://arxiv.org/abs/2209.02032
Cardoso, M.J., et al.: Geodesic information flows: spatially-variant graphs and their application to segmentation and fusion. IEEE Trans. Med. Imaging 34(9), 1976–1988 (2015)
Chen, S., Ma, K., Zheng, Y.: MED3D: Transfer Learning for 3D Medical Image Analysis. https://github.com/Tencent/MedicalNet
Consortium, M.: MONAI: Medical Open Network for AI, March 2020
Deng, J., Dong, W., Socher, R., Li, L.J., Kai, L., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database, pp. 248–255 (2010). https://doi.org/10.1109/CVPR.2009.5206848
Esteva, A., et al.: A guide to deep learning in healthcare. Nat. Med. 25(1), 24–29 (2019). https://doi.org/10.1038/s41591-018-0316-z, https://www.nature.com/articles/s41591-018-0316-z
Fernandez, V., et al.: Can segmentation models be trained with fully synthetically generated data? In: Zhao, C., Svoboda, D., Wolterink, J.M., Escobar, M. (eds.) SASHIMI 2022. LNCS, vol. 13570, pp. 79–90. Springer International Publishing, Cham (2022). https://doi.org/10.1007/978-3-031-16980-9_8
Foroozandeh, M., Eklund, A.: Synthesizing brain tumor images and annotations by combining progressive growing GAN and SPADE (2020). https://doi.org/10.48550/arxiv.2009.05946, https://arxiv.org/abs/2009.05946v1
Hoogeboom, E., Heek, J., Salimans, T.: simple diffusion: end-to-end diffusion for high resolution images
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning (2015). https://doi.org/10.1016/B978-0-12-391420-0.09987-X
Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18(2), 203–211 (2021). https://doi.org/10.1038/s41592-020-01008-z
Jack, C.R., et al.: The Alzheimer’s Disease Neuroimaging Initiative (ADNI): MRI methods. J. Magn. Resonan. Imaging (JMRI) 27(4), 685–691 (2008). https://doi.org/10.1002/JMRI.21049, https://pubmed.ncbi.nlm.nih.gov/18302232/
Jones, S., et al.: Cohort profile update: southall and brent revisited (SABRE) study: a UK population-based comparison of cardiovascular disease and diabetes in people of European, South Asian and African Caribbean heritage. Int. J. Epidemiol. 49(5), 1441–1442 (2020). https://doi.org/10.1093/ije/dyaa135
Liu, L., Ren, Y., Lin, Z., Zhao, Z.: Pseudo numerical methods for diffusion models on manifolds (2022). https://doi.org/10.48550/arxiv.2202.09778, https://arxiv.org/abs/2202.09778v2
Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging 34(10), 1993–2024 (2015). https://doi.org/10.1109/TMI.2014.2377694, https://pubmed.ncbi.nlm.nih.gov/25494501, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4833122/
Park, T., et al.: Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of IEEE CVPR, June 2019, pp. 2332–2341 (2019)
Pinaya, W.H.L., et al.: Brain Imaging Generation with Latent Diffusion Models. In: Mukhopadhyay, A., Oksuz, I., Engelhardt, S., Zhu, D., Yuan, Y. (eds.) DGM4MICCAI 2022. LNCS, vol. 13609, pp. 117–126. Springer Nature Switzerland, Cham (2022). https://doi.org/10.1007/978-3-031-18576-2_12
Qasim, A.B., et al.: Red-GAN: attacking class imbalance via conditioned generation. Yet another medical imaging perspective (2020). https://proceedings.mlr.press/v121/qasim20a.html
Rieke, N., et al.: The future of digital health with federated learning. NPJ Digit. Med. 3(1), 119 (2020)
Salimans, T., Ho, J.: Progressive distillation for fast sampling of diffusion models
Sudre, C.H., Cardoso, M.J., Bouvy, W.H., Biessels, G.J., Barnes, J., Ourselin, S.: Bayesian model selection for pathological neuroimaging data applied to white matter lesion segmentation. IEEE Trans. Med. Imaging 34(10), 2079–2102 (2015). https://doi.org/10.1109/TMI.2015.2419072
Tudosiu, P.D., et al.: Morphology-preserving autoregressive 3d generative modelling of the brain. In: Zhao, C., Svoboda, D., Wolterink, J.M., Escobar, M. (eds.) SASHIMI 2022. LNCS, vol. 13570, pp. 66–78. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16980-9_7
Wachinger, C., et al.: BrainPrint: a discriminative characterization of brain morphology. NeuroImage 109, 232–248 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Fernandez, V., Pinaya, W.H.L., Borges, P., Graham, M.S., Vercauteren, T., Cardoso, M.J. (2024). A 3D Generative Model of Pathological Multi-modal MR Images and Segmentations. In: Mukhopadhyay, A., Oksuz, I., Engelhardt, S., Zhu, D., Yuan, Y. (eds) Deep Generative Models. MICCAI 2023. Lecture Notes in Computer Science, vol 14533. Springer, Cham. https://doi.org/10.1007/978-3-031-53767-7_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-53767-7_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-53766-0
Online ISBN: 978-3-031-53767-7
eBook Packages: Computer ScienceComputer Science (R0)