Skip to main content
SpringerLink
Log in

Brain Imaging Generation with Latent Diffusion Models

  • Conference paper
  • First Online:
Deep Generative Models (DGM4MICCAI 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13609))

Included in the following conference series:

Abstract

Deep neural networks have brought remarkable breakthroughs in medical image analysis. However, due to their data-hungry nature, the modest dataset sizes in medical imaging projects might be hindering their full potential. Generating synthetic data provides a promising alternative, allowing to complement training datasets and conducting medical image research at a larger scale. Diffusion models recently have caught the attention of the computer vision community by producing photorealistic synthetic images. In this study, we explore using Latent Diffusion Models to generate synthetic images from high-resolution 3D brain images. We used T1w MRI images from the UK Biobank dataset (N = 31,740) to train our models to learn about the probabilistic distribution of brain images, conditioned on covariates, such as age, sex, and brain structure volumes. We found that our models created realistic data, and we could use the conditioning variables to control the data generation effectively. Besides that, we created a synthetic dataset with 100,000 brain images and made it openly available to the scientific community.

W. H. L. Pinaya and P.-D. Tudosiu—Equal contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/brudfors/UniRes.

  2. 2.

    https://github.com/BBillot/SynthSeg.

  3. 3.

    https://academictorrents.com/details/63aeb864bbe2115ded0aa0d7d36334c026f0660b.

  4. 4.

    https://figshare.com/.

  5. 5.

    https://www.healthdatagateway.org/.

References

  1. Billot, B., et al.: Synthseg: domain randomisation for segmentation of brain MRI scans of any contrast and resolution. arXiv preprint arXiv:2107.09559 (2021)

  2. Brudfors, M., Balbastre, Y., Nachev, P., Ashburner, J.: MRI super-resolution using multi-channel total variation. In: Nixon, M., Mahmoodi, S., Zwiggelaar, R. (eds.) MIUA 2018. CCIS, vol. 894, pp. 217–228. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-95921-4_21

    Chapter  Google Scholar 

  3. Brudfors, M., Balbastre, Y., Nachev, P., Ashburner, J.: A tool for super-resolving multimodal clinical MRI. arXiv preprint arXiv:1909.01140 (2019)

  4. Chen, G.H., Yang, C.L., Xie, S.L.: Gradient-based structural similarity for image quality assessment. In: 2006 International Conference on Image Processing, pp. 2929–2932. IEEE (2006)

    Google Scholar 

  5. Chen, S., Ma, K., Zheng, Y.: Med3d: transfer learning for 3d medical image analysis. arXiv preprint arXiv:1904.00625 (2019)

  6. Cole, J.H., et al.: Predicting brain age with deep learning from raw imaging data results in a reliable and heritable biomarker. Neuroimage 163, 115–124 (2017)

    Article  Google Scholar 

  7. Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., Bharath, A.A.: Generative adversarial networks: an overview. IEEE Signal Process. Mag. 35(1), 53–65 (2018)

    Article  Google Scholar 

  8. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

    Google Scholar 

  9. Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Adv. Neural. Inf. Process. Syst. 34, 8780–8794 (2021)

    Google Scholar 

  10. Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873–12883 (2021)

    Google Scholar 

  11. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local Nash equilibrium. Adv. Neural Inf. Process. Syst. 30 (2017)

    Google Scholar 

  12. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Adv. Neural. Inf. Process. Syst. 33, 6840–6851 (2020)

    Google Scholar 

  13. Ho, J., Salimans, T.: Classifier-free diffusion guidance. In: NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications (2021)

    Google Scholar 

  14. Jordon, J., et al.: Synthetic data-what, why and how? arXiv preprint arXiv:2205.03257 (2022)

  15. Jordon, J., Wilson, A., van der Schaar, M.: Synthetic data: opening the data floodgates to enable faster, more directed development of machine learning methods. arXiv preprint arXiv:2012.04580 (2020)

  16. Kodali, N., Abernethy, J., Hays, J., Kira, Z.: On convergence and stability of gans. arXiv preprint arXiv:1705.07215 (2017)

  17. Kwon, G., Han, C., Kim, D.: Generation of 3D brain MRI using auto-encoding generative adversarial networks. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11766, pp. 118–126. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32248-9_14

    Chapter  Google Scholar 

  18. Larsen, A.B.L., Sønderby, S.K., Larochelle, H., Winther, O.: Autoencoding beyond pixels using a learned similarity metric. In: International Conference on Machine Learning, pp. 1558–1566. PMLR (2016)

    Google Scholar 

  19. Li, C., Bovik, A.C.: Content-partitioned structural similarity index for image quality assessment. Signal Process. Image Commun. 25(7), 517–526 (2010)

    Article  Google Scholar 

  20. Lundervold, A.S., Lundervold, A.: An overview of deep learning in medical imaging focusing on MRI. Z. Med. Phys. 29(2), 102–127 (2019)

    Article  Google Scholar 

  21. Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2794–2802 (2017)

    Google Scholar 

  22. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022)

  23. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)

    Google Scholar 

  24. Rouse, D.M., Hemami, S.S.: Analyzing the role of visual structure in the recognition of natural image content with multi-scale SSIM. In: Human Vision and Electronic Imaging XIII, vol. 6806, pp. 410–423. SPIE (2008)

    Google Scholar 

  25. Saharia, C., et al.: Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 (2022)

  26. Schuhmann, C., et al.: Laion-400m: open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 (2021)

  27. Shen, D., Wu, G., Suk, H.I.: Deep learning in medical image analysis. Annu. Rev. Biomed. Eng. 19, 221 (2017)

    Article  Google Scholar 

  28. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: International Conference on Machine Learning, pp. 2256–2265. PMLR (2015)

    Google Scholar 

  29. Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)

  30. Sudlow, C., et al.: Uk biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med. 12(3), e1001779 (2015)

    Google Scholar 

  31. Sun, L., Chen, J., Xu, Y., Gong, M., Yu, K., Batmanghelich, K.: Hierarchical amortized gan for 3D high resolution medical image synthesis. IEEE J. Biomed. Health Inform. 26(8), 3966–3975 (2022)

    Article  Google Scholar 

  32. Wang, L., Chen, W., Yang, W., Bi, F., Yu, F.R.: A state-of-the-art review on image synthesis with generative adversarial networks. IEEE Access 8, 63514–63537 (2020)

    Article  Google Scholar 

  33. Wang, T., et al.: A review on medical imaging synthesis using deep learning and its clinical applications. J. Appl. Clin. Med. Phys. 22(1), 11–36 (2021)

    Article  Google Scholar 

  34. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)

    Google Scholar 

Download references

Acknowledgements

WHLP and MJC are supported by Wellcome Innovations [WT213038/Z/18/Z]. PTD is supported by the EPSRC Research Council, part of the EPSRC DTP, grant Ref: [EP/R513064/1]. JD is supported by the Intramural Research Program of the NIMH (ZIC-MH002960 and ZIC-MH002968). PFDC is supported by the European Union’s HORIZON 2020 Research and Innovation Programme under the Marie Sklodowska-Curie Grant Agreement No 814302. PN is supported by Wellcome Innovations [WT213038/Z/18/Z] and the UCLH NIHR Biomedical Research Centre. This research has been conducted using the UK Biobank Resource (Project number: 58292).

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, School of Biomedical Engineering and Imaging Sciences, King’s College London, London, UK

    Walter H. L. Pinaya, Petru-Daniel Tudosiu, Virginia Fernandez, Sebastien Ourselin & M. Jorge Cardoso

  2. Data Science and Sharing Team, Functional Magnetic Resonance Imaging Facility, National Institute of Mental Health, Bethesda, MD, 20892, USA

    Jessica Dafflon

  3. Machine Learning Team, Functional Magnetic Resonance Imaging Facility, National Institute of Mental Health, Bethesda, MD, 20892, USA

    Jessica Dafflon

  4. Psychology and Neuroscience, Institute of Psychiatry, King’s College London, London, UK

    Pedro F. Da Costa

  5. Centre for Brain and Cognitive Development, Birkbeck College, London, UK

    Pedro F. Da Costa

  6. Institute of Neurology, University College London, London, UK

    Parashkev Nachev

Authors
  1. Walter H. L. Pinaya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Petru-Daniel Tudosiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jessica Dafflon
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pedro F. Da Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Virginia Fernandez
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Parashkev Nachev
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Sebastien Ourselin
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. M. Jorge Cardoso
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Walter H. L. Pinaya .

Editor information

Editors and Affiliations

  1. TU Darmstadt, Darmstadt, Germany

    Anirban Mukhopadhyay

  2. Istanbul Technical University, Istanbul, Turkey

    Ilkay Oksuz

  3. University Hospital Heidelberg, Heidelberg, Germany

    Sandy Engelhardt

  4. The University of Texas at Arlington, Arlington, TX, USA

    Dajiang Zhu

  5. University of Hong Kong, Hong Kong, Hong Kong

    Yixuan Yuan

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 2 (mp4 8397 KB)

Supplementary material 1 (pdf 78 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pinaya, W.H.L. et al. (2022). Brain Imaging Generation with Latent Diffusion Models. In: Mukhopadhyay, A., Oksuz, I., Engelhardt, S., Zhu, D., Yuan, Y. (eds) Deep Generative Models. DGM4MICCAI 2022. Lecture Notes in Computer Science, vol 13609. Springer, Cham. https://doi.org/10.1007/978-3-031-18576-2_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-18576-2_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-18575-5

  • Online ISBN: 978-3-031-18576-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation