Skip to main content
SpringerLink
Log in

Inpainting Cropped Diffusion MRI Using Deep Generative Models

  • Conference paper
  • First Online:
Predictive Intelligence in Medicine (PRIME 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12329))

Included in the following conference series:

Abstract

Minor artifacts introduced during image acquisition are often negligible to the human eye, such as a confined field of view resulting in MRI missing the top of the head. This cropping artifact, however, can cause suboptimal processing of the MRI resulting in data omission or decreasing the power of subsequent analyses. We propose to avoid data or quality loss by restoring these missing regions of the head via variational autoencoders (VAE), a deep generative model that has been previously applied to high resolution image reconstruction. Based on diffusion weighted images (DWI) acquired by the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA), we evaluate the accuracy of inpainting the top of the head by common autoencoder models (U-Net, VQVAE, and VAE-GAN) and a custom model proposed herein called U-VQVAE. Our results show that U-VQVAE not only achieved the highest accuracy, but also resulted in MRI processing producing lower fractional anisotropy (FA) in the supplementary motor area than FA derived from the original MRIs. Lower FA implies that inpainting reduces noise in processing DWI and thus increases the quality of the generated results. The code is available at https://github.com/RdoubleA/DWI-inpainting.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Soares, J.M., Marques, P., Alves, V., Sousa, N.: A hitchhiker’s guide to diffusion tensor imaging. Front. Neurosci. 7(7), 1–14 (2013). https://doi.org/10.3389/fnins.2013.00031

    Article  Google Scholar 

  2. Le Bihan, D., Poupon, C., Amadon, A., Lethimonnier, F.: Artifacts and pitfalls in diffusion MRI. J. Magn. Reson. Imaging 24(3), 478–488 (2006). https://doi.org/10.1002/jmri.20683

    Article  Google Scholar 

  3. Greve, D.N., Fischl, B.: Accurate and robust brain image alignment using boundary-based registration. NeuroImage 48(1), 63–72 (2009). https://doi.org/10.1016/j.neuroimage.2009.06.060. https://linkinghub.elsevier.com/retrieve/pii/S1053811909006752

    Article  Google Scholar 

  4. Elharrouss, O., Almaadeed, N., Al-Maadeed, S., Akbari, Y.: Image inpainting: a review. Neural Process. Lett. 51(2), 2007–2028 (2020). https://doi.org/10.1007/s11063-019-10163-0. http://link.springer.com/10.1007/s11063-019-10163-0

    Article  Google Scholar 

  5. Lu, H., Liu, Q., Zhang, M., Wang, Y., Deng, X.: Gradient-based low rank method and its application in image inpainting. Multimed. Tools Appl. 77(5), 5969–5993 (2017). https://doi.org/10.1007/s11042-017-4509-0

    Article  Google Scholar 

  6. Jin, K.H., Ye, J.C.: Annihilating filter-based low-rank Hankel matrix approach for image inpainting. IEEE Trans. Image Process. 24(11), 3498–3511 (2015). https://doi.org/10.1109/TIP.2015.2446943

    Article  MathSciNet  MATH  Google Scholar 

  7. Guo, Q., Gao, S., Zhang, X., Yin, Y., Zhang, C.: Patch-based image inpainting via two-stage low rank approximation. IEEE Trans. Vis. Comput. Graph. 24(6), 2023–2036 (2018). https://doi.org/10.1109/TVCG.2017.2702738

    Article  Google Scholar 

  8. Kozhekin, N., Savchenko, V., Senin, M., Hagiwara, I.: An approach to surface retouching and mesh smoothing. Vis. Comput. 19(7–8), 549–564 (2003). https://doi.org/10.1007/s00371-003-0218-y

    Article  Google Scholar 

  9. Chan, T.F., Shen, J.: Nontexture inpainting by curvature-driven diffusions. J. Vis. Commun. Image Represent. 12(4), 436–449 (2001). https://doi.org/10.1006/jvci.2001.0487

  10. Alsalamah, M., Amin, S.: Medical image inpainting with RBF interpolation technique. Int. J. Adv. Comput. Sci. Appl. 7(8), 91–99 (2016). https://doi.org/10.14569/ijacsa.2016.070814

  11. Yan, Z., Li, X., Li, M., Zuo, W., Shan, S.: Shift-net: image inpainting via deep feature rearrangement. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11218, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_1

    Chapter  Google Scholar 

  12. Shelhamer, E., Long, J., Darrell, T.: Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 640–651 (2017). https://doi.org/10.1109/TPAMI.2016.2572683

    Article  Google Scholar 

  13. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  14. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, vol. 2017, pp. 5967–5976, January 2017. https://doi.org/10.1109/CVPR.2017.632

  15. Armanious, K., Mecky, Y., Gatidis, S., Yang, B.: Adversarial inpainting of medical image modalities. In: ICASSP 2019 – 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3267–3271. IEEE, May 2019. https://doi.org/10.1109/ICASSP.2019.8682677. https://ieeexplore.ieee.org/document/8682677/

  16. Armanious, K., Kumar, V., Abdulatif, S., Hepp, T., Gatidis, S., Yang, B.: ipA-MedGAN: inpainting of arbitrary regions in medical imaging (2019). http://arxiv.org/abs/1910.09230

  17. Armanious, K., Gatidis, S., Nikolaou, K., Yang, B., Kustner, T.: Retrospective correction of rigid and non-rigid MR motion artifacts using GANs. In: Proceedings of the International Symposium on Biomedical Imaging, vol. 2019, pp. 1550–1554, April 2019. https://doi.org/10.1109/ISBI.2019.8759509

  18. Sabokrou, M., Pourreza, M., Fayyaz, M., Entezari, R., Fathy, M., Gall, J., Adeli, E.: AVID: adversarial visual irregularity detection. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11366, pp. 488–505. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20876-9_31

    Chapter  Google Scholar 

  19. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: 2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings (Ml), pp. 1–14 (2014)

    Google Scholar 

  20. Xie, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: Advances in Neural Information Processing Systems, vol. 1, pp. 341–349 (2012)

    Google Scholar 

  21. Dosovitskiy, A., Brox, T.: Generating images with perceptual similarity metrics based on deep networks. In: Advances in Neural Information Processing Systems, pp. 658–666 (2016)

    Google Scholar 

  22. Van Den Oord, A., Vinyals, O., Kavukcuoglu, K.: Neural discrete representation learning. In: Advances in Neural Information Processing Systems, vol. 2017 (NIPS), pp. 6307–6316, December 2017

    Google Scholar 

  23. Razavi, A., van den Oord, A., Vinyals, O.: Generating diverse high-fidelity images with VQ-VAE-2 (2019). http://arxiv.org/abs/1906.00446

  24. Tudosiu, P.D., et al.: Neuromorphologicaly-preserving Volumetric data encoding using VQ-VAE, pp. 1–13 (2020). http://arxiv.org/abs/2002.05692

  25. Larsen, A.B.L., Sønderby, S.K., Larochelle, H., Winther, O.: Autoencoding beyond pixels using a learned similarity metric. In: 33rd International Conference on Machine Learning, ICML 2016, vol. 4, pp. 2341–2349 (2016)

    Google Scholar 

  26. Van Essen, D.C., et al.: The human connectome project: a data acquisition perspective. NeuroImage 62(4), 2222–2231 (2012). https://doi.org/10.1016/j.neuroimage.2012.02.018

    Article  Google Scholar 

  27. Hodge, M.R., et al.: ConnectomeDB-sharing human brain connectivity data. NeuroImage 124(3), 1102–1107 (2016). https://doi.org/10.1016/j.neuroimage.2015.04.046. https://linkinghub.elsevier.com/retrieve/pii/S1053811915003468

    Article  Google Scholar 

  28. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  29. Pohl, K.M., et al.: Harmonizing DTI measurements across scanners to examine the development of white matter microstructure in 803 adolescents of the NCANDA study. NeuroImage 130, 194–213 (2016). https://doi.org/10.1016/j.neuroimage.2016.01.061

    Article  Google Scholar 

  30. Rohlfing, T., Zahr, N.M., Sullivan, E.V., Pfefferbaum, A.: The SRI24 multichannel atlas of normal adult human brain structure. Hum. Brain Mapp. 31(5), 798–819 (2010). https://doi.org/10.1002/hbm.20906

    Article  Google Scholar 

  31. Cook, P.a., Bai, Y., Seunarine, K.K., Hall, M.G., Parker, G.J., Alexander, D.C.: Camino: open-source diffusion-MRI reconstruction and processing. In: 14th Scientific Meeting of the International Society for Magnetic Resonance in Medicine, vol. 14, p. 2759 (2006)

    Google Scholar 

  32. Farrell, J.A.D., et al.: Effects of SNR on the accuracy and reproducibility of DTI-derived fractional anisotropy, mean diffusivity, and principal eigenvector measurements at 1.5T. J. Magn. Reson. 26(3), 756–767 (2010). https://doi.org/10.1002/jmri.21053.Effects

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by NIH Grants AA021697, AA005965, and AA010723. This work was also supported by the National Science Foundation Graduate Research Fellowship and the 2020 HAI-AWS Cloud Credits Award.

Author information

Authors and Affiliations

  1. Stanford University, Stanford, CA, USA

    Rafi Ayub, Qingyu Zhao, Edith V. Sullivan, Adolf Pfefferbaum, Ehsan Adeli & Kilian M. Pohl

  2. SRI International, Menlo Park, CA, USA

    Adolf Pfefferbaum & Kilian M. Pohl

  3. University of California, San Diego, La Jolla, CA, USA

    M. J. Meloy

Authors
  1. Rafi Ayub
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qingyu Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. J. Meloy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Edith V. Sullivan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Adolf Pfefferbaum
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ehsan Adeli
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Kilian M. Pohl
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kilian M. Pohl .

Editor information

Editors and Affiliations

  1. Istanbul Technical University, Istanbul, Turkey

    Islem Rekik

  2. Stanford University, Stanford, CA, USA

    Ehsan Adeli

  3. Daegu Gyeongbuk Institute of Science and Technology, Daegu, Korea (Republic of)

    Sang Hyun Park

  4. The University of Edinburgh, Edinburgh, UK

    Maria del C. Valdés Hernández

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ayub, R. et al. (2020). Inpainting Cropped Diffusion MRI Using Deep Generative Models. In: Rekik, I., Adeli, E., Park, S.H., Valdés Hernández, M.d.C. (eds) Predictive Intelligence in Medicine. PRIME 2020. Lecture Notes in Computer Science(), vol 12329. Springer, Cham. https://doi.org/10.1007/978-3-030-59354-4_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-59354-4_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-59353-7

  • Online ISBN: 978-3-030-59354-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation