Skip to main content
SpringerLink
Log in

3D Isotropic Super-resolution Prostate MRI Using Generative Adversarial Networks and Unpaired Multiplane Slices

  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

We developed a deep learning–based super-resolution model for prostate MRI. 2D T2-weighted turbo spin echo (T2w-TSE) images are the core anatomical sequences in a multiparametric MRI (mpMRI) protocol. These images have coarse through-plane resolution, are non-isotropic, and have long acquisition times (approximately 10–15 min). The model we developed aims to preserve high-frequency details that are normally lost after 3D reconstruction. We propose a novel framework for generating isotropic volumes using generative adversarial networks (GAN) from anisotropic 2D T2w-TSE and single-shot fast spin echo (ssFSE) images. The CycleGAN model used in this study allows the unpaired dataset mapping to reconstruct super-resolution (SR) volumes. Fivefold cross-validation was performed. The improvements from patch-to-volume reconstruction (PVR) to SR are 80.17%, 63.77%, and 186% for perceptual index (PI), RMSE, and SSIM, respectively; the improvements from slice-to-volume reconstruction (SVR) to SR are 72.41%, 17.44%, and 7.5% for PI, RMSE, and SSIM, respectively. Five ssFSE cases were used to test for generalizability; the perceptual quality of SR images surpasses the in-plane ssFSE images by 37.5%, with 3.26% improvement in SSIM and a higher RMSE by 7.92%. SR images were quantitatively assessed with radiologist Likert scores. Our isotropic SR volumes are able to reproduce high-frequency detail, maintaining comparable image quality to in-plane TSE images in all planes without sacrificing perceptual accuracy. The SR reconstruction networks were also successfully applied to the ssFSE images, demonstrating that high-quality isotropic volume achieved from ultra-fast acquisition is feasible.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Ghai, S. & Haider, M. A. Multiparametric-MRI in diagnosis of prostate cancer. Indian J. Urol. 31, 194–201 (2015).

    Article  Google Scholar 

  2. Fütterer, J. J. et al. Can Clinically Significant Prostate Cancer Be Detected with Multiparametric Magnetic Resonance Imaging? A Systematic Review of the Literature. Eur. Urol. 68, 1045–1053 (2015).

    Article  Google Scholar 

  3. Tsai, W. C., Field, L., Stewart, S. & Schultz, M. Review of the accuracy of multi-parametric MRI prostate in detecting prostate cancer within a local reporting service. J. Med. Imaging Radiat. Oncol. (2020) https://doi.org/10.1111/1754-9485.13029.

    Article  PubMed  Google Scholar 

  4. Stabile, A. et al. Multiparametric MRI for prostate cancer diagnosis: current status and future directions. Nat. Rev. Urol. 17, 41–61 (2020).

    Article  Google Scholar 

  5. Bittencourt, L. K., de Hollanda, E. S. & de Oliveira, R. V. Multiparametric MR Imaging for Detection and Locoregional Staging of Prostate Cancer. Top. Magn. Reson. Imaging 25, 109–117 (2016).

    Article  Google Scholar 

  6. Polanec, S. H. et al. 3D T2-weighted imaging to shorten multiparametric prostate MRI protocols. Eur. Radiol. 28, 1634–1641 (2018).

    Article  Google Scholar 

  7. Rosenkrantz, A. B. et al. Prostate cancer: Comparison of 3D T2-weighted with conventional 2D T2-weighted imaging for image quality and tumor detection. AJR Am. J. Roentgenol. 194, 446–452 (2010).

    Article  Google Scholar 

  8. Westphalen, A. C. et al. High-Resolution 3-T Endorectal Prostate MRI: A Multireader Study of Radiologist Preference and Perceived Interpretive Quality of 2D and 3D T2-Weighted Fast Spin-Echo MR Images. AJR Am. J. Roentgenol. 206, 86–91 (2016).

    Article  Google Scholar 

  9. Caglic, I. et al. Defining the incremental value of 3D T2-weighted imaging in the assessment of prostate cancer extracapsular extension. Eur. Radiol. 29, 5488–5497 (2019).

    Article  Google Scholar 

  10. Alansary, A. et al. PVR: Patch-to-Volume Reconstruction for Large Area Motion Correction of Fetal MRI. IEEE Trans. Med. Imaging 36, 2031–2044 (2017).

    Article  Google Scholar 

  11. Uus, A. et al. Deformable Slice-to-Volume Registration for Motion Correction of Fetal Body and Placenta MRI. IEEE Trans. Med. Imaging (2020) https://doi.org/10.1109/TMI.2020.2974844.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Ledig, C. et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition 4681–4690 (2017).

  13. Lyu, Q., You, C., Shan, H., Zhang, Y. & Wang, G. Super-resolution MRI and CT through GAN-CIRCLE. In Developments in X-Ray Tomography XII vol. 11113 111130X (International Society for Optics and Photonics, 2019).

  14. He, X. et al. Super-resolution magnetic resonance imaging reconstruction using deep attention networks. In Medical Imaging 2020: Image Processing vol. 11313 113132J (International Society for Optics and Photonics, 2020).

  15. Lyu, Q., You, C., Shan, H. & Wang, G. Super-resolution MRI through Deep Learning. arXiv [physics.med-ph] (2018).

  16. Sood, R. & Rusu, M. Anisotropic Super Resolution In Prostate Mri Using Super Resolution Generative Adversarial Networks. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) 1688–1691 (2019).

  17. McCann, M. T., Jin, K. H. & Unser, M. Convolutional Neural Networks for Inverse Problems in Imaging: A Review. IEEE Signal Process. Mag. 34, 85–95 (2017).

    Article  Google Scholar 

  18. Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. 2017 IEEE International Conference on Computer Vision (ICCV) (2017) https://doi.org/10.1109/iccv.2017.244.

  19. Geert Litjens, Oscar Debats, Jelle Barentsz, Nico Karssemeijer, and Henkjan Huisman. ProstateX Challenge data. (2017) https://doi.org/10.7937/K9TCIA.2017.MURS5CL.

    Article  Google Scholar 

  20. Tustison, N. J. et al. N4ITK: improved N3 bias correction. IEEE Trans. Med. Imaging 29, 1310–1320 (2010).

    Article  Google Scholar 

  21. Goodfellow, I. et al. Generative Adversarial Nets. in Advances in Neural Information Processing Systems 27 (eds. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D. & Weinberger, K. Q.) 2672–2680 (Curran Associates, Inc., 2014).

  22. Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. in Proceedings of the IEEE conference on computer vision and pattern recognition 1125–1134 (2017).

  23. Klein, A. et al. Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. Neuroimage 46, 786–802 (2009).

    Article  Google Scholar 

  24. Yi, X., Walia, E. & Babyn, P. Generative adversarial network in medical imaging: A review. Med. Image Anal. 58, 101552 (2019).

  25. Blau, Y., Mechrez, R., Timofte, R., Michaeli, T. & Zelnik-Manor, L. The 2018 PIRM Challenge on Perceptual Image Super-resolution. arXiv [cs.CV] (2018).

  26. Ma, C., Yang, C.-Y., Yang, X. & Yang, M.-H. Learning a no-reference quality metric for single-image super-resolution. Comput. Vis. Image Underst. 158, 1–16 (2017).

    Article  Google Scholar 

  27. Mittal, A., Soundararajan, R. & Bovik, A. C. Making a ‘Completely Blind’ Image Quality Analyzer. IEEE Signal Process. Lett. 20, 209–212 (2013).

    Article  Google Scholar 

  28. Moorthy, A. K. & Bovik, A. C. A Two-Step Framework for Constructing Blind Image Quality Indices. IEEE Signal Process. Lett. 17, 513–516 (2010).

    Article  Google Scholar 

  29. Giganti, F. et al. Understanding PI-QUAL for prostate MRI quality: a practical primer for radiologists. Insights Imaging 12, 59 (2021).

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Radiology, Columbia University Irving Medical Center, 622 W 168 11th St, New York, NY, USA

    Yucheng Liu, Michael Liu, Hao-Yun Hsu, Richard Ha, Hiram Shaish & Sachin Jambawalikar

  2. Department of Information and Computer Engineering, Chung Yuan Christian University, Chung Li District, 200 Chung Pei Road, Taoyuan City, Taiwan

    Yulin Liu

  3. Computational Oncology Service, Department of Epidemiology & Biostatistics, Memorial Sloan, Kettering Cancer Center 485 Lexington Ave, New York, NY, 10017, USA

    Rami Vanguri

  4. Global MR Applications and Workflow, GE Healthcare, New York, NY, USA

    Daniel Litwiller

Authors
  1. Yucheng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yulin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rami Vanguri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniel Litwiller
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hao-Yun Hsu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Richard Ha
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Hiram Shaish
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Sachin Jambawalikar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yucheng Liu.

Ethics declarations

Conflict of Interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Liu, Y., Vanguri, R. et al. 3D Isotropic Super-resolution Prostate MRI Using Generative Adversarial Networks and Unpaired Multiplane Slices. J Digit Imaging 34, 1199–1208 (2021). https://doi.org/10.1007/s10278-021-00510-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-021-00510-w

Keywords

Search

Navigation