Skip to main content
SpringerLink
Log in

MedSRGAN: medical images super-resolution using generative adversarial networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Super-resolution (SR) in medical imaging is an emerging application in medical imaging due to the needs of high quality images acquired with limited radiation dose, such as low dose Computer Tomography (CT), low field magnetic resonance imaging (MRI). However, because of its complexity and higher visual requirements of medical images, SR is still a challenging task in medical imaging. In this study, we developed a deep learning based method called Medical Images SR using Generative Adversarial Networks (MedSRGAN) for SR in medical imaging. A novel convolutional neural network, Residual Whole Map Attention Network (RWMAN) was developed as the generator network for our MedSRGAN in extracting the useful information through different channels, as well as paying more attention on meaningful regions. In addition, a weighted sum of content loss, adversarial loss, and adversarial feature loss were fused to form a multi-task loss function during the MedSRGAN training. 242 thoracic CT scans and 110 brain MRI scans were collected for training and evaluation of MedSRGAN. The results showed that MedSRGAN not only preserves more texture details but also generates more realistic patterns on reconstructed SR images. A mean opinion score (MOS) test on CT slices scored by five experienced radiologists demonstrates the efficiency of our methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. K Armanious, C Jiang, M Fischer, T Küstner, K Nikolaou, S Gatidis, B Yang 2018. “MedGAN: Medical Image Translation using GANs”. In: arXiv:1806.06397 [cs.CV]

  2. Berrington GA, Kim KP, Berg CD (2008) Low-dose lung CT screening before age 55: estimates of the mortality reduction required to outweigh the radiation-induced cancer risk. In: J Med Screen 15(3):153–158. https://doi.org/10.1258/jms.2008.008052

    Article  Google Scholar 

  3. Chen Y, Xie Y, Zhou Z, Shi F, Christodoulou AG, Li D (2018) Brain MRI super resolution using 3D deep densely connected neural networks. In: In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), Washington, pp 739–742

  4. Y Chen, F Shi, A G Christodoulou, Y Xie, Z Zhou, D Li 2018. “Efficient and Accurate MRI Super-Resolution Using a Generative Adversarial Network and 3D Multi-level Densely Connected Network”. In: Frangi A., Schnabel J., Davatzikos C., Alberola-López C., Fichtinger G. (eds) Medical Image Computing and Computer Assisted Intervention - MICCAI. MICCAI 2018. Lecture notes in computer science, vol 11070. Springer, Cham

  5. Z Cheng, Y Ding, X He, L Zhu, X Song, M Kankanhalli 2018. “A3NCF: an adaptive aspect attention model for rating prediction”. In: IJCAI'18 Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 3748–3754, July

  6. K Cho et al. 2014 “Learning phrase representations using RNN encoder-decoder for statistical machine translation”. In: arXiv:1406.1078 [cs.CL]

  7. Christe A, Torrente JC, Lin M, Yen A, Hallett R, Roychoudhury K, Schmitzberger F, Vock P, Roos J (2011) CT screening and follow-up of lung nodules: Effects of tube current-time setting and nodule size and density on detectability and of tube current-time setting on apparent size. AJR am J Roentgenol 197(3):623–630. https://doi.org/10.2214/AJR.10.5288

    Article  Google Scholar 

  8. C Dong, C C Loy, K He, X Tang 2014. “Learning a Deep Convolutional Network for Image Super-Resolution”. In: Fleet D., Pajdla T., Schiele B., Tuytelaars T. (eds) Computer Vision – ECCV 2014. ECCV 2014. Lecture notes in computer science, vol 8692. Springer, Cham

  9. Gatys LA, Ecker AS, Bethge M (2016) Image Style Transfer Using Convolutional Neural Networks. In: IEEE conference on computer vision and pattern recognition (CVPR), vol 2016, Las Vegas, pp 2414–2423

  10. Gondara L (2016) Medical Image Denoising Using Convolutional Denoising Autoencoders. In: IEEE 16th international conference on data mining workshops (ICDMW), vol 2016, Barcelona, pp 241–246

  11. I Goodfellow, J Pouget-Abadie, M Mirza, B Xu, D Warde-Farley, S Ozair, A Courville, Y Bengio 2014. “Generative Adversarial Nets”. In: Z. Ghahramani and M. Welling and C. Cortes and N. D. Lawrence and K. Q. Weinberger (eds) Advances in Neural Information Processing Systems 27, , Curran associates, Inc pp 2672–2680.

  12. Greenspan H (2008) Super-resolution in medical imaging. In: Comput J 52(1):43–63

    Google Scholar 

  13. Guo Y, Cheng Z, Nie L, Wang Y, Ma J, Kankanhalli M (2019) Attentive Long Short-Term Preference Modeling for Personalized Product Search. ACM Transactions on Information Systems (TOIS) 37(2). https://doi.org/10.1145/3295822

  14. He K, Zhang X, Ren S, Sun J (2016) Deep Residual Learning for Image Recognition. In: IEEE conference on computer vision and pattern recognition (CVPR), vol 2016, Las Vegas, pp 770–778

  15. J Hu, L Shen, G Sun n.d.. “Squeeze-and-excitation networks”. In: IEEE Transactions on Pattern Analysis and Machine Intelligence

  16. Huang G, Liu Z, Maaten LVD, Weinberger KQ (2017) Densely Connected Convolutional Networks. In: IEEE conference on computer vision and pattern recognition (CVPR), vol 2017, Honolulu, pp 2261–2269

  17. Hunink MGM, Gazelle GS (2003) CT screening: A trade-off of risks, benefits, and costs. the journal of clinical investigation 111(11):1612–1619. https://doi.org/10.1172/JCI200318842

    Article  Google Scholar 

  18. Isola P, Zhu J, Zhou T, Efros AA (2017) Image-to-Image Translation with Conditional Adversarial Networks. In: IEEE conference on computer vision and pattern recognition (CVPR), vol 2017, Honolulu, pp 5967–5976

  19. G Jiang, Y Lu, J Wei, Y Xu 2019. “Synthesize mammogram from digital breast tomosynthesis with gradient guided cGANs”. In: Shen D. et al. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI. MICCAI 2019. Lecture notes in computer science, vol 11769. Springer, Cham.

  20. Jifara W, Jiang F, Rho S, Cheng M, Liu S (2019) Medical image denoising using convolutional neural network: a residual learning approach. In: J Supercomput 75:704–718. https://doi.org/10.1007/s11227-017-2080-0

    Article  Google Scholar 

  21. Y Jing, Y Yang, Z Feng, J Ye, Y Yu, M Song 2017. “Neural Style Transfer: A Review”. In: arXiv:1705.04058 [cs.CV]

  22. J Johnson, A Alahi, Fei-Fei Li 2016 “Perceptual Losses for Real-Time Style Transfer and Super-Resolution”. In: Leibe B., Matas J., Sebe N., Welling M. (eds) Computer Vision – ECCV. ECCV 2016. Lecture notes in computer science, vol 9906. Springer, Cham.

  23. A Jolicoeur-Martineau 2018. “The relativistic discriminator: a key element missing from standard GAN”. In: arXiv:1807.00734 [cs.LG]

  24. Karadi RL (2006) Radiation risk of screening with low dose CT. Thorax 61(6):548

    Google Scholar 

  25. Kavanagh J, Liu G, Menezes R, O’Kane GM, McGregor M, Tsao M, Shepherd FA, Schmidt H (2018) Importance of Long-term Low-Dose CT Follow-up after Negative Findings at Previous Lung Cancer Screening. Radiology 289(1). https://doi.org/10.1148/radiol.2018180053

  26. Kim J, Lee JK, Lee KM (2016) Deeply-Recursive Convolutional Network for Image Super-Resolution. In: IEEE conference on computer vision and pattern recognition (CVPR), vol 2016, Las Vegas, pp 1637–1645

  27. Kim J, Lee JK, Lee KM (2016) Accurate Image Super-Resolution Using Very Deep Convolutional Networks. In: IEEE conference on computer vision and pattern recognition (CVPR), vol 2016, Las Vegas, pp 1646–1654

  28. D P Kingma, J Ba 2014. “Adam: A method for stochastic optimization”. In: arXiv:1412.6980 [cs.LG]

  29. D Kouame, M Ploquin 2009. “Super-resolution in medical imaging: an illustrative approach through ultrasound”. In: IEEE International Symposium on Biomedical Imaging pp 249-252

  30. Ledig C et al (2017) Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In: IEEE conference on computer vision and pattern recognition (CVPR), vol 2017, Honolulu, pp 105–114

  31. D Lee, J Yoo, J C Ye 2017 “Compressed sensing and parallel MRI using deep residual learning”. In: Proc. 25st Annu. Meeting ISMRM

  32. Libby DM, Wu N, Lee IJ, Farooqi A, Smith JP, Pasmantier MW, McCauley D, Yankelevitz DF, Henschke CI (2006) CT screening for lung cancer: The value of short-term CT follow-up. chest 129(4):1039–1042

    Google Scholar 

  33. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced Deep Residual Networks for Single Image Super-Resolution. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), vol 2017, Honolulu, pp 1132–1140

  34. Lingala SG, Hu Y, DiBella E, Jacob M (2011) Accelerated dynamic MRI exploiting sparsity and low-rank structure: k-t SLR. In: IEEE Transactions on Medical Imaging 30(5):1042–1054

    Google Scholar 

  35. Mardani M et al (2019) Deep generative adversarial neural networks for compressive sensing MRI. IEEE Transactions on Medical Imaging 38(1):167–179

    Google Scholar 

  36. M Mirza, S Osindero 2014. “Conditional Generative Adversarial Nets”. In: arXiv:1411.1784 [cs.LG]

  37. Mohammed RH, Sabry YY, Nasef AA (2011) Brain MRI screening showing evidences of early central nervous system involvement in patients with systemic sclerosis. Rheumatol Int. 31(5):667–671. https://doi.org/10.1007/s00296-009-1325-5

    Article  Google Scholar 

  38. Rampinelli C, Origgi D, Bellomi M (2013) Low-dose CT: technique, reading methods and image interpretation. Cancer Imaging 12:548–556. https://doi.org/10.1102/1470-7330.2012.0049

    Article  Google Scholar 

  39. Ravishankar S, Bresler Y (2011) MR image reconstruction from highly undersampled k-space data by dictionary learning. In: IEEE Transactions on Medical Imaging 30(5):1028–1041

    Google Scholar 

  40. Setio AAA, Traverso A, de Bel T, Berens MSN, Bogaard C, Cerello P, Chen H, Dou Q, Fantacci ME, Geurts B, Gugten R, Heng PA, Jansen B, de Kaste MMJ, Kotov V, Lin JYH, Manders JTMC, Sóñora-Mengana A, García-Naranjo JC, Papavasileiou E, Prokop M, Saletta M, Schaefer-Prokop CM, Scholten ET, Scholten L, Snoeren MM, Torres EL, Vandemeulebroucke J, Walasek N, Zuidhof GCA, Ginneken B, Jacobs C (2017) Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge. Med Image Anal 42:1–13. https://doi.org/10.1016/j.media.2017.06.015

    Article  Google Scholar 

  41. Shi W et al (2016) Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In: IEEE conference on computer vision and pattern recognition (CVPR), vol 2016, Las Vegas, pp 1874–1883

  42. K Simonyan, A Zisserman. 2014 “Very Deep Convolutional Networks for Large-Scale Image Recognition”. In: arXiv:1409.1556 [cs.CV]

  43. Ulus S, Suleyman E, Ozcan UA, Karaarslan E (2016) Whole-Body MRI Screening in Asymptomatic Subjects; Preliminary Experience and Long-Term Follow-Up Findings. Polish Journal of Radiology 81:407–414. https://doi.org/10.12659/PJR.897570

    Article  Google Scholar 

  44. Wang F et al (2017) Residual Attention Network for Image Classification. In: IEEE conference on computer vision and pattern recognition (CVPR), vol 2017, Honolulu, pp 6450–6458

  45. X Wang et al. 2018 “ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks”. In: Leal-Taixé L., Roth S. (eds) Computer Vision – ECCV Workshops. ECCV 2018. Lecture notes in computer science, vol 11133. Springer, Cham

  46. S Woo, J Park, J Y Lee, I S Kweon 2018. “CBAM: Convolutional Block Attention Module”. In: Ferrari V., Hebert M., Sminchisescu C., Weiss Y. (eds) Computer Vision – ECCV 2018. ECCV. Lecture notes in computer science, vol 11211. Springer, Cham.

  47. Yasuda S, Ide M (2005) PET and cancer screening. Annals of nuclear medicine 19(3):167–177

    Google Scholar 

  48. C You et al. 2018 “CT Super-resolution GAN Constrained by the Identical, Residual, and Cycle Learning Ensemble(GAN-CIRCLE)”. In: arXiv:1808.04256 [eess.IV]

  49. Y Zhang, K Li, K Li, L Wang, B Zhong, Y Fu 2018. “Image Super-Resolution Using Very Deep Residual Channel Attention Networks”. In: Ferrari V., Hebert M., Sminchisescu C., Weiss Y. (eds) Computer Vision – ECCV 2018. ECCV 2018. Lecture notes in computer science, vol 11211. Springer, Cham

  50. Zhu J, Park T, Isola P, Efros AA (2017) Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), Venice, pp 2242–2251

Download references

Acknowledgments

This work was supported in part by the National Key R&D Program of China under Grant 2018YFC1704206, Grant 2016YFB0200602, in part by the NSFC under Grant 81971691, Grant 81801809, Grant 81830052, Grant 81827802, Grant U1811461, and Grant 11401601, in part by the Science and Technology Program of Guangzhou under Grant 20180420053, in part by the Science and Technology Innovative Project of Guangdong Province under Grant 2016B030307003, Grant 2015B010110003, and Grant 2015B020233008, in part by the Science and Technology Planning Project of Guangdong Province under Key Grant 2017B020210001, in part by the Guangzhou Science and Technology Creative Project under Key Grant 201604020003, in part by the Guangdong Province Key Laboratory of Computational Science Open Grant 2018009, in part by the Construction Project of Shanghai Key Laboratory of Molecular Imaging 18DZ2260400, and in part by China postdoctoral science foundation No.2019M653185.

Author information

Authors and Affiliations

  1. School of Data and Computer Science, Sun Yat-sen University, Guangzhou, China

    Yuchong Gu, Zitao Zeng, Haibin Chen, Zhuoren Jiang & Yao Lu

  2. Department of Radiology, University of Michigan, Ann Arbor, MI, USA

    Jun Wei

  3. Department of Radiology, The Fifth Affiliated Hospital of Sun Yat-sen University, Zhuhai, China

    Yaqin Zhang, Binghui Chen, Yingqin Li, Yujuan Qin & Qing Xie

  4. Guangdong Province Key Laboratory of Computational Science, Guangzhou, China

    Yao Lu

Authors
  1. Yuchong Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zitao Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haibin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yaqin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Binghui Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yingqin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yujuan Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Qing Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Zhuoren Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Yao Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Haibin Chen, Zhuoren Jiang or Yao Lu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gu, Y., Zeng, Z., Chen, H. et al. MedSRGAN: medical images super-resolution using generative adversarial networks. Multimed Tools Appl 79, 21815–21840 (2020). https://doi.org/10.1007/s11042-020-08980-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08980-w

Keywords

Search

Navigation