Skip to main content

Advertisement

SpringerLink
Log in

Capsule GAN for prostate MRI super-resolution

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Prostate cancer is a prevalent disease among adult men. One in seven Canadian men is diagnosed with this cancer in their lifetime. Super-Resolution (SR) can facilitate early diagnosis and potentially save many lives. In this paper, a robust and accurate model is proposed for prostate MRI SR. For the first time, MSG-GAN and CapsGAN are utilized simultaneously for high-scale medical SR. The model is trained on the Prostate-Diagnosis and PROSTATEx datasets. The proposed model outperformed the state-of-the-art prostate SR model in all similarity metrics with substantial margins. For \(8 \times\) SR, 19.77, 0.60, and 0.79 are achieved for Peak Signal-to-Noise Ratio (PSNR), Structural SIMilarity index metric (SSIM), and Multi-Scale Structural SIMilarity index metric (MS-SSIM), respectively. A new task-specific similarity assessment is introduced as well. A classifier is trained for severe cancer detection. The drop in the accuracy of this model when dealing with super-resolved images is used to evaluate the ability of medical detail reconstruction of the SR models. The proposed model surpassed state-of-the-art work with a 6% margin. The model is also more compact in comparison with the related architecture and has 45% less number of trainable parameters. The proposed SR model is a step towards an efficient and accurate general medical SR platform.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. https://www.cancerimagingarchive.net/

Abbreviations

L :

Loss

N :

batch size

y :

Ground truth label

p(y):

Predicted probability

\(\delta\) :

Coefficient

\(\alpha\) :

Coefficient

\(s_i\) :

Super-resolved image with the scale of \(2^i\)

\(f_vgg19\) :

Feature vector extracted from VGG19

\(y^{img}_{s_i}\) :

Ground truth image with the \(2^{i-3}\) of the original size

I :

Reference image

\(\hat{I}\) :

Reconstructed image

\(\mu\) :

Mean

\(\sigma\) :

Variance

\(\sigma _{I \hat{I}}\) :

Covarience

References

  1. Akbari H, Halig L, Schuster DM, Fei B, Osunkoya A, Master V, Nieh P, Chen G (2012) Hyperspectral imaging and quantitative analysis for prostate cancer detection. Journal of biomedical optics 17(7):076005

    Article  Google Scholar 

  2. Amaranageswarao G, Deivalakshmi S, Ko SB (2020a) Residual learning based densely connected deep dilated network for joint deblocking and super resolution. Applied Intelligence 50(7):2177–2193

  3. Amaranageswarao G, Deivalakshmi S, Ko SB (2020b) Wavelet based medical image super resolution using cross connected residual-in-dense grouped convolutional neural network. Journal of Visual Communication and Image Representation 70:102819

  4. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein generative adversarial networks. In: International conference on machine learning, PMLR, pp 214–223

  5. Bloch, Nicolas B, Jain, Ashali, Carl JC (2015) Data from prostate-diagnosis. The Cancer Imaging Archive

  6. Brenner DR, Weir HK, Demers AA, Ellison LF, Louzado C, Shaw A, Turner D, Woods RR, Smith LM (2020) Projected estimates of cancer in canada in 2020. Cmaj 192(9):E199–E205

    Article  Google Scholar 

  7. Brock A, Donahue J, Simonyan K (2018) Large scale gan training for high fidelity natural image synthesis. arXiv preprint arxiv:180911096

  8. Castro-Zunti R, Park EH, Choi Y, Jin GY, Sb Ko (2020) Early detection of ankylosing spondylitis using texture features and statistical machine learning, and deep learning, with some patient age analysis. Computerized Medical Imaging and Graphics 82:101718

    Article  Google Scholar 

  9. Chae KJ, Jin GY, Ko SB, Wang Y, Zhang H, Choi EJ, Choi H (2020) Deep learning for the classification of small (\(\leq\) 2 cm) pulmonary nodules on ct imaging: a preliminary study. Academic radiology 27(4):e55–e63

    Article  Google Scholar 

  10. Chaudhari AS, Fang Z, Kogan F, Wood J, Stevens KJ, Gibbons EK, Lee JH, Gold GE, Hargreaves BA (2018) Super-resolution musculoskeletal mri using deep learning. Magnetic resonance in medicine 80(5):2139–2154

    Article  Google Scholar 

  11. Chen Y, Xie Y, Zhou Z, Shi F, Christodoulou AG, Li D (2018) Brain mri super resolution using 3d deep densely connected neural networks. In: IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), IEEE, pp 739–742

  12. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, Ieee, pp 248–255

  13. Dong C, Loy CC, He K, Tang X (2015) Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(2):295–307

    Article  Google Scholar 

  14. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  15. Haghanifar A, Majdabadi MM, Ko SB (2020a) Automated teeth extraction from dental panoramic x-ray images using genetic algorithm. In: 2020 IEEE International Symposium on Circuits and Systems (ISCAS), IEEE, pp 1–5

  16. Haghanifar A, Majdabadi MM, Ko SB (2020b) Paxnet: Dental caries detection in panoramic x-ray using ensemble transfer learning and capsule classifier. arXiv preprint arXiv:201213666

  17. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  18. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  19. Ibrahim D (2020) Normal prostate (mri) | radiology case | radiopaedia.org URL https://radiopaedia.org/cases/normal-prostate-mri-1

  20. Islam MS, Kaabouch N, Hu WC (2013) A survey of medical imaging techniques used for breast cancer detection. In: IEEE International Conference on Electro-Information Technology, EIT 2013, IEEE, pp 1–5

  21. Jolicoeur-Martineau A (2018) The relativistic discriminator: a key element missing from standard gan. arXiv preprint arXiv:180700734

  22. Karnewar A, Wang O (2020) Msg-gan: Multi-scale gradients for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 7799–7808

  23. Karras T, Aila T, Laine S, Lehtinen J (2017) Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:171010196

  24. Kasivisvanathan V, Rannikko AS, Borghi M, Panebianco V, Mynderse LA, Vaarala MH, Briganti A, Budäus L, Hellawell G, Hindley RG et al (2018) Mri-targeted or standard biopsy for prostate-cancer diagnosis. New England J Medicine 378(19):1767–1777

  25. Kim J, Kwon Lee J, Mu Lee K (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654

  26. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, et al. (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4681–4690

  27. Li Z, Wang Y, Yu J (2017) Reconstruction of thin-slice medical images using generative adversarial network. In: International workshop on machine learning in medical imaging, Springer, pp 325–333

  28. Liau J, Goldberg D, Arif-Tiwari H (2019) Prostate cancer detection and diagnosis: role of ultrasound with mri correlates. Current Radiology Reports 7(3):7

    Article  Google Scholar 

  29. Litjens G, Debats O, Barentsz J, Karssemeijer N, Huisman H (2017) Prostatex challenge data. The Cancer Imaging Archive

  30. Ma Y, Liu K, Xiong H, Fang P, Li X, Chen Y, Yan Z, Zhou Z, Liu C (2021) Medical image super-resolution using a relativistic average generative adversarial network. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 992:165053

    Article  Google Scholar 

  31. Majdabadi MM, Ko SB (2020) Msg-capsgan: Multi-scale gradient capsule gan for face super resolution. In: 2020 International conference on electronics, information, and communication (ICEIC), IEEE, pp 1–3

  32. Marini A, Mirmohammadsadegh A, Nambiar S, Gustrau A, Ruzicka T, Hengge UR (2006) Epigenetic inactivation of tumor suppressor genes in serum of patients with cutaneous melanoma. Journal of Investigative Dermatology 126(2):422–431

    Article  Google Scholar 

  33. Molahasani Majdabadi M, Ko SB (2020) Capsule GAN for robust face super resolution. Multimedia Tools and Applications 79(41–42):31205–31218. https://doi.org/10.1007/s11042-020-09489-y

  34. Park J, Hwang D, Kim KY, Kang SK, Kim YK, Lee JS (2018) Computed tomography super-resolution using deep convolutional neural network. Physics in Medicine & Biology 63(14):145011

    Article  Google Scholar 

  35. Peeters RR, Kornprobst P, Nikolova M, Sunaert S, Vieville T, Malandain G, Deriche R, Faugeras O, Ng M, Van Hecke P (2004) The use of super-resolution techniques to reduce slice thickness in functional mri. International Journal of Imaging Systems and Technology 14(3):131–138

    Article  Google Scholar 

  36. Pizer SM, Amburn EP, Austin JD, Cromartie R, Geselowitz A, Greer T, ter Haar Romeny B, Zimmerman JB, Zuiderveld K (1987) Adaptive histogram equalization and its variations. Computer vision, graphics, and image processing 39(3):355–368

    Article  Google Scholar 

  37. Rajpurkar P, Irvin J, Zhu K, Yang B, Mehta H, Duan T, Ding D, Bagul A, Langlotz C, Shpanskaya K, et al. (2017) Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint 171105225

  38. Reda I, Khalil A, Elmogy M, Abou El-Fetouh A, Shalaby A, Abou El-Ghar M, Elmaghraby A, Ghazal M, El-Baz A (2018) Deep learning role in early diagnosis of prostate cancer. Technology in cancer research & treatment 17:1533034618775530

    Article  Google Scholar 

  39. Rousseau F, Initiative ADN et al (2010) A non-local approach for image super-resolution using intermodality priors. Medical image analysis 14(4):594–605

    Article  Google Scholar 

  40. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. In: Advances in neural information processing systems, pp 3856–3866

  41. Shi W, Ledig C, Wang Z, Theis L, Huszar F (2018) Super resolution using a generative adversarial network. US Patent App. 15/706,428

  42. Sood R, Rusu M (2019) Anisotropic super resolution in prostate mri using super resolution generative adversarial networks. In: IEEE 16th international symposium on biomedical imaging (ISBI 2019), IEEE, pp 1688–1691

  43. Sood R, Topiwala B, Choutagunta K, Sood R, Rusu M (2018) An application of generative adversarial networks for super resolution medical imaging. In: 17th IEEE international conference on machine learning and applications (ICMLA), IEEE, pp 326–331

  44. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  45. Tong T, Li G, Liu X, Gao Q (2017) Image super-resolution using dense skip connections. In: Proceedings of the IEEE international conference on computer vision, pp 4799–4807

  46. Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Qiao Y, Change Loy C (2018) Esrgan: Enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV) workshops, pp 0–0

  47. Wang Y, Choi EJ, Choi Y, Zhang H, Jin GY, Ko SB (2020) Breast cancer classification in automated breast ultrasound using multiview convolutional neural network with transfer learning. Ultrasound in medicine & biology 46(5):1119–1132

    Article  Google Scholar 

  48. Yang X, Zhan S, Hu C, Liang Z, Xie D (2016) Super-resolution of medical image using representation learning. In: 8th International Conference on Wireless Communications & Signal Processing (WCSP), IEEE, pp 1–6

Download references

Author information

Authors and Affiliations

  1. Department of Electrical & Computer Engineering, University of Saskatchewan, Saskatoon, Canada

    Mahdiyar Molahasani Majdabadi & Seokbum Ko

  2. International Road Dynamics, Saskatoon, Canada

    Younhee Choi

  3. Department of Electrical & Computer Engineering, National Institute of Technology, Trichy, India

    S. Deivalakshmi

Authors
  1. Mahdiyar Molahasani Majdabadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Younhee Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. S. Deivalakshmi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Seokbum Ko
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seokbum Ko.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Table 7 Table of abbreviations
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Molahasani Majdabadi, M., Choi, Y., Deivalakshmi, S. et al. Capsule GAN for prostate MRI super-resolution. Multimed Tools Appl 81, 4119–4141 (2022). https://doi.org/10.1007/s11042-021-11697-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11697-z

Keywords

Search

Navigation