Abstract
Learning based single image super resolution (SISR) is well investigated in 2D images. However, SISR for 3D magnetic resonance images (MRI) is more challenging compared to 2D, mainly due to the increased number of neural network parameters, the larger memory requirement, and the limited amount of available training data. Current SISR methods for 3D volumetric images are based on generative adversarial networks (GANs), especially Wasserstein GANs. Other common architectures in the 2D domain, e.g. transformer models, require large amounts of training data and are therefore not suitable for the limited 3D data. As a popular GAN variant, Wasserstein GANs sometimes produce blurry results as they may not converge to a global (or local) optimum. In this paper, we propose a new method for 3D SISR using a GAN framework. Our approach incorporates instance noise to improve the training stability of the GAN, as well as a relativistic GAN loss function and an updating feature extractor as the third player during the training process. Our experiments demonstrate that our method produces highly accurate results using very few training samples. Specifically, we show that we need less than 30 training samples, which is a significant reduction compared to the thousands typically required in previous studies. Moreover, our method improves out-of-sample results, demonstrating its effectiveness. The code is available in https://github.com/wqlevi/threeplayerGANSR.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Balduzzi, D., Racaniere, S., Martens, J., Foerster, J., Tuyls, K., Graepel, T.: The mechanics of n-player differentiable games. In: Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 354–363. PMLR, 10–15 July 2018
Chen, Y., Xie, Y., Zhou, Z., Shi, F., Christodoulou, A.G., Li, D.: Brain MRI super resolution using 3d deep densely connected neural networks. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE, April 2018. https://doi.org/10.1109/isbi.2018.8363679
Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of Wasserstein GANs. In: Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.: Densely Connected Convolutional Networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017
Jolicoeur-Martineau, A.: The relativistic discriminator: a key element missing from standard GAN. In: International Conference on Learning Representations (2019)
Justin, J., Alexandre, A., Li, F.: Perceptual losses for real-time style transfer and super-resolution (2016). https://doi.org/10.48550/ARXIV.1603.08155
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings (2015)
Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. CoRR abs/1609.04802 (2016)
Li, C., Xu, K., Zhu, J., Zhang, B.: Triple generative adversarial nets. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 4091–4101. NIPS 2017, Curran Associates Inc. (2017)
Li, G., et al.: Transformer-empowered multi-scale contextual matching and aggregation for multi-contrast MRI super-resolution. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20604–20613 (2022). https://doi.org/10.1109/CVPR52688.2022.01998
Ma, C., Rao, Y., Cheng, Y., Chen, C., Lu, J., Zhou, J.: Structure-preserving super resolution with gradient guidance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Ma, C., Yang, C., Yang, X., Yang, M.: Learning a no-reference quality metric for single-image super-resolution. Comput. Vis. Image Underst. 158, 1–16 (2017). https://doi.org/10.1016/j.cviu.2016.12.009
Martin, A., Soumith, C., Léon, B.: Wasserstein generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 214–223. PMLR, 06–11 August 2017
Mescheder, L., Nowozin, S., Geiger, A.: Which training methods for GANs do actually converge? In: International Conference on Machine Learning (ICML) (2018)
Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library (2019). https://doi.org/10.48550/ARXIV.1912.01703
Pham, C., Ducournau, A., Fablet, R., Rousseau, F.: Brain MRI super-resolution using deep 3d convolutional networks. In: 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pp. 197–200 (2017). https://doi.org/10.1109/ISBI.2017.7950500
Sawyer, A.M., et al.: Creation of fully sampled MR data repository for compressed sensing of the knee. In: Proceedings of Society for MR Radiographers & Technologists (SMRT) 22nd Annual Meeting, Salt Lake City, UT, USA (2013)
Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
Sønderby, C.K., Caballero, J., Theis, L., Shi, W., Huszár, F.: Amortised MAP inference for image super-resolution. In: 5th International Conference on Learning Representations, ICLR 2017, OpenReview.net (2017). https://openreview.net/forum?id=S1RP6GLle
Sui, Y., Afacan, O., Gholipour, A., Warfield, S.K.: Learning a gradient guidance for spatially isotropic MRI super-resolution reconstruction. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12262, pp. 136–146. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_14
Van Essen, D.C., Smith, S.M., Barch, D.M., Behrens, T.E.J., Yacoub, E., Ugurbil, K.: The WU-Minn human connectome project: an overview. Neuroimage 80, 64–79 (2013)
Vandenhende, S., Brabandere, B.D., Neven, D., Gool, L.V.: A three-player GAN: generating hard samples to improve classification networks. CoRR abs/1903.03496 (2019)
Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 63–79. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_5
Wu, Q., et al.: An arbitrary scale super-resolution approach for 3d MR images via implicit neural representation. IEEE J. Biomed. Health Inform. 27, 1–12 (2022)
Xu, K., Li, C., Zhu, J., Zhang, B.: Understanding and stabilizing GANs’ training dynamics using control theory. In: Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 10566–10575. PMLR, 13–18 July 2020
Zhang, R., Isola, P., Efros, A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)
Zhang, Y., Li, K., Li, K., Fu, Y.: MR image super-resolution with squeeze and excitation reasoning attention network. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13420–13429 (2021). https://doi.org/10.1109/CVPR46437.2021.01322
Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 294–310. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_18
Acknowledgements
This work is supported by the Deutsche Forschungsgemeinschaft (DFG - German Research Foundation), the grant numbers are DFG LO1728/2-1 and DFG HE 9297/1-1.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, Q., Mahler, L., Steiglechner, J., Birk, F., Scheffler, K., Lohmann, G. (2023). A Three-Player GAN for Super-Resolution in Magnetic Resonance Imaging. In: Abdulkadir, A., et al. Machine Learning in Clinical Neuroimaging. MLCN 2023. Lecture Notes in Computer Science, vol 14312. Springer, Cham. https://doi.org/10.1007/978-3-031-44858-4_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-44858-4_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44857-7
Online ISBN: 978-3-031-44858-4
eBook Packages: Computer ScienceComputer Science (R0)
