Skip to main content
SpringerLink
Log in

Adaptive one-stage generative adversarial network for unpaired image super-resolution

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Existing deep learning-based image super-resolution (SR) algorithms have made great achievements on synthetic LR–HR pairs in a fully supervised manner. However, the real image degradation process is complicated and unclear, leading to a domain gap between the synthetic LR images and real LR images. To address this issue, the most widely used alternatives are GAN-based methods trained with unpaired data. But most GAN-based methods tend to follow complex multi-stage pipelines and still suffer from the vast cost of hyperparameter tuning on the loss function. In this paper, in contrast to complicated two-stage or multi-stage GAN, a straightforward and simple one-stage GAN-based framework is proposed. Different from the current one-stage GAN, our approach exclusively fits the CycleGAN pattern without any other restrictions, fully leveraging images from unpaired datasets. Specifically, our framework consists of a DegradeNet and an SRNet, which implicitly models the degradation and SR process between the source LR domain and target HR domain. Such a simple one-stage GAN allows us to flexibly learn the data distribution without any assumptions, greatly reducing the model complexity and training difficulty. To further address the inherent practice of empirically setting hyperparameters in previous methods, we propose an adaptive weighting network to automatically learn the contribution weights of different losses in a meta-learning manner. This flexible strategy could shorten the gap between various losses and meanwhile help the model to adaptively optimize in an efficient way. Extensive experiments demonstrate the superiority of our proposed method in terms of indicators and perceptual quality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  2. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 1646–1654

  3. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 1132–1140

  4. Lai WS, Huang JB, Ahuja N, Yang MH (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 624–632

  5. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision, pp 294–310

  6. Chen H, Wang Y, Guo T, Xu C, Deng Y, Liu Z, Ma S, Xu C, Xu C, Gao W (2021) Pre-trained image processing transformer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 12299–12310

  7. Que Y, Lee HJ (2022) Single image super-resolution via deep progressive multi-scale fusion networks. Neural Comput Appl 34:10707–10717

    Article  Google Scholar 

  8. Wang B, Li S, Chen Q, Zuo C (2023) Learning-based single-shot long-range synthetic aperture Fourier ptychographic imaging with a camera array. Opt Lett 48(2):263–266

    Article  Google Scholar 

  9. Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken AP, Tejani A, Totz J, Wang Z (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 105–114

  10. Sajjadi MSM, Scholkopf B, Hirsch M (2017) EnhanceNet: single image super-resolution through automated texture synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 4501–4510

  11. Park S-J, Son H, Cho S, Hong K-S, Lee S (2018) SRFeat: single image super-resolution with feature discrimination. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 439–455

  12. Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Qiao Y, Change Loy C (2018) ESRGAN: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision Workshops, pp 63–79

  13. Jolicoeur-Martineau A (2018) The relativistic discriminator: a key element missing from standard GAN. In: Proceedings of the International Conference on Learning Representations, pp 1–25

  14. Liu A, Liu Y, Gu J, Qiao Y, Dong C (2022) Blind image super-resolution: a survey and beyond. IEEE Trans Pattern Anal Mach Intell 45:5461–5480

    Google Scholar 

  15. Yue Z, Zhao Q, Xie J, Zhang L, Meng D, Wong K-YK (2022) Blind image super-resolution with elaborate degradation modeling on noise and kernel. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 2128–2138

  16. Wei Y, Gu S, Li Y, Timofte R, Jin L, Song H (2021) Unsupervised real-world image super resolution via domain-distance aware training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 13385–13394

  17. Gazdieva M, Rout L, Korotin A, Kravchenko A, Filippov A, Burnaev E (2022) An optimal transport perspective on unpaired image super-resolution. arXiv preprint arXiv:2202.01116

  18. Wang X, Xie L, Dong C, Shan Y (2021) Real-ESRGAN: training real-world blind super-resolution with pure synthetic data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 1905–1914

  19. Zhang K, Zuo W, Zhang L (2018) Learning a single convolutional super-resolution network for multiple degradations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3262–3271

  20. Xu Y-S, Tseng S-YR, Tseng Y, Kuo H-K, Tsai Y-M (2020) Unified dynamic convolutional network for super-resolution with variational degradations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 12496–12505

  21. Luo Z, Huang Y, Li S, Wang L, Tan T (2022) Learning the degradation distribution for blind image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 6063–6072

  22. Gu J, Lu H, Zuo W, Dong C (2019) Blind super-resolution with iterative kernel correction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 1604–1613

  23. Huang Y, Li S, Wang L, Tan T (2020) Unfolding the alternating optimization for blind super resolution. Adv Neural Inf Process Syst 33:5632–5643

    Google Scholar 

  24. Guo Y, Chen J, Wang J, Chen Q, Cao J, Deng Z, Xu Y, Tan M (2020) Closed-loop matters: dual regression networks for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5407–5416

  25. Tao G, Ji X, Wang W, Chen S, Lin C, Cao Y, Lu T, Luo D, Tai Y (2021) Spectrum-to-kernel translation for accurate blind image super-resolution. Adv Neural Inf Process Syst 34:22643–22654

    Google Scholar 

  26. Yuan Y, Liu S, Zhang J, Zhang Y, Dong C, Lin L (2018) Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 701–710

  27. Bharti V, Biswas B, Shukla KK (2021) EMOCGAN: a novel evolutionary multiobjective cyclic generative adversarial network and its application to unpaired image translation. Neural Comput Appl, 1–15

  28. Sun W, Gong D, Shi Q, van den Hengel A, Zhang Y (2021) Learning to zoom-in via learning to zoom-out: real-world super-resolution by generating and adapting degradation. IEEE Trans Image Process 30:2947–2962

    Article  Google Scholar 

  29. Lugmayr A, Danelljan M, Timofte R (2019) Unsupervised learning for real-world super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop, pp 3408–3416

  30. Maeda S (2020) Unpaired image super-resolution using pseudo-supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 291–300

  31. Chen S, Han Z, Dai E, Jia X, Liu Z, Xing L, Zou X, Xu C, Liu J, Tian Q (2020) Unsupervised image super-resolution with an indirect supervised path. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 468–469

  32. Wang W, Zhang H, Yuan Z, Wang C (2021) Unsupervised real-world super-resolution: A domain adaptation perspective. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 4318–4327

  33. Kim G, Park J, Lee K, Lee J, Min J, Lee B, Han DK, Ko H (2020) Unsupervised real-world super resolution with cycle generative adversarial network and domain discriminator. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 456–457

  34. Romero A, Van Gool L, Timofte R (2022) Unpaired real-world super-resolution with pseudo controllable restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 798–807

  35. Shen C, Yin Y, Wang X, Li X, Song J, Song M (2021) Training generative adversarial networks in one stage. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3350–3360

  36. Mei Y, Fan Y, Zhou Y, Huang L, Huang TS, Shi H (2020) Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5690–5699

  37. Yang Y, Zhuang Y, Pan Y (2021) Multiple knowledge representation for big data artificial intelligence: framework, applications, and case studies. Front Inf Technol Electron Eng 22(12):1551–1558

    Article  Google Scholar 

  38. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 2472–2481

  39. Zhou S, Zhang J, Zuo W, Loy CC (2020) Cross-scale internal graph neural network for image super-resolution. Adv Neural Inf Process Syst 33:3499–3509

    Google Scholar 

  40. Shocher A, Cohen N, Irani M (2018) Zero-shot super-resolution using deep internal learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3118–3126

  41. Park S, Yoo J, Cho D, Kim J, Kim TH (2020) Fast adaptation to super-resolution networks via meta-learning. In: Proceedings of the European Conference on Computer Vision, pp 754–769

  42. Soh JW, Cho S, Cho NI (2020) Meta-transfer learning for zero-shot super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3516–3525

  43. Bell-Kligler S, Shocher A, Irani M (2019) Blind super-resolution kernel estimation using an Internal-GAN. Adv Neural Inf Process Syst 32:284–293

    Google Scholar 

  44. Cheng X, Fu Z, Yang J (2020) Zero-shot image super-resolution with depth guided internal degradation learning. In: Proceedings of the European Conference on Computer Vision, pp 265–280

  45. Emad M, Peemen M, Corporaal H (2021) DualSR: zero-shot dual learning for real-world super-resolution. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp 1630–1639

  46. Ulyanov D, Vedaldi A, Lempitsky V (2018) Deep image prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 9446–9454

  47. Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2223–2232

  48. Bulat A, Yang J, Tzimiropoulos G (2018) To learn image super-resolution, use a gan to learn how to do image degradation first. In: Proceedings of the European Conference on Computer Vision, pp 185–200

  49. Zhao T, Ren W, Zhang C, Ren D, Hu Q (2018) Unsupervised degradation learning for single image super-resolution. arXiv preprint arXiv:1812.04240

  50. Prajapati K, Chudasama V, Patel H, Upla K, Ramachandra R, Raja K, Busch C (2020) Unsupervised single image super-resolution network (USISResNet) for real-world data using generative adversarial network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 464–465

  51. Li L, Jamieson K, DeSalvo G, Rostamizadeh A, Talwalkar A (2017) Hyperband: a novel bandit-based approach to hyperparameter optimization. J Mach Learn Res 18(1):6765–6816

    MathSciNet  MATH  Google Scholar 

  52. Malkiel I, Wolf L (2020) Mtadam: automatic balancing of multiple training loss terms. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp 10713–10729

  53. Shu J, Xie Q, Yi L, Zhao Q, Meng D (2019) Meta-weight-net: learning an explicit mapping for sample weighting. Adv Neural Inf Process Syst 32:1917–1928

    Google Scholar 

  54. Isola P, Zhu J, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 5967–5976

  55. Lugmayr A, Danelljan M, Timofte R, Fritsche M, Gu S, Purohit K, Kandula P, Suin M, Rajagoapalan A, Joon NH (2019) Aim 2019 challenge on real-world image super-resolution: methods and results. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop, pp 3575–3583

  56. Lugmayr A, Danelljan M, Timofte R (2020) Ntire 2020 challenge on real-world image super-resolution: methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 494–495

  57. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations, pp 1–13

  58. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  59. Zhang R, Isola P, Efros AA, Shechtman E, Wang O (2018) The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 586–595

  60. Timofte R, Agustsson E, Van Gool L, Yang M, Zhang L, Lim B, Son S, Kim H, Nah S, Lee KM (2017) Ntire 2017 challenge on single image super-resolution: methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 1110–1121

  61. Liu Z, Luo P, Wang X, Tang X (2015) Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp 3730–3738

  62. Cao Q, Shen L, Xie W, Parkhi OM, Zisserman A (2018) VGGFace2: a dataset for recognising faces across pose and age. In: Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition, pp 67–74

  63. Yang S, Luo P, Loy C-C, Tang X (2016) Wider face: a face detection benchmark. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5525–5533

Download references

Acknowledgements

This work was supported by National Key Research and development Program of China (2021YFA1000102), and in part by the grants from the National Natural Science Foundation of China (Nos. 61673396, 61976245), Natural Science Foundation of Shandong Province, China (No. ZR2022MF260).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, China University of Petroleum, Qingdao, 266580, China

    Mingwen Shao, Huan Liu & Jianxin Yang

  2. College of Sciences, China Jiliang University, Hangzhou, 310018, China

    Feilong Cao

Authors
  1. Mingwen Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianxin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Feilong Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huan Liu.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shao, M., Liu, H., Yang, J. et al. Adaptive one-stage generative adversarial network for unpaired image super-resolution. Neural Comput & Applic 35, 20909–20922 (2023). https://doi.org/10.1007/s00521-023-08888-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08888-0

Keywords

Search

Navigation