Skip to main content
SpringerLink
Log in

E2-RealSR: efficient and effective real-world super-resolution network based on partial degradation modulation

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The goal of efficient and effective real-world image super-resolution (Real-ISR) is to recover the high-resolution image from the given low-resolution image with unknown degradation by limited computation resources. Prior research has attempted to design a fully degradation-adaptive network, where the entire backbone is a nonlinear combination of several sub-networks which can handle different degradation subspaces. However, these methods heavily rely on expensive dynamic convolution operations and are inefficient in super-resolving images of different degradation levels. To address this issue, we propose an efficient and effective real-world image super-resolution network (E2-RealSR) based on partial degradation modulation, which is consisted of a small regression and a lightweight super-resolution network. The former accurately predicts the individual degradation parameters of input images, while the latter only modulates its partial parameters based on the degradation information. The extensive experiments validate that our proposed method is capable of recovering the rich details in real-world images with varying degradation levels. Moreover, our approach also has an advantage in terms of efficiency, compared to state-of-the-art methods. Our method shows improved performance while only using 20% of the parameters and 60% of the FLOPs of DASR. The relevant code is made available on this link as open source.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The datasets analyzed during the current study are referred in this paper.

References

  1. Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 126–135, (2017)

  2. Bell-Kligler, S., Shocher, A., Irani, M.: Blind super-resolution kernel estimation using an internal-GAN. Adv. Neural Inform. Process. Syst. 32, 284–293 (2019)

    Google Scholar 

  3. Chen, H., He, X., Qing, L., Wu, Y., Ren, C., Sheriff, R.E., Zhu, C.: Real-world single image super-resolution: a brief review. Inform. Fus. 79(22), 124–145 (2022)

    Article  Google Scholar 

  4. Chen, X., Wang, X., Zhou, J., Dong, C.: Activating more pixels in image super-resolution transformer. arxiv 2022. arXiv preprintarXiv:2205.04437, (2022)

  5. Chen, Y., Xia, R., Yang, K., Zou, K.: Mffn: Image super-resolution via multi-level features fusion network. Vis. Comput., 1–16, (2023)

  6. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Proceedings of the European Conference on Computer Vision, 184–199. Springer, (2014)

  7. Fritsche, M., Gu, S., Timofte, R.: Frequency separation for real-world super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 3599–3608. IEEE, (2019)

  8. Gu, J., Lu, H., Zuo, W., Dong, C.: Blind super-resolution with iterative kernel correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1604–1613, (2019)

  9. Hua, X., Wang, M., Su, B., Liu, X.: Hierarchical feature fusion network for light field spatial super-resolution. Vis. Comput. 39(1), 267–279 (2023)

    Article  Google Scholar 

  10. Huang, C.-K., Nien, H.-H.: Multi chaotic systems based pixel shuffle for image encryption. Opt. Commun. 282(11), 2123–2127 (2009)

    Article  Google Scholar 

  11. Huang, Y., Li, S., Wang, L., Tan, T., et al.: Unfolding the alternating optimization for blind super resolution. Adv. Neural Inform. Process. Syst. 33(473), 5632–5643 (2020)

    Google Scholar 

  12. Kang, X., Li, J., Duan, P., Ma, F., Li, S.: Multilayer degradation representation-guided blind super-resolution for remote sensing images. IEEE Trans. Geosci. Remote Sens. 60, 1–12 (2022)

    Google Scholar 

  13. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprintarXiv:1412.6980, (2014)

  14. Kong, F., Li, M., Liu, S., Liu, D., He, J., Bai, Y., Chen, F., Fu, L.: Residual local feature network for efficient super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 766–776, (2022)

  15. Li, X., Jin, X., Yu, T., Sun, S., Pang, Y., Zhang, Z., Chen, Z.: Learning omni-frequency region-adaptive representations for real image super-resolution. In: Proceedings of the AAAI Conference on Artificial Intelligence 35, 1975–1983 (2021)

  16. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Swinir: Image restoration using swin transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 1833–1844, (2021)

  17. Liang, J., Zeng, H., Zhang, L.: Efficient and degradation-adaptive network for real-world image super-resolution. In: Proceedings of the European Conference on Computer Vision, 574–591. Springer, (2022)

  18. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 136–144, (2017)

  19. Liu, A., Li, S., Chang, Y.: Cross-resolution feature attention network for image super-resolution. Vis. Comput., 1–13, (2022)

  20. Liu, A., Liu, Y., Gu, J., Qiao, Y., Dong, C.: Blind image super-resolution: a survey and beyond. IEEE Trans. Pattern Anal. Mach. Intell. 45(5), 5461–5480 (2022)

    Google Scholar 

  21. Lu, X., Xie, X., Ye, C., Xing, H., Liu, Z., Cai, C.: A lightweight generative adversarial network for single image super-resolution. Vis. Comput., 1–12, (2023)

  22. Luo, Z., Huang, H., Yu, L., Li, Y., Fan, H., Liu, S.: Deep constrained least squares for blind image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 17642–17652, (2022)

  23. Luo, Z., Huang, Y., Li, S., Wang, L., Tan, T.: End-to-end alternating optimization for blind super resolution. arXiv preprint arXiv:2105.06878, (2021)

  24. Luo, Z., Huang, Y., Li, S., Wang, L., Tan, T.: Learning the degradation distribution for blind image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6063–6072, (2022)

  25. Mahapatra, D., Bozorgtabar, B., Garnavi, R.: Image super-resolution using progressive generative adversarial networks for medical image analysis. Computer. Med. Imaging Graph. 71, 30–39 (2019)

    Article  Google Scholar 

  26. Mou, C., Wu, Y., Wang, X., Dong, C., Zhang, J., Shan, Y.: Metric learning based interactive modulation for real-world super-resolution. In: Proceedings of the European Conference on Computer Vision, 723–740. Springer, (2022)

  27. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, 234–241. Springer, (2015)

  28. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, (2014)

  29. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.-H., Zhang, L.: Ntire 2017 challenge on single image super-resolution: Methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 114–125, (2017)

  30. Trockman, A., Zico Kolter, J.: Patches are all you need? arXiv e-prints, (2022)

  31. Wang, J., Zou, Y., Wu, H.: Image super-resolution method based on attention aggregation hierarchy feature. Vis. Comput., 1–12, (2023)

  32. Wang, X., Xie, L., Dong, C., Shan, Y.: Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 1905–1914, (2021)

  33. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, C.: Esrgan: Enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV) workshops, (2018)

  34. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  35. Wei, P., Xie, Z., Lu, H., Zhan, Z., Ye, Q., Zuo, W., Lin, L.: Component divide-and-conquer for real-world image super-resolution. In: Proceedings of the European Conference on Computer Vision, 101–117. Springer, (2020)

  36. Xie, L., Wang, X., Dong, C., Qi, Z., Shan, Y.: Finding discriminative filters for specific degradations in blind super-resolution. Adv. Neural Inform. Process. Syst. 34, 51–61 (2021)

    Google Scholar 

  37. Xu, Y.-S., Tseng, S.-Y. R., Tseng, Y., Kuo, H.-K., Tsai, Y.-M.: Unified dynamic convolutional network for super-resolution with variational degradations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12496–12505, (2020)

  38. Yoo, J., Kim, T., Lee, S., Kim, S. H., Lee, H., Kim, T. H.: Rich cnn-transformer feature aggregation networks for super-resolution. arXiv preprint arXiv:2203.07682, (2022)

  39. Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M.-H.: Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5728–5739, (2022)

  40. Zhang, K., Liang, J., Van Gool, L., Timofte, R.: Designing a practical degradation model for deep blind image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 4791–4800, (2021)

  41. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3262–3271, (2018)

  42. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision, 286–301, (2018)

  43. Zhou, Y., Deng, W., Tong, T., Gao, Q.: Guided frequency separation network for real-world super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 428–429, (2020)

Download references

Funding

This work was supported by National Natural Science Foundation of China under Grant 62171133, in part by the Artificial Intelligence and Economy Integration Platform of Fujian Province, and the Fujian Health Commission under Grant 2022ZD01003.

Author information

Author notes

  1. Jiajun Zhang and Yuanbo Zhou have equally contributed to this work.

Authors and Affiliations

  1. The College of Physics and Information Engineering, Fuzhou University, Fuzhou, 350108, Fujian, China

    Jiajun Zhang, Yuanbo Zhou, Tong Tong & Qinquan Gao

  2. The Chongqing Dadukou District Archives, Chongqing, 400000, China

    Hongjun Liu, Tian Tian & Xingmei Hu

  3. Xiamen University of Technology, Xiamen, 361021, Fujian, China

    Xiaoyong Lin

Authors
  1. Jiajun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuanbo Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tong Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hongjun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tian Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xingmei Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Qinquan Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xiaoyong Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JZ analyzed the results and wrote the manuscript. YZ designed the research framework. HL, TT and XH revised the manuscript. XL provided support for the research. TT and QG contributed to review and supervision. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Qinquan Gao.

Ethics declarations

Conflict of interest

There is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Zhou, Y., Tong, T. et al. E2-RealSR: efficient and effective real-world super-resolution network based on partial degradation modulation. Vis Comput (2024). https://doi.org/10.1007/s00371-024-03279-5

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-024-03279-5

Keywords

Search

Navigation