Skip to main content

Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13678))

Included in the following conference series:

Abstract

Efficient and effective real-world image super-resolution (Real-ISR) is a challenging task due to the unknown complex degradation of real-world images and the limited computation resources in practical applications. Recent research on Real-ISR has achieved significant progress by modeling the image degradation space; however, these methods largely rely on heavy backbone networks and they are inflexible to handle images of different degradation levels. In this paper, we propose an efficient and effective degradation-adaptive super-resolution (DASR) network, whose parameters are adaptively specified by estimating the degradation of each input image. Specifically, a tiny regression network is employed to predict the degradation parameters of the input image, while several convolutional experts with the same topology are jointly optimized to specify the network parameters via a non-linear mixture of experts. The joint optimization of multiple experts and the degradation-adaptive pipeline significantly extend the model capacity to handle degradations of various levels, while the inference remains efficient since only one adaptively specified network is used for super-resolving the input image. Our extensive experiments demonstrate that DASR is not only much more effective than existing methods on handling real-world images with different degradation levels but also efficient for easy deployment. Codes, models and datasets are available at https://github.com/csjliang/DASR.

This work is supported by the Hong Kong RGC RIF grant (R5001-18) and the PolyU-OPPO Joint Innovation Lab.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    Chapter  Google Scholar 

  2. Wang, X., et al.: ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. In: Leal-Taixé, Laura, Roth, Stefan (eds.) ECCV 2018. LNCS, vol. 11133, pp. 63–79. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_5

    Chapter  Google Scholar 

  3. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 294–310. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_18

    Chapter  Google Scholar 

  4. Ma, C., Rao, Y., Cheng, Y., Chen, C., Lu, J., Zhou, J.: Structure-preserving super resolution with gradient guidance. In CVPR (2020)

    Google Scholar 

  5. Sun, J., Zongben, X., Shum, H.-Y.: Gradient profile prior and its applications in image super-resolution and enhancement. IEEE Trans. Image Process. 20(6), 1529–1542 (2010)

    MathSciNet  MATH  Google Scholar 

  6. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13

    Chapter  Google Scholar 

  7. Sajjadi, M.S.M., Scholkopf, B., Hirsch, M.: EnhanceNet: single image super-resolution through automated texture synthesis. In: ICCV, (2017)

    Google Scholar 

  8. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution, In : CVPR (2016)

    Google Scholar 

  9. Soh, J.W., Park, G.Y., Jo, J., Cho, N.I.: Natural and realistic single image super-resolution with explicit natural manifold discrimination, In: CVPR (2019)D

    Google Scholar 

  10. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: CVPR (2018)

    Google Scholar 

  11. Jo, Y., Oh, S.W., Vajda, P., Kim, S.J.: Tackling the ill-posedness of super-resolution through adaptive target generation, In: CVPR (2021)

    Google Scholar 

  12. Liu, A., Liu, Y., Gu, J., Qiao, Y., Dong. C.: Blind image super-resolution: a survey and beyond. arXiv preprint arXiv:2107.03055 (2021)

  13. Zhang, J., Liang, K., Van Gool, L., Timofte, R.: Designing a practical degradation model for deep blind image super-resolution. In : ICCV (2021)

    Google Scholar 

  14. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: CVPR (2018)

    Google Scholar 

  15. Luo, Z., Huang, Y., Li, S., Wang, L., Tan, T.: Unfolding the alternating optimization for blind super resolution. In: NeurIPS (2020)

    Google Scholar 

  16. Gu, J., Lu, H., Zuo, W., Dong, C.: Blind super-resolution with iterative kernel correction. In: CVPR (2019)

    Google Scholar 

  17. Zhou, R., Susstrunk. Kernel modeling super-resolution on real low-resolution images. In: ICCV (2019)

    Google Scholar 

  18. Ignatov, A., Kobyshev, N., Timofte, R., Vanhoey, N., Van Gool, U.C.: DSLR-quality photos on mobile devices with deep convolutional networks. In ICCV (2017)

    Google Scholar 

  19. Cai, J., Zeng, H., Yong, H., Cao, Z., Zhang, L.: Toward real-world single image super-resolution: a new benchmark and a new model. In ICCV (2019)

    Google Scholar 

  20. Wei, P., et al.: Component divide-and-conquer for real-world image super-resolution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12353, pp. 101–117. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58598-3_7

    Chapter  Google Scholar 

  21. Lugmayr, A., Danelljan, M., Timofte, R.: NTIRE 2020 challenge on real-world image super-resolution: methods and results. In: CVPRW (2020)

    Google Scholar 

  22. Lugmayr, A., et al:. AIM 2019 challenge on real-world image super-resolution: methods and results. In ICCVW (2019)

    Google Scholar 

  23. Fritsche, M., Gu, S., Timofte, R.: Frequency separation for real-world super-resolution. In: ICCVW (2019)

    Google Scholar 

  24. Lugmayr, A., Danelljan, M., Timofte, R.: Unsupervised learning for real-world super-resolution. In ICCVW (2019)D

    Google Scholar 

  25. Ji, X., Cao, Y., Tai, Y., Wang, C., Li, J., Huang, F.: Real-world super-resolution via kernel estimation and noise injection. In:: CVPRW (2020

    Google Scholar 

  26. Ji, X., Cao, Y., Tai, Y., Wang, C., Li, J., Huang, F.: Real-world super-resolution via kernel estimation and noise injection. In: CVPRW (2020)

    Google Scholar 

  27. Bulat, Adrian, Yang, Jing, Tzimiropoulos, Georgios: To learn image super-resolution, use a gan to learn how to do image degradation first. In: Ferrari, Vittorio, Hebert, Martial, Sminchisescu, Cristian, Weiss, Yair (eds.) ECCV 2018. LNCS, vol. 11210, pp. 187–202. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01231-1_12

    Chapter  Google Scholar 

  28. Wei, Y., Gu, S., Li, Y., Timofte, R., Jin, L., Song, H.: Unsupervised real-world image super resolution via domain-distance aware training. In: CVPR (2021)

    Google Scholar 

  29. Wang, X., Xie, L., Dong, C., Shan, Y.: Real-ESRGAN: training real-world blind super-resolution with pure synthetic data. In: ICCVW (2021)

    Google Scholar 

  30. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_25

    Chapter  Google Scholar 

  31. Zhang, X., Zeng, H., Zhang, L.: Edge-oriented convolution block for real-time super resolution on mobile devices. In: ACM Multimedia (2021)

    Google Scholar 

  32. Ahn, N., Kang, B., Sohn, K.-A.: Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 256–272. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01249-6_16

    Chapter  Google Scholar 

  33. Yang, W., Wang, W., Zhang, X., Sun, S., Liao, Q.: Lightweight feature fusion network for single image super-resolution. IEEE Signal Process. Lett. 26(4), 538–542 (2019)

    Article  Google Scholar 

  34. Song, D., Wang, Y., Chen, H., Chang, X., Chunjing, X., Tao, D.: AdderSR: towards energy efficient image super-resolution. In: CVPR (2021)

    Google Scholar 

  35. Zhang, X., Chen, Q., Ng, R., Koltun, V.: Zoom to learn, learn to zoom. In: CVPR (2019)

    Google Scholar 

  36. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: SwinIR: image restoration using swin transformer. In: ICCVW (2021)

    Google Scholar 

  37. Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: ICCV (2021)

    Google Scholar 

  38. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR (2017)

    Google Scholar 

  39. Wang, X., Yu, K., Dong, C., Loy, C.C.: Recovering realistic texture in image super-resolution by deep spatial feature transform. In: CVPR (2018)

    Google Scholar 

  40. Fuoli, D., Van Gool, L., Timofte, R.: Fourier space losses for efficient perceptual image super-resolution. In: ICCV (2021)

    Google Scholar 

  41. Xu, Y-S., Roy Tseng, S.-Y., Tseng, Y., Kuo, H.-K., Tsai, Y.M Unified dynamic convolutional network for super-resolution with variational degradations In: CVPR (2020)

    Google Scholar 

  42. Zhang, K., Van Gool, L., Timofte, R.: Deep unfolding network for image super-resolution. In: CVPR (2020)

    Google Scholar 

  43. Zhang, K., Zuo, W., Zhang, L.: Deep plug-and-play super-resolution for arbitrary blur kernels. In: CVPR (2019)

    Google Scholar 

  44. Wang, L., et al.: Unsupervised degradation representation learning for blind super-resolution. In: CVPR (2021)

    Google Scholar 

  45. Hui, Z., Li, J., Wang, X., Gao, X.: Learning the non-differentiable optimization for blind super-resolution. In: CVPR (2021)

    Google Scholar 

  46. Liu, P., Zhang, H., Cao, Y., Liu, S., Ren, D., Zuo, W.: Learning cascaded convolutional networks for blind single image super-resolution. Neurocomputing 417, 371–383 (2020)

    Article  Google Scholar 

  47. Maeda, S.: Unpaired image super-resolution using pseudo-supervision. In: CVPR (2020)

    Google Scholar 

  48. Yuan, Y., et al.: Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In: CVPRW (2018)

    Google Scholar 

  49. Goodfellow, I., et al.: Generative adversarial nets. In: NeurIPS (2014)

    Google Scholar 

  50. Cornillere, V., Djelouah, A., Yifan, W., Sorkine-Hornung, O., Schroers, C.: Blind image super-resolution with spatially variant degradations. ACM Trans. Graph. 38(6), 1–13 (2019)

    Article  Google Scholar 

  51. Kim, S.Y., Sim, H., Kim, M.: KOALAnet: Blind super-resolution using kernel-oriented adaptive local adjustment. In: CVPR (2021)

    Google Scholar 

  52. Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive mixtures of local experts. Neural Comput. 3(1), 79–87 (1991)

    Google Scholar 

  53. Jordan, M.I., Jacobs, R.J.: Hierarchical mixtures of experts and the em algorithm. Neural Comput. 6(2), 181–214 (1994)

    Google Scholar 

  54. Gross, S., Ranzato, M.A., Szlam, A.: Hard mixtures of experts for large scale weakly supervised vision. In: CVPR (2017)

    Google Scholar 

  55. Aljundi, R., Chakravarty, P., Tuytelaars, T.: Expert gate: lifelong learning with a network of experts. In: CVPR (2017)

    Google Scholar 

  56. Maeda, S.: Fast and flexible image blind denoising via competition of experts. In: CVPRW (2020)

    Google Scholar 

  57. Wang, Y., Wang, L., Wang, H., Li, P., Huchuan, L.: Blind single image super-resolution with a mixture of deep networks. Pattern Recogn. 102, 107169 (2020)

    Article  Google Scholar 

  58. Chen, Y., et al.: Dynamic convolution: attention over convolution kernels. In: CVPR (2020)

    Google Scholar 

  59. Li, Y., et al.: Revisiting dynamic convolution via matrix decomposition. In: ICLR (2021)

    Google Scholar 

  60. Li, C., Zhou, A., Yao, A.: Omni-dimensional dynamic convolution. In: ICLR (2021)

    Google Scholar 

  61. Yang, B., Bender, G., Le, Q.V., Ngiam, J.: CondConv: conditionally parameterized convolutions for efficient inference. In: NeurIPS (2019)

    Google Scholar 

  62. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: CVPRW (2017)

    Google Scholar 

  63. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  64. Zhang, K., et al.: AIM 2020 challenge on efficient super-resolution: methods and results. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12537, pp. 5–40. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-67070-2_1

    Chapter  Google Scholar 

  65. Zhang, K., et al.: Aim 2019 challenge on constrained super-resolution: methods and results. In: ICCVW (2019)

    Google Scholar 

  66. Emad, M., Peemen, M., Corporaal, H.: MoESR: blind super-resolution using kernel-aware mixture of experts. In: WACV (2022)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Lei Zhang .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1874 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liang, J., Zeng, H., Zhang, L. (2022). Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13678. Springer, Cham. https://doi.org/10.1007/978-3-031-19797-0_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19797-0_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19796-3

  • Online ISBN: 978-3-031-19797-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics