Skip to main content
SpringerLink
Log in

End-to-End Alternating Optimization for Real-World Blind Super Resolution

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Blind super-resolution (SR) usually involves two sub-problems: (1) estimating the degradation of the given low-resolution (LR) image; (2) super-resolving the LR image to its high-resolution (HR) counterpart. Both problems are ill-posed due to the information loss in the degrading process. Most previous methods try to solve the two problems independently, but often fall into a dilemma: a good super-resolved HR result requires an accurate degradation estimation, which however, is difficult to be obtained without the help of original HR information. To address this issue, instead of considering these two problems independently, we adopt an alternating optimization algorithm, which can estimate the degradation and restore the SR image in a single model. Specifically, we design two convolutional neural modules, namely Restorer and Estimator. Restorer restores the SR image based on the estimated degradation, and Estimator estimates the degradation with the help of the restored SR image. We alternate these two modules repeatedly and unfold this process to form an end-to-end trainable network. In this way, both Restorer and Estimator could get benefited from the intermediate results of each other, and make each sub-problem easier. Moreover, Restorer and Estimator are optimized in an end-to-end manner, thus they could get more tolerant of the estimation deviations of each other and cooperate better to achieve more robust and accurate final results. Extensive experiments on both synthetic datasets and real-world images show that the proposed method can largely outperform state-of-the-art methods and produce more visually favorable results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Agustsson, E., & Timofte, R. (2017). Ntire 2017 challenge on single image super-resolution: Dataset and study, 1122–1131.

  • Ahn, N., Kang, B., & Sohn, K.-A. (2018).Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European conference on computer vision (pp. 252–268).

  • Baker, S., & Kanade, T. (2002). Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9), 1167–1183.

    Article  Google Scholar 

  • Bell-Kligler, S., Shocher, A., & Irani, M.(2019). Blind super-resolution kernel estimation using an internal-gan. In Advances in neural information processing systems.

  • Blau, Y., Mechrez, R., Timofte, R., Michaeli, T., & Zelnik-Manor, L.(2018). The 2018 pirm challenge on perceptual image super-resolution. In Proceedings of the European conference on computer vision (ECCV) workshops.

  • Brooks, T., Mildenhall, B., Xue, T., Chen, J., Sharlet, D., & Barron, J.T.(2019). Unprocessing images for learned raw denoising. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11036–11045).

  • Cai, J., Zeng, H., Yong, H., Cao, Z., & Zhang, L. (2019).Toward real-world single image super-resolution: A new benchmark and a new model. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 3086–3095).

  • Cai, J., Zuo, W., & Zhang, L. (2020). Dark and bright channel prior embedded network for dynamic scene deblurring. IEEE Transactions on Image Processing, 29, 6885–6897.

    Article  MATH  Google Scholar 

  • Chen, S., Han, Z., Dai, E., Jia, X., Liu, Z., Xing, L., Zou, X., Xu, C., Liu, J., & Tian, Q. (2020). Unsupervised image super-resolution with an indirect supervised path. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp. 468–469).

  • Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021). Pre-trained image processing transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 12299–12310).

  • Dai, T., Cai, J., Zhang, Y., Xia, S.-T., & Zhang, L. (2019).Second-order attention network for single image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11065–11074).

  • Dong, C., Loy, C. C., & Tang, X. (2016). Accelerating the super-resolution convolutional neural network. In Proceedings of the European conference on computer vision (pp. 391–407). Springer.

  • Dong, C., Loy, C. C., He, K., & Tang, X. (2015). Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295–307.

    Article  Google Scholar 

  • Glasner, D., Bagon, S., & Irani, M.(2009). Super-resolution from a single image. In 2009 IEEE 12th international conference on computer vision (pp. 349–356).

  • Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., & Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems.

  • Gu, J., Lu, H., Zuo, W., & Dong, C.(2019). Blind super-resolution with iterative kernel correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1604–1613).

  • Haris, M., Shakhnarovich, G., & Ukita, N. (2018). Deep back-projection networks for super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1664–1673).

  • Hu, X., Mu, H., Zhang, X., Wang, Z., Tan, T., & Sun, J.(2019). Meta-sr: A magnification-arbitrary network for super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1575–1584).

  • Hui, Z., Gao, X., Yang, Y., & Wang, X. (2019). Lightweight image super-resolution with information multi-distillation network. In Proceedings of the 27th ACM international conference on multimedia (pp. 2024–2032).

  • Hui, Z., Wang, X., & Gao, X. (2018). Fast and accurate single image super-resolution via information distillation network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 723–731).

  • Hussein, S. A., Tirer, T., & Giryes, R. (2020). Correction filter for single image super-resolution: Robustifying off-the-shelf deep super-resolvers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1428–1437).

  • Ignatov, A., Kobyshev, N., Timofte, R., Vanhoey, K., & Van Gool, L. (2017). Dslr-quality photos on mobile devices with deep convolutional networks. In Proceedings of the IEEE international conference on computer vision (pp. 3277–3285).

  • Ji, X., Cao, Y., Tai, Y., Wang, C., Li, J., & Huang, F. (2020). Real-world super-resolution via kernel estimation and noise injection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp. 466–467).

  • Jo, Y., Oh, S.W., Vajda, P., & Kim, S.J. (2021). Tackling the ill-posedness of super-resolution through adaptive target generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16236–16245).

  • Kaufman, A., & Fattal, R. (2020). Deblurring using analysis-synthesis networks pair. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5811–5820).

  • Kim, J., Kwon Lee, J., & Mu Lee, K. (2016). Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1646–1654).

  • Kim, S. Y., Sim, H., & Kim, M. (2021). Koalanet: Blind super-resolution using kernel-oriented adaptive local adjustment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10611–10620).

  • Kim, K., & Kwon, Y. (2010). Single-image super-resolution using sparse regression and natural image prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(6), 1127–1133.

    Article  Google Scholar 

  • Kingma, D. P., & Ba, J.(2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

  • Köhler, T., Bätz, M., Naderi, F., Kaup, A., Maier, A., & Riess, C. (2019). Toward bridging the simulated-to-real gap: Benchmarking super-resolution on real data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(11), 2944–2959.

    Google Scholar 

  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 1097–1105.

    Google Scholar 

  • Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., & Wang, Z. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4681–4690.

  • Levin, Anat, Weiss, Yair, Durand, Fredo, & Freeman, William T.(2011). Efficient marginal likelihood optimization in blind deconvolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2657–2664). IEEE.

  • Levin, A., Weiss, Y., Durand, F., & Freeman, W.T. (2009).Understanding and evaluating blind deconvolution algorithms. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1964–1971). IEEE.

  • Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., & Timofte, R. (2021). Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 1833–1844).

  • Liang, J., Zhang, K., Gu, S., Van Gool, L., & Timofte, R.(2021). Flow-based kernel prior with application to blind super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10601–10610).

  • Lim, B., Son, S., Kim, H., Nah, S., & Mu Lee, K. (2017).Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp. 136–144).

  • Lugmayr, A., Danelljan, M., & Timofte, R. (2020). Ntire 2020 challenge on real-world image super-resolution: Methods and results. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp. 494–495).

  • Luo, Z., Huang, Y., Li, S., Wang, L., & Tan, T.(2022). Learning the degradation distribution for blind image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

  • Luo, Z., Huang, Y., Li, S., Wang, L., & Tan, T.(2020). Unfolding the alternating optimization for blind super resolution. Adv. Neural Inf. Process. Syst. 33.

  • Luo, Z., Huang, H., Yu, L., Li, Y., Fan, H., & Liu, S. (2022). Deep constrained least squares for blind image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

  • Martin, D., Fowlkes, C., Tal, D., & Malik, J.(2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE international conference on computer vision. ICCV 2001 (Vol. 2, pp. 416–423). IEEE.

  • Matsui, Y., Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., & Aizawa, K. (2016). Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications, 76, 21811–21838.

    Article  Google Scholar 

  • Ma, C., Yang, C.-Y., Yang, X., & Yang, M.-H. (2017). Learning a no-reference quality metric for single-image super-resolution. Computer Vision and Image Understanding, 158, 1–16.

    Article  Google Scholar 

  • Michaeli, T., & Irani, M.(2013). Nonparametric blind super-resolution. In IEEE international conference on computer vision (pp. 945–952).

  • Mittal, A., Soundararajan, R., & Bovik, A. C. (2012). Making a “completely blind’’ image quality analyzer. IEEE Signal Processing Letters, 20(3), 209–212.

    Article  Google Scholar 

  • Pan, J., Hu, Z., Su, Z., & Yang, M.-H. (2014). Deblurring text images via l0-regularized intensity and gradient prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2901–2908).

  • Pan, J., Sun, D., Pfister, H., & Yang, M.H. (2016).Blind image deblurring using dark channel prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1628–1636).

  • Pan, J., Sun, D., Pfister, H., & Yang, M.-H. (2018). Deblurring images via dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40, 2315–2328.

    Article  Google Scholar 

  • Pascal, F., Bombrun, L., Tourneret, J.-Y., & Berthoumieu, Y. (2013). Parameter estimation for multivariate generalized gaussian distributions. IEEE Transactions on Signal Processing, 61(23), 5960–5971.

    Article  MathSciNet  MATH  Google Scholar 

  • Ren, D., Zhang, K., Wang, Q., Hu, Q., & Zuo, W. (2020). Neural blind deconvolution using deep priors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3341–3350).

  • Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1874–1883).

  • Shin, R., & Song, D. (2017). Jpeg-resistant adversarial images. In NIPS 2017 workshop on machine learning and computer security (vol. 1).

  • Shocher, A., Cohen, N., & Irani, M.(2018). “zero-shot” super-resolution using deep internal learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

  • Simonyan, K., & Zisserman, A.(2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

  • Soh, J.W., Cho, S., & Cho, N.I.(2020). Meta-transfer learning for zero-shot super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3516–3525).

  • Timofte, R., De Smet, V., & Van Gool, L.(2013). Anchored neighborhood regression for fast example-based super-resolution. In Proceedings of the IEEE international conference on computer vision (pp. 1920–1927).

  • Timofte, R. et al., (2017). Ntire 2017 challenge on single image super-resolution: Methods and results pp. 1110–1121

  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems30.

  • Wang, Z., Chen, J., & Hoi, S.C.(2020). Deep learning for image super-resolution: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence.

  • Wang, X., Xie, L., Dong, C., & Shan, Y. (2021). Real-esrgan: Training real-world blind super-resolution with pure synthetic data. arXiv preprint arXiv:2107.10833.

  • Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., & Change Loy, C.(2018). Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision workshops.

  • Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Article  Google Scholar 

  • Wei, Y., Gu, S., Li, Y., Timofte, R., Jin, L., & Song, H.(2021). Unsupervised real-world image super resolution via domain-distance aware training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 13385–13394).

  • Wei, P., Xie, Z., Lu, H., Zhan, Z., Ye, Q., Zuo, W., &Lin, L. (2020).Component divide-and-conquer for real-world image super-resolution. In European conference on computer vision (pp. 101–117). Springer.

  • Xie, J., Zhan, X., Liu, Z., Ong, Y.-S., & Loy, C.C.(2022). Delving into inter-image invariance for unsupervised visual representations. International Journal of Computer Vision, 1–20.

  • Yan, Y., Ren, W., Guo, Y., Wang, R., & Cao, X.(2017). Image deblurring via extreme channels prior. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4003–4011).

  • Yang, J., Wright, J., Huang, T. S., & Ma, Y.(2010). Image super-resolution via sparse representation. IEEE Transactions on Image Processing 19(11), 2861–2873.

  • Zhang, K., Gool, L.V., & Timofte, R. (2020). Deep unfolding network for image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3217–3226).

  • Zhang, R., Isola, P., Efros, A.A., Shechtman, E., & Wang, O.(2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

  • Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., & Fu, Y. (2018). Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision (pp. 286–301).

  • Zhang, K., Liang, J., Van Gool, L., &Timofte, R.(2021). Designing a practical degradation model for deep blind image super-resolution. arXiv preprint arXiv:2103.14006.

  • Zhang, K., Zuo, W., & Zhang, L.(2018). Ffdnet: Toward a fast and flexible solution for cnn-based image denoising. IEEE Transactions on Image Processing27(9), 4608–4622.

  • Zhang, K., Zuo, W., & Zhang, L. (2018).Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3262–3271).

  • Zhang, K., Zuo, W., & Zhang, L. (2019). Deep plug-and-play super-resolution for arbitrary blur kernels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1671–1681).

  • Zhang, K., Zuo, W., Gu, S., & Zhang, L.(2017). Learning deep cnn denoiser prior for image restoration. In Proceedings of the IEEE/CVF conference on computer cision and pattern recognition (pp. 3929–3938).

  • Zhang, H., Li, Y., Chen, H., Gong, C., Bai, Z., & Shen, C. (2022). Memory-efficient hierarchical neural architecture search for image restoration. International Journal of Computer Vision, 130(1), 157–178.

    Article  Google Scholar 

  • Zhou, M., Yan, K., Pan, J., Ren, W., Xie, Q., & Cao, X.(2022). Memory-augmented deep unfolding network for guided image super-resolution. International Journal of Computer Vision.

Download references

Author information

Authors and Affiliations

  1. Artificial Intelligence School, University of Chinese Academy of Sciences, Beijing, 100049, China

    Zhengxiong Luo & Shang Li

  2. Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Zhengxiong Luo, Yan Huang, Shang Li, Liang Wang & Tieniu Tan

  3. Center for Research on Intelligent Perception and Computing, Beijing, China

    Zhengxiong Luo, Yan Huang, Liang Wang & Tieniu Tan

  4. Center for Excellence in Brain Science and Intelligence Technology, Beijing, China

    Liang Wang

  5. Nanjing University, Nanjing, China

    Tieniu Tan

Authors
  1. Zhengxiong Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tieniu Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yan Huang.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Communicated by Rynson W. H. Lau.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luo, Z., Huang, Y., Li, S. et al. End-to-End Alternating Optimization for Real-World Blind Super Resolution. Int J Comput Vis 131, 3152–3169 (2023). https://doi.org/10.1007/s11263-023-01833-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-023-01833-7

Keywords

Search

Navigation