Skip to main content
SpringerLink
Log in

Self-supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13678))

Included in the following conference series:

Abstract

In this paper, we consider two challenging issues in reference-based super-resolution (RefSR), (i) how to choose a proper reference image, and (ii) how to learn real-world RefSR in a self-supervised manner. Particularly, we present a novel self-supervised learning approach for real-world image SR from observations at dual camera zooms (SelfDZSR). Considering the popularity of multiple cameras in modern smartphones, the more zoomed (telephoto) image can be naturally leveraged as the reference to guide the SR of the lesser zoomed (short-focus) image. Furthermore, SelfDZSR learns a deep network to obtain the SR result of short-focus image to have the same resolution as the telephoto image. For this purpose, we take the telephoto image instead of an additional high-resolution image as the supervision information and select a center patch from it as the reference to super-resolve the corresponding short-focus image patch. To mitigate the effect of the misalignment between short-focus low-resolution (LR) image and telephoto ground-truth (GT) image, we design an auxiliary-LR generator and map the GT to an auxiliary-LR while keeping the spatial position unchanged. Then the auxiliary-LR can be utilized to deform the LR features by the proposed adaptive spatial transformer networks (AdaSTN), and match the Ref features to GT. During testing, SelfDZSR can be directly deployed to super-solve the whole short-focus image with the reference of telephoto image. Experiments show that our method achieves better quantitative and qualitative performance against state-of-the-arts. Codes are available at https://github.com/cszhilu1998/SelfDZSR.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bell-Kligler, S., Shocher, A., Irani, M.: Blind super-resolution kernel estimation using an internal-gan. In: NeurIPS, pp. 284–293 (2019)

    Google Scholar 

  2. Cai, J., Gu, S., Timofte, R., Zhang, L.: Ntire 2019 challenge on real image super-resolution: Methods and results. In: CVPR Workshops (2019)

    Google Scholar 

  3. Cai, J., Zeng, H., Yong, H., Cao, Z., Zhang, L.: Toward real-world single image super-resolution: A new benchmark and a new model. In: ICCV, pp. 3086–3095 (2019)

    Google Scholar 

  4. Chan, K.C., Wang, X., Yu, K., Dong, C., Loy, C.C.: Understanding deformable alignment in video super-resolution. In: AAAI, pp. 973–981 (2021)

    Google Scholar 

  5. Chen, C., Xiong, Z., Tian, X., Zha, Z.J., Wu, F.: Camera lens super-resolution. In: CVPR, pp. 1652–1660 (2019)

    Google Scholar 

  6. Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., Wei, Y.: Deformable convolutional networks. In: ICCV, pp. 764–773 (2017)

    Google Scholar 

  7. Delbracio, M., Talebi, H., Milanfar, P.: Projected distribution loss for image enhancement. arXiv preprint arXiv:2012.09289 (2020)

  8. Deshpande, I., Zhang, Z., Schwing, A.G.: Generative modeling using the sliced wasserstein distance. In: CVPR, pp. 3483–3491 (2018)

    Google Scholar 

  9. Dinh, L., Sohl-Dickstein, J., Bengio, S.: Density estimation using real nvp. In: ICLR (2017)

    Google Scholar 

  10. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE PAMI 38(2), 295–307 (2015)

    Article  Google Scholar 

  11. Dosovitskiy, A., et al.: Flownet: Learning optical flow with convolutional networks. In: ICCV, pp. 2758–2766 (2015)

    Google Scholar 

  12. Geng, Z., Sun, K., Xiao, B., Zhang, Z., Wang, J.: Bottom-up human pose estimation via disentangled keypoint regression. In: CVPR, pp. 14676–14686 (2021)

    Google Scholar 

  13. Gu, J., Lu, H., Zuo, W., Dong, C.: Blind super-resolution with iterative kernel correction. In: CVPR, pp. 1604–1613 (2019)

    Google Scholar 

  14. He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: CVPR, pp. 9729–9738 (2020)

    Google Scholar 

  15. Heitz, E., Vanhoey, K., Chambon, T., Belcour, L.: A sliced wasserstein loss for neural texture synthesis. In: CVPR, pp. 9412–9420 (2021)

    Google Scholar 

  16. Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image super-resolution with information multi-distillation network. In: ACM MM, pp. 2024–2032 (2019)

    Google Scholar 

  17. Hussein, S.A., Tirer, T., Giryes, R.: Correction filter for single image super-resolution: Robustifying off-the-shelf deep super-resolvers. In: CVPR, pp. 1428–1437 (2020)

    Google Scholar 

  18. Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: NeurIPS, pp. 2017–2025 (2015)

    Google Scholar 

  19. Jiang, Y., Chan, K.C., Wang, X., Loy, C.C., Liu, Z.: Robust reference-based super-resolution via c2-matching. In: CVPR, pp. 2103–2112 (2021)

    Google Scholar 

  20. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)

    Google Scholar 

  21. Kong, X., Zhao, H., Qiao, Y., Dong, C.: Classsr: A general framework to accelerate super-resolution networks by data characteristic. In: CVPR, pp. 12016–12025 (2021)

    Google Scholar 

  22. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, pp. 4681–4690 (2017)

    Google Scholar 

  23. Liang, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Mutual affine network for spatially variant kernel estimation in blind image super-resolution. In: ICCV, pp. 4096–4105 (2021)

    Google Scholar 

  24. Liang, J., Zhang, K., Gu, S., Van Gool, L., Timofte, R.: Flow-based kernel prior with application to blind super-resolution. In: CVPR, pp. 10601–10610 (2021)

    Google Scholar 

  25. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: CVPR Workshops, pp. 136–144 (2017)

    Google Scholar 

  26. Liu, M., Zhang, Z., Hou, L., Zuo, W., Zhang, L.: Deep adaptive inference networks for single image super-resolution. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12538, pp. 131–148. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-66823-5_8

    Chapter  Google Scholar 

  27. Lu, L., Li, W., Tao, X., Lu, J., Jia, J.: Masa-sr: Matching acceleration and spatial adaptation for reference-based image super-resolution. In: CVPR, pp. 6368–6377 (2021)

    Google Scholar 

  28. Lugmayr, A., Danelljan, M., Timofte, R.: Ntire 2020 challenge on real-world image super-resolution: Methods and results. In: CVPR Workshops, pp. 494–495 (2020)

    Google Scholar 

  29. Lugmayr, A., Danelljan, M., Timofte, R., Fritsche, M., et al.: Aim 2019 challenge on real-world image super-resolution: Methods and results. In: ICCV Workshops, pp. 3575–3583. IEEE (2019)

    Google Scholar 

  30. Luo, Z., Huang, Y., Li, S., Wang, L., Tan, T.: Unfolding the alternating optimization for blind super resolution. In: NeurIPS (2020)

    Google Scholar 

  31. Paszke, A., et al.: Pytorch: An imperative style, high-performance deep learning library. In: NeurIPS, pp. 8024–8035 (2019)

    Google Scholar 

  32. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR, pp. 1874–1883 (2016)

    Google Scholar 

  33. Shim, G., Park, J., Kweon, I.S.: Robust reference-based super-resolution with similarity-aware deformable convolution. In: CVPR, pp. 8425–8434 (2020)

    Google Scholar 

  34. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2014)

    Google Scholar 

  35. Sun, D., Yang, X., Liu, M.Y., Kautz, J.: Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In: CVPR, pp. 8934–8943 (2018)

    Google Scholar 

  36. Wang, L., et al.: Exploring sparsity in image super-resolution for efficient inference. In: CVPR, pp. 4917–4926 (2021)

    Google Scholar 

  37. Wang, L., et al.: Unsupervised degradation representation learning for blind super-resolution. In: CVPR, pp. 10581–10590 (2021)

    Google Scholar 

  38. Wang, T., Xie, J., Sun, W., Yan, Q., Chen, Q.: Dual-camera super-resolution with aligned attention modules. In: ICCV, pp. 2001–2010 (2021)

    Google Scholar 

  39. Wang, X., Xie, L., Dong, C., Shan, Y.: Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In: ICCV Workshops, pp. 1905–1914 (2021)

    Google Scholar 

  40. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE TIP 13(4), 600–612 (2004)

    Google Scholar 

  41. Wei, P., et al.: AIM 2020 challenge on real image super-resolution: methods and results. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12537, pp. 392–422. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-67070-2_24

    Chapter  Google Scholar 

  42. Wei, P., et al.: Component divide-and-conquer for real-world image super-resolution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12353, pp. 101–117. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58598-3_7

    Chapter  Google Scholar 

  43. Wei, Y., Gu, S., Li, Y., Timofte, R., Jin, L., Song, H.: Unsupervised real-world image super resolution via domain-distance aware training. In: CVPR, pp. 13385–13394 (2021)

    Google Scholar 

  44. Wu, J., et al.: Sliced wasserstein generative models. In: CVPR, pp. 3713–3722 (2019)

    Google Scholar 

  45. Xie, W., Song, D., Xu, C., Xu, C., Zhang, H., Wang, Y.: Learning frequency-aware dynamic network for efficient super-resolution. In: ICCV, pp. 4308–4317 (2021)

    Google Scholar 

  46. Xie, Y., Xiao, J., Sun, M., Yao, C., Huang, K.: Feature representation matters: End-to-end learning for reference-based image super-resolution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 230–245. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_14

    Chapter  Google Scholar 

  47. Yang, F., Yang, H., Fu, J., Lu, H., Guo, B.: Learning texture transformer network for image super-resolution. In: CVPR, pp. 5791–5800 (2020)

    Google Scholar 

  48. Zhang, K., Liang, J., Van Gool, L., Timofte, R.: Designing a practical degradation model for deep blind image super-resolution. In: ICCV, pp. 4791–4800 (2021)

    Google Scholar 

  49. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: CVPR, pp. 3262–3271 (2018)

    Google Scholar 

  50. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR, pp. 586–595 (2018)

    Google Scholar 

  51. Zhang, X., Chen, Q., Ng, R., Koltun, V.: Zoom to learn, learn to zoom. In: CVPR, pp. 3762–3770 (2019)

    Google Scholar 

  52. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 294–310. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_18

    Chapter  Google Scholar 

  53. Zhang, Y., Zhang, Z., DiVerdi, S., Wang, Z., Echevarria, J., Fu, Y.: Texture hallucination for large-factor painting super-resolution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12352, pp. 209–225. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58571-6_13

    Chapter  Google Scholar 

  54. Zhang, Z., Wang, Z., Lin, Z., Qi, H.: Image super-resolution by neural texture transfer. In: CVPR, pp. 7982–7991 (2019)

    Google Scholar 

  55. Zhang, Z., Wang, H., Liu, M., Wang, R., Zhang, J., Zuo, W.: Learning raw-to-srgb mappings with inaccurately aligned supervision. In: ICCV, pp. 4348–4358 (2021)

    Google Scholar 

  56. Zheng, H., et al.: Learning cross-scale correspondence and patch-based synthesis for reference-based super-resolution. In: BMVC, vol. 1, p. 2 (2017)

    Google Scholar 

  57. Zheng, H., Ji, M., Wang, H., Liu, Y., Fang, L.: Crossnet: An end-to-end reference-based super resolution network using cross-scale warping. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11210, pp. 87–104. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01231-1_6

    Chapter  Google Scholar 

  58. Zhu, X., Hu, H., Lin, S., Dai, J.: Deformable convnets v2: More deformable, better results. In: CVPR, pp. 9308–9316 (2019)

    Google Scholar 

Download references

Acknowledgement

This work was supported by Alibaba Group through Alibaba Innovative Research Program, the Major Key Project of Peng Cheng Laboratory (PCL2021A12), and the National Natural Science Foundation of China (NSFC) under Grants No.s 61872118 and U19A2073.

Author information

Authors and Affiliations

  1. Harbin Institute of Technology, Harbin, China

    Zhilu Zhang, Ruohao Wang, Hongzhi Zhang, Yunjin Chen & Wangmeng Zuo

  2. Peng Cheng Laboratory, Shenzhen, China

    Yunjin Chen & Wangmeng Zuo

Authors
  1. Zhilu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruohao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongzhi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yunjin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wangmeng Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongzhi Zhang .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 3139 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, Z., Wang, R., Zhang, H., Chen, Y., Zuo, W. (2022). Self-supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13678. Springer, Cham. https://doi.org/10.1007/978-3-031-19797-0_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19797-0_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19796-3

  • Online ISBN: 978-3-031-19797-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation