Skip to main content
SpringerLink
Log in

Reference Image Guided Super-Resolution via Progressive Channel Attention Networks

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

In recent years, the convolutional neural networks (CNNs) for single image super-resolution (SISR) are becoming more and more complex, and it is more challenging to improve the SISR performance. In contrast, the reference image guided super-resolution (RefSR) is an effective strategy to boost the SR (super-resolution) performance. In RefSR, the introduced high-resolution (HR) references can facilitate the high-frequency residual prediction process. According to the best of our knowledge, the existing CNN-based RefSR methods treat the features from the references and the low-resolution (LR) input equally by simply concatenating them together. However, the HR references and the LR inputs contribute differently to the final SR results. Therefore, we propose a progressive channel attention network (PCANet) for RefSR. There are two technical contributions in this paper. First, we propose a novel channel attention module (CAM), which estimates the channel weighting parameter by weightedly averaging the spatial features instead of using global averaging. Second, considering that the residual prediction process can be improved when the LR input is enriched with more details, we perform super-resolution progressively, which can take advantage of the reference images in multi-scales. Extensive quantitative and qualitative evaluations on three benchmark datasets, which represent three typical scenarios for RefSR, demonstrate that our method is superior to the state-of-the-art SISR and RefSR methods in terms of PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Dong C, Loy C C, He K et al. Learning a deep convolutional network for image super-resolution. In Proc. the 13th European Conference on Computer Vision, September 2014, pp.184-199.

  2. Wang Z, Liu D, Yang J et al. Deep networks for image super-resolution with sparse prior. In Proc. the 2015 IEEE International Conference on Computer Vision, December 2015, pp.370-378.

  3. Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.1646-1654.

  4. Lai WS, Huang J B, Ahuja N et al. Deep Laplacian pyramid networks for fast and accurate super-resolution. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.5835-5843.

  5. Tong T, Li G, Liu X et al. Image super-resolution using dense skip connections. In Proc. the IEEE International Conference on Computer Vision, October 2017, pp.4809-4817.

  6. Lim B, Son S, Kim H et al. Enhanced deep residual networks for single image super-resolution. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, July 2017, pp.1132-1140.

  7. Hu Y, Li J, Huang Y et al. Channel-wise and spatial feature modulation network for single image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology. doi:https://doi.org/10.1109/TCSVT.2019.2915238.

  8. Zhang Y, Li K, Li K et al. Image super-resolution using very deep residual channel attention networks. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.294-310.

  9. Liu S, Gang R, Li C et al. Adaptive deep residual network for single image super-resolution. Computational Visual Media, 2019, 5(4): 391-401.

    Article  Google Scholar 

  10. Yue H, Sun X, Yang J et al. Landmark image super-resolution by retrieving web images. IEEE Transactions on Image Processing, 2013, 22(12): 4865-4878.

    Article  MathSciNet  Google Scholar 

  11. Zhang Z, Wang Z, Lin Z et al. Image super-resolution by neural texture transfer. In Proc. the 2019 IEEE Conference on Computer Vision and Pattern Recognition, June 2019, pp.7982-7991.

  12. Zheng H, Ji M, Wang H et al. CrossNet: An end-to-end reference-based super resolution network using cross-scale warping. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.87-104.

  13. Li Y, Dong W, Shi G et al. Learning parametric distributions for image super-resolution: Where patch matching meets sparse coding. In Proc. the 2015 IEEE International Conference on Computer Vision, December 2015, pp.450-458.

  14. Liu J, Yang W, Zhang X et al. Retrieval compensated group structured sparsity for image super-resolution. IEEE Transactions on Multimedia, 2017, 19(2): 302-316.

    Article  Google Scholar 

  15. Yang W, Xia S, Liu J et al. Reference-guided deep super-resolution via manifold localized external compensation. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 29(5): 1270-1283.

    Article  Google Scholar 

  16. Yue H, Liu J, Yang J et al. IENet: Internal and external patch matching ConvNet for web image guided denoising. IEEE Transactions on Circuits and Systems for Video Technology. doi: https://doi.org/10.1109/TCSVT.2019.2930305.

  17. Zhang J, Gai D, Zhang X et al. Multi-example feature-constrained back-projection method for image super-resolution. Computational Visual Media, 2017, 3(1): 73-82.

    Article  Google Scholar 

  18. Zhao X, Wu Y, Tian J et al. Single image super-resolution via blind blurring estimation and anchored space mapping. Computational Visual Media, 2016, 2(1): 71-85.

    Article  Google Scholar 

  19. Glasner D, Bagon S, Irani M. Super-resolution from a single image. In Proc. the 12th IEEE International Conference on Computer Vision, September 2009, pp.349-356.

  20. Freeman W T, Jones T R, Pasztor E C. Example-based super-resolution. IEEE Computer Graphics and Applications, 2002, 22(2): 56-65.

    Article  Google Scholar 

  21. Yang J, Wright J, Huang T et al. Image super-resolution as sparse representation of raw image patches. In Proc. the 2008 IEEE Conference on Computer Vision and Pattern Recognition, June 2008.

  22. Yang J, Wright J, Huang T S et al. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873.

    Article  MathSciNet  Google Scholar 

  23. Kim J, Lee J K, Lee K M. Deeply-recursive convolutional network for image super-resolution. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.1637-1645.

  24. Shi W, Caballero J, Huszár F et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.1874-1883.

  25. Ledig C, Theis L, Huszár F et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.105-114.

  26. He K, Zhang X, Ren S et al. Deep residual learning for image recognition. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.770-778.

  27. Goodfellow I, Pouget-Abadie J, Mirza M et al. Generative adversarial nets. In Proc. the 2014 Annual Conference on Neural Information Processing Systems, December 2014, pp.2672-2680.

  28. Johnson J, Alahi A, Li F F. Perceptual losses for real-time style transfer and super-resolution. In Proc. the 14th European Conference on Computer Vision, October 2016, pp.694-711.

  29. Hu J, Shen L, Sun G. Squeeze-and-excitation networks. In Proc. the 2018 IEEE Conference on Computer Vision and Pattern Recognition, June 2018, pp.7132-7141.

  30. Mansimov E, Parisotto E, Ba J L et al. Generating images from captions with attention. arXiv:1511.02793, 2015. https://arxiv.org/abs/1511.02793, Jan. 2020.

  31. Kim J H, Choi J H, Cheon M et al. Ram: Residual attention module for single image super-resolution. arXiv:1811.12043, 2018. https://arxiv.org/pdf/1811.12043.pdf, Jan. 2020.

  32. Woo S, Park J, Lee J Y et al. CBAM: Convolutional block attention module. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.3-19.

  33. Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91-110.

    Article  Google Scholar 

  34. Fischler M A, Bolles R C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981, 24(6): 381-395.

    Article  MathSciNet  Google Scholar 

  35. Kingma D P, Ba J. ADAM: A method for stochastic optimization. arXiv:1412.6980, 2014. https://arxiv.org/pdf/1412.6980.pdf, Jan. 2020.

  36. Cao Q, Shen L, Xie W et al. VGGFace2: A dataset for recognising faces across pose and age. In Proc. the 13th IEEE International Conference on Automatic Face & Gesture Recognition, May 2018, pp.67-74.

  37. Wang Z, Bovik A C, Sheikh H R et al. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China

    Huan-Jing Yue, Sheng Shen, Jing-Yu Yang & Yan-Fang Chen

  2. School of Precision Instrument and Opto-Electronics Engineering, Tianjin University, Tianjin, 300072, China

    Hao-Feng Hu

Authors
  1. Huan-Jing Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sheng Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jing-Yu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hao-Feng Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yan-Fang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jing-Yu Yang.

Electronic supplementary material

ESM 1

(PDF 543 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yue, HJ., Shen, S., Yang, JY. et al. Reference Image Guided Super-Resolution via Progressive Channel Attention Networks. J. Comput. Sci. Technol. 35, 551–563 (2020). https://doi.org/10.1007/s11390-020-0270-3

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-020-0270-3

Keywords

Search

Navigation