Skip to main content
SpringerLink
Log in

Pixel-level self-paced adversarial network with multiple attention in single image super-resolution

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Image super-resolution (SR) is an important image processing technique in computer vision. Although the convolutional neural network has developed rapidly and made some breakthroughs in the field of super-division, there are still some problems when images are magnified at large upscaling factors. Recently, generative adversarial network is popular, but the structural similarity (SSIM) between the super-resolution (SR) image generated by GAN network and high-resolution (HR) image is always unsatisfactory. In this paper, we propose a pixel-level self-paced adversarial network with multiple attention (PSPA) method to reduce the noise of SR image and increase its structural similarity with HR image. The combination of multiple attentions makes the model grasp the global information and restore the detail texture more accurately. The PSPA network can make the model notice the position with a large difference between the pixel values of SR and HR images and speed up the gradient descent speed. Our method shows excellent performance on Set5, Set14 and BSD100 datasets and overcomes many popular algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and materials

All data generated or analyzed during this study are included in this published article and its supplementary information files.

References

  1. Lanaras, C., Bioucas-Dias, J., Galliani, S., Baltsavias, E., Schindler, K.: Super-resolution of sentinel-2 images: learning a globally applicable deep neural network. ISPRS J. Photogr. Remote Sens. 146, 305–319 (2018)

    Article  Google Scholar 

  2. Basak, H., Kundu, R., Singh, P.K., Ijaz, M.F., Woźniak, M., Sarkar, R.: A union of deep learning and swarm-based optimization for 3d human action recognition. Scient. Reports 12(1), 1–17 (2022)

    Article  Google Scholar 

  3. Yan, G., Woźniak, M.: Accurate key frame extraction algorithm of video action for aerobics online teaching. Mobile Networks and Applications, 1–10 (2022)

  4. Shao, J., Cheng, Q.: E-fcnn for tiny facial expression recognition. Appl Intell 51(1), 549–559 (2021)

    Article  Google Scholar 

  5. Zhang, L., Zhang, H., Shen, H., Li, P.: A super-resolution reconstruction algorithm for surveillance images. Signal Process 90(3), 848–859 (2010)

    Article  MATH  Google Scholar 

  6. Wieczorek, M., Siłka, J., Woźniak, M., Garg, S., Hassan, M.M.: Lightweight convolutional neural network model for human face detection in risk situations. IEEE Trans. Ind. Infor 18, 4820–4829 (2021)

    Article  Google Scholar 

  7. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1637–1645 (2016)

  8. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3147–3155 (2017)

  9. Li, Z., Yang, J., Liu, Z., Yang, X., Jeon, G., Wu, W.: Feedback network for image super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3867–3876 (2019)

  10. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681–4690 (2017)

  11. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, C.: Esrgan: Enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV) Workshops, (2018)

  12. Rakotonirina, N.C., Rasoanaivo, A.: Esrgan+: Further improving enhanced super-resolution generative adversarial network. In: ICASSP 2020-2020 IEEE International conference on acoustics, speech and signal processing (ICASSP), pp. 3637–3641 (2020). IEEE

  13. Wang, X., Xie, L., Dong, C., Shan, Y.: Real-esrgan: training real-world blind super-resolution with pure synthetic data. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 1905–1914 (2021)

  14. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4), 600–612 (2004)

    Article  Google Scholar 

  15. Bevilacqua, M., Roumy, A., Guillemot, C., Morel, A.: Low-complexity single image super-resolution based on nonnegative neighbor embedding. BMVC (2012)

  16. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: International conference on curves and surfaces, pp. 711–730 (2010). Springer

  17. Yang, W., Zhang, X., Tian, Y., Wang, W., Xue, J.-H., Liao, Q.: Deep learning for single image super-resolution: a brief review. IEEE Trans Multim 21(12), 3106–3121 (2019)

    Article  Google Scholar 

  18. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European conference on computer vision, pp. 184–199 (2014). Springer

  19. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans Patt Anal Mach Intell 38(2), 295–307 (2015)

    Article  Google Scholar 

  20. Talab, M.A., Awang, S., Najim, S.A.-d.M.: Super-low resolution face recognition using integrated efficient sub-pixel convolutional neural network (ESPCN) and convolutional neural network (cnn). In: 2019 IEEE International conference on automatic control and intelligent systems (I2CACIS), pp. 331–335 (2019)

  21. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646–1654 (2016)

  22. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 770–778 (2016)

  23. Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: Proceedings of the IEEE international conference on computer vision, pp. 4799–4807 (2017)

  24. Dharejo, F.A., Deeba, F., Zhou, Y., Das, B., Jatoi, M.A., Zawish, M., Du, Y., Wang, X.: Twist-gan: towards wavelet transform and transferred gan for spatio-temporal single image super resolution. ACM Trans Intell Sys Technol (TIST) 12(6), 1–20 (2021)

    Article  Google Scholar 

  25. Shi, Y., Han, L., Han, L., Chang, S., Hu, T., Dancey, D.: A latent encoder coupled generative adversarial network (le-gan) for efficient hyperspectral image super-resolution. IEEE Trans Geosci Remote Sens 60, 1–19 (2022)

    Google Scholar 

  26. Gong, Y., Liao, P., Zhang, X., Zhang, L., Chen, G., Zhu, K., Tan, X., Lv, Z.: Enlighten-gan for super resolution reconstruction in mid-resolution remote sensing images. Remote Sens 13(6), 1104 (2021)

    Article  Google Scholar 

  27. de Farias, E.C., Di Noia, C., Han, C., Sala, E., Castelli, M., Rundo, L.: Impact of gan-based lesion-focused medical image super-resolution on the robustness of radiomic features. Scientif Reports 11(1), 1–12 (2021)

    Article  Google Scholar 

  28. Cheng, W., Zhao, M., Ye, Z., Gu, S.: Mfagan: A compression framework for memory-efficient on-device super-resolution gan. arXiv preprint arXiv:2107.12679 (2021)

  29. Lu, Y., Zhou, Y., Jiang, Z., Guo, X., Yang, Z.: Channel attention and multi-level features fusion for single image super-resolution. In: 2018 IEEE visual communications and image processing (VCIP), IEEE, pp. 1–4 (2018)

  30. Liu, Y., Wang, Y., Li, N., Cheng, X., Zhang, Y., Huang, Y., Lu, G.: An attention-based approach for single image super resolution. In: 2018 24Th International conference on pattern recognition (ICPR), pp. 2777–2784 (2018). IEEE

  31. Yang, F., Yang, H., Fu, J., Lu, H., Guo, B.: Learning texture transformer network for image super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5791–5800 (2020)

  32. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: Proceedings of the european conference on computer vision (ECCV), pp. 3–19 (2018)

  33. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)

  34. Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., Tang, X.: Residual attention network for image classification. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 3156–3164 (2017)

  35. Lin, W., Gao, J., Wang, Q., Li, X.: Pixel-level self-paced learning for super-resolution. In: ICASSP 2020-2020 IEEE International conference on acoustics, speech and signal processing (ICASSP), IEEE, pp. 2538–2542 (2020)

  36. Jolicoeur-Martineau, A.: The relativistic discriminator: a key element missing from standard gan. arXiv preprint arXiv:1807.00734 (2018)

  37. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision, pp. 740–755 (2014). Springer

  38. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 136–144 (2017)

  39. Dai, T., Cai, J., Zhang, Y., Xia, S.-T., Zhang, L.: Second-order attention network for single image super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11065–11074 (2019)

  40. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Swinir: image restoration using swin transformer. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 1833–1844 (2021)

Download references

Funding

This work was supported by the Local the College Capacity Building Project of Shanghai Municipal Science and Technology Commission under Grant No. 20020500700.

Author information

Authors and Affiliations

  1. Department of Information and Communication Engineering, Shanghai University of Electric Power, Shanghai, 200120, China

    Jie Shao, Xuecheng Zhuang, Zhengqi Wang & Wenzhong Shen

Authors
  1. Jie Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuecheng Zhuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhengqi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenzhong Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors have the same contribution. All authors reviewed the manuscript.

Corresponding author

Correspondence to Xuecheng Zhuang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shao, J., Zhuang, X., Wang, Z. et al. Pixel-level self-paced adversarial network with multiple attention in single image super-resolution. SIViP 17, 1863–1872 (2023). https://doi.org/10.1007/s11760-022-02397-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02397-8

Keywords

Search

Navigation