Skip to main content
SpringerLink
Log in

Exploiting local detail in single image super-resolution via hypergraph convolution

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Exploiting both local detail features and global correlation information in low-resolution (LR) images is vital for single-image super-resolution (SISR) reconstruction. Current deep CNN methods often overlook the interaction of local details while focusing on global features, rendering them incapable of accommodating both global and local features of LR images, therefore compromising the reconstruction performance. To address this, we propose a hypergraph convolution super-resolution (HCSR) network, which integrates local detail feature information and global correlation information. Hypergraph convolution is innovatively incorporated into SISR tasks, facilitating the modeling of complex spatial relationships among local features. We devise an incidence matrix for hypergraph convolution and propose a spatial feature interaction module (SFIM) for enriched local texture detail. We also use multi-hyperplane locally sensitive hash for efficient non-local hash attention (NLHA), optimally extracting correlations among global features. Based on this, a feature forward module (FFM) is developed that integrates global dependencies of multi-level features, improving the overall image detail recovery. Evaluated across five benchmark datasets, the proposed demonstrates substantial performance in both quantitative metrics and visual appeal.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

Data will be made available on request.

References

  1. Georgescu, M.-I., Ionescu, R.T., Miron, A.-I., Savencu, O., Ristea, N.-C., Verga, N., Khan, F.S.: Multimodal multi-head convolutional attention with various kernel sizes for medical image super-resolution. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2195–2205 (2023). https://doi.org/10.48550/arXiv.2204.04218

  2. Lei, S., Shi, Z.: Hybrid-scale self-similarity exploitation for remote sensing image super-resolution. IEEE Trans. Geosci. Remote Sens. 60, 1–10 (2021). https://doi.org/10.1109/TGRS.2021.3069889

    Article  Google Scholar 

  3. Pang, Y., Cao, J., Wang, J., Han, J.: Jcs-net: Joint classification and super-resolution network for small-scale pedestrian detection in surveillance images. IEEE Trans. Inf. Forensics Secur. 14(12), 3322–3331 (2019). https://doi.org/10.1109/TIFS.2019.2916592

    Article  Google Scholar 

  4. Dong, C., Chen, C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV vol. 13, pp. 184–199 (2014). Springer. https://doi.org/10.48550/arXiv.1501.00092

  5. Lai, W.-S., Huang, J.-B., Ahuja, N., Yang, M.-H.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 624–632 (2017). https://doi.org/10.1109/CVPR.2017.618

  6. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017). https://doi.org/10.48550/arXiv.1707.02921

  7. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018). https://doi.org/10.48550/arXiv.1802.08797

  8. Anwar, S., Barnes, N.: Densely residual laplacian super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 44(3), 1192–1204 (2020). https://doi.org/10.48550/arXiv.1906.12021

    Article  Google Scholar 

  9. Zhao, H., Kong, X., He, J., Qiao, Y., Dong, C.: Efficient image super-resolution using pixel attention. In: Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part III vol. 16, pp. 56–72 (2020). Springer. https://doi.org/10.48550/arXiv.2210.05960

  10. Niu, B., Wen, W., Ren, W., Zhang, X., Yang, L., Wang, S., Zhang, K., Cao, X., Shen, H.: Single image super-resolution via a holistic attention network. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII vol. 16, pp. 191–207 (2020). Springer. https://doi.org/10.48550/arXiv.2008.08767

  11. Chen, H., Gu, J., Zhang, Z.: Attention in attention network for image super-resolution. arXiv preprint arXiv:2104.09497 (2021). https://doi.org/10.48550/arXiv.2104.09497

  12. Xia, B., Hang, Y., Tian, Y., Yang, W., Liao, Q., Zhou, J.: Efficient non-local contrastive attention for image super-resolution. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 2759–2767 (2022). https://doi.org/10.48550/arXiv.2201.03794

  13. Zhou, S., Zhang, J., Zuo, W., Loy, C.C.: Cross-scale internal graph neural network for image super-resolution. Adv. Neural Inf. Process. Syst. 33, 3499–3509 (2020)

    Google Scholar 

  14. Maeda, S.: Image super-resolution with deep dictionary. In: European Conference on Computer Vision, pp. 464–480 (2022). Springer. https://doi.org/10.48550/arXiv.2207.09228

  15. Lee, J., Jin, K.H.: Local texture estimator for implicit representation function. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 929–1938 (2022). https://doi.org/10.48550/arXiv.2111.08918

  16. Ma, C., Zhang, J., Zhou, J., Lu, J.: Learning series-parallel lookup tables for efficient image super-resolution. In: European Conference on Computer Vision, pp. 305–321 (2022). Springer. https://doi.org/10.48550/arXiv.2207.12987

  17. Guo, B., Zhang, X., Wu, H., Wang, Y., Zhang, Y., Wang, Y.-F.: Lar-sr: a local autoregressive model for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1909–1918 (2022). https://doi.org/10.1109/CVPR52688.2022.00195

  18. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016). https://doi.org/10.1109/CVPR.2016.182

  19. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 286–301 (2018). https://doi.org/10.48550/arXiv.1807.02758

  20. Yang, L., Tang, J., Niu, B., Fu, H., Zhu, H., Jiang, W., Wang, X.: Single image super-resolution via a ternary attention network. Appl. Intell. 53(11), 13067–13081 (2023). https://doi.org/10.1007/s10489-022-04129-4

    Article  Google Scholar 

  21. Chai, X., Shao, F., Jiang, Q., Ying, H.: Tccl-net: transformer-convolution collaborative learning network for omnidirectional image super-resolution. Knowl.-Based Syst. 274, 110625 (2023). https://doi.org/10.1016/j.knosys.2023.110625

    Article  Google Scholar 

  22. Gou, Y., Hu, P., Lv, J., Zhu, H., Peng, X.: Rethinking image super resolution from long-tailed distribution learning perspective. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14327–14336 (2023)

  23. Zhang, Y., Chen, H., Chen, X., Deng, Y., Xu, C., Wang, Y.: Data-free knowledge distillation for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7852–7861 (2021). https://doi.org/10.1109/CVPR46437.2021.00776

  24. Oh, J., Kim, H., Nah, S., Hong, C., Choi, J., Lee, K.M.: Attentive fine-grained structured sparsity for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17673–17682 (2022). https://doi.org/10.48550/arXiv.2204.12266

  25. Zhang, M., Wu, Q., Zhang, J., Gao, X., Guo, J., Tao, D.: Fluid micelle network for image super-resolution reconstruction. IEEE Trans. Cybern. 53(1), 578–591 (2022). https://doi.org/10.1109/TCYB.2022.3163294

    Article  Google Scholar 

  26. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018). https://doi.org/10.48550/arXiv.1709.01507

  27. Yan, Y., Ren, W., Hu, X., Li, K., Shen, H., Cao, X.: Srgat: Single image super-resolution with graph attention network. IEEE Trans. Image Process. 30, 4905–4918 (2021)

    Article  Google Scholar 

  28. He, Z., Chen, D., Cao, Y., Yang, J., Cao, Y., Li, X., Tang, S., Zhuang, Y., Lu, Z.-M.: Single image super-resolution based on progressive fusion of orientation-aware features. Pattern Recogn. 133, 109038 (2023). https://doi.org/10.1016/j.patcog.2022.109038

    Article  Google Scholar 

  29. Zhou, L., Cai, H., Gu, J., Li, Z., Liu, Y., Chen, X., Qiao, Y., Dong, C.: Efficient image super-resolution using vast-receptive-field attention. In: European Conference on Computer Vision, pp. 256–272 (2022). Springer. https://doi.org/10.48550/arXiv.2210.05960

  30. Zhang, X., Zeng, H., Guo, S., Zhang, L.: Efficient long-range attention network for image super-resolution. In: European Conference on Computer Vision, pp. 649–667 (2022). Springer. https://doi.org/10.48550/arXiv.2203.06697

  31. Wu, H., Gui, J., Zhang, J., Kwok, J.T., Wei, Z.: Feedback pyramid attention networks for single image super-resolution. IEEE Trans. Circuits Syst. Video Technol. (2023). https://doi.org/10.48550/arXiv.2106.06966

  32. Du, Z., Liu, D., Liu, J., Tang, J., Wu, G., Fu, L.: Fast and memory-efficient network towards efficient image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 853–862 (2022). https://doi.org/10.48550/arXiv.2204.08397

  33. Zhu, Q., Li, P., Li, Q.: Attention retractable frequency fusion transformer for image super resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1756–1763 (2023)

  34. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021).https://doi.org/10.48550/arXiv.2103.14030

  35. Chen, X., Wang, X., Zhou, J., Qiao, Y., Dong, C.: Activating more pixels in image super-resolution transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22367–22377 (2023). https://doi.org/10.48550/arXiv.2205.04437

  36. Valsesia, D., Fracastoro, G., Magli, E.: Deep graph-convolutional image denoising. IEEE Trans. Image Process. 29, 8226–8237 (2020)

    Article  MathSciNet  Google Scholar 

  37. Xu, B., Yin, H.: Graph convolutional networks in feature space for image deblurring and super-resolution. In: 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2021). IEEE. https://doi.org/10.1109/IJCNN52387.2021.9534213

  38. Zhao, Y., Deng, F., Pei, J., Yang, X.: Progressive deep non-negative matrix factorization architecture with graph convolution-based basis image reorganization. Pattern Recogn. 132, 108984 (2022). https://doi.org/10.1016/j.patcog.2022.108984

    Article  Google Scholar 

  39. Ma, Z., Jiang, Z., Zhang, H.: Hyperspectral image classification using feature fusion hypergraph convolution neural network. IEEE Trans. Geosci. Remote Sens. 60, 1–14 (2021). https://doi.org/10.1109/TGRS.2021.3123423

    Article  Google Scholar 

  40. Lim, J., Yun, S., Park, S., Choi, J.Y.: Hypergraph-induced semantic tuplet loss for deep metric learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 212–222 (2022). https://doi.org/10.1109/CVPR52688.2022.00031

  41. Wang, J., Zhang, Y., Wang, L., Hu, Y., Piao, X., Yin, B.: Multitask hypergraph convolutional networks: a heterogeneous traffic prediction framework. IEEE Trans. Intell. Transp. Syst. 23(10), 18557–18567 (2022). https://doi.org/10.1109/TITS.2022.3168879

    Article  Google Scholar 

  42. Feng, Y., You, H., Zhang, Z., Ji, R., Gao, Y.: Hypergraph neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 3558–3565 (2019). https://doi.org/10.48550/arXiv.1809.09401

  43. Bai, S., Zhang, F., Torr, P.H.: Hypergraph convolution and hypergraph attention. Pattern Recogn. 110, 107637 (2021)

    Article  Google Scholar 

  44. Jiang, J., Wei, Y., Feng, Y., Cao, J., Gao, Y.: Dynamic hypergraph neural networks. In: IJCAI, pp. 2635–2641 (2019)

  45. Gao, Y., Feng, Y., Ji, S., Ji, R.: Hgnn+: general hypergraph neural networks. IEEE Trans. Pattern Anal. Mach. Intell. 45(3), 3181–3199 (2022). https://doi.org/10.1109/tpami.2022.3182052

    Article  Google Scholar 

  46. Liu, H., Xiong, R., Song, Q., Wu, F., Gao, W.: Image super-resolution based on adaptive joint distribution modeling. In: 2017 IEEE Visual Communications and Image Processing (VCIP), pp. 1–4 (2017). IEEE. https://doi.org/10.1109/VCIP.2017.8305142

  47. Mei, Y., Fan, Y., Zhou, Y., Huang, L., Huang, T.S., Shi, H.: Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5690–5699 (2020). https://doi.org/10.48550/arXiv.2006.01424

  48. Li, X., Dong, J., Tang, J., Pan, J.: Dlgsanet: Lightweight dynamic local and global self-attention networks for image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12792–12801 (2023). https://doi.org/10.48550/arXiv.2301.02031

  49. Hu, Y., Li, J., Huang, Y., Gao, X.: Image super-resolution with self-similarity prior guided network and sample-discriminating learning. IEEE Trans. Circuits Syst. Video Technol. 32(4), 1966–1985 (2021). https://doi.org/10.1109/TCSVT.2021.3093483

    Article  Google Scholar 

  50. Li, X., Yang, Y., Zhao, Q., Shen, T., Lin, Z., Liu, H.: Spatial pyramid based graph reasoning for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8950–8959 (2020). https://doi.org/10.1109/CVPR42600.2020.00897

  51. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

  52. Sajjadi, M.S., Scholkopf, B., Hirsch, M.: Enhancenet: Single image super-resolution through automated texture synthesis. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4491–4500 (2017). https://doi.org/10.48550/arXiv.1612.07919

  53. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017). https://doi.org/10.48550/arXiv.1609.04802

  54. Zontak, M., Irani, M.: Internal statistics of a single natural image. In: CVPR 2011, pp. 977–984 (2011). IEEE. https://doi.org/10.1109/CVPR.2011.5995401

  55. Yadati, N., Nimishakavi, M., Yadav, P., Nitin, V., Louis, A., Talukdar, P.: Hypergcn: A new method for training graph convolutional networks on hypergraphs. Adv. Neural Inf. Process. Syst. (2019). https://doi.org/10.48550/arXiv.1809.02589

  56. Li, Z., Yang, J., Liu, Z., Yang, X., Jeon, G., Wu, W.: Feedback network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3867–3876 (2019). https://doi.org/10.48550/arXiv.1903.09814

  57. Xu, X., Wang, Z., Shi, H.: Ultrasr: Spatial encoding is a missing key for implicit image function-based arbitrary-scale super-resolution. arXiv preprint arXiv:2103.12716 (2021). https://doi.org/10.48550/arXiv.2103.12716

  58. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.-H., Zhang, L.: Ntire 2017 challenge on single image super-resolution: Methods and results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 114–125 (2017). https://doi.org/10.1109/CVPRW.2017.149

  59. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012). https://doi.org/10.5244/C.26.135

  60. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Curves and Surfaces: 7th International Conference, Avignon, France, June 24–30, 2010, Revised Selected Papers 7, pp. 711–730 (2012). Springer

  61. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 2, pp. 416–423 (2001). IEEE. https://doi.org/10.1109/ICCV.2001.937655

  62. Huang, J.-B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015). https://doi.org/10.1109/CVPR.2015.7299156

  63. Matsui, Y., Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., Aizawa, K.: Sketch-based manga retrieval using manga109 dataset. Multimedia Tools Appl. 76, 21811–21838 (2017). https://doi.org/10.1007/s11042-016-4020-z

    Article  Google Scholar 

  64. Peleg, T., Elad, M.: A statistical prediction model based on sparse representations for single image super-resolution. IEEE Trans. Image Process. 23(6), 2569–2582 (2014). https://doi.org/10.1109/TIP.2014.2305844

    Article  MathSciNet  Google Scholar 

  65. Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep cnn denoiser prior for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3929–3938 (2017). https://doi.org/10.48550/arXiv.1704.03264

  66. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3262–3271 (2018). https://doi.org/10.1109/CVPR.2018.00344

  67. He, H., Du, Z., Li, W., Tang, J., Wu, G.: Pyramid fusion attention network for single image super-resolution. In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2165–2169 (2022). IEEE. https://doi.org/10.1109/ICASSP43922.2022.9747609

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang, 550025, Guizhou, China

    Bufan Wang, Yongjun Zhang, Weihao Gao & He Yao

  2. North China Institute of Science and Technology, Langfang, Hebei, China

    Ruzhong Chen

Authors
  1. Bufan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongjun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weihao Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. He Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ruzhong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Bufan Wang: Conceptualization, Methodology, Software, Validation, Data curation.Yongjun Zhang: Conceptualization, Resources, Supervision, Project administration.Weihao Gao: Writing—review and editing.He Yao: Writing—review and editing.Ruzhong Cheng: Writing—review and editing.

Corresponding author

Correspondence to Yongjun Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Communicated by Y. Zhang.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, B., Zhang, Y., Gao, W. et al. Exploiting local detail in single image super-resolution via hypergraph convolution. Multimedia Systems 30, 157 (2024). https://doi.org/10.1007/s00530-024-01355-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00530-024-01355-3

Keywords

Search

Navigation