Skip to main content
SpringerLink
Log in

Image super-resolution reconstruction based on instance spatial feature modulation and feedback mechanism

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Image super-resolution reconstruction is a research hotspot in the field of computer vision. Traditional image super-resolution reconstruction methods based on deep learning mostly up-sample low-resolution images ignoring categories and instances, which will cause some problems such as unrealistic texture in the reconstructed images or sawtooth phenomenon on the edge of instance. In this manuscript, we propose an image super-resolution reconstruction method based on instance spatial feature modulation and feedback mechanism. First, the prior knowledge of instance spatial features is introduced in the reconstruction process. Instance spatial features of low-resolution images are extracted to modulate super-resolution reconstruction features. Then, based on the feedback mechanism, the modulated low-resolution image features are iteratively optimized for the reconstruction results, so that the model can finally learn instance-level reconstruction ability. Experiments on COCO-2017 show that, compared with traditional deep learning-based image super-resolution reconstruction methods, the proposed method can obtain better image reconstruction results, and the reconstructed images have more realistic instance textures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The data that support the findings of this study are available on request from the corresponding author Hanxu Jiang.

References

  1. Tong T, Li G, Liu X et al (2017) Image super-resolution using dense skip connections[C]// 2017 IEEE international conference on computer vision (ICCV), Venice, pp. 4799-4807

  2. Lim B, Son S, Kim H et al (2017) Enhanced deep residual networks for single image super-resolution[C]// 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), Honolulu, pp. 1132-1140

  3. Haris M, Shakhnarovich G, Ukita N (2018) Deep back-projection networks for super-resolution[C]// 2018 IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, pp. 3606–3616

  4. Wang X, Yu K, Dong C et al (2018) Recovering realistic texture in image super-resolution by deep spatial feature transform[C]// 2018 IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, pp. 606–615

  5. Zhen L, Jing L Y, Zheng L et al (2019) Feedback network for image super-resolution[C]// 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Long Beach, pp 3867–3876

  6. Tai Y W, Liu S, Brown M S et al (2010) Super-resolution using edge prior and single image detail synthesis[C]// IEEE conference on computer vision and pattern recognition(CVPR), San Francisco, pp. 2400-2407

  7. Timofte R, De S V, Van G L. (2013) Anchored neighborhood regression for fast example-based super-resolution[C]// IEEE international conference on computer vision (ICCV), Sydney, pp. 1920-1927

  8. Timofte R, De SV, Van GL (2014) A+: adjusted anchored neighborhood regression for fast super-resolution[C]// Asian conference on computer vision(ACCV), Singapore, pp. 111-126

  9. Huang JB, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars[C]// IEEE conference on computer vision and pattern recognition (CVPR), Boston, pp. 5197-5206

  10. Peleg T, Elad M (2014) A statistical prediction model based on sparse representations for single image super-resolution[J]. IEEE Trans Image Process 23(6):2569–2582

    Article  MathSciNet  MATH  Google Scholar 

  11. Schulter S, Leistner C, Bischof H. (2015) Fast and accurate image upscaling with super-resolution forests[C]// IEEE conference on computer vision and pattern recognition (CVPR), Boston, pp 3791–3799

  12. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks[J]. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  13. Dong C, Loy C C, He K et al (2014) Learning a deep convolutional network for image super-resolution [C]// European conference on computer vision (ECCV), Zurich, pp 184–199

  14. Dong C, Loy C C, Tang X (2016) Accelerating the super-resolution convolutional neural network[C]// European conference on computer vision (ECCV), Amsterdam, pp 391–407

  15. Shi W, Caballero J, Huszár F et al (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]// IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, pp 1874–1883

  16. Ledig C, Theis L, Huszar F et al (2017) Photo-realistic single image super-resolution using a generative adversarial network[C]// IEEE conference on computer vision and pattern recognition(CVPR), Honolulu, pp 105–114

  17. Yu J, Fan Y, Yang J et al (2018) Wide activation for efficient and accurate image super-resolution[C]// IEEE conference on computer vision and pattern recognition(CVPR), Salt Lake City, pp 2621–2624

  18. Hui Z, Li J, Gao X et al (2021) Progressive perception-oriented network for single image super-resolution[J]. Inf Sci 546:769–786

    Article  MathSciNet  Google Scholar 

  19. Du X (2022) Single image super-resolution using global enhanced upscale network[J]. Appl Intell 52:2813–2819

    Article  Google Scholar 

  20. Wang Z, Lu Y, Li W et al (2021) Single image super-resolution with attention-based densely connected module[J]. Neurocomputing 453:876–884

    Article  Google Scholar 

  21. Zhang Y, Tian Y, Kong Y et al (2018) Residual dense network for image super-resolution[C]// IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, pp. 2472–2481

  22. Kim J, Lee J K, Lee K M (2016) Deeply-recursive convolutional network for image super-resolution[C]// IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, pp. 1637–1645

  23. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network[C]// IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, pp. 3147–3155

  24. Zhang Y, Li K, Li K et al (2018) Image super-resolution using very deep residual channel attention networks[C]// European conference on computer vision (ECCV), Munich, pp. 286–301

  25. Wang X, Yu K, Wu S et al (2018) ESRGAN: enhanced super-resolution generative adversarial networks[C]// European conference on computer vision (ECCV), Munich, pp. 63-79

  26. Lin TY, Maire M, Belongie S et al (2014) Microsoft COCO: common objects in context[C]// European conference on computer vision (ECCV), Zurich, pp. 740–755

  27. Tai Y, Yang J, Liu X et al (2017) MemNet: a persistent memory network for image restoration[C]// IEEE international conference on computer vision (ICCV), Venice, pp. 4539-4547

  28. Niu B, Wen W, Ren W et al (2020) Single image super-resolution via a holistic attention network[C]// European conference on computer vision(ECCV), Cham, pp 191–207

  29. Ying T, Jian Y, Liu X (2017) Image super-resolution via deep recursive residual network[C]// IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, pp. 3147–3155

  30. Woo S, Park J, Lee JY et al (2018) CBAM: convolutional block attention module[C]// European conference on computer vision (ECCV), Munich, pp 3–19

  31. Wang L, Wang Y, Lin Z et al (2021) Learning a single network for scale-arbitrary super-resolution[C]// Proceedings of the IEEE/CVF international conference on computer vision (ICCV), Montreal, pp. 4801-4810

  32. Kong X, Zhao H, Qiao Y et al (2021) ClassSR: a general framework to accelerate super-resolution networks by data characteristic[C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 12016-12025

Download references

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Beijing University of Technology, Beijing, China

    Lihua Fu, Hanxu Jiang, Huixian Wu, Shaoxing Yan, Junxiang Wang & Dan Wang

Authors
  1. Lihua Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hanxu Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huixian Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shaoxing Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Junxiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hanxu Jiang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fu, L., Jiang, H., Wu, H. et al. Image super-resolution reconstruction based on instance spatial feature modulation and feedback mechanism. Appl Intell 53, 601–615 (2023). https://doi.org/10.1007/s10489-022-03625-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03625-x

Keywords

Search

Navigation