Skip to main content
SpringerLink
Log in

Video super-resolution reconstruction method based on deep Back projection and motion feature fusion

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

How to effectively utilize inter-frame redundancies is the key to improve the accuracy and speed of video super-resolution reconstruction methods. Previous methods usually process every frame in the whole video in the same way, and do not make full use of redundant information between frames, resulting in low accuracy or long reconstruction time. In this paper, we propose the idea of reconstructing key frames and non-key frames respectively, and give a video super-resolution reconstruction method based on deep back projection and motion feature fusion. Key-frame reconstruction subnet can obtain key frame features and reconstruction results with high accuracy. For non-key frames, key frame features can be reused by fusing them and motion features, so as to obtain accurate non-key frame features and reconstruction results quickly. Experiments on several public datasets show that the proposed method performs better than the state-of-the-art methods, and has good robustness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Agustsson E, Timofte R (2017) NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study[C] // IEEE. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE Computer Society, Honolulu, pp 1122–1131

    Google Scholar 

  2. Caballero J, Ledig C, Aitken A et al (2017) Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation[C]. // IEEE. 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). IEEE Computer Society, Honolulu, pp 2848–2857

    Google Scholar 

  3. Chao D, Loy CC, Kaiming H et al (2014) Learning a deep convolutional network for image super-Resolution [C]// David Fleet et al. Computer Vision–ECCV 2014. Springer International Publishing, Zurich, pp 184–199

    Google Scholar 

  4. Dong C, Loy CC, Tang X (2016) Accelerating the Super-Resolution Convolutional Neural Network[C]// Bastian Leibe et al. Computer Vision–ECCV 2016. Springer International Publishing, Amsterdam, pp 391–407

    Google Scholar 

  5. Fu L-h, Sun X-B, Zhao Y et al (2019) Fast Video Super-resolution Reconstruction Method based on Motion Feature Fusion[J]. Pattern Recognit Artificial Intell 32(11):1022–1035

    Google Scholar 

  6. Haris M, Shakhnarovich G, Ukita N (2018) Deep back-projection networks for super-resolution[C]. In: IEEE. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Salt Lake City, pp 3606–3616

  7. Haris M, Shakhnarovich G, Ukita N (2019) Recurrent Back-Projection Network for Video Super-Resolution[C] // IEEE. 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Long Beach, pp 59–62

    Google Scholar 

  8. Huang Y, Wang W, Wang L (2018) Video Super-Resolution via Bidirectional Recurrent Convolutional Networks[J]. IEEETrans Pattern Anal Machine Intell 40(4):1015–1028

    Article  Google Scholar 

  9. Kappeler A, Yoo S, Dai Q et al (2016) Video Super-Resolution with Convolutional Neural Networks[J]. IEEE Trans Computation Imaging 2(2):109–122

    Article  MathSciNet  Google Scholar 

  10. Kingma D, Ba J (2014) Adam: a method for stochastic optimization[J]. arXiv preprint arXiv:1412.6980

  11. Ledig C, Theis L, Huszar F et al (2017) Photo-realistic single image super-resolution using a generative adversarial network[C]. In: IEEE. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Honolulu, pp 105–114

  12. Li X, Orchard MT (2001) New edge-directed interpolation[J]. IEEE Trans Image Process: Public IEEE Signal Process Soc 10(10):1521–1527

    Article  Google Scholar 

  13. Li Y, Shi J, Lin D (2018) Low-Latency Video Semantic Segmentation [C]. // IEEE. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Salt Lake City, pp 5997–6005

    Book  Google Scholar 

  14. Lim B, Son S, Kim H et al (2017) Enhanced deep residual networks for single image super-resolution[C]. In: IEEE. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE Computer Society, Honolulu, pp 1132–1140

  15. Liu C, Sun D (2011) A bayesian approach to adaptive video super resolution[C]. In: IEEE. 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Providence, pp 209–216

  16. Liu H, Han J, Hou S et al (2018) Single Image Super-Resolution Using a Deep Encoder-Decoder Symmetrical Network with Iterative Back Projection[J]. Neurocomputing 282:52–59

    Article  Google Scholar 

  17. Liu H, Fu Z, Han J et al (2019) Single Image Super-Resolution Using Multi-Scale Deep Encoder-Decoder with Phase Congruency Edge Map Guidance[J]. Information Sci 473:44–58

    Article  MathSciNet  MATH  Google Scholar 

  18. Perazzi F, Pont-Tuset J, Mcwilliams B et al (2016) A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation[C] // IEEE. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Comput Soc, Las Vegas, pp 724–732

    Book  Google Scholar 

  19. Shi W, Caballero J, Huszár F et al (2016) Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network [C] // IEEE. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Las Vegas, pp 1874–1883

    Google Scholar 

  20. Sun J, Sun J, Xu Z, et al (2008) Image super-resolution using gradient profile prior[C]. In: IEEE. 2008 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). IEEE Computer Society, Anchorage, pp 1–8

  21. Tai YW , Liu S , Brown MS  et al (2010) Super resolution using edge prior and single image detail synthesis[C]. In: IEEE. 2010 Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, San Francisco, pp 2400–2407

  22. Tao X, Gao H, Liao R, Wang J, Jia J (2017) Detail-revealing deep video super-resolution[C]// IEEE 2017 IEEE International Conference on Computer Vision (ICCV). IEEE Computer Society, Venice, pp 22–29

    Google Scholar 

  23. Tong T, Li G, Liu X et al (2017) Image super-resolution using dense skip connections[C]. In: IEEE. 2017 IEEE International Conference on Computer Vision (ICCV). IEEE Computer Society, Venice, pp 2380–7504

  24. Wang Z, Bovik AC, Sheikh HR et al (2004) Image Quality Assessment: From Error Visibility to Structural Similarity[J]. IEEE Trans Image Process:13, 600–14, 612

  25. Wang Z et al (2019) Multi-memory convolutional neural network for video super-resolution[J]. IEEE Trans Image Process 28(5):2530–2544

    Article  MathSciNet  Google Scholar 

  26. Yi P et al (2019) Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations. IEEE International Conference on Computer Vision (ICCV), Seoul

    Book  Google Scholar 

  27. Yi P, Wang Z, Jiang K et al (2020) Multi-temporal ultra dense memory network for video super-resolution[J]. IEEE Trans Circuits Syst Video Technol 30(8):2503–2516

    Article  Google Scholar 

  28. Yu J, Fan Y, Yang J et al (2018) Wide Activation for Efficient and Accurate Image Super-Resolution[C] // IEEE. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Salt Lake City, pp 2621–2624

    Google Scholar 

  29. Zhang L, Wu X (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion[J]. IEEE Trans Image Process 15(8):2226–2238

    Article  Google Scholar 

  30. Zheng H, Ji M, Wang H et al (2018) CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping[C] // Vittorio Ferrari et al. Computer Vision–ECCV 2018. Springer International Publishing, Munich, pp 88–104

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Beijing University of Technology, Beijing, 100124, China

    Li-hua Fu, Xiao-wei Sun, Yu Zhao, Ren-jie Chen, Hui Chen & Ru Zhao

Authors
  1. Li-hua Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiao-wei Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ren-jie Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hui Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ru Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li-hua Fu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fu, Lh., Sun, Xw., Zhao, Y. et al. Video super-resolution reconstruction method based on deep Back projection and motion feature fusion. Multimed Tools Appl 80, 11423–11441 (2021). https://doi.org/10.1007/s11042-020-10337-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10337-2

Keywords

Search

Navigation