Skip to main content
SpringerLink
Log in

Efficient lightweight network for video super-resolution

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Recently, video super-resolution has achieved an outstanding performance. However, many existing methods to solve video super-resolution usually make use of complex strategies, such as explicit optical flow, deformable convolution, which increase complexity and computation. In this paper, we propose a lightweight network for video super-resolution, namely Efficient Lightweight Network for Video Super-Resolution (ELNVSR). We design a Multi-group Block extracting long-distance spatial information to construct a lightweight Bidirection Alignment Module which is implicitly capable of fusing and propagating spatial-temporal information in a bidirectional way. Meanwhile, a Multi-scale Pyramid Block is built as a lightweight reconstruction module to extract different levels of information layer by layer. Comprehensive experiments are conducted on public benchmarks. The results demonstrate a promising performance with fewer parameters.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data Availibility

The data are available from the corresponding author on reasonable request.

References

  1. Wang Z, Chen J, Hoi SC (2020) Deep learning for image super-resolution: a survey. IEEE Trans Pattern Anal Mach Intell 43(10):3365–3387

    Article  Google Scholar 

  2. Sun L, Liu Z, Sun X, Liu L, Lan R, Luo X (2021) Lightweight image super-resolution via weighted multi-scale residual network. IEEE/CAA J Autom Sin 8(7):1271–1280

    Article  Google Scholar 

  3. Ma Q, Jiang J, Liu X, Ma J (2022) Deep unfolding network for spatiospectral image super-resolution. IEEE Trans Comput Imaging 8:28–40

    Article  MathSciNet  Google Scholar 

  4. Liu H, Ruan Z, Zhao P, Dong C, Shang F, Liu Y, Yang L (2020) Video super resolution based on deep learning: a comprehensive survey. arXiv preprint arXiv:2007.12928

  5. Belekos SP, Galatsanos NP, Katsaggelos AK (2010) Maximum a posteriori video super-resolution using a new multichannel image prior. IEEE Trans Image Process 19(6):1451–1464

    Article  MathSciNet  Google Scholar 

  6. Farsiu S, Robinson MD, Elad M, Milanfar P (2004) Fast and robust multiframe super resolution. IEEE Trans Image Process 13(10):1327–1344

    Article  Google Scholar 

  7. Liao R, Tao X, Li R, Ma Z, Jia J (2015) Video super-resolution via deep draft-ensemble learning. In: Proceedings of the IEEE international conference on computer vision, pp 531–539

  8. Liu C, Sun D (2013) On Bayesian adaptive video super resolution. IEEE Trans Pattern Anal Mach Intell 36(2):346–360

    Article  Google Scholar 

  9. Zhang D, Yao L, Chen K, Wang S, Chang X, Liu Y (2019) Making sense of spatio-temporal preserving representations for EEG-based human intention recognition. IEEE Trans Cybern 50(7):3033–3044

    Article  Google Scholar 

  10. Luo M, Chang X, Nie L, Yang Y, Hauptmann AG, Zheng Q (2017) An adaptive semisupervised feature analysis for video semantic recognition. IEEE Trans Cybern 48(2):648–660

    Article  Google Scholar 

  11. Chen K, Yao L, Zhang D, Wang X, Chang X, Nie F (2019) A semisupervised recurrent convolutional attention model for human activity recognition. IEEE Trans Neural Netw Learn Syst 31(5):1747–1756

    Article  Google Scholar 

  12. Lucas A, Lopez-Tapia S, Molina R, Katsaggelos AK (2019) Generative adversarial networks and perceptual losses for video super-resolution. IEEE Trans Image Process 28(7):3312–3327

    Article  MathSciNet  Google Scholar 

  13. Kim SY, Lim J, Na T, Kim M (2019) Video super-resolution based on 3D-CNNS with consideration of scene change. In: 2019 IEEE international conference on image processing. IEEE, pp 2831–2835

  14. Huang Y, Wang W, Wang L (2015) Bidirectional recurrent convolutional networks for multi-frame super-resolution. In: Proceedings of the 28th international conference on neural information processing systems, vol 1, pp 235–243

  15. Huang Y, Wang W, Wang L (2017) Video super-resolution via bidirectional recurrent convolutional networks. IEEE Trans Pattern Anal Mach Intell 40(4):1015–1028

    Article  Google Scholar 

  16. Yi P, Wang Z, Jiang K, Jiang J, Lu T, Ma J (2022) A progressive fusion generative adversarial network for realistic and consistent video super-resolution. IEEE Trans Pattern Anal Mach Intell 44(5):2264–2280

    Google Scholar 

  17. Girshick X, Wang R, Gupta A, He K (2018) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794–7803

  18. Dosovitskiy A, Fischer P, Ilg E, Hausser P, Hazirbas C, Golkov V, Van Der Smagt P, Cremers D, Brox T (2015) FlowNet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 2758–2766

  19. Ranjan A, Black MJ (2017) Optical flow estimation using a spatial pyramid network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4161–4170

  20. Dai J, Qi H, Xiong Y, Li Y, Zhang G, H, H, Wei Y (2017) Deformable convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 764–773

  21. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30:6000–6010

    Google Scholar 

  22. Cao J, Li Y, Zhang K, Liang J, Van Gool L (2021) Video super-resolution transformer. arXiv preprint arXiv:2106.06847

  23. Liang J, Cao J, Fan Y, Zhang K, Ranjan R, Li Y, Timofte R, Van Gool L (2022) VRT: a video restoration transformer. arXiv preprint arXiv:2201.12288

  24. Krizhevsky A, Sutskever I, Hinton GE (2017) ImageNet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  25. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861

  26. Yu F, Koltun V (2015) Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122

  27. Dong C, Loy CC, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: European conference on computer vision. Springer, pp 184–199

  28. Kappeler A, Yoo S, Dai Q, Katsaggelos AK (2016) Video super-resolution with convolutional neural networks. IEEE Trans Comput Imaging 2(2):109–122

    Article  MathSciNet  Google Scholar 

  29. Shi W, Caballero J, Huszár F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1874–1883

  30. Caballero J, Ledig, Aitken A, Acosta A, Totz J, Wang Z, Shi W (2017) Real-time video super-resolution with spatio-temporal networks and motion compensation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4778–4787

  31. Wei L, Guo Y, Lin Z, Deng X, An W (2018) Learning for video super-resolution through HR optical flow estimation. In: Asian conference on computer vision. Springer, pp 514–529

  32. Tian Y, Zhang Y, Fu Y, Xu C (2020) Tdan: Temporally-deformable alignment network for video super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3360–3369

  33. Wang X, Chan, KC, Yu K, Dong C, Change Loy C (2019) EDVR: video restoration with enhanced deformable convolutional networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 1954–1963

  34. Jo Y, Oh SW, Kang J, Kim SJ (2018) Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3224–3232

  35. Guo J, Chao H (2017) Building an end-to-end spatial-temporal convolutional network for video super-resolution. In: Proceedings of the thirty-first AAAI conference on artificial intelligence, pp 4053–4060

  36. Li W, Tao X, Guo T, Qi L, Lu J, Jia J (2020) MuCAN: multi-correspondence aggregation network for video super-resolution. In: European conference on computer vision. Springer, pp 335–351

  37. Chan KCK, Wang X, Yu K, Dong C, Loy CC (2021) BasicVSR: the search for essential components in video super-resolution and beyond. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4947–4956

  38. Chan KC, Zhou S, Xu,X, Loy CC (2021) BasicVSR++: improving video super-resolution with enhanced propagation and alignment. arXiv preprint arXiv:2104.13371

  39. Zhang X, Zhou X, Lin M, Sun J (2018) ShuffleNet: an extremely efficient convolutional neural network for mobile devices. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 6848–6856

  40. Zeng Y, Xiao Z, Hung K-W, Lui S (2021) Real-time video super resolution network using recurrent multi-branch dilated convolutions. Signal Process Image Commun 93:116167

    Article  Google Scholar 

  41. Wang Z, Yi P, Jiang K, Jiang J, Han Z, Lu T, Ma J (2018) Multi-memory convolutional neural network for video super-resolution. IEEE Trans Image Process 28(5):2530–2544

    Article  MathSciNet  Google Scholar 

  42. Nah S, Baik S, Hong S, Moon G, Son S, Timofte R, Lee KM (2019) NTIRE 2019 challenge on video deblurring and super-resolution: dataset and study. In: 2019 IEEE/CVF conference on computer vision and pattern recognition workshops. IEEE, pp 1996–2005

  43. Tao X, Gao H, Liao R, Wang J, Jia, J (2017) Detail-revealing deep video super-resolution. In: Proceedings of the IEEE international conference on computer vision, pp 4472–4480

  44. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

Download references

Funding

Funding was provided by National Natural Science Foundation of China (Grant nos. 62071339, U1903214), Natural Science Foundation of Hubei Province (Grant no. 2021CFB464).

Author information

Authors and Affiliations

  1. The National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, Wuhan, 430072, China

    Zhongyuan Wang, Peng Yi & Zheng He

  2. The Electronics Information School, Wuhan University, Wuhan, 430072, China

    Laigan Luo & Benshun Yi

Authors
  1. Laigan Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Benshun Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhongyuan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Peng Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zheng He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Benshun Yi or Zhongyuan Wang.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luo, L., Yi, B., Wang, Z. et al. Efficient lightweight network for video super-resolution. Neural Comput & Applic 36, 883–896 (2024). https://doi.org/10.1007/s00521-023-09065-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09065-z

Keywords

Search

Navigation