Skip to main content
SpringerLink
Log in

Multi-frame spatio-temporal super-resolution

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Increasing the resolution of digital images and videos using digital super-resolution (SR) techniques has been of great interest in industry and academia over the past three decades. Most SR methods target improving only the spatial resolution of images and videos, whereas improving the temporal resolution could be more critical for some videos. Motion blur is a temporal artifact by nature, so removing it using spatial SR techniques would be highly challenging and often unsuccessful. This paper proposes a multi-frame motion-based video super-resolution method to increase both spatial and temporal resolutions of a single input video. Our optimization problem is based on a maximum a posteriori estimator that estimates each high-resolution (HR) frame by fusing multiple low-resolution frames. The form of the image prior used in the optimization framework is based on the assumption that natural HR frames are piecewise smooth. We introduce a new method to enhance the sharpness of edges in the video frames during the optimization process. We also involve a temporal constraint that improves temporal consistency in the estimated video. Moreover, we propose a new scheme for motion estimation that better suits video frame rate upsampling. Our results are compared with state-of-the-art SR methods, including ML-based ones, which confirm the effectiveness of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

No dataset is used by this research.

Notes

  1. \(\left\lfloor \cdot \right\rfloor\) is the floor operator.

  2. A few first and last frames of the video may have less number of adjacent frames. Also, for real-time applications, \(b\) should be set to zero.

  3. Generative Adversarial Networks.

References

  1. Park, S.C., Park, M.K., Kang, M.G.: Super-resolution image reconstruction: a technical overview. IEEE Signal Process. Mag. Inst. Electr. Electron. Eng. Inc 20(3), 21–36 (2003). https://doi.org/10.1109/MSP.2003.1203207

    Article  Google Scholar 

  2. Tian, J., Ma, K.K.: A survey on super-resolution imaging. Signal Image Video Process 5(3), 329–342 (2011). https://doi.org/10.1007/s11760-010-0204-6

    Article  Google Scholar 

  3. Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L.: Image super-resolution: the techniques, applications, and future. Signal Process. 128, 389–408 (2016). https://doi.org/10.1016/j.sigpro.2016.05.002

    Article  Google Scholar 

  4. Liu, C., Sun, D.: A Bayesian approach to adaptive video super resolution. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 209–216. (2011). https://doi.org/10.1109/CVPR.2011.5995614

  5. Faramarzi, E., Rajan, D., Fernandes, F.C.A., Christensen, M.P.: Blind super resolution of real-life video sequences. IEEE Trans. Image Process. 25(4), 1544–1555 (2016). https://doi.org/10.1109/TIP.2016.2523344

    Article  MathSciNet  MATH  Google Scholar 

  6. Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Gr. Appl. 22(2), 56–65 (2002). https://doi.org/10.1109/38.988747

    Article  Google Scholar 

  7. Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25, 1423–1468 (2014). https://doi.org/10.1007/s00138-014-0623-4

    Article  Google Scholar 

  8. Singh, A., Singh, J.: Survey on single image based super-resolution—implementation challenges and solutions. Multimed. Tools Appl. 79(3), 1641–1672 (2019). https://doi.org/10.1007/s11042-019-08254-0

    Article  MathSciNet  Google Scholar 

  9. Shahar, O., Faktor, A., Irani, M.: Space-time super-resolution from a single video. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3353–3360, (2011). https://doi.org/10.1109/CVPR.2011.5995360

  10. Son, S. et al.: NTIRE 2021 challenge on video super-resolution. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 166–181, (2021). https://doi.org/10.1109/CVPRW53098.2021.00026

  11. Xiang, X., Tian, Y., Zhang, Y., Fu, Y., Allebach, J.P., Xu, C.: Zooming slow-mo: fast and accurate one-stage space-time video super-resolution. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society, pp. 3367–3376, (2020). https://doi.org/10.1109/CVPR42600.2020.00343

  12. Xue, T., Chen, B., Wu, J., Wei, D., Freeman, W.T.: Video enhancement with task-oriented flow. Int. J. Comput. Vis. 127(8), 1106–1125 (2017). https://doi.org/10.1007/s11263-018-01144-2

    Article  Google Scholar 

  13. Kappeler, A., Yoo, S., Dai, Q., Katsaggelos, A.K.: Video super-resolution with convolutional neural networks. IEEE Trans. Comput. Imaging 2(2), 109–122 (2016). https://doi.org/10.1109/TCI.2016.2532323

    Article  MathSciNet  Google Scholar 

  14. Caballero, J. et al.: Real-time video super-resolution with spatio-temporal networks and motion compensation. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, vol. 2017-January, pp. 2848–2857, (2016), https://doi.org/10.1109/CVPR.2017.304

  15. Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1664–1673, (2018). https://doi.org/10.1109/CVPR.2018.00179

  16. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. (2018), Accessed: May 14, 2023. [Online]. Available: https://arxiv.org/abs/1802.08797v2

  17. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Yun, F.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pp. 294–310. Springer International Publishing, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_18

    Chapter  Google Scholar 

  18. Tian, Y., Zhang, Y., Fu, Y., Xu, C.: TDAN: temporally deformable alignment network for video super-resolution. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3357–3366, (2018). https://doi.org/10.1109/CVPR42600.2020.00342

  19. Wang, Z., Chen, J., Hoi, S.C.H.: Deep learning for image super-resolution: a survey. IEEE Trans. Pattern. Anal. Mach. Intell. (2020). https://doi.org/10.1109/tpami.2020.2982166

    Article  Google Scholar 

  20. Jo, Y., Oh, W.S., Kang, J., Kim, S.J.: Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3224–3232, (2018). https://doi.org/10.1109/CVPR.2018.00340

  21. Isobe, T., et al.: Video super-resolution with temporal group attention. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 8005–8014, (2020). https://doi.org/10.1109/CVPR42600.2020.00803

  22. Kawulok, M., Benecki, P., Piechaczek, S., Hrynczenko, K., Kostrzewa, D., Nalepa, J.: Deep learning for multiple-image super-resolution. IEEE Geosci. Remote Sens. Lett. 17(6), 1062–1066 (2020). https://doi.org/10.1109/LGRS.2019.2940483

    Article  Google Scholar 

  23. Salvetti, F., Mazzia, V., Khaliq, A., Chiaberge, M.: Multi-image super resolution of remotely sensed images using residual attention deep neural networks. Remote Sens. 12(14), 2207 (2020). https://doi.org/10.3390/rs12142207

    Article  Google Scholar 

  24. Molini, A.B., Valsesia, D., Fracastoro, G., Magli, E.: DeepSUM: deep neural network for super-resolution of unregistered multitemporal images. IEEE Trans. Geosci. Remote Sens. 58(5), 3644–3656 (2019). https://doi.org/10.1109/TGRS.2019.2959248

    Article  Google Scholar 

  25. Shechtman, E., Caspi, Y., Irani, M.: Space-time super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. (2005). https://doi.org/10.1109/TPAMI.2005.85

    Article  MATH  Google Scholar 

  26. Faramarzi, E., Rajan, D., Christensen, M.P.: Space-time super-resolution from multiple-videos. In: 2012 11th International Conference on Information Science, Signal Processing and their Applications, ISSPA 2012, (2012). https://doi.org/10.1109/ISSPA.2012.6310553

  27. Mudenagudi, U., Banerjee, S., Kalra, P.K.: Space-time super-resolution using graph-cut optimization. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 995–1008 (2011). https://doi.org/10.1109/TPAMI.2010.167

    Article  Google Scholar 

  28. Caspi, Y., Irani, M.: Spatio-temporal alignment of sequences. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1409–1424 (2002). https://doi.org/10.1109/TPAMI.2002.1046148

    Article  Google Scholar 

  29. Takeda, H., Milanfar, P., Protter, M., Elad, M.: Super-resolution without explicit subpixel motion estimation. IEEE Trans. Image Process. 18(9), 1958–1975 (2009). https://doi.org/10.1109/TIP.2009.2023703

    Article  MathSciNet  MATH  Google Scholar 

  30. Faramarzi, E., Rajan, D., Christensen, M.P.: Space-time super-resolution from multiple-videos. In: The 11th International Conference on Information Sciences, Signal Processing and their Applications, pp. 23–28, (2012)

  31. Gonzalez, R.C., Woods, R.E.: Digital image processing, 4th edn. Pearson, London (2018)

    Google Scholar 

  32. Burrus, C.S.: Iterative reweighted least squares. OpenStax CNX module: m45285, (2014)

  33. Shewchuk, J.R.: An introduction to the conjugate gradient method without the agonizing Pain, Edition 1 ¼. In: School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213, (1994)

  34. Tao, X., Gao, H., Liao, R., Wang, J., Jia, J.: Detail-revealing Deep Video Super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision, vol. 2017-October, pp. 4482–4490, (2017). https://doi.org/10.1109/ICCV.2017.479

  35. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  36. Takeda, H., Milanfar, P.: Removing motion blur with space-time processing. IEEE Trans. Image Process. 20(10), 2990–3000 (2011). https://doi.org/10.1109/TIP.2011.2131666

    Article  MathSciNet  MATH  Google Scholar 

  37. Cho, S., Lee, S.: Fast motion deblurring, p. 1. ACM Transactions on Graphics, ACM Press, New York (2009)

    Google Scholar 

  38. Deblurring–imageupscaler. https://imageupscaler.com/deblurring/. Accessed 01 Jan 2021

Download references

Funding

There is no funding or grant supporting this manuscript.

Author information

Authors and Affiliations

  1. Department of Operations, and Supply Chain Management, Califonia State University San Marcos, San Marcos, CA, USA

    Zahra Gharibi

  2. San Marcos, CA, USA

    Sam Faramarzi

Authors
  1. Zahra Gharibi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sam Faramarzi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Both authors contributed to all parts of the manuscripts.

Corresponding author

Correspondence to Zahra Gharibi.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest associated with this study.

Ethical approval

This research is not sponsored by any industrial company and does not aim to endorse any commercial product.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (MP4 925 KB)

Supplementary file2 (MP4 1444 KB)

Supplementary file3 (MP4 62 KB)

Supplementary file4 (MP4 681 KB)

Supplementary file5 (MP4 2712 KB)

Supplementary file6 (MP4 13960 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gharibi, Z., Faramarzi, S. Multi-frame spatio-temporal super-resolution. SIViP 17, 4415–4424 (2023). https://doi.org/10.1007/s11760-023-02675-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02675-z

Keywords

Search

Navigation