Skip to main content
SpringerLink
Log in

Optical flow for video super-resolution: a survey

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Video super-resolution is currently one of the most active research topics in computer vision as it plays an important role in many visual applications. Generally, video super-resolution contains a significant component, i.e., motion compensation, which is used to estimate the displacement between successive video frames for temporal alignment. Optical flow, which can supply dense and sub-pixel motion between consecutive frames, is among the most common ways for this task. To obtain a good understanding of the effect that optical flow acts in video super-resolution, in this work, we conduct a comprehensive review on this subject for the first time. This investigation covers the following major topics: the function of super-resolution (i.e., why we require super-resolution); the concept of video super-resolution (i.e., what is video super-resolution); the description of evaluation metrics (i.e., how (video) super-resolution performs); the introduction of optical flow based video super-resolution; the investigation of using optical flow to capture temporal dependency for video super-resolution. Prominently, we give an in-depth study of the deep learning based video super-resolution method, where some representative algorithms are analyzed and compared. Additionally, we highlight some promising research directions and open issues that should be further addressed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28

Similar content being viewed by others

References

  • Anwar S, Khan S, Barnes N (2020) A deep journey into super-resolution: a survey. ACM Comput Surv 53(3):1–34

    Article  Google Scholar 

  • Babacan SD, Molina R, Katsaggelos AK (2011) Variational Bayesian super resolution. IEEE Trans Image Process 20(4):984–999

    Article  MathSciNet  MATH  Google Scholar 

  • Baker S, Kanade T (1999) Super-resolution optical flow. Carnegie Mellon University, Pittsburgh, pp 36–99

    Google Scholar 

  • Baker S, Schar D, Lewis J, Roth S, Black M, Szeliski R (2011) A database and evaluation methodology for optical flow. Int J Comput Vision 92(1):1–31

    Article  Google Scholar 

  • Bao W, Lai W, Zhang X, Gao Z, Yang M (2019) MEMC-Net: motion estimation and motion compensation driven neural network for video interpolation and enhancement. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2019.2941941

    Article  Google Scholar 

  • Bian W, Ding S, Xue Y (2017) Fingerprint image super resolution using sparse representation with ridge pattern prior by classification coupled dictionaries. IET Biometr 6(5):342–350

    Article  Google Scholar 

  • Blau Y, Mechrez R, Timofte R, Michaeli T, Zelnik-Manor L (2018) The 2018 pirm challenge on perceptual image super-resolution. In: Proceedings ECCVW, pp 334–355

  • Borsoi RA, Costa GH, Bermudez JCM (2019) A new adaptive video super-resolution algorithm with improved robustness to innovations. IEEE Trans Image Process 28(2):673–686

    Article  MathSciNet  MATH  Google Scholar 

  • Caballero J, Ledig C, Aitken A, Acosta A, Totz J, Wang Z, Shi W (2017) Real-time video super-resolution with spatio-temporal networks and motion compensation. In: Proceedings CVPR, pp 78–4787

  • Cai D, Chen K, Qian Y, Kamarainen J (2019) Convolutional low-resolution fine-grained classification. Pattern Recogn Lett 119:166–171

    Article  Google Scholar 

  • Chan K, Wang X, Yu K, Dong C, Loy C (2021) BasicVSR: the search for essential components in video super-resolution and beyond. In: Proceedings CVPR, pp 4947–4956

  • Chen Y, Tai Y, Liu X, Shen C, Yang J (2018) FSRNet: end-to-end learning face super-resolution with facial priors. In: Proceedings CVPR, pp 2492–2501

  • Choi K, Kim C, Kang M, Ra J (2011) Resolution improvement of infrared images using visible image information. IEEE Trans Image Process 18(10):611–614

    Google Scholar 

  • Cruz C, Mehta R, Katkovnik V, Egiazarian KO (2018) Single image super-resolution based on Wiener filter in similarity domain. IEEE Trans Image Process 27(3):1376–1389

    Article  MathSciNet  MATH  Google Scholar 

  • Dai Q, Yoo S, Kappeler A, Katsaggelos AK (2017) Sparse representation-based multiple frame video super-resolution. IEEE Trans Image Process 26(2):765–781

    Article  MathSciNet  MATH  Google Scholar 

  • Daithankar MV, Ruikar SD (2020) Video super resolution: a review. In: Proceedings first international conference on data science, machine learning and applications, vol 601, pp 488–495

  • Dong C, Chen CL, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: Proceedings ECCV, pp 184–199

  • Dosovitskiy A, Fischer P, Ilg E, Husser P, Hazirbas C, Golkov V, Smagt P, Cremers D, Brox T (2015) FlowNet: learning optical flow with convolutional networks. In: Proceedings ICCV, pp 2758–2766

  • Drulea M, Nedevschi S (2011) Total variation regularization of local global optical flow. In: Proceedings ITSC, pp 318–323

  • Eekeren AWM, Schutte K, Vliet LJ (2010) Multiframe super-resolution reconstruction of small moving objects. IEEE Trans Image Process 19(11):2901–2912

    Article  MathSciNet  MATH  Google Scholar 

  • Farsiu S, Robinson MD, Elad M, Milanfar P (2004) Fast and robust multiframe super resolution. IEEE Trans Image Process 13(10):1327–1344

    Article  Google Scholar 

  • Fookes C, Lin F, Chandran V, Sridharan S (2004) Super-Resolved face images using robust optical flow. In: IEEE workshop on the internet, telecommunications and signal processing, pp 391–396

  • Girshick R, Donahue J, Darrell T, Malik J (2016) Region-based convolutional networks for accurate object detection and segmentation. IEEE Trans Pattern Anal Mach Intell 38(1):142–158

    Article  Google Scholar 

  • Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings international conference on artificial intelligence and statistics, pp 249–256

  • Han T, Kim D, Lee S, Song B (2018) Resolution improvement of infrared images using visible image information. J Vis Commun Image Represent 51:191–200

    Article  Google Scholar 

  • Haris M, Shakhnarovich G, Ukita N (2018) Deep back-projection networks for super-resolution. In: Proceedings CVPR, pp 1664–1673

  • Haris M, Shakhnarovich G, Ukita N (2019) Recurrent back-projection network for video super-resolution. In: Proceedings CVPR, pp 3897–3906

  • He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings CVPR, pp 770–778

  • Hong C, Yu J, Wan J, Tao D, Wang M (2015) Multimodal deep autoencoder for human pose recovery. IEEE Trans Image Process 24(12):5659–5670

    Article  MathSciNet  MATH  Google Scholar 

  • Hore A, Ziou D (2010) Image quality metrics: PSNR vs. SSIM. In: Proceedings international conference on pattern recognition, pp 2358-2369

  • Hou L, Yu C, Samaras D (2016) Squared Earth mover’s distance-based loss for training deep neural networks, pp 1–9. arXiv:1611.05916

  • Huang J, Ma L, Tan T, Wang Y (2003) Learning based resolution enhancement of iris images. Proc BMVC 1:1–10

    Google Scholar 

  • Huang Y, Wang W, Wang L (2015) Bidirectional recurrent convolutional networks for multi-frame super-resolution. In: Proceedings NIPS, pp 235–243

  • Huang Y, Shao L, Frangi A (2017) Simultaneous super-resolution and cross-modality synthesis of 3D medical images using weakly-supervised joint convolutional sparse coding. In: Proceedings CVPR, pp 6070–6079

  • Huang Y, Wang W, Wang L (2018) Video super-resolution via bidirectional recurrent convolutional networks. IEEE Trans Pattern Anal Mach Intell 40(4):1015–1028

    Article  Google Scholar 

  • Huang Y, Lu Z, Shao Z, Ran M, Zhou J, Fang L, Zhang Y (2019) Simultaneous denoising and super-resolution of optical coherence tomography images based on generative adversarial network. Opt Express 27(9):12289–12307

    Article  Google Scholar 

  • Ilg E, Mayer N, Saikia T, Keuper M, Dosovitskiy A, Brox T (2017) Flownet 2.0: evolution of optical flow estimation with deep networks. In: Proceedings CVPR, pp 2462–2470

  • Isobe T, Li S, Jia X, Yuan S, Slabaugh G, Xu C, Li Y, Wang S, Tian Q (2020) Video super-resolution with temporal group attention. In: Proceedings CVPR, pp 8008–8017

  • Jing X, Zhu X, Wu F, Hu R, You X, Wang Y, Feng H, Yang J (2017) On Bayesian adaptive video super resolution. IEEE Trans Image Process 26(3):1363–1378

    Article  MathSciNet  Google Scholar 

  • Jo Y, Oh S, Kang J, Kim S (2018) Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In: Proceedings CVPR, pp 3224–3232

  • Kalarot R, Porikli F (2019) MultiBoot VSR: multistage multi-reference bootstrapping for video superresolution. In: Proceedings CVPRW, pp 2060–2069

  • Kappeler A, Yoo S, Dai Q, Katsaggelos A (2016) Video super-resolution with convolutional neural networks. IEEE Trans Comput Imaging 2(2):109–122

    Article  MathSciNet  Google Scholar 

  • Keller S, Lauze F, Nielsen M (2011) Video super-resolution using simultaneous motion and intensity calculations. IEEE Trans Image Process 20(7):1870–1884

    Article  MathSciNet  MATH  Google Scholar 

  • Khrulkov V, Babenko A (2021) Neural side-by-side: predicting human preferences for no-reference super-resolution evaluation. In: Proceedings CVPR, pp 4988–4997

  • Kim K, Kwon Y (2010) Single-image super-resolution using sparse regression and natural image prior. IEEE Trans Pattern Anal Mach Intell 32(6):1127–1133

    Article  Google Scholar 

  • Kim J, Lee S (2017) Deep learning of human visual sensitivity in image quality assessment framework. In: Proceedings CVPR, pp 1969–1977

  • Kim T, Sajjadi M, Hirsch M, Scholkopf B (2018) Spatio-temporal transformer network for video restoration. In: Proceedings ECCV, pp 111–127

  • Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, Shi W (2017) Photorealistic single image super-resolution using a generative adversarial network. In: Proceedings CVPR, pp 4681–4690

  • Lei S, Shi Z, Zou Z (2020) Coupled adversarial training for remote sensing image super-resolution. IEEE Trans Geosci Remote Sens 58(5):3633–3643

    Article  Google Scholar 

  • Li X, Hu Y, Gao X, Tao D, Ning B (2010) A multi-frame image super-resolution method. Signal Process 90:405–414

    Article  MATH  Google Scholar 

  • Li K, Zhu Y, Yang J, Jiang J (2016) Video super-resolution using an adaptive superpixel-guided auto-regressive model. Pattern Recogn 51:59–71

    Article  Google Scholar 

  • Li D, Liu Y, Wang Z (2019) Video super-resolution using non-simultaneous fully recurrent convolutional network. IEEE Trans Image Process 28(3):1342–1355

    Article  MathSciNet  Google Scholar 

  • Li F, Bai H, Zhao Y (2020) Learning a deep dual attention network for video super-resolution. IEEE Trans Image Process 29:4474–4488

    Article  MATH  Google Scholar 

  • Liao R, Tao X, Li R, Ma Z, Jia J (2015) Video super-resolution via deep draft-ensemble learning. In: Proceedings ICCV, pp 531–539

  • Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: Proceedings CVPRW, pp 1132–1140

  • Lin K, Wang G (2018) Hallucinated-IQA: no-reference image quality assessment via adversarial learning. In: Proceedings CVPR, pp 732–741

  • Lin F, Fookes C, Chandran V, Sridharan S (2005) Investigation into optical flow super-resolution for surveillance applications. In: Proceedings APRS workshop on digital image computing, pp 73–78

  • Liu C, Sun D (2014) On Bayesian adaptive video super resolution. IEEE Trans Pattern Anal Mach Intell 36(2):346–360

    Article  Google Scholar 

  • Liu D, Wang Z, Fan Y, Liu X, Wang Z, Chang S, Huang T (2017) Robust video super-resolution with learned temporal dynamics. In: Proceedings ICCV, pp 2507–2515

  • Liu D, Wang Z, Fan Y, Liu X, Wang Z, Chang S, Wang X, Huang T (2018) Learning temporal dynamics for video super-resolution: a deep learning approach. IEEE Trans Image Process 27(7):3432–3445

    Article  MathSciNet  MATH  Google Scholar 

  • Liu H, Ruan Z, Zhao P, Shang F, Yang L, Liu Y (2020) Video super resolution based on deep learning: a comprehensive survey. arXiv:2007.12928

  • Lucas A, Lopez-Tapia S, Molina R, Katsaggelos AK (2019) Generative adversarial networks and perceptual losses for video super-resolution. IEEE Trans Image Process 28(7):3312–3327

    Article  MathSciNet  MATH  Google Scholar 

  • Ma Z, Liao R, Tao X, Xu L, Jia J, Wu E (2015) Handling motion blur in multi-frame super-resolution. In: Proceedings CVPR, pp 5224–5232

  • Ma C, Yang C, Yang X, Yang M (2017) Learning a no-reference quality metric for single-image super-resolution. Comput Vis Image Underst 158:1–16

    Article  Google Scholar 

  • Ma C, Jiang Z, Rao Y, Lu J, Zhou J (2020) Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation. In: Proceedings CVPR, pp 5569–5578

  • Meur OL, Ebdelli M, Guillemot C (2013) Hierarchical super-resolution-based inpainting. IEEE Trans Image Process 22(10):3779–3790

    Article  MathSciNet  MATH  Google Scholar 

  • Mitzel D, Pock T, Schoenemann T, Cremers D (2009) Video super resolution using duality based TV-L1 optical flow. In: Proceedings DAGM 2009: pattern recognition, pp 432–441

  • Mudenagudi U, Banerjee S, Kalra PK (2011) Space-time super-resolution using graph-cut optimization. IEEE Trans Pattern Anal Mach Intell 33(5):995–1008

    Article  Google Scholar 

  • Mudunuri S, Biswas S (2016) Low resolution face recognition across variations in pose and illumination. IEEE Trans Pattern Anal Mach Intell 38(5):1034–1040

    Article  Google Scholar 

  • Na B, Fox GC (2017) Object detection by a super-resolution method and a convolutional neural networks. In: Proceedings IEEE international conference on big data, pp 2263–2269

  • Nasrollahi K, Moeslund TB (2014) Super-resolution: a comprehensive survey. Mach Vis Appl 25:1423–1468

    Article  Google Scholar 

  • Nguyen K, Fookes C, Sridharan S, Tistarelli M, Nixon M (2018) Super-resolution for biometrics: a comprehensive survey. Pattern Recogn 78:23–42

    Article  Google Scholar 

  • Noor DF, Li Y, Li Z, Bhattacharyya S, York G (2019) Multi-scale gradient image super-resolution for preserving SIFT key points in low-resolution images. Signal Process Image Commun 78:236–245

    Article  Google Scholar 

  • Pan J, Bai H, Dong J, Zhang J, Tang J (2021) Deep blind video super-resolution. In: Proceedings ICCV, pp 4811–4820

  • Papenberg N, Bruhn A, Brox T, Didas S, Weickert J (2006) Highly accurate optic flow computation with theoretically justified warping. Int J Comput Vis 67(2):141–158

    Article  Google Scholar 

  • Park S, Park M, Kang M (2003) Super-resolution image reconstruction: a technical overview. IEEE Signal Process Mag 20(3):21–35

    Article  Google Scholar 

  • Peleg S, Keren D, Schweitzer L (1987) Improving image resolution using subpixel motion. Pattern Recogn Lett 5(3):223–226

    Article  Google Scholar 

  • Picku LC (2007) Machine learning in multi-frame image super-resolution. Ph.D thesis, University of Oxford, Britain

  • Prashnani E, Cai H, Mostofi Y, Sen P (2018) PieAPP: perceptual image-error assessment through pairwise preference. In: Proceedings CVPR, pp 1808–1817

  • Ranjan A, Black M (2017) Optical flow estimation using a spatial pyramid network. In: Proceedings CVPR, pp 4161–4170

  • Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Proceedings MICCAI, pp 234–241

  • Ryoo MS, Rothrock B, Fleming C, Yang HJ (2017) Privacy-preserving human activity recognition from extreme low resolution. In: Proceedings AAAI, pp 4255–4262

  • Sajjadi M, Scholkopf B, Hirsch M (2017) Enhancenet: single image super-resolution through automated texture synthesis. In: Proceedings ICCV, pp 4491–4500

  • Sajjadi M, Vemulapalli R, Brown M (2018) Frame-recurrent video super-resolution. In: Proceedings CVPR, pp 6626–6634

  • Schoenemann T, Cremers D (2012) A coding-cost framework for super-resolution motion layer decomposition. IEEE Trans Image Process 21(3):1097–1110

    Article  MathSciNet  MATH  Google Scholar 

  • Shen CT, Liu HH, Yang MH, Hung YP, Pei SC (2015) Viewing-distance aware super-resolution for high-definition display. IEEE Trans Image Process 24(1):403–418

    Article  MathSciNet  MATH  Google Scholar 

  • Shi W, Caballero J, Huszar F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings CVPR, pp 1874–1883

  • Singh A, Singh J (2020) Survey on single image based super-resolution-implementation challenges and solutions. Multimed Tools Appl 79(3):1641–1672

    Article  Google Scholar 

  • Su H, Wu Y, Zhou J (2012) Super-resolution without dense flow. IEEE Trans Image Process 21(4):1782–999

    Article  MathSciNet  MATH  Google Scholar 

  • Sun K, Xiao B, Liu D, Wang J (2019) Deep high-resolution representation learning for human pose estimation. In: Proceedings CVPR, pp 5693–5703

  • Sun D, Roth S, Black M (2010) Secrets of optical flow estimation and their principles. In: Proceedings CVPR, pp 2432–2439

  • Talebi H, Milanfar P (2018) NIMA: neural image assessment. IEEE Trans Image Process 27(8):3998–4011

    Article  MathSciNet  MATH  Google Scholar 

  • Tang L, Sun K, Liu L, Wang G, Liu Y (2019) A reduced-reference quality assessment metric for super-resolution reconstructed images with information gain and texture similarity. Signal Process Image Commun 79:32–39

    Article  Google Scholar 

  • Tan W, Yan B, Bare B (2018) Feature super-resolution: make machine see more clearly. In: Proceedings CVPR, pp 3994–4002

  • Tao X, Gao H, Liao R, Wang J, Jia J (2017) Detail-revealing deep video super-resolution. In: Proceedings ICCV, pp 4472–4480

  • Tatem AJ, Lewis HG, Atkinson PM, Nixon MS (2001) Super-resolution target identification from remotely sensed images using a Hopfield neural network. IEEE Trans Geosci Remote Sens 39(4):781–796

    Article  Google Scholar 

  • Thapa D, Raahemifar K, Bobier W, Lakshminarayanan V (2016) A performance comparison among different super-resolution techniques. Comput Electr Eng 54:313–329

    Article  Google Scholar 

  • Timofte R, Gu S, Wu J, Van Gool L et al (2018) Ntire 2018 challenge on single image super-resolution: methods and results. In: Proceedings CVPRW, pp 965–976

  • Tsai R, Huang TS (1984) Multiframe image restoration and registration. In: Advances in computer vision and image processing. pp 317–339

  • Tu Z, Aa Nico, Gemeren C, Veltkamp R (2014) A combined post-filtering method to improve accuracy of variational optical flow estimation. Pattern Recogn 47(5):1926–1940

    Article  Google Scholar 

  • Tu Z, Xie W, Zhang D, Poppe R, Veltkamp R, Li B, Yuan J (2019) A survey of variational and CNN-based optical flow techniques. Signal Process Image Commun 72:9–24

    Article  Google Scholar 

  • Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  • Wang Z, Yi P, Jiang K, Jiang J, Han Z, Lu T, Ma J (2019) Multi-memory convolutional neural network for video super-resolution. IEEE Trans Image Process 28(5):2530–2544

    Article  MathSciNet  Google Scholar 

  • Wang L, Li D, Zhu Y, Tian L, Shan Y (2020a) Dual super-resolution learning for semantic segmentation. In: Proceedings CVPR, pp 3774–3783

  • Wang L, Guo Y, Liu L, Lin Z, Deng X, An W (2020b) Deep video super-resolution using HR optical flow estimation. IEEE Trans Image Process 29:4323–4336

    Article  MATH  Google Scholar 

  • Wang Z, Chen J, Hoi S (2021a) Deep learning for image super-resolution: a survey. IEEE Trans Pattern Anal Mach Intell 43(10):3365–3387

    Article  Google Scholar 

  • Wang W, Zhang H, Yuan Z, Wang C (2021b) Unsupervised real-world super-resolution: a domain adaptation perspective. In: Proceedings ICCV, pp 4318–4327

  • Xiang X, Tian Y, Zhang Y, Fu Y, Allebach J, Xu C (2020) Zooming Slow-Mo: fast and accurate one-stage space-time video super-resolution. In: Proceedings CVPR, pp 3370–3379

  • Xiong Z, Sun X, Wu F (2010) Robust web image/video super-resolution. IEEE Trans Image Process 19(8):2017–2028

    Article  MathSciNet  MATH  Google Scholar 

  • Xue T, Chen B, Wu J, Wei D, Freeman W (2019) Video enhancement with task-oriented flow. Int J Comput Vis 127(8):1106–1125

    Article  Google Scholar 

  • Xu G, Xu J, Li Z, Wang L, Sun X, Cheng M (2021) Temporal modulation network for controllable space-time video super-resolution. In: Proceedings CVPR, pp 6388–6397

  • Yang W, Feng J, Xie G, Liu J, Guo Z, Yan S (2018) Video super-resolution based on spatial-temporal recurrent residual networks. Comput Vis Image Underst 168:79–92

    Article  Google Scholar 

  • Yang W, Zhang X, Tian Y, Wang W, Xue JH, Liao Q (2019) Deep learning for single image superresolution: a brief review. IEEE Trans Multimed 21(12):3106–3121

    Article  Google Scholar 

  • Yang X, Xiang W, Zeng H, Zhang L (2021) Real-world video super-resolution: a benchmark dataset and a decomposition based learning scheme. In: Proceedings ICCV, pp 4781–4790

  • Yi P, Wang Z, Jiang K, Jiang J, Lu T, Tian X, Ma J (2021) Omniscient video super-resolution. In: Proceedings ICCV, pp 4429–4438

  • You C, Li G, Zhang Y, Zhang X, Shan H, Li M, Ju S, Zhao Z, Zhang Z, Cong W, Vannier M, Saha P, Hoffman E, Wang G (2020) CT super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (GAN-CIRCLE). IEEE Trans Med Imaging 39(1):188–203

    Article  Google Scholar 

  • Yuan Z, Wu J, Kamata S, Ahrary A, Yan P (2009) Fingerprint image enhancement by super resolution with early stopping. Proc ICIS 4:527–531

    Google Scholar 

  • Yuan Q, Zhang L, Shen H, Li P (2010) Adaptive multiple-frame image super-resolution based on U-curve. IEEE Trans Image Process 19(12):3157–3170

    Article  MathSciNet  MATH  Google Scholar 

  • Zhang K, Zuo W, Zhang L (2018a) Learning a single convolutional super-resolution network for multiple degradations. In: Proceedings CVPR, pp 3262–3271

  • Zhang R, Isola P, Efros AA, Shechtman E, Wang O (2018b) The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings CVPR, pp 586–595

  • Zhang H, Liu D, Xiong Z (2019) Two-stream action recognition-oriented video super-resolution. In: Proceedings ICCV, pp 8799–8808

  • Zhao W, Sawhney H (2002) Is super-resolution with optical flow feasible? In: Proceedings ECCV, pp 599–613

  • Zhao H, Qi X, Shen X, Shi J, Jia J (2018) ICNet for real-time semantic segmentation on high-resolution images. In: Proceedings ECCV, pp 418–434

  • Zhu H, Li L, Wu J, Dong W, Shi G (2020) MetaIQA: deep meta-learning for no-reference image quality assessment. In: Proceedings CVPR, pp 14143–14152

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 62106177. It was also supported by the Central University Basic Research Fund of China (Nos. 2042020KF0016, CCNU20TS028), the Teaching research project of CCNU (202013), and the Wuhan University-Infinova project Nos. 2019010019. The numerical calculation was supported by the supercomputing system in the Super-computing Center of Wuhan University.

Author information

Authors and Affiliations

  1. State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China

    Zhigang Tu, Yuanzhong Liu & Shifu Zhang

  2. School of Information Engineering, Hubei University of Economics, Wuhan, China

    Hongyan Li & Shifu Zhang

  3. School of Computer, Central China Normal University, Wuhan, China

    Wei Xie & Shifu Zhang

  4. School of Computing, Informatics, Decision System Engineering, Arizona State University, Tempe, USA

    Shifu Zhang & Baoxin Li

  5. Computer Science and Engineering Department, State University of New York at Buffalo, Buffalo, USA

    Shifu Zhang & Junsong Yuan

Authors
  1. Zhigang Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongyan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuanzhong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shifu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Baoxin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Junsong Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhigang Tu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tu, Z., Li, H., Xie, W. et al. Optical flow for video super-resolution: a survey. Artif Intell Rev 55, 6505–6546 (2022). https://doi.org/10.1007/s10462-022-10159-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-022-10159-8

Keywords

Search

Navigation