Skip to main content

DeMFI: Deep Joint Deblurring and Multi-frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13667))

Included in the following conference series:

Abstract

We propose a novel joint deblurring and multi-frame interpolation (DeMFI) framework in a two-stage manner, called DeMFI-Net, which converts blurry videos of lower-frame-rate to sharp videos at higher-frame-rate based on flow-guided attentive-correlation-based feature bolstering (FAC-FB) module and recursive boosting (RB), in terms of multi-frame interpolation (MFI). Its baseline version performs feature-flow-based warping with FAC-FB module to obtain a sharp-interpolated frame as well to deblur two center-input frames. Its extended version further improves the joint performance based on pixel-flow-based warping with GRU-based RB. Our FAC-FB module effectively gathers the distributed blurry pixel information over blurry input frames in feature-domain to improve the joint performances. RB trained with recursive boosting loss enables DeMFI-Net to adequately select smaller RB iterations for a faster runtime during inference, even after the training is finished. As a result, our DeMFI-Net achieves state-of-the-art (SOTA) performances for diverse datasets with significant margins compared to recent joint methods. All source codes, including pretrained DeMFI-Net, are publicly available at https://github.com/JihyongOh/DeMFI.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bahat, Y., Efrat, N., Irani, M.: Non-uniform blind deblurring by reblurring. In: ICCV, pp. 3286–3294 (2017)

    Google Scholar 

  2. Bao, W., Lai, W.S., Ma, C., Zhang, X., Gao, Z., Yang, M.H.: Depth-aware video frame interpolation. In: CVPR, pp. 3703–3712 (2019)

    Google Scholar 

  3. Bao, W., Lai, W.S., Zhang, X., Gao, Z., Yang, M.H.: MEMC-Net: motion estimation and motion compensation driven neural network for video interpolation and enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019)

    Google Scholar 

  4. Chi, Z., Wang, Y., Yu, Y., Tang, J.: Test-time fast adaptation for dynamic scene deblurring via meta-auxiliary learning. In: CVPR, pp. 9137–9146 (2021)

    Google Scholar 

  5. Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: EMNLP (2014)

    Google Scholar 

  6. Choi, M., Kim, H., Han, B., Xu, N., Lee, K.M.: Channel attention is all you need for video frame interpolation. In: AAAI, pp. 10663–10671 (2020)

    Google Scholar 

  7. Chu, M., You, X., Jonas, M., Laura, L.T., Nils, T.: Learning temporal coherence via self-supervision for GAN-based video generation. ACM ToG 39(4), 1–75 (2020)

    Article  Google Scholar 

  8. Dai, J., et al.: Deformable convolutional networks. In: CVPR, pp. 764–773 (2017)

    Google Scholar 

  9. Dutta, S., Shah, N.A., Mittal, A.: Efficient space-time video super resolution using low-resolution flow and mask upsampling. In: CVPR, pp. 314–323 (2021)

    Google Scholar 

  10. Gao, H., Tao, X., Shen, X., Jia, J.: Dynamic scene deblurring with parameter selective sharing and nested skip connections. In: CVPR, pp. 3848–3856 (2019)

    Google Scholar 

  11. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS, pp. 249–256 (2010)

    Google Scholar 

  12. Gui, S., Wang, C., Chen, Q., Tao, D.: Featureflow: robust video interpolation via structure-to-texture generation. In: CVPR, pp. 14004–14013 (2020)

    Google Scholar 

  13. Gupta, A., Aich, A., Roy-Chowdhury, A.K.: Alanet: adaptive latent attention network for joint video deblurring and interpolation. In: ACMMM, pp. 256–264 (2020)

    Google Scholar 

  14. Gupta, A., Joshi, N., Lawrence Zitnick, C., Cohen, M., Curless, B.: Single image deblurring using motion density functions. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 171–184. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15549-9_13

    Chapter  Google Scholar 

  15. Haris, M., Shakhnarovich, G., Ukita, N.: Space-time-aware multi-resolution video enhancement. In: CVPR, pp. 2859–2868 (2020)

    Google Scholar 

  16. Harmeling, S., Michael, H., Schölkopf, B.: Space-variant single-image blind deconvolution for removing camera shake. NeurIPS 23, 829–837 (2010)

    Google Scholar 

  17. Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. In: NeurIPS, pp. 2017–2025 (2015)

    Google Scholar 

  18. Jiang, H., Sun, D., Jampani, V., Yang, M.H., Learned-Miller, E., Kautz, J.: Super slomo: high quality estimation of multiple intermediate frames for video interpolation. In: CVPR, pp. 9000–9008 (2018)

    Google Scholar 

  19. Jin, M., Hu, Z., Favaro, P.: Learning to extract flawless slow motion from blurry videos. In: CVPR, pp. 8112–8121 (2019)

    Google Scholar 

  20. Jin, M., Meishvili, G., Favaro, P.: Learning to extract a video sequence from a single motion-blurred image. In: CVPR (2018)

    Google Scholar 

  21. Kang, J., Jo, Y., Oh, S.W., Vajda, P., Kim, S.J.: Deep space-time video upsampling networks. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12355, pp. 701–717. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58607-2_41

    Chapter  Google Scholar 

  22. Kim, S.Y., Oh, J., Kim, M.: FISR: deep joint frame interpolation and super-resolution with a multi-scale temporal loss. In: AAAI, pp. 11278–11286 (2020)

    Google Scholar 

  23. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)

    Google Scholar 

  24. Kuroki, Y., Nishi, T., Kobayashi, S., Oyaizu, H., Yoshimura, S.: A psychophysical study of improvements in motion-image quality by using high frame rates. J. Soc. Inform. Display 15(1), 61–68 (2007)

    Article  Google Scholar 

  25. Kuroki, Y., Takahashi, H., Kusakabe, M., Yamakoshi, K.i.: Effects of motion image stimuli with normal and high frame rates on EEG power spectra: comparison with continuous motion image stimuli. J. Soc. Inf. Display 22(4), 191–198 (2014)

    Google Scholar 

  26. Lee, H., Kim, T., Chung, T.y., Pak, D., Ban, Y., Lee, S.: AdaCoF: adaptive collaboration of flows for video frame interpolation. In: CVPR, pp. 5316–5325 (2020)

    Google Scholar 

  27. Liu, Y., Xie, L., Siyao, L., Sun, W., Qiao, Yu., Dong, C.: Enhanced quadratic video interpolation. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12538, pp. 41–56. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-66823-5_3

    Chapter  Google Scholar 

  28. Liu, Z., Yeh, R.A., Tang, X., Liu, Y., Agarwala, A.: Video frame synthesis using deep voxel flow. In: CVPR, pp. 4463–4471 (2017)

    Google Scholar 

  29. Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: CVPR, pp. 3883–3891 (2017)

    Google Scholar 

  30. Niklaus, S., Liu, F.: Context-aware synthesis for video frame interpolation. In: CVPR, pp. 1701–1710 (2018)

    Google Scholar 

  31. Niklaus, S., Liu, F.: Softmax splatting for video frame interpolation. In: CVPR, pp. 5437–5446 (2020)

    Google Scholar 

  32. Pan, J., Sun, D., Pfister, H., Yang, M.H.: Blind image deblurring using dark channel prior. In: CVPR, pp. 1628–1636 (2016)

    Google Scholar 

  33. Park, D., Kang, D.U., Kim, J., Chun, S.Y.: Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12351, pp. 327–343. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58539-6_20

    Chapter  Google Scholar 

  34. Park, J., Ko, K., Lee, C., Kim, C.-S.: BMBC: bilateral motion estimation with bilateral cost volume for video interpolation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12359, pp. 109–125. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58568-6_7

    Chapter  Google Scholar 

  35. Park, J., Lee, C., Kim, C.S.: Asymmetric bilateral motion estimation for video frame interpolation. In: ICCV (2021)

    Google Scholar 

  36. Peleg, T., Szekely, P., Sabo, D., Sendik, O.: IM-Net for high resolution video frame interpolation. In: CVPR, pp. 2398–2407 (2019)

    Google Scholar 

  37. Purohit, K., Rajagopalan, A.: Region-adaptive dense network for efficient motion deblurring. In: AAAI, vol. 34, pp. 11882–11889 (2020)

    Google Scholar 

  38. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  39. Shen, W., Bao, W., Zhai, G., Chen, L., Min, X., Gao, Z.: Blurry video frame interpolation. In: CVPR, pp. 5114–5123 (2020)

    Google Scholar 

  40. Shen, W., Bao, W., Zhai, G., Chen, L., Min, X., Gao, Z.: Video frame interpolation and enhancement via pyramid recurrent framework. IEEE Trans. Image Process. 30, 277–292 (2020)

    Article  MathSciNet  Google Scholar 

  41. Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.: Convolutional lstm network: a machine learning approach for precipitation nowcasting. In: NeurIPS (2015)

    Google Scholar 

  42. Sim, H., Oh, J., Kim, M.: XVFI: extreme video frame interpolation. In: ICCV (2021)

    Google Scholar 

  43. Su, S., Delbracio, M., Wang, J., Sapiro, G., Heidrich, W., Wang, O.: Deep video deblurring for hand-held cameras. In: CVPR, pp. 1279–1288 (2017)

    Google Scholar 

  44. Sun, D., Yang, X., Liu, M.Y., Kautz, J.: PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: CVPR, pp. 8934–8943 (2018)

    Google Scholar 

  45. Tao, X., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In: CVPR, pp. 8174–8182 (2018)

    Google Scholar 

  46. Teed, Z., Deng, J.: RAFT: recurrent all-pairs field transforms for optical flow. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 402–419. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_24

    Chapter  Google Scholar 

  47. Telleen, J., et al.: Synthetic shutter speed imaging. In: Computer Graphics Forum, vol. 26, pp. 591–598. Wiley Online Library (2007)

    Google Scholar 

  48. Tian, Y., Zhang, Y., Fu, Y., Xu, C.: TDAN: temporally-deformable alignment network for video super-resolution. In: CVPR, pp. 3360–3369 (2020)

    Google Scholar 

  49. Wang, X., Chan, K.C., Yu, K., Dong, C., Change Loy, C.: EDVR: video restoration with enhanced deformable convolutional networks. In: CVPRW (2019)

    Google Scholar 

  50. Wu, J., Yuen, C., Cheung, N.M., Chen, J., Chen, C.W.: Modeling and optimization of high frame rate video transmission over wireless networks. IEEE Trans. Wireless Commun. 15(4), 2713–2726 (2015)

    Article  Google Scholar 

  51. Xiang, X., Tian, Y., Zhang, Y., Fu, Y., Allebach, J.P., Xu, C.: Zooming slow-MO: fast and accurate one-stage space-time video super-resolution. In: CVPR, pp. 3370–3379 (2020)

    Google Scholar 

  52. Xiao, Z., Xiong, Z., Fu, X., Liu, D., Zha, Z.J.: Space-time video super-resolution using temporal profiles. In: ACM MM, pp. 664–672 (2020)

    Google Scholar 

  53. Xu, G., Xu, J., Li, Z., Wang, L., Sun, X., Cheng, M.M.: Temporal modulation network for controllable space-time video super-resolution. In: CVPR, pp. 6388–6397 (2021)

    Google Scholar 

  54. Xu, X., Siyao, L., Sun, W., Yin, Q., Yang, M.H.: Quadratic video interpolation. In: NeurIPS, pp. 1647–1656 (2019)

    Google Scholar 

  55. Zhang, H., Dai, Y., Li, H., Koniusz, P.: Deep stacked hierarchical multi-patch network for image deblurring. In: CVPR, pp. 5978–5986 (2019)

    Google Scholar 

  56. Zhang, K., Luo, W., Zhong, Y., Ma, L., Liu, W., Li, H.: Adversarial spatio-temporal learning for video deblurring. IEEE Trans. Image Process. 28(1), 291–301 (2018)

    Article  MathSciNet  Google Scholar 

  57. Zhang, K., Luo, W., Zhong, Y., Ma, L., Stenger, B., Liu, W., Li, H.: Deblurring by realistic blurring. In: CVPR, pp. 2737–2746 (2020)

    Google Scholar 

  58. Zhang, Y., Wang, C., Tao, D.: Video frame interpolation without temporal priors. In: NeurIPS 33 (2020)

    Google Scholar 

Download references

Acknowledgement

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017–0-00419, Intelligent High Realistic Visual Processing for Smart Broadcasting Media).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Munchurl Kim .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 4947 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Oh, J., Kim, M. (2022). DeMFI: Deep Joint Deblurring and Multi-frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13667. Springer, Cham. https://doi.org/10.1007/978-3-031-20071-7_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20071-7_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20070-0

  • Online ISBN: 978-3-031-20071-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics