Skip to main content
Log in

Real-World Video Deblurring: A Benchmark Dataset and an Efficient Recurrent Neural Network

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Real-world video deblurring in real time still remains a challenging task due to the complexity of spatially and temporally varying blur itself and the requirement of low computational cost. To improve the network efficiency, we adopt residual dense blocks into RNN cells, so as to efficiently extract the spatial features of the current frame. Furthermore, a global spatio-temporal attention module is proposed to fuse the effective hierarchical features from past and future frames to help better deblur the current frame. Another issue that needs to be addressed urgently is the lack of a real-world benchmark dataset. Thus, we contribute a novel dataset (BSD) to the community, by collecting paired blurry/sharp video clips using a co-axis beam splitter acquisition system. Experimental results show that the proposed method (ESTRNN) can achieve better deblurring performance both quantitatively and qualitatively with less computational cost against state-of-the-art video deblurring methods. In addition, cross-validation experiments between datasets illustrate the high generality of BSD over the synthetic datasets. The code and dataset are released at https://github.com/zzh-tech/ESTRNN.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  • Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv:1409.0473

  • Bar, L., Berkels, B., Rumpf, M., Sapiro, G. (2007). A variational framework for simultaneous motion estimation and restoration of Motion-Blurred video. In 2007 IEEE 11th International Conference on Computer Vision, IEEE, pp. 1–8.

  • Cao, M., Zhong, Z., Fan, Y., Wang, J., Zhang, Y., Wang, J., Yang, Y., & Zheng, Y. (2022). Towards real-world video deblurring by exploring blur formation process. arXiv:2208.13184

  • Cao, M., Zhong, Z., Wang, J., Zheng, Y., & Yang, Y. (2022). Learning adaptive warping for real-world rolling shutter correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17785–17793.

  • Chakrabarti, A. (2016). A neural approach to blind motion deblurring. In European conference on computer vision (pp. 221–235). Springer.

  • Cho, S., Wang, J., & Lee, S. (2012). Video deblurring for hand-held cameras using patch-based synthesis. ACM Transactions on Graphics (TOG), 31(4), 1–9.

    Article  Google Scholar 

  • Dumoulin, V., & Visin, F. (2016). A guide to convolution arithmetic for deep learning. ArXiv e-prints

  • Goldstein, A., & Fattal, R. (2012). Blur-kernel estimation from spectral irregularities. In European Conference on Computer Vision (pp. 622–635). Springer.

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.

  • Hore, A., Ziou, D. (2010). Image quality metrics: PSNR vs. SSIM. In 2010 20th International Conference on Pattern Recognition, IEEE, pp. 2366–2369.

  • Hu, J., Shen, L., & Sun, G. (2018) Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141.

  • Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K.Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708.

  • Hui, T.W., Tang, X., & Loy, C.C. (2018). Liteflownet: A lightweight convolutional neural network for optical flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8981–8989.

  • Jin, M., Meishvili, G., & Favaro, P. (2018). Learning to extract a video sequence from a single Motion-Blurred image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6334–6342.

  • Kim, T.H., & Lee, K.M. (2014). Segmentation-free dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2766–2773.

  • Kim, T.H., Lee, K.M., Scholkopf, B., & Hirsch, M. (2017). Online video deblurring via dynamic temporal blending network. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4038–4047.

  • Kim, T. H., Nah, S., & Lee, K. M. (2018). Dynamic video deblurring using a locally adaptive blur model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10), 2374–2387. https://doi.org/10.1109/TPAMI.2017.2761348

    Article  Google Scholar 

  • Kingma, D.P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv:1412.6980

  • Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., & Matas, J. (2018). Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8183–8192.

  • Lee, H.S., Kwon, J., & Lee, K.M. (2011). Simultaneous localization, mapping and deblurring. In 2011 International Conference on Computer Vision, IEEE, pp. 1203–1210.

  • Lee, H.S., & Lee, K.M. (2013). Dense 3d reconstruction from severely blurred images using a single moving camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 273–280.

  • Lei, T., Zhang, Y., Wang, S.I., Dai, H., & Artzi, Y. (2017). Simple recurrent units for highly parallelizable recurrence. arXiv:1709.02755

  • Lin, M., Chen, Q., & Yan, S. (2013). Network in network. arXiv:1312.4400

  • Michaeli, T., & Irani, M. (2014). Blind deblurring using internal patch recurrence. In European Conference on Computer Vision (pp. 783–798). Springer.

  • Nah, S., Baik, S., Hong, S., Moon, G., Son, S., Timofte, R., Lee, K.M. (2019a). Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 0–0.

  • Nah, S., Kim, T.H., & Lee, K.M. (2017). Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3883–3891.

  • Nah, S., Son, S., & Lee, K.M. (2019b). Recurrent neural networks with intra-frame iterations for video deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8102–8111.

  • Nair, V., & Hinton, G.E. (2010). Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807–814.

  • Niklaus, S., Mai, L., & Liu, F. (2017). Video frame interpolation via adaptive separable convolution. In Proceedings of the IEEE International Conference on Computer Vision, pp. 261–270.

  • Nimisha, T.M., Kumar Singh, A., & Rajagopalan, A.N. (2017). Blur-invariant deep learning for blind-deblurring. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4752–4760.

  • Pan, J., Bai, H., & Tang, J. (2020). Cascaded deep video deblurring using temporal sharpness prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3043–3051.

  • Ren, W., Pan, J., Cao, X., & Yang, M.H. (2017) Video deblurring via semantic segmentation and pixel-wise non-linear kernel. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1077–1085.

  • Rim, J., Lee, H., Won, J., & Cho, S. (2020) Real-world blur dataset for learning and benchmarking deblurring algorithms. In European Conference on Computer Vision (pp. 184–201). Springer.

  • Schuler, C.J., Christopher Burger, H., Harmeling, S., & Scholkopf, B. (2013). A machine learning approach for non-blind image deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1067–1074.

  • Shen, W., Bao, W., Zhai, G., Chen, L., Min, X., & Gao, Z. (2020). Blurry video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5114–5123.

  • Son, H., Lee, J., Lee, J., Cho, S., & Lee, S. (2021). Recurrent video deblurring with blur-invariant motion estimation and pixel volumes. ACM Transactions on Graphics (TOG), 40(5), 1–18.

    Article  Google Scholar 

  • Su, S., Delbracio, M., Wang, J., Sapiro, G., Heidrich, W., & Wang, O. (2017). Deep video deblurring for hand-held cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1279–1288.

  • Sun, D., Yang, X., Liu, M.Y., & Kautz, J. (2018). Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8934–8943.

  • Sun, J., Cao, W., Xu, Z., & Ponce, J. (2015). Learning a convolutional neural network for non-uniform motion blur removal. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 769–777.

  • Sun, L., Cho, S., Wang, J., & Hays, J. (2014). Good image priors for non-blind deconvolution. In European Conference on Computer Vision (pp. 231–246). Springer.

  • Tao, X., Gao, H., Shen, X., Wang, J., Jia, J. (2018). Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8174–8182.

  • Wang, X., Chan, K.C., Yu, K., Dong, C., & Change Loy, C. (2019). Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 0–0.

  • Wang, Y., Lu, Y., Gao, Y., Wang, L., Zhong, Z., Zheng, Y., & Yamashita, A. (2022). Efficient video deblurring guided by motion magnitude. arXiv:2207.13374

  • Wieschollek, P., Hirsch, M., Scholkopf, B., & Lensch, H. (2017). Learning blind motion deblurring. In Proceedings of the IEEE International Conference on Computer Vision, pp. 231–240.

  • Wu, Y., Ling, H., Yu, J., Li, F., Mei, X., & Cheng, E. (2011). Blurred target tracking by blur-driven tracker. In 2011 International Conference on Computer Vision, IEEE, pp. 1100–1107.

  • Wulff, J., & Black, M.J. (2014). Modeling blurred video with layers. In European Conference on Computer Vision (pp. 236–252). Springer.

  • Xu, L., & Jia, J. (2010). Two-phase kernel estimation for robust motion deblurring. In European conference on computer vision (pp. 157–170). Springer.

  • Xu, L., Ren, J. S., Liu, C., & Jia, J. (2014). Deep convolutional neural network for image deconvolution. Advances in Neural Information Processing Systems, 27, 1790–1798.

    Google Scholar 

  • Yu, Z., & Liu, G. (2018). Sliced recurrent neural networks. arXiv:1807.02291

  • Zhang, X., Chen, Q., Ng, R., Koltun, V. (2019). Zoom to learn, learn to zoom. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3762–3770.

  • Zhang, Y., Tian, Y., Kong, Y., Zhong, B., & Fu, Y. (2018). Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481.

  • Zhang, Y., Tian, Y., Kong, Y., Zhong, B., & Fu, Y. (2020). Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2020.2968521

    Article  Google Scholar 

  • Zhong, Z., Gao, Y., Zheng, Y., & Zheng, B. (2020). Efficient spatio-temporal recurrent neural network for video deblurring. In European Conference on Computer Vision (pp. 191–207). Springer.

  • Zhong, Z., Sun, X., Wu, Z., Zheng, Y., Lin, S., & Sato, I. (2022). Animation from blur: Multi-modal blur decomposition with motion guidance. arXiv:2207.10123

  • Zhong, Z., Zheng, Y., & Sato, I. (2021). Towards rolling shutter correction and deblurring in dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9219–9228.

  • Zhou, S., Zhang, J., Pan, J., Xie, H., Zuo, W., & Ren, J. (2019). Spatio-temporal filter adaptive network for video deblurring. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2482–2491.

  • Zhu, X., Hu, H., Lin, S., & Dai, J. (2019). Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9308–9316.

  • Zoran, D., & Weiss, Y. (2011). From learning models of natural image patches to whole image restoration. In 2011 International Conference on Computer Vision, IEEE, pp. 479–486.

Download references

Acknowledgements

This work was supported in part by JSPS KAKENHI Grant Numbers JP22H00529, JP20H05951 and JP20H05953.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yinqiang Zheng.

Additional information

Communicated by Jiaya Jia.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhong, Z., Gao, Y., Zheng, Y. et al. Real-World Video Deblurring: A Benchmark Dataset and an Efficient Recurrent Neural Network. Int J Comput Vis 131, 284–301 (2023). https://doi.org/10.1007/s11263-022-01705-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-022-01705-6

Keywords

Navigation