Skip to main content
SpringerLink
Log in

RT-Deblur: real-time image deblurring for object detection

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

After a long period of research and development, 2D image object detection technology has been greatly improved in terms of efficiency and accuracy, but it is still a great challenge in dealing with motion blurred images. Most of the current deblurring algorithms are too computationally intensive to meet the demands of real-time tasks. To address this problem, a real-time deblur network (RT-Deblur) based on adversarial generative networks has been proposed. Specifically, a fast Fourier transform residual (FFTRes) block was designed to allow the model to achieve better performance while using fewer residual blocks, and to reduce the number of parameters to meet real-time requirements. In order to improve the deblurring performance and the accuracy of object detection after deblurring, a weighted loss function was developed, which includes the discriminator loss, MSE loss, multi-scale frequency reconstruction loss and perceptual YOLO loss. To validate the effectiveness of deblurring for object detection, we produced two blurred object detection datasets based on REDS and GoPro. A large number of comparative experiments on object detection before and after deblurring using YOLOv5s have been done, and the results show that our network achieves leading \(mAP\) scores. RT-Deblur improved the \(mAP\) scores from 0.486 to 0.566 and from 0.17 to 0.433 on the two blurred datasets, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Nishiyama, M., Hadid, A., Takeshima, H., Shotton, J., Kozakaya, T., Yamaguchi, O.: Facial deblur inference using subspace analysis for recognition of blurred faces. IEEE Trans. Pattern Anal. Mach. Intell. 33, 838–845 (2011)

    Article  Google Scholar 

  2. Ben-Ezra, M., Nayar, S.K.: Motion-based motion deblurring. IEEE Trans. Pattern Anal. Mach. Intell. 26, 689–698 (2004)

    Article  Google Scholar 

  3. Fortunato, H.E., Oliveira, M.M.: Fast high-quality non-blind deconvolution using sparse adaptive priors. Vis. Comput. 30, 661–671 (2014)

    Article  Google Scholar 

  4. Dong, J.X., Roth, S., Schiele, B.: DWDN: deep wiener deconvolution network for non-blind image deblurring. IEEE Trans. Pattern Anal. Mach. Intell. 44, 9960–9976 (2022)

    Article  Google Scholar 

  5. Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T.: Removing camera shake from a single photograph. Acm Trans. Graph. 25, 787–794 (2006)

    Article  Google Scholar 

  6. Kim, T.H., Lee, K.M.: IEEE: Generalized video deblurring for dynamic scenes. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5426–5434. (2015)

  7. Dong, J., Pan, J., Su, Z.: Blur kernel estimation via salient edges and low rank prior for blind image deblurring. Signal Process.-Image Commun. 58, 134–145 (2017)

    Article  Google Scholar 

  8. Bai, H., Cheng, S., Tang, J., Pan, J., Soc, I.C.: Learning a cascaded non-local residual network for super-resolving blurry images. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 223–232. (2021)

  9. Chan, K.C.K., Wang, X., Yu, K., Dong, C., Loy, C.C.: IEEE Comp, S.O.C.: BasicVSR: the search for essential components in video super-resolution and beyond. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4945–4954. (2021)

  10. Zou, W., Jiang, M., Zhang, Y., Chen, L., Lu, Z., Wu, Y., Soc, I.C.: SDWNet: A Straight Dilated Network with Wavelet Transformation for image Deblurring. In: 18th IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1895–1904. (2021)

  11. Zhang, Z., Chen, H., Yin, X., Deng, J., Li, W.: Dynamic selection of proper kernels for image deblurring: a multistrategy design. Vis. Comput. 39, 1375–1390 (2022)

  12. Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., Matas, J.: IEEE: DeblurGAN: blind motion deblurring using conditional adversarial networks. In: 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8183–8192. (2018)

  13. Kupyn, O., Martyniuk, T., Wu, J., Wang, Z.: IEEE: DeblurGAN-v2: Deblurring (Orders-of-Magnitude) faster and better. In: IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8877–8886. (2019)

  14. Liu, X., Du, H., Xu, J., Qiu, B.: DBGAN: a dual-branch generative adversarial network for undersampled MRI reconstruction. Magn. Reson. Imaging 89, 77–91 (2022)

    Article  Google Scholar 

  15. Cho, S.-J., Ji, S.-W., Hong, J.-P., Jung, S.-W., Ko, S.-J.: IEEE: rethinking coarse-to-fine approach in single image deblurring. In: 18th IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4621–4630. (2021)

  16. Nah, S., Kim, T.H., Lee, K.M.: IEEE: deep multi-scale convolutional neural network for dynamic scene deblurring. In: 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 257–265. (2017)

  17. Nah, S., Baik, S., Hong, S., Moon, G., Son, S., Timofte, R., Lee, K.M.: IEEE: NTIRE 2019 challenge on video deblurring and super-resolution: dataset and study. In: 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1996–2005. (2019)

  18. Zhang, H., Dai, Y., Li, H., Koniusz, P., Soc, I.C.: Deep stacked hierarchical multi-patch network for image deblurring. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5971–5979. (2019)

  19. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: IEEE Comp, S.O.C.: multi-stage progressive image restoration. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14816–14826. (2021)

  20. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Commun. ACM 63, 139–144 (2020)

    Article  Google Scholar 

  21. Pan, J.S., Dong, J.X., Liu, Y., Zhang, J.W., Ren, J.M., Tang, J.H., Tai, Y.W., Yang, M.H.: Physics-based generative adversarial models for image restoration and beyond. IEEE Trans. Pattern Anal. Mach. Intell. 43, 2449–2462 (2021)

    Article  Google Scholar 

  22. Zhang, K., Luo, W., Zhong, Y., Ma, L., Stenger, B., Liu, W., Li, H.: IEEE: deblurring by realistic blurring. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2734–2743. (2020)

  23. He, K.M., Zhang, X.Y., Ren, S.Q., Sun, J.: IEEE: deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. (2016)

  24. Suin, M., Purohit, K., Rajagopalan, A.N.: IEEE: spatially-attentive patch-hierarchical network for adaptive motion deblurring. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3603–3612. (2020)

  25. Zhou, L.H., Min, W.D., Lin, D.Y., Han, Q., Liu, R.K.: Detecting motion blurred vehicle logo in IoV Using filter-DeblurGAN and VL-YOLO. IEEE Trans. Veh. Technol. 69, 3604–3614 (2020)

    Article  Google Scholar 

  26. Batchuluun, G., Kang, J.K., Nguyen, D.T., Pham, T.D., Arsalan, M., Park, K.R.: Deep learning-based thermal image reconstruction and object detection. IEEE Access 9, 5951–5971 (2021)

    Article  Google Scholar 

  27. Truong, N.Q., Lee, Y.W., Owais, M., Nguyen, D.T., Batchuluun, G., Pham, T.D., Park, K.R.: SlimDeblurGAN-based motion deblurring and marker detection for autonomous drone landing. Sensors 20, 3918 (2020)

    Article  Google Scholar 

  28. Xie, G., Li, Z., Bhattacharyya, S., Mehmood, A.: IEEE: plug-and-play deblurring for robust object detection. In: IEEE International Conference on Visual Communications and Image Processing (VCIP) - Visual Communications in the Era of AI and Limited Resources. (2021)

  29. Tao, X., Gao, H.Y., Shen, X.Y., Wang, J., Jia, J.Y.: IEEE: scale-recurrent network for deep image deblurring. In: 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8174–8182. (2018)

  30. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: 18th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 234–241. (2015)

  31. Isola, P., Zhu, J.Y., Zhou, T.H., Efros, A.A.: IEEE: image-to-image translation with conditional adversarial networks. In: 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976. (2017)

  32. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: 34th International Conference on Machine Learning. (2017)

  33. Wu, Y.L., Shuai, H.H., Tam, Z.R., Chiu, H.Y.: IEEE: Gradient normalization for generative adversarial networks. In: 18th IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6353–6362. (2021)

  34. Johnson, J., Alahi, A., Li, F.F.: Perceptual losses for real-time style transfer and super-resolution. In: 14th European Conference on Computer Vision (ECCV), pp. 694–711. (2016)

  35. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.M., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J.J., Chintala, S.: PyTorch: An imperative style, high-performance deep learning library. In: 33rd Conference on Neural Information Processing Systems (NeurIPS). (2019)

  36. Nah, S., Son, S., Lee, S., Timofte, R., Lee, K.M., Chen, L.Y., Zhang, J., Lu, X., Chu, X.J., Chen, C.P., Xiong, Z.W., Xu, R.K., Xiao, Z.Y., Huang, J., Zhang, Y.Y., Xi, S., Wei, J., Bai, H.R., Cheng, S.S., Wei, H., Sun, L., Tang, J.H., Pan, J.S., Lee, D., Lee, C., Kim, T., Wang, X.B., Zhang, D.F., Pan, Z.H., Lin, T.W., Wu, W.H., He, D.L., Li, B.P., Li, B.Y., Xi, T., Zhang, G., Liu, J.T., Han, J.Y., Ding, E.R., Tao, G.P., Chu, W.Q., Cao, Y., Luo, D.H., Tai, Y., Lu, T., Wang, C.J., Li, J.L., Huang, F.Y., Chen, H.T., Chen, S.J., Guo, T.Y., Wang, Y.H., Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Shao, L., Zuo, Y.S., Ou, Y.M., Chai, Y.J., Shi, L., Liu, S., Lei, L., Feng, C.Y., Zeng, K., Yao, Y.Y., Liu, X.R., Zhang, Z.Z., Huang, H.C., Zhang, Y.C., Jiang, M.C., Zou, W.B., Miao, S., Kim, Y., Sun, Y.J., Deng, S.Y., Ren, W.Q., Cao, X.C., Wang, T., Suin, M., Rajagopalan, A.N., Duong, V.V., Nguyen, T.H., Yim, J., Jeon, B., Li, R., Xie, J.W., Han, J.W., Choi, J.H., Kim, J.H., Lee, J.S., Zhang, J.X., Peng, F., Svitov, D., Pakulich, D., Kim, J., Jeong, J., Soc, I.C.: NTIRE 2021 challenge on image deblurring. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 149–165. (2021)

  37. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004)

    Article  Google Scholar 

  38. Xu, R.K., Xiao, Z.Y., Huang, J., Zhang, Y.Y., Xiong, Z.W., Soc, I.C.: EDPN: Enhanced deep pyramid network for blurry image restoration. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 414–423. (2021)

  39. Purohit, K., Rajagopalan, A.N.: Assoc Advancement Artificial, I.: Region-adaptive dense network for efficient motion deblurring. In: 34th AAAI Conference on Artificial Intelligence / 32nd Innovative Applications of Artificial Intelligence Conference / 10th AAAI Symposium on Educational Advances in Artificial Intelligence, pp. 11882–11889. (2020)

  40. Chen, L., Chu, X., Zhang, X., Sun, J.: Simple baselines for image restoration. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII, pp. 17–33. (2022)

  41. Blalock, D., Gonzalez Ortiz, J.J., Frankle, J., Guttag, J.: What is the state of neural network pruning? Proc. Mach. Learn. Syst. 2, 129–146 (2020)

    Google Scholar 

  42. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

  43. Heikkila, J., Silvén, O.: A four-step camera calibration procedure with implicit image correction. In: Proceedings of IEEE computer society conference on computer vision and pattern recognition, pp. 1106–1112. (1997)

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (62072246).

Funding

National Natural Science Foundation of China, 62072246.

Author information

Authors and Affiliations

  1. College of Information Science and Technology, Nanjing Forestry University, Nanjing, 21003, Jiangsu Province, China

    Hanzhao Wang, Chunhua Hu, Weijie Qian & Qian Wang

Authors
  1. Hanzhao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chunhua Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weijie Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chunhua Hu.

Ethics declarations

Conflict of interest

All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, H., Hu, C., Qian, W. et al. RT-Deblur: real-time image deblurring for object detection. Vis Comput 40, 2873–2887 (2024). https://doi.org/10.1007/s00371-023-02991-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02991-y

Keywords

Search

Navigation