Skip to main content
SpringerLink
Log in

A deraining with detail-recovery network via context aggregation

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

As one representative object of bad weather, rain streaks can have a bad influence on image capture. Worse still, computer vision tasks can be badly influenced when working in rainy scenes. Therefore, it makes sense to conduct research on rain removal. This paper introduces a new deraining network (DDRNet). Specifically, rain streaks contain both frequency and spatial information, so we propose a rain removal module (RRM) consisting of a rain channel attention module (RCAM) and a rain spatial attention module (RSAM) for extracting rain information. Furthermore, we design a detail-recovery module (DRM) to extract the background feature so that the original background image details deleted by mistake during the deraining process can be made up by it. Moreover, a two-branch aggregation (TBA) mechanism is adopted to promote the process of information flow that effectively enhances the execution of our DDRNet. The evaluation of our DDRNet on various benchmark datasets has shown the superiority of our algorithm. In addition, a range of low-level experiments, such as object detection and semantic segmentation, provide further evidence of the effectiveness of the proposed DDRNet.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Zamanakos, G., Tsochatzidis, L., Amanatiadis, A., Pratikakis, I.: A comprehensive survey of lidar-based 3d object detection methods with deep learning for autonomous driving. Comput. Grap. 99, 153–181 (2021)

    Article  Google Scholar 

  2. Mei, R., Wang, H., Men, A.: Attention-enhanced and more balanced r-cnn for object detection. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 2136–2140 . IEEE (2020)

  3. Vanian, V., Zamanakos, G., Pratikakis, I.: Improving performance of deep learning models for 3d point cloud semantic segmentation via attention mechanisms. Comput. Graph. 106, 277–287 (2022)

    Article  Google Scholar 

  4. Kang, L.-W., Lin, C.-W., Fu, Y.-H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  5. Huang, D.-A., Kang, L.-W., Wang, Y.-C.F., Lin, C.-W.: Self-learning based image decomposition with applications to single image denoising. IEEE Trans. Multimedia 16(1), 83–93 (2013)

    Article  Google Scholar 

  6. Kim, J.-H., Lee, C., Sim, J.-Y., Kim, C.-S.: Single-image deraining using an adaptive nonlocal means filter. In: 2013 IEEE International Conference on Image Processing, pp. 914–917. IEEE (2013)

  7. Chen, D.-Y., Chen, C.-C., Kang, L.-W.: Visual depth guided color image rain streaks removal using sparse coding. IEEE Trans. Circuits Syst. Video Technol. 24(8), 1430–1455 (2014)

    Article  Google Scholar 

  8. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3397–3405 (2015)

  9. Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2736–2744 (2016)

  10. Chang, Y., Yan, L., Zhong, S.: Transformed low-rank model for line pattern noise removal. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1726–1734 (2017)

  11. Zhu, L., Fu, C.-W., Lischinski, D., Heng, P.-A.: Joint bi-layer optimization for single-image rain streak removal. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2526–2534 (2017)

  12. Fu, X., Huang, J., Ding, X., Liao, Y., Paisley, J.: Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans. Image Process. 26(6), 2944–2956 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  13. Li, R., Cheong, L.-F., Tan, R.T.: Single image deraining using scale-aware multi-stage recurrent network. arXiv preprint arXiv:1712.06830 (2017)

  14. Fan, Z., Wu, H., Fu, X., Huang, Y., Ding, X.: Residual-guide network for single image deraining. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 1751–1759 (2018)

  15. Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 254–269 (2018)

  16. Liu, J., Yang, W., Yang, S., Guo, Z.: Erase or fill? deep joint recurrent rain removal and reconstruction in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3233–3242 (2018)

  17. Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 695–704 (2018)

  18. Li, R., Cheong, L.-F., Tan, R.T.: Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1633–1642 (2019)

  19. Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., Lau, R.W.: Spatial attentive single-image deraining with a high quality real rain dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12270–12279 (2019)

  20. Wei, W., Meng, D., Zhao, Q., Xu, Z., Wu, Y.: Semi-supervised transfer learning for image rain removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3877–3886 (2019)

  21. Zhang, H., Sindagi, V., Patel, V.M.: Image de-raining using a conditional generative adversarial network. IEEE Trans. Circuits Syst. Video Technol. 30(11), 3943–3956 (2019)

    Article  Google Scholar 

  22. Zheng, Y., Yu, X., Liu, M., Zhang, S.: Residual multiscale based single image deraining. In: Bmvc, p. 147 (2019)

  23. Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., Ma, J., Jiang, J.: Multi-scale progressive fusion network for single image deraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8346–8355 (2020)

  24. Wang, G., Sun, C., Sowmya, A.: Context-enhanced representation learning for single image deraining. Int. J. Comput. Vision 129, 1650–1674 (2021)

    Article  Google Scholar 

  25. Liang, S., Meng, X., Su, Z., Zhou, F.: Multi-receptive field aggregation network for single image deraining. J. Vis. Commun. Image Represent. 84, 103469 (2022)

    Article  Google Scholar 

  26. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  27. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, Springer. pp. 234–241 (2015)

  28. Cao, M., Gao, Z., Ramesh, B., Mei, T., Cui, J.: Single image deraining integrating physics model and density-oriented conditional gan refinement. IEEE Signal Process. Lett. 28, 1635–1639 (2021)

    Article  Google Scholar 

  29. Xu, J., Ai, B., Chen, W., Yang, A., Sun, P., Rodrigues, M.: Wireless image transmission using deep source channel coding with attention modules. IEEE Trans. Circuits Syst. Video Technol. 32(4), 2315–2328 (2021)

    Article  Google Scholar 

  30. Charbonnier, P., Blanc-Feraud, L., Aubert, G., Barlaud, M.: Two deterministic half-quadratic regularization algorithms for computed imaging. In: Proceedings of 1st International Conference on Image Processing, vol. 2, pp. 168–172 (1994). IEEE

  31. Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1357–1366 (2017)

  32. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3855–3863 (2017)

  33. Li, S., Araujo, I.B., Ren, W., Wang, Z., Tokuda, E.K., Junior, R.H., Cesar-Junior, R., Zhang, J., Guo, X., Cao, X.: Single image deraining: A comprehensive benchmark analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3838–3847 (2019)

  34. Hu, X., Fu, C.-W., Zhu, L., Heng, P.-A.: Depth-attentional features for single-image rain removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8022–8031 (2019)

  35. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  36. Mittal, A., Soundararajan, R., Bovik, A.C.: Making a completely blind" image quality analyzer. IEEE Signal Process. Lett. 20(3), 209–212 (2012)

    Article  Google Scholar 

  37. Liu, L., Liu, B., Huang, H., Bovik, A.C.: No-reference image quality assessment based on spatial and spectral entropies. Signal Process. 29(8), 856–863 (2014)

    Google Scholar 

  38. Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)

  39. Yasarla, R., Patel, V.M.: Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8405–8414 (2019)

  40. Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: A better and simpler baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3937–3946 (2019)

  41. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.-H., Shao, L.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14821–14831 (2021)

  42. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 801–818 (2018)

Download references

Acknowledgements

This work was supported by Natural science research project of Guizhou Provincial Department of Education (QianJiaoJi[2022]029, QianJiaoHeKY[2021]022).

Author information

Authors and Affiliations

  1. State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang, 550025, China

    Weihao Gao, Yongjun Zhang & Wei Long

  2. School of Mathematics and Big Data, Guizhou Education University, Guiyang, 550018, Guizhou, China

    Zhongwei Cui

Authors
  1. Weihao Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongjun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Long
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhongwei Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yongjun Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, W., Zhang, Y., Long, W. et al. A deraining with detail-recovery network via context aggregation. Multimedia Systems 29, 2591–2601 (2023). https://doi.org/10.1007/s00530-023-01116-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-023-01116-8

Keywords

Search

Navigation