Skip to main content
SpringerLink
Log in

Dual-attention U-Net and multi-convolution network for single-image rain removal

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Images taken on rainy days have rain streaks of varying degrees of intensity, which seriously affect the visibility of the background scene. Aiming at the above problems, we propose a rain mark removal algorithm based on the combination of dual-attention mechanism U-Net and multi-convolution. First, we add a double attention mechanism to the encoder of U-Net. It can give different weights to the rain mark features that need to be extracted in different channels and spaces so that sufficient rain mark features can be obtained. With different dilation factors, we can obtain rain mark characteristics of different depths. Secondly, the multi-convolutional channel integrates the characteristics of rain streaks and prepares sufficient rain mark information for the task of clearing rain streaks. By introducing a cyclic rain streaks detection and removal mechanism into the network architecture, it can achieve gradual removal of rain streaks. Even in the case of heavy rain, our algorithm can get good results. Finally, we tested on both synthetic and real datasets to obtain subjective results and objective evaluations. Experimental results show that for the rainy day image de-rain task with different intensities of rain streaks, our algorithm is more robust. Moreover, the ability of our algorithm to remove rain streaks is better than that of the other five different classical algorithms. The de-raining images produced by our algorithm are visually sharper, and its visibility enhancements are effective for computer vision applications (Google Vision API).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Data availability

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

  1. Hang, Y., Derong, C., Jiulu, G.: Object tracking using both a kernel and a non-parametric active contour model. Neurocomputing 295, 108–117 (2018)

    Article  Google Scholar 

  2. Collins, R.T., Lipton, A.J., Kanade, T., Fujiyoshi, H., Duggins, D., Tsin,Y., Tolliver, D., Enomoto, N., Hasegawa, O., Burt, P., et al.: A system for video surveillance and monitoring. VSAM final report 2000(1–68), 1 (2000)

  3. Dollar, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: an evaluation of the state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 34(4), 743–761 (2011)

    Article  Google Scholar 

  4. Luo, Y., Wu, M., Huang, Q., Zhu, J., Ling, J., Sheng, B.: Joint feedback and recurrent deraining network with ensemble learning. The Vis. Comput. 268, 1–11 (2022)

    Google Scholar 

  5. Yang, H., Zhou, D., Li, M., Zhao, Q.: A two-stage network with wavelet transformation for single-image deraining. Vis. Comput. 39(9), 3887–3903 (2023)

    Article  Google Scholar 

  6. Fan, L., Zhang, F., Fan, H., Zhang, C.: Brief review of image denoising techniques. Vis. Comput. Ind. Biomed. Art 2(1), 1–12 (2019)

    Article  Google Scholar 

  7. Zhang, L., Li, Y., Wang, P., Wei, W., Xu, S., Zhang, Y.: A separation–aggregation network for image denoising. Appl. Soft Comput. 83, 105603 (2019)

    Article  Google Scholar 

  8. Garg, K., Nayar, S.K.: Detection and removal of rain from videos. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2004, vol. 1, IEEE (2004)

  9. Wei, W., Yi, L., Xie, Q., Zhao, Q., Meng, D., Xu, Z.: Should we encode rain streaks in video as deterministic or stochastic? In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2516–2525 (2017)

  10. Zhang, H., Patel, V.M.: Convolutional sparse and low-rank coding-based rain streak removal. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1259–1267, IEEE (2017)

  11. Zhu, L., Fu, C.-W., Lischinski, D., Heng, P.-A.: Joint bi-layer optimization for single-image rain streak removal. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2526–2534 (2017)

  12. Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2736–2744 (2016)

  13. He, X., Feng, Y., Xie, Q., Wang, J.: Low-rank appearance-preserving rain streak removal from single images. Commun. Inf. Syst. 22(1), 79–102 (2022)

    Article  Google Scholar 

  14. Jayaraman, T., Chinnusamy, S.: Performance analysis of a dual stage deep rain streak removal convolution neural network module with a modified deep residual dense network. Int. J. Appl. Math. Comput. Sci. (2022). https://doi.org/10.34768/amcs-2022-0009

    Article  Google Scholar 

  15. Zheng, Z., Chen, Z., Wang, S., Wang, W., Wang, H.: Memory-efficient multi-scale residual dense network for single image rain removal. Comput. Vis. Image Underst. 235, 103766 (2023). https://doi.org/10.1016/j.cviu.2023.103766

    Article  Google Scholar 

  16. Fu, X., Huang, J., Ding, X., Liao, Y., Paisley, J.: Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans. Image Process. 26(6), 2944–2956 (2017)

    Article  MathSciNet  Google Scholar 

  17. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 234–241. Springer (2015)

  18. Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

  19. Kim, J.-H., Lee, C., Sim, J.-Y., Kim, C.-S.: Single-image deraining using an adaptive nonlocal means filter. In: 2013 IEEE International Conference on Image Processing, pp. 914–917. IEEE (2013)

  20. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3397–3405 (2015)

  21. Wang, Y., Liu, S., Chen, C., Zeng, B.: A hierarchical approach for rain or snow removing in a single color image. IEEE Trans. Image Process. 26(8), 3936–3950 (2017)

    Article  MathSciNet  Google Scholar 

  22. Li, G., He, X., Zhang, W., Chang, H., Dong, L., Lin, L.: Non-locally enhanced encoder-decoder network for single image de-raining. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 1056–1064 (2018)

  23. Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: A better and simpler baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3937–3946 (2019)

  24. Pan, J., Liu, S., Sun, D., Zhang, J., Liu, Y., Ren, J., Li, Z., Tang, J., Lu, H., Tai, Y.-W., et al.: Learning dual convolutional neural networks for low-level vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3070–3079 (2018)

  25. Chen, C., Li, H.: Robust representation learning with feedback for single image deraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7742–7751 (2021)

  26. Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 695–704 (2018)

  27. Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., Lau, R.W.: Spatial attentive single-image deraining with a high quality real rain dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12270–12279 (2019)

  28. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: Cbam: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  29. Garg, K., Nayar, S.K.: Photorealistic rendering of rain streaks. ACM Trans. Graph. (TOG) 25(3), 996–1002 (2006)

    Article  Google Scholar 

  30. Collobert, R., Kavukcuoglu, K., Farabet, C.: Torch7: a matlab-like environment for machine learning. In: BigLearn, NIPS Workshop (2011)

  31. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  32. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  33. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3855–3863 (2017)

  34. Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 254–269 (2018)

  35. Yang, W., Tan, R.T., Feng, J., Guo, Z., Yan, S., Liu, J.: Joint rain detection and removal from a single image with contextualized deep networks. IEEE Trans. Pattern Anal. Mach. Intell. 42(6), 1377–1393 (2019)

    Article  Google Scholar 

  36. Ran, W., Yang, Y., Lu, H.: Single image rain removal boosting via directional gradient. In: 2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6, IEEE (2020)

  37. Fu, X., Liang, B., Huang, Y., Ding, X., Paisley, J.: Lightweight pyramid networks for image deraining. IEEE Trans. Neural Netw. Learn. Syst. 31(6), 1794–1807 (2019)

    Article  Google Scholar 

  38. Yadav, S., Mehra, A., Rohmetra, H., Ratnakumar, R., Narang, P.: Deraingan: Single image deraining using wasserstein gan. Multimed. Tools Appl. 80(30), 36491–36507 (2021)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Natural Science Foundation of Zhangzhou under Grant ZZ2020J33 and the Natural Science Foundation of Fujian under Grant 2023J01924.

Author information

Authors and Affiliations

  1. School of Physics and Information Engineering, Minnan Normal University, Zhangzhou, 363000, Fujian, China

    Ziyang Zheng, Zhixiang Chen & Wenpeng Wang

  2. School of Computer, Minnan Normal University, Zhangzhou, 363000, Fujian, China

    Shuqi Wang

  3. Key Laboratory of Data Science and Intelligence Application, Minnan Normal University, Zhangzhou, 363000, Fujian, China

    Shuqi Wang

Authors
  1. Ziyang Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhixiang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuqi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenpeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

ZZ contributed to conceptualization, methodology, data curation, writing—original draft, and writing—review and editing, and provided software. ZC was involved in conceptualization, methodology, supervision, and writing—review and editing. SW contributed to writing—review and editing. WW was involved in writing—review and editing.

Corresponding author

Correspondence to Zhixiang Chen.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical approval

I testify on behalf of all the co-authors: This material has not been published in whole or in part elsewhere. The manuscript is not currently being considered for publication in another journal. All authors have been personally and actively involved in substantive work leading to the manuscript and will hold themselves jointly and individually responsible for its content.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, Z., Chen, Z., Wang, S. et al. Dual-attention U-Net and multi-convolution network for single-image rain removal. Vis Comput (2023). https://doi.org/10.1007/s00371-023-03198-x

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-023-03198-x

Keywords

Search

Navigation