Skip to main content
SpringerLink
Log in

Recurrent multi-level residual and global attention network for single image deraining

  • S.I. : Deep Geospatial Data Understanding
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Deraining is an essential preprocess for many computer vision tasks, e.g., vision-based autonomous driving. The existing methods usually depend on the prior information or specified network structures and correspondingly suffer from high computational costs. To enhance the performance to meet the real-time requirement of autonomous driving, we propose a novel end-to-end recurrent multi-level residual learning deraining network featured with the global attention mechanism and residual network architecture. In the proposed Recurrent Multi-level Residual and Global Attention Network (RMRGN in short), we employ a recurrent stage scheme to gradually utilize global contextual information and image details to remove the rain streaks progressively. The global-attention mechanism enables us to focus on the meaningful context in every recurrent stage, which further benefits the network to distinguish the rain streaks and the rain-free images. By exploring the attention information, we further propose a deep multi-level residual learning network to eliminate rain streaks in a single image. Comprehensive experimental results demonstrate that RMRGN performs favorably against the state-of-the-art methods for removing rain streaks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Cao Y, Xu J, Lin S, Wei F, Hu H (2019) Gcnet: Non-local networks meet squeeze-excitation networks and beyond. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp 00,

  2. Chen YL, Hsu CT (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: Proceedings of the IEEE international conference on computer vision, pp 1968–1975

  3. Collins Robert T, Lipton Alan J, Takeo K, Hironobu F, David D, Yanghai T, David T, Nobuyoshi E, Osamu H, Peter B et al (2000) A system for video surveillance and monitoring. VSAM Final Rep 1(2000):1–68

    Google Scholar 

  4. Eigen D, Krishnan D, Fergus R (2013) Restoring an image taken through a window covered with dirt or rain. In: Proceedings of the IEEE international conference on computer vision, pp 633–640

  5. Fan Z, Wu H, Fu X , Hunag Y, Ding X (2018) Residual-guide feature fusion network for single image deraining. arXiv preprint arXiv: 1804.07493

  6. Farenzena M , Bazzani L , Perina A , Murino V , Cristani M (2010) Person re-identification by symmetry-driven accumulation of local features. In: 2010 IEEE computer society conference on computer vision and pattern recognition, pp 2360–2367. IEEE

  7. Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans Image Process 26(6):2944–2956

    Article  MATH  Google Scholar 

  8. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3855–3863

  9. Fu X, Liang B, Huang Y, Ding X, Paisley J (2019) Lightweight pyramid networks for image deraining. IEEE Trans Neural Netw Learn Syst 31(6):1794–1807

    Article  Google Scholar 

  10. Han P, Liu G, Huang S, Yuan W, Lu Z (2016) Segmentation with selectively propagated constraints. In: International conference on neural information processing, pp 585–592. Springer

  11. Han P, Shang S, Sun A, Zhao P, Zheng K, Zhang X (2021) Point-of-interest recommendation with global and local context. IEEE Trans Knowl Data Eng

  12. He K, Sun J, Tang X (2012) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35(6):1397–1409

    Article  Google Scholar 

  13. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  14. Hu H, Gu J, Zhang Z, Dai J, Wei Y (2018) Relation networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3588–3597

  15. Huynh-Thu Q, Ghanbari M (2008) Scope of validity of PSNR in image/video quality assessment. Electron Lett 44(13):800–801

    Article  Google Scholar 

  16. Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259

    Article  Google Scholar 

  17. Kang LW, Lin CW, Fu YH (2011) Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans Image Process 21(4):1742–1755

    Article  MATH  Google Scholar 

  18. Kingma DP, Ba J (2014) A method for stochastic optimization. arXiv:1412.6980

  19. Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of the European Conference on Computer Vision (ECCV)

  20. Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2736–2744

  21. Lin H, Li Y, Fu X, Ding X, Huang Y, Paisley J (2020) Rain o’er me: Synthesizing real rain to derain with data distillation. IEEE Trans Image Process 29:7668–7680

  22. Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE International Conference on Computer Vision, pp 3397–3405

  23. Maji S, Berg AC, Malik J (2008) Classification using intersection kernel support vector machines is efficient. In: 2008 IEEE conference on computer vision and pattern recognition, pp 1–8. IEEE

  24. Mandic D, Chambers J (2001) Recurrent neural networks for prediction: learning algorithms, architectures and stability. Wiley, New York

    Book  Google Scholar 

  25. Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv:1411.1784

  26. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Icml

  27. Niu Y, Lu Z, Huang S, Han P, Wen JR (2015) Weakly supervised matrix factorization for noisily tagged image parsing. In: IJCAI

  28. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A, Automatic differentiation in pytorch, Alban Desmaison

  29. Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) Progressive image deraining networks: A better and simpler baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3937–3946

  30. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.1556

  31. Sun H, Ang MH, Rus D (2019) A convolutional network for joint deraining and dehazing from a single image for autonomous driving in rain. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 962–969. IEEE

  32. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  33. Tomasi C, Manduchi R (1998) Bilateral filtering for gray and color images. In: Sixth international conference on computer vision (IEEE Cat. No. 98CH36271), pp 839–846. IEEE

  34. Wang X, Girshick R, Gupta A, Kaiming H (2018) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794–7803

  35. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  36. Wei W, Meng D, Zhao Q, Xu Z, Wu Y (2019) Semi-supervised transfer learning for image rain removal. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3877–3886

  37. Xingjian SH, Chen Z, Wang H, Yeung DY, Wong WK, Woo WC (2015) Convolutional lstm network: A machine learning approach for precipitation nowcasting. In: Advances in neural information processing systems, pp 802–810

  38. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1357–1366

  39. Yu F, Koltun V (2015) Multi-scale context aggregation by dilated convolutions. arXiv: 1511.07122

  40. Zhang H, Dana K, Shi J, Zhang Z, Wang X, Tyagi A (2018) Context encoding for semantic segmentation. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp 7151–7160

  41. Zhang H, Sindagi V, Patel VM (2019) Image de-raining using a conditional generative adversarial network. IEEE Trans Circuits Syst Video Technol 30(11):3943–3956

    Article  Google Scholar 

  42. Zhu L, Fu CW, Lischinski D, Heng PA (2017) Joint bi-layer optimization for single-image rain streak removal. In: Proceedings of the IEEE international conference on computer vision, pp 2526–2534

Download references

Author information

Authors and Affiliations

  1. College of Mathematics and Informatics, South China Agricultural University, Guangzhou, 510642, China

    Meihua Wang, Chao Li & Fanhui Ke

Authors
  1. Meihua Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fanhui Ke
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Meihua Wang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, M., Li, C. & Ke, F. Recurrent multi-level residual and global attention network for single image deraining. Neural Comput & Applic 35, 3697–3708 (2023). https://doi.org/10.1007/s00521-021-06814-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06814-w

Keywords

Search

Navigation