Skip to main content
SpringerLink
Log in

Factorized multi-scale multi-resolution residual network for single image deraining

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The performance of vision systems can be affected when used in severe weather conditions such as heavy rain or snow. Rain streak removal is an ill posed problem as they can vary in shape, size, and density across the image. In this paper, a single-image deraining network named Factorized Multi-scale Multi-resolution Residual Network (FMMRNet), which follows a U-Net backbone, is proposed. As rain streaks affect non-local regions of the image, larger receptive fields are beneficial to capture these non-local dependencies. We propose the use of multi-scale grouped convolutions to integrate information from global and local scales. In the proposed FMMRNet, multi-scale convolutions are factorized so that they can have large effective kernel sizes while reducing the computational complexity. During training, intermediate multi-resolution outputs are produced for loss computation which improves gradient flow in the deeper layers of the network and promotes better learning. A channel-wise attention mechanism is included to recalibrate feature maps before fusion instead of direct concatenation at each stage of the decoder. A higher-level reconstruction loss called perceptual loss is included for effective training to improve the visual quality of the derained images. The performance of the proposed FMMRNet is quantitatively and qualitatively compared on public benchmark datasets, and it outperforms the state-of-the-art deraining methods. Furthermore, to show the practical applicability of the proposed network, we demonstrate that when FMMRNet is used as a preprocessing step for object-detection methods such as Faster-RCNN and YOLO, it improves their performance on images degraded by rain streaks by 12.6% and 75.2% respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Code Availability

The code for this project will be made available.

References

  1. Cai L, Li SY, Ren D, Wang P (2019) Dual recursive network for fast image deraining. In: 2019 IEEE international conference on image processing (ICIP), pp 2756–2760. https://doi.org/10.1109/ICIP.2019.8803308

  2. Everingham M, Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88(2). https://doi.org/10.1007/s11263-009-0275-4

  3. Fan W, Wu Y, Wang C (2021) Single image rain streak removal via layer similarity prior. Appl Intell. https://doi.org/10.1007/s10489-020-02056-w

  4. Fan Z, Wu H, Fu X, Huang Y, Ding X (2018) Residual-guide network for single image deraining. In: 2018 ACM multimedia conference on multimedia conference - MM ’18, pp. 1751–1759. ACM Press. https://doi.org/10.1145/3240508.3240694. http://dl.acm.org/citation.cfm?doid=3240508.3240694

  5. Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: A deep network architecture for single-image rain removal. IEEE Trans Image Process 26(6):2944–2956. https://doi.org/10.1109/TIP.2017.2691802

    Article  MathSciNet  Google Scholar 

  6. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 1715–1723. https://doi.org/10.1109/CVPR.2017.186. http://ieeexplore.ieee.org/document/8099669/

  7. Gou Y, Li B, Liu Z, Yang S, Peng X (2020) Clearer: Multi-scale neural architecture search for image restoration. In: Larochelle H., Ranzato M., Hadsell R., Balcan M.F. , Lin H. (eds) Advances in neural information processing systems. https://proceedings.neurips.cc/paper/2020/file/c6e81542b125c36346d9167691b8bd09-Paper.pdf, vol 33. Curran Associates, Inc., pp 17129–17140

  8. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  9. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  10. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision. Springer, pp 694–711

  11. Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. International Conference on Learning Representations. arXiv:1412.6980

  12. Li B, Gou Y, Gu S, Liu JZ, Zhou JT, Peng X (2021) You only look yourself: Unsupervised and untrained single image dehazing neural network. Int J Comput Vis 129(5):1754– 1767

    Article  Google Scholar 

  13. Li P, Tian J, Tang Y, Wang G, Wu C (2020) Model-based deep network for single image deraining. IEEE Access 8:14036–14047. https://doi.org/10.1109/ACCESS.2020.2965545

    Article  Google Scholar 

  14. Li R, Cheong LF, Tan RT (2019) Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, pp. 1633–1642. https://doi.org/10.1109/CVPR.2019.00173. https://ieeexplore.ieee.org/document/8953352/

  15. Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Computer vision – ECCV 2018. ECCV 2018. Lecture Notes in Computer Science, vol 11211. Springer, Cham, pp 254–269

  16. Li Y, Tan RT, Guo X, Lu J, Brown MS (2017) Single image rain streak decomposition using layer priors. IEEE Trans Image Process 26(8):3874–3885. https://doi.org/10.1109/TIP.2017.2708841

    Article  MathSciNet  Google Scholar 

  17. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P., Zitnick CL (2014) Microsoft coco: Common objects in context. In: European conference on computer vision. Springer, pp 740–755

  18. Luo Y, Zhu J, Ling J, Wu E (2018) Fast removal of rain streaks from a single image via a shape prior. IEEE Access 6:60069–60078. https://doi.org/10.1109/ACCESS.2018.2875171

    Article  Google Scholar 

  19. Odena A, Dumoulin V, Olah C (2016) Deconvolution and checkerboard artifacts. Distill. https://doi.org/10.23915/distill.00003. http://distill.pub/2016/deconv-checkerboard

  20. Qian R, Tan RT, Yang W, Su J, Liu J (2018) Attentive generative adversarial network for raindrop removal from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2482–2491

  21. Redmon J, Farhadi A (2018) Yolov3: An incremental improvement. arXiv:1804.02767

  22. Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) Progressive image deraining networks: A better and simpler baseline. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, pp 3932–3941. https://doi.org/10.1109/CVPR.2019.00406. https://ieeexplore.ieee.org/document/8953349/

  23. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. In: Cortes C., Lawrence N., Lee D., Sugiyama M., Garnett R. (eds) Advances in neural information processing systems. https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf, vol 28. Curran Associates, Inc.

  24. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  25. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9. https://doi.org/10.1109/CVPR.2015.7298594

  26. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  27. Wang C, Fan W, Zhu H, Su Z (2020) Single image deraining via nonlocal squeeze-and-excitation enhancing network. Appl Intell 50(9):2932–2944. https://doi.org/10.1007/s10489-020-01693-5

    Article  Google Scholar 

  28. Wang T, Yang X, Xu K, Chen S, Zhang Q, Lau RW (2019) Spatial attentive single-image deraining with a high quality real rain dataset. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, pp. 12262–12271. https://doi.org/10.1109/CVPR.2019.01255. https://ieeexplore.ieee.org/document/8953571/

  29. Wei Y, Zhang Z, Wang Y, Xu M, Yang Y, Yan S, Wang M (2021) Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking. IEEE Trans Image Process 30:4788–4801

    Article  Google Scholar 

  30. Wei Y, Zhang Z, Wang Y, Zhang H, Zhao M, Xu M, Wang M (2021) Semi-deraingan: A new semi-supervised single image deraining. In: 2021 IEEE international conference on multimedia and expo (ICME). IEEE, pp. 1–6

  31. Wei Y, Zhang Z, Zhang H, Hong R, Wang M (2019) A coarse-to-fine multi-stream hybrid deraining network for single image deraining. In: 2019 IEEE international conference on data mining (ICDM), pp 628–637. https://doi.org/10.1109/ICDM.2019.00073

  32. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp. 1685–1694. https://doi.org/10.1109/CVPR.2017.183. http://ieeexplore.ieee.org/document/8099666/

  33. Yasarla R, Patel VM (2020) Confidence measure guided single image de-raining. IEEE Trans Image Process 29:4544–4555. https://doi.org/10.1109/TIP.2020.2973802

    Article  Google Scholar 

  34. Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 695–704. https://doi.org/10.1109/CVPR.2018.00079

  35. Zhang H, Sindagi V, Patel VM (2020) Image de-raining using a conditional generative adversarial network. IEEE Trans Circuits Syst Video Technol 30 (11):3943–3956. https://doi.org/10.1109/TCSVT.2019.2920407

    Article  Google Scholar 

  36. Zhang X, Wang T, Wang J, Tang G, Zhao L (2020) Pyramid channel-based feature attention network for image dehazing. Comput Vision Image Underst 197–198:103003. https://doi.org/10.1016/j.cviu.2020.103003. https://www.sciencedirect.com/science/article/pii/S1077314220300709

    Article  Google Scholar 

Download references

Funding

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and Affiliations

  1. Department of Instrumentation and Control Engineering, National Institute of Technology, Tiruchirappalli, India

    Shivakanth Sujit

  2. Department of Electronics and Communication Engineering, National Institute of Technology, Tiruchirappalli, India

    Deivalakshmi S

  3. Department of Electrical and Computer Engineering, University of Saskatchewan, Saskatoon, Canada

    Seok-Bum Ko

Authors
  1. Shivakanth Sujit
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Deivalakshmi S
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seok-Bum Ko
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shivakanth Sujit.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sujit, S., Deivalakshmi S & Ko, SB. Factorized multi-scale multi-resolution residual network for single image deraining. Appl Intell 52, 7582–7598 (2022). https://doi.org/10.1007/s10489-021-02772-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02772-x

Keywords

Search

Navigation