Skip to main content
SpringerLink
Log in

Recursive residual Fourier transformation for single image deraining

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Reconstructing a rain-free image from its degraded counterpart requires transforms regarding information from diverse frequency levels. On the frequency domain perspective, despite current CNNs-based methods exhibit excellent abilities in capturing the high-frequency components of images, they often fail to adequately consider or overlook the low-frequency information. To address this challenge, we introduce a fast Fourier transform block (FFTB) which can effectively capture both long-term and short-term interactions, while integrating high- and low-frequency residual information. Our FFTB is a conceptually simple yet computationally efficient block, leading to remarkable performance gains. Based on FFTB, we further develop a multi-stage architecture termed recursive residual fourier network (RRFNet) to enhance the ability of capturing and modeling spatial and frequency domain visual cues. To fully maximize the performance of RRFNet, a novel global–local convert test strategy is employed to alleviate the training–testing inconsistency. Experimental results on the synthetic and real-world datasets demonstrate that our RRFNet performs favorably against state-of-the-art methods while enjoying faster inference speed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

Data is openly available in a public repository.

References

  1. Comaniciu D, Ramesh V, Meer P (2003) Kernel-based object tracking. IEEE Trans Pattern Anal Mach Intell 25(5):564–577

    Article  Google Scholar 

  2. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 2961–2969

  3. Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2736–2744

  4. Zhu L, Fu C-W, Lischinski D, Heng P-A (2017) Joint bi-layer optimization for single-image rain streak removal. In: Proceedings of the IEEE international conference on computer vision, pp 2526–2534

  5. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3855–3863

  6. Jiang K, Wang Z, Yi P, Chen C, Huang B, Luo Y, Ma J, Jiang J (2020) Multi-scale progressive fusion network for single image deraining. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8346–8355

  7. Yang W, Tan RT, Wang S, Fang Y, Liu J (2020) Single image deraining: from model-based to data-driven and beyond. IEEE Trans Pattern Anal Mach Intell 43(11):4059–4077

    Article  Google Scholar 

  8. Deng L-J, Huang T-Z, Zhao X-L, Jiang T-X (2018) A directional global sparse model for single image rain removal. Appl Math Model 59:662–679

    Article  MathSciNet  Google Scholar 

  9. Wang Y, Liu S, Chen C, Zeng B (2017) A hierarchical approach for rain or snow removing in a single color image. IEEE Trans Image Process 26(8):3936–3950

    Article  MathSciNet  Google Scholar 

  10. Li S, Ren W, Zhang J, Yu J, Guo X (2019) Single image rain removal via a deep decomposition–composition network. Comput Vis Image Underst 186:48–57

    Article  Google Scholar 

  11. Kang L-W, Lin C-W, Fu Y-H (2011) Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans Image Process 21(4):1742–1755

    Article  MathSciNet  Google Scholar 

  12. Chen Y-L, Hsu C-T (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: Proceedings of the IEEE international conference on computer vision, pp 1968–1975

  13. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1357–1366

  14. Fan Z, Wu H, Fu X, Huang Y, Ding X (2018) Residual-guide network for single image deraining. In: Proceedings of the 26th ACM international conference on multimedia, pp 1751–1759

  15. Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang M-H, Shao L (2021) Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14821–14831

  16. Chen L, Lu X, Zhang J, Chu X, Chen C (2021) Hinet: Half instance normalization network for image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 182–192

  17. Wang H, Wu X, Huang Z, Xing EP (2020) High-frequency component helps explain the generalization of convolutional neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8684–8694

  18. Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE international conference on computer vision, pp 3397–3405

  19. Yasarla R, Sindagi VA, Patel VM (2020) Syn2real transfer learning for image deraining using gaussian processes. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2726–2736

  20. Deng S, Wei M, Wang J, Feng Y, Liang L, Xie H, Wang FL, Wang M (2020) Detail-recovery image deraining via context aggregation networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14560–14569

  21. Wang H, Xie Q, Zhao Q, Meng D (2020) A model-driven deep neural network for single image rain removal. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3103–3112

  22. Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang M-H (2022) Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5728–5739

  23. Ehrlich M, Davis LS (2019) Deep residual learning in the jpeg transform domain. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 3484–3493

  24. Guo X, Fu X, Zhou M, Huang Z, Peng J, Zha Z-J (2022) Exploring fourier prior for single image rain removal. In: Proceedings of the 30th international joint conferences on artificial intelligence, pp 935–941

  25. Mao X, Liu Y, Shen W, Li Q, Wang Y (2021) Deep residual Fourier transformation for single image deblurring. arXiv preprint arXiv:2111.11745

  26. Suvorov R, Logacheva E, Mashikhin A, Remizova A, Ashukha A, Silvestrov A, Kong N, Goka H, Park K, Lempitsky V (2022) Resolution-robust large mask inpainting with Fourier convolutions. In: Proceedings of the IEEE/cvf winter conference on applications of computer vision, pp 2149–2159

  27. Shao M, Qiao Y, Meng D, Zuo W (2023) Uncertainty-guided hierarchical frequency domain transformer for image restoration. Knowl-Based Syst 263:110306

  28. Cao J, Li Y, Sun M, Chen Y, Lischinski D, Cohen-Or D, Chen B, Tu C (2020) Do-conv: Depthwise over-parameterized convolutional layer. arXiv preprint arXiv:2006.12030

  29. Chi L, Jiang B, Mu Y (2020) Fast Fourier convolution. Adv Neural Inf Process Syst 33:4479–4488

    Google Scholar 

  30. Brigham EO, Morrow R (1967) The fast Fourier transform. IEEE spectrum 4(12):63–70

    Article  Google Scholar 

  31. Chu X, Chen L, Chen C, Lu X (2021) Revisiting global statistics aggregation for improving image restoration. arXiv preprint arXiv:2112.04491

  32. Zhang H, Sindagi V, Patel VM (2019) Image de-raining using a conditional generative adversarial network. IEEE Trans Circuits Syst Video Technol 30(11):3943–3956

    Article  Google Scholar 

  33. Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 695–704

  34. Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans Image Process 26(6):2944–2956

    Article  MathSciNet  Google Scholar 

  35. Wei W, Meng D, Zhao Q, Xu Z, Wu Y (2019) Semi-supervised transfer learning for image rain removal. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3877–3886

  36. Yasarla R, Patel VM (2019) Uncertainty guided multi-scale residual learning-using a cycle spinning CNN for single image de-raining. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8405–8414

  37. Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of the European conference on computer vision (ECCV), pp 254–269

  38. Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) Progressive image deraining networks: A better and simpler baseline. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3937–3946

  39. Purohit K, Suin M, Rajagopalan A, Boddeti VN (2021) Spatially-adaptive image restoration using distortion-guided networks. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 2309–2319

  40. Tu Z, Talebi H, Zhang H, Yang F, Milanfar P, Bovik A, Li Y (2022) Maxim: Multi-axis mlp for image processing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5769–5780

  41. Lai W-S, Huang J-B, Ahuja N, Yang M-H (2018) Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Trans Pattern Anal Mach Intell 41(11):2599–2613

    Article  Google Scholar 

  42. Kamgar-Parsi B, Rosenfeld A (1999) Optimally isotropic laplacian operator. IEEE Trans Image Process 8(10):1467–1472

    Article  MathSciNet  Google Scholar 

  43. Loshchilov I, Hutter F (2016) Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983

  44. Kingma D, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.69803

  45. Quan R, Yu X, Liang Y, Yang Y (2021) Removing raindrops and rain streaks in one go. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9147–9156

  46. Huynh-Thu Q, Ghanbari M (2008) Scope of validity of PSNR in image/video quality assessment. Electron Lett 44(13):800–801

    Article  Google Scholar 

  47. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

Download references

Acknowledgments

The authors are very indebted to the anonymous referees for their critical comments and suggestions for the improvement of this paper. This work was supported by National Key Research and development Program of China (2021YFA1000102), and in part by the grants from the National Natural Science Foundation of China (Nos. 62376285, 62272375, 61673396), Natural Science Foundation of Shandong Province, China (No. ZR2022MF260).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, China University of Petroleum (East China), Qingdao, 266000, Shandong, China

    Zhiyuan Bao, Mingwen Shao, Yecong Wan & Yuanjian Qiao

Authors
  1. Zhiyuan Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingwen Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yecong Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuanjian Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingwen Shao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bao, Z., Shao, M., Wan, Y. et al. Recursive residual Fourier transformation for single image deraining. Int. J. Mach. Learn. & Cyber. 15, 1743–1754 (2024). https://doi.org/10.1007/s13042-023-01994-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-023-01994-4

Keywords

Search

Navigation