Skip to main content
SpringerLink
Log in

DeTformer: A Novel Efficient Transformer Framework for Image Deraining

  • Published:
Circuits, Systems, and Signal Processing Aims and scope Submit manuscript

Abstract

Captured rainy images severely degrade outdoor vision systems performance, such as semi-autonomous or autonomous driving systems and video surveillance systems. Consequently, removing heavy and complex rain streaks, i.e. undesirable rainy artifacts from a rainy image, plays a crucial role for many high-level computer vision tasks and has drawn researchers’ attention over the past few years. The main drawbacks of convolutional neural networks are: have smaller receptive field, lack of model’s ability to capture long-range dependencies and complicated rainy artifacts, non-adaptive to input content and also increase in computational complexity quadratically with input image size. The aforementioned issues limit the performance of deraining model improvement further. Recently, transformer has achieved better performance in terms of both natural language processing (NLP) and high-level computer vision (CV). We cannot adopt transformer directly to image deraining as it has the following limitations: (a) although the transformer possesses powerful long-range computational capability, it lacks the ability to model local features, and (b) to process input image, transformer uses fixed patch size; therefore, pixels at the patch edges cannot use local features of surrounding pixels while removing heavy rain streaks. To address these issues, in single image deraining, we propose a novel and efficient deraining transformer (DeTformer). In DeTformer, we designed a “gated depth-wise convolution feed-forward network” (GDWCFN) to address the first issue and applied depth-wise convolution to improve the modelling capability of local features and suppress unnecessary features and allow only useful information to higher layers. Also, the second issue was addressed, by introducing multi-resolution features in our network, where we applied progressive learning in the transformer, and thus, it allows the edge pixels to utilize local features effectively. Furthermore, to integrate the extracted multi-scale features and provide feature interaction across channel dimensions, we introduced a “multi-head depth-wise convolution transposed attention” (MDWCTA) module. The proposed network was experimented with on various derained datasets and compared with state-of-the-art networks. The experimental results show that DeTformer network achieves superior performance compared to state-of-the-art networks on synthetic and real-world rain datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability Statement

All data generated or analysed during this study are included in this published articles Rain100L [38], Rain100H [38], Rain1800 [38], Rain12 [20], Rain1200 [42], Rain14000 [11] and Rain800 [44].

References

  1. M.W. Ahmed, A.A. Abdulla, Quality improvement for exemplar-based image inpainting using a modified searching mechanism. UHD J Sci Technol 4(1), 1–8 (2020)

    Article  Google Scholar 

  2. J.L. Ba, J.R. Kiros, G.E. Hinton, Layer normalization. arXiv preprint arXiv:1607.06450 (2016)

  3. S. Cao, L. Liu, L. Zhao, Y. Xu, J. Xu, X. Zhang, Deep feature interactive aggregation network for single image deraining. IEEE Access 10, 103872–103879 (2022)

    Article  Google Scholar 

  4. C. Chen, H. Li, Robust representation learning with feedback for single image deraining, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7742–7751 (2021)

  5. H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, et al., Pre-trained image processing transformer, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12299–12310 (2021)

  6. L. Chen, X. Lu, J. Zhang, X. Chu, C. Chen, Hinet: half instance normalization network for image restoration, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 182–192

  7. Y.L. Chen, C.T. Hsu, A generalized low-rank appearance model for spatio-temporally correlated rain streaks, in Proceedings of the IEEE International Conference on Computer Vision (2013), pp. 1968–1975

  8. S. Deng, M. Wei, J. Wang, Y. Feng, L. Liang, H. Xie, et al., Detail-recovery image deraining via context aggregation networks, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020), pp. 14560–14569

  9. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, et al., An image is worth 16 × 16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  10. X. Fu, J. Huang, X. Ding, Y. Liao, J. Paisley, Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans. Image Process. 26(6), 2944–2956 (2017)

    Article  MathSciNet  Google Scholar 

  11. X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, J. Paisley, Removing rain from single images via a deep detail network, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 3855–3863

  12. X. Fu, B. Liang, Y. Huang, X. Ding, J. Paisley, Lightweight pyramid networks for image deraining. IEEE Trans Neural Netw Learn Syst 31(6), 1794–1807 (2019)

    Article  Google Scholar 

  13. X. Fu, Q. Qi, Z.J. Zha, Y. Zhu, X. Ding, Rain streak removal via dual graph convolutional network, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 2 (2021), pp. 1352–1360

  14. D. Hendrycks, K. Gimpel, Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)

  15. K. Jiang, Z. Wang, P. Yi, C. Chen, B. Huang, Y. Luo, et al., Multi-scale progressive fusion network for single image deraining, in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020), pp. 8346–8355

  16. H. Ji, X. Feng, W. Pei, J. Li, G. Lu, U2-former: a nested u-shaped transformer for image restoration. arXiv preprint arXiv:2112.02279 (2021)

  17. L.W. Kang, C.W. Lin, Y.H. Fu, Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2011)

    Article  MathSciNet  Google Scholar 

  18. X. Li, J. Wu, Z. Lin, H. Liu, H. Zha, Recurrent squeeze-and-excitation context aggregation net for single image deraining, in Proceedings of the European Conference on Computer Vision (ECCV) (2018), pp. 254–269

  19. Y. Li, Y. Monno, M. Okutomi, Single image deraining network with rain embedding consistency and layered LSTM, in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2022), pp. 4060–4069

  20. Y. Li, R.T. Tan, X. Guo, J. Lu, M.S. Brown, Rain streak removal using layer priors, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 2736–2744

  21. Y. Li, K. Zhang, J. Cao, R. Timofte, L. Van Gool, Localvit: bringing locality to vision transformers. arXiv preprint arXiv:2104.05707 (2021)

  22. X. Liu, M. Suganuma, Z. Sun, T. Okatani, Dual residual networks leveraging the potential of paired operations for image restoration, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), pp. 7007–7016

  23. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, et al., Swin transformer: hierarchical vision transformer using shifted windows, in Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 10012–10022

  24. Y. Luo, Y. Xu, H. Ji, Removing rain from a single image via discriminative sparse coding, in Proceedings of the IEEE International Conference on Computer Vision (2015), pp. 3397–3405

  25. C.B. Murthy, M.F. Hashmi, A.G. Keskar, EfficientLiteDet: a real-time pedestrian and vehicle detection algorithm. Mach. Vis. Appl. 33(3), 47 (2022)

    Article  Google Scholar 

  26. T. Ragini, K. Prakash, Progressive multi-scale deraining network, in 2022 IEEE International Symposium on Smart Electronic Systems (iSES) (IEEE, 2022), pp. 231–235

  27. D. Ren, W. Zuo, Q. Hu, P. Zhu, D. Meng, Progressive image deraining networks: a better and simpler baseline, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), pp. 3937–3946

  28. W. Shi, J. Caballero, F. Huszár, J. Totz, A.P. Aitken, R. Bishop, et al., Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 1874–1883

  29. F. Tan, Y. Kong, Y. Fan, F. Liu, D. Zhou, L. Chen, et al., SDNet: mutil-branch for single image deraining using swin. arXiv preprint arXiv:2105.15077 (2021)

  30. J.M.J. Valanarasu, R. Yasarla, V.M. Patel, Transweather: transformer-based restoration of images degraded by adverse weather conditions, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 2353–2363

  31. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, et al., Attention is all you need. in Advances in Neural Information Processing Systems (NIPS) (2017), pp. 5998–6008

  32. H. Wang, Q. Xie, Q. Zhao, D. Meng, A model-driven deep neural network for single image rain removal, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020), pp. 3103–3112

  33. T. Wang, L. Zhao, P. Huang, X. Zhang, J. Xu, Haze concentration adaptive network for image dehazing. Neurocomputing 439, 75–85 (2021)

    Article  Google Scholar 

  34. Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, H. Li, Uformer: a general u-shaped transformer for image restoration, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 17683–17693

  35. W. Wei, D. Meng, Q. Zhao, Z. Xu, Y. Wu, Semi-supervised transfer learning for image rain removal, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), pp. 3877–3886

  36. H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, L. Zhang, Cvt: introducing convolutions to vision transformers, in Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 22–31

  37. H. Yang, D. Zhou, M. Li, Q. Zhao, A two-stage network with wavelet transformation for single-image deraining. Vis. Comput. 39, 3887–3903 (2023)

  38. W. Yang, R.T. Tan, J. Feng, J. Liu, Z. Guo, S. Yan, Deep joint rain detection and removal from a single image, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 1357–1366

  39. W. Yang, R.T. Tan, S. Wang, Y. Fang, J. Liu, Single image deraining: From model-based to data-driven and beyond. IEEE Trans. Pattern Anal. Mach. Intell. 43(11), 4059–4077 (2020)

    Article  Google Scholar 

  40. R. Yasarla, V.M. Patel, Uncertainty guided multi-scale residual learning-using a cycle spinning CNN for single image de-raining, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), pp. 8405–8414

  41. R. Yasarla, J.M.J. Valanarasu, V.M. Patel, Exploring overcomplete representations for single image deraining using cnns. IEEE J Sel Top Signal Process 15(2), 229–239 (2020)

    Article  Google Scholar 

  42. H. Zhang, V.M. Patel, Density-aware single image de-raining using a multi-stream dense network, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), pp. 695–704

  43. H. Zhang, V.M. Patel, Convolutional sparse and low-rank coding-based rain streak removal, in 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) (IEEE, 2017), pp. 1259–1267

  44. H. Zhang, V. Sindagi, V.M. Patel, Image de-raining using a conditional generative adversarial network. IEEE Trans. Circuits Syst. Video Technol. 30(11), 3943–3956 (2019)

    Article  Google Scholar 

  45. S. Zheng, C. Lu, Y. Wu, G. Gupta, SAPNet: segmentation-aware progressive network for perceptual contrastive deraining, in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2022), pp.52–62

  46. X. Zhou, Q. Qiu, X. Huang, Deep joint rain detection and removal from a single image. IEEE Trans. Image Process. 28(2), 865–878 (2019)

    Google Scholar 

  47. Y. Zhu, Y. Su, P. Tan, A fast single image haze removal algorithm using color attenuation prior. IEEE Trans. Image Process. 24(11), 3522–3533 (2018)

    MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Communication Engineering, National Institute of Technology Warangal, Hanamkonda, Telangana, India

    Thatikonda Ragini & Kodali Prakash

  2. Department of Computer Science and Engineering, National Institute of Technology Warangal, Hanamkonda, Telangana, India

    Ramalingaswamy Cheruku

Authors
  1. Thatikonda Ragini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kodali Prakash
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ramalingaswamy Cheruku
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kodali Prakash.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ragini, T., Prakash, K. & Cheruku, R. DeTformer: A Novel Efficient Transformer Framework for Image Deraining. Circuits Syst Signal Process 43, 1030–1052 (2024). https://doi.org/10.1007/s00034-023-02499-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00034-023-02499-9

Keywords

Search

Navigation