Skip to main content
SpringerLink
Log in

Raindrop removal from a single image using a two-step generative adversarial network

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Existing methods in removing raindrops from images have encountered a key challenge, i.e., removing raindrops of different sizes and shapes while recovering the lost details. These approaches are based on a pixel-wise regression process. However, they are lacking in maintaining the balance between raindrop removal and image inpainting. To resolve the problem, we propose a method based on a hierarchical supervision network (HSNet). This network is a fusion of dense network blocks and a U-Net network. Our proposed method works in two stages. In the first stage, the HSNet is used to extract raindrop features to make a residual between raindrop image samples, i.e., input data and output of HSNet to obtain an image without raindrops. In the second stage, image inpainting is combined with an attention mechanism through HSNet to recover the areas covered by raindrops in the input image. Experimental results show that our proposed method outperforms the existing methods proposed for image deraining.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Lan, X., Ye, M., Zhang, S., et al.: Modality-correlation-aware sparse representation for RGB-infrared object tracking. Pattern Recognit. Lett. 130, 12–20 (2020)

    Article  Google Scholar 

  2. Zhang, H., Patel, V. M. Density-aware single image de-raining using a multi-stream dense network. Computer Vision and Pattern Recognition, pp. 695–704 (2018)

  3. Qian, R., Tan, R. T., Yang, W., Su, J., Liu, J.: Attentive generative adversarial network for raindrop removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2482–2491 (2018)

  4. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: International Conference on Computer Vision, pp. 3397–3405 (2015)

  5. Li, Y., Tan, R. T., Guo, X., Lu, J., Brown, M. S.: Rain streak removal using layer priors. Computer Vision and Pattern Recognition, pp. 2736–2744 (2016)

  6. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. Computer Vision and Pattern Recognition, pp. 1715–1723 (2017)

  7. Liu, H., Ye, H., Li, X., Shi, W., Liu, M., Sun, Q.: Self-refining deep symmetry enhanced network for rain removal. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 2786–2790 (2019)

  8. Lin, C., Zhang, Q., Cao, Y.: Multi-scale contour detection model based on fixational eye movement mechanism. SIViP 14, 57–65 (2020)

    Article  Google Scholar 

  9. ElTantawy, A., Shehata, M.S.: Local null space pursuit for real-time moving object detection in aerial surveillance. SIViP 14, 87–95 (2020)

    Article  Google Scholar 

  10. Lan, X., Ye, M., Shao, R., et al.: Learning modality-consistency feature templates: a robust RGB-infrared tracking system. IEEE Trans. Ind. Electron. 66(12), 9887–9897 (2019)

    Article  Google Scholar 

  11. Gao, X.: A post-processing scheme for the performance improvement of vehicle detection in wide-area aerial imagery. SIViP 14, 625–633 (2020)

    Article  Google Scholar 

  12. Lan, X., Zhang, W., Zhang, S., et al.: Robust multi-modality anchor graph-based label prediction for RGB-infrared tracking. IEEE Trans. Ind. Inform. (2019)

  13. Liu, D.S.M., Chen, Y.J.: Rain removal system for dynamic scene in diminished reality. SIViP 14, 945–953 (2020)

    Article  Google Scholar 

  14. Kurihata, H., Takahashi, T., Ide, I., Mekada, Y., Murase, H., Tamatsu, Y., Miyahara, T.: Rainy weather recognition from in-vehicle camera images for driver assistance. In: IEEE Proceedings of the Intelligent Vehicles Symposium, pp. 205–210 (2005)

  15. Roser, M., Geiger, A.: Video-based raindrop detection for improved image registration. In: 2009 IEEE 12th International Conference on Computer Vision Workshops, pp. 570–577 (2009)

  16. You, S., Tan, R.T., Kawakami, R., Mukaigawa, Y., Ikeuchi, K.: Adherent raindrop modeling, detectionand removal in video. IEEE Trans. Pattern Anal. Mach. Intell. 38(9), 1721–1733 (2015)

    Article  Google Scholar 

  17. Eigen, D., Krishnan, D., Fergus, R.: Restoring an image taken through a window covered with dirt or rain. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 633–640 (2013)

  18. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems, pp. 2672–2680 (2014)

  19. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 234–241 (2015)

  20. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K. Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

  21. Krizhevsky, A., Sutskever, I., Hinton, G. E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

  22. Roser, M., Kurz, J., Geiger, A.: Realistic modeling of water droplets for monocular adherent raindrop recognition using Bezier curves. In: Asian Conference on Computer Vision, pp. 235–244 (2010)

  23. Zhang, H., Sindagi, V., Patel, V.M.: Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology (2019)

  24. Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1357–1366 (2017)

  25. Kingma, D., Ba, J.: Adam: A Method for Stochastic Optimization. Computer Science (2014)

  26. Kang, L.W., Lin, C.W., Fu, Y.H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2011)

    Article  MathSciNet  Google Scholar 

  27. Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 254–269 (2018)

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 61762014 and Grant 62106054, and in part by the Science and Technology Project of Guangxi under Grant 2018GXNSFAA281351.

Author information

Authors and Affiliations

  1. College of Electronics Engineering, Guangxi Normal University, Guilin, 541004, China

    Haiying Xia, Yang Lan, Shuxiang Song & Haisheng Li

Authors
  1. Haiying Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Lan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuxiang Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haisheng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yang Lan or Shuxiang Song.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xia, H., Lan, Y., Song, S. et al. Raindrop removal from a single image using a two-step generative adversarial network. SIViP 16, 677–684 (2022). https://doi.org/10.1007/s11760-021-02007-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-021-02007-z

Keywords

Search

Navigation