Skip to main content

Image based rainfall amount estimation for auto-wiping of vehicles

Neural Computing and Applications (2022)Cite this article

Abstract

With rapid development of deep learning, a lot of computer vision tasks, such as object detection and semantic segmentation, have been applied to various Advanced Driver Assistance Systems. However, few computer vision solutions to estimate rainfall amount have been developed so far. So, we propose a rainfall amount estimation method based on deep learning and computer vision. The proposed method mainly consists of two steps. The first step is raindrop segmentation, and the second step is rainfall amount estimation. The raindrop segmentation is specifically based on three techniques: A relational ASPP to explore the correlation between raindrop features, a height attention module to consider the features that vary depending on raindrop locations, and a masking loss to further improve the performance of raindrop segmentation. Second, using the segmented raindrops, we present a rainfall amount estimation algorithm for auto-wiping. We experimentally achieved acceptable raindrop segmentation performance, i.e., mean IoU (mIoU) score of 70.6% that is much higher than other algorithms. This verifies that the proposed network is good at segmenting raindrops on a windshield. And, we demonstrate that the proposed rainfall amount estimation scheme provides sufficiently high accuracy of about 93%. In addition, we have built a rainy driving dataset for computer vision-based auto-wiping purpose and published it publicly on https://github.com/jjh930910/raindrop-segmentation.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

References

  1. Ishikawa J (10 Aug. 2010) Raindrop detecting device and method of selecting wiping mode for vehicle. U.S. Patent No. 7,772,793

  2. VanDam SA (14 Jul 1998) Windshield wiper rain sensor system. U.S. Patent No. 5,780,719

  3. Cord A, Aubert D (2011) Towards rain detection through use of in-vehicle multipurpose cameras In: Proc IEEE IV Symp, Baden-Baden, Germany, pp 399–404

  4. Vijay CS, Bhat R, Ragavan V (2018) Raindrop detection considering extremal regions and salient features In: Proceedings electronic imaging, pp 348-1-348-6

  5. Qian R, Tan RT, Yang W, Su J, Liu J (2018) Attentive generative adversarial network for raindrop removal from a single image In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2482-2491

  6. Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Transactions Syst Man Cybern 9(1):62–66

    Article  Google Scholar 

  7. Dhanachandra N, Manglem K, Chanu YJ (2015) Image segmentation using K-means clustering algorithm and subtractive clustering algorithm. Procedia Computer Sci 54:764–771

    Article  Google Scholar 

  8. Najman L, Schmitt M (1994) Watershed of a continuous function. Signal Process 38(1):99–112

    Article  Google Scholar 

  9. Kass M, Witkin A, Terzopoulos D (1988) Snakes: active contour models. Int J Computer V 1(4):321–331

    MATH  Google Scholar 

  10. Boykov Y, Veksler O, Zabih R (2001) Fast approximate energy minimization via graph cuts. IEEE Transactions Pattern Anal Mach Intell 23(11):1222–1239

    Article  Google Scholar 

  11. Plath N, Toussaint M, Nakajima S (2009) Multi-class image segmentation using conditional random fields and global classification In: Proceedings of the 26th Annual International Conference on Machine Learning, pp 817–824

  12. Minaee S, Boykov YY, Porikli F, Plaza AJ, Kehtarnavaz N, Terzopoulos D (2021) Image segmentation using deep learning: a survey IEEE Transactions Pattern Anal Mach Intell (Early Access)

  13. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation In: International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, pp 234–241

  14. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431-3440

  15. Fan DP, Ji GP, Zhou T, Chen G, Fu H, Shen J, Shao L (2020) Pranet: parallel reverse attention network for polyp segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention Springer, Cham, pp 263–273

  16. Jha D, Riegler MA, Johansen D, Halvorsen P, Johansen HD (2020) DoubleU-Net: a deep convolutional neural network for medical image segmentation In: arXiv preprint arXiv:2006.04868

  17. Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, Liu J (2019) Ce-net: context encoder network for 2d medical image segmentation. Proc IEEE Transactions Med Imag 38(10):2281–2292

    Article  Google Scholar 

  18. Paluru N et al (2021) Anam-Net: anamorphic depth embedding-based lightweight CNN for segmentation of anomalies in COVID-19 chest CT images. IEEE Transactions Neural Netw Learn Syst 32(3):932–946

    Article  Google Scholar 

  19. Yeung M, Sala E, Schönlieb C-B, Rundo L (2021) Focus U-Net: a novel dual attention-gated CNN for polyp segmentation during colonoscopy. Computers Biol Med 137:104815

    Article  Google Scholar 

  20. Yu F, Koltun V (2015) Multi-scale context aggregation by dilated convolutions. In: arXiv preprint arXiv:1511.07122

  21. Chen LC, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation In: Proceedings of the European conference on computer vision, pp 801-818

  22. Liu W, Rabinovich A, Berg AC (2015) Parsenet: looking wider to see better. In: arXiv preprint arXiv:1506.04579

  23. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, ... Polosukhin I (2017) Attention is all you need Adv Neural Information Process Syst pp 5998-6008

  24. Meng Z, Gaur Y, Li J, Gong Y (2019) Character-aware attention-based end-to-end speech recognition In: Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, pp 949-955

  25. Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2019) Dual attention network for scene segmentation In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3146-3154

  26. Zhang F, Chen Y, Li Z, Hong Z, Liu J, Ma F, ... Ding E, (2019) Acfnet: attentional class feature network for semantic segmentation In: Proceedings of the IEEE International Conference on Computer Vision, pp 6798-6807

  27. Strudel R, Garcia R, Laptev I, Schmid C (2021) Segmenter: transformer for semantic segmentation In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 7262–7272

  28. Vaswani A et al. (2017) Attention is all you need Advances in neural information processing systems. In: Proceedings of the advances in neural information processing systems, pp 5998–6008

  29. Dosovitskiy A et al. (2021) An Image is Worth 16x16 Words: transformers for image recognition at scale ICLR. In: Proceedings of the international conference on learning representations, pp 1–21

  30. Choi S, Kim JT, Choo J (2020) Cars can’t fly up in the sky: improving urban-scene segmentation via height-driven attention networks In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

  31. You S, Tan RT, Kawakami R, Ikeuchi K (2013) Adherent raindrop detection and removal in video In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1035–1042

  32. Ito K, Noro K, Aoki T (2015) An adherent raindrop detection method using MSER In: 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pp 105–109

  33. Ishizuka J, Onoguchi K (2016) Detection of raindrop with various shapes on aWindshield ICPRAM, pp 475–483

  34. Shi X, Chen Z, Wang H, Yeung DY, Wong WK, Woo WC (2015) Convolutional LSTM network: a machine learning approach for precipitation nowcasting. Proc Adv Neural Information Process Syst 28:802–810

    Google Scholar 

  35. Lin J, Dai L (2020) X-NET for single image raindrop removal. In: Proceeding of the IEEE International Conference on Image Processing, pp 1003-1007

  36. Alletto S, Carlin C, Rigazio L, Ishii Y, Tsukizawa S (2019) Adherent raindrop removal with self-supervised attention maps and spatio-temporal generative adversarial networks In: Proceedings of the IEEE International Conference on Computer Vision Workshops

  37. Welch G, Bishop G (1995) An introduction to the Kalman filter. In: Proceedings of the SIGGRAPH, Course 8, pp 127-132

  38. Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, ... Glocker B (2018) Attention u-net: learning where to look for the pancreas In: arXiv preprint arXiv:1804.03999

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Inha University, Incheon, 22212, South Korea

    Seung Hoon Lee, Jung Ho Jeon, Dong Yoon Choi & Byung Cheol Song

  2. Advanced Body Technology Development Team, Hyundai Motor Company, Seoul, 06797, South Korea

    Jong Min Park

Authors
  1. Seung Hoon Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jung Ho Jeon
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dong Yoon Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jong Min Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Byung Cheol Song
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Byung Cheol Song.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lee, S.H., Jeon, J.H., Choi, D.Y. et al. Image based rainfall amount estimation for auto-wiping of vehicles. Neural Comput & Applic (2022). https://doi.org/10.1007/s00521-022-07269-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00521-022-07269-3

Keywords

  • Rainfall amount estimation
  • Raindrop segmentation
  • ADAS
  • Auto-wiping