Abstract
Raindrop blur or rain streaks can severely degrade the visual quality of the images, which causes many practical vision systems to fail to work, such as autonomous driving and video surveillance. Hence, it is important to address the problem of single image de-raining. In this paper, we propose a novel deep network for single image de-raining. The proposed network consists of three stages, including encoder stage, Dense Non-Local Residual Block (DNLRB) stage, and decoder stage. As spatial contextual information has been analyzed to be meaningful for image de-raining (Huang et al. ??), we adopt squeeze-and-excitation enhancing on feature maps in each convolution layer for capturing spatial contextual information. In addition, to better leverage spatial contextual information for extracting rain components, the non-local mean operation has been embed in DNLRB. Both quantitative and qualitative experimental results demonstrate the proposed method performs favorably against the state-of-the-art de-raining methods. The source codes will be available at https://supercong94.wixsite.com/supercong94.
Similar content being viewed by others
References
Bossu J, Hautière N, Tarel J (2011) Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int J Comput Vis 93(3):348–367. https://doi.org/10.1007/s11263-011-0421-7
Burger HC, Schuler CJ, Harmeling S (2012) Image denoising with multi-layer perceptrons, part 1: comparison with existing algorithms and with bounds. arXiv:1211.1544
Burger HC, Schuler CJ, Harmeling S (2012) Image denoising with multi-layer perceptrons, part 2: training trade-offs and analysis of their mechanisms. arXiv:1211.1552
Cai B, Xu X, Jia K, Qing C, Tao D (2016) Dehazenet: an end-to-end system for single image haze removal. IEEE Trans Image Process 25(11):5187–5198. https://doi.org/10.1109/TIP.2016.2598681
Chen C, Chen Q, Xu J, Koltun V (2018) Learning to see in the dark. In: CVPR., pp 3291–3300 https://doi.org/10.1109/CVPR.2018.00347
Chen J, Tan C, Hou J, Chau L, Li H (2018) Robust video content alignment and compensation for rain removal in a CNN framework. In: CVPR, pp 6286–6295. https://doi.org/10.1109/CVPR.2018.00658
Chen Y, Hsu C (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: ICCV, pp 1968–1975. https://doi.org/10.1109/ICCV.2013.247
Cui Z, Chang H, Shan S, Zhong B, Chen X (2014) Deep network cascade for image super-resolution. In: ECCV, pp 49–64. https://doi.org/10.1007/978-3-319-10602-1_4
Dong C, Loy CC, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: ECCV, pp 184–199. https://doi.org/10.1007/978-3-319-10593-2_13
Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans Image Process 26(6):2944–2956. https://doi.org/10.1109/TIP.2017.2691802
Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: CVPR, pp 1715–1723. https://doi.org/10.1109/CVPR.2017.186
Garg K, Nayar SK (2004) Detection and removal of rain from videos. In: CVPR, pp 528–535. https://doi.org/10.1109/CVPR.2004.79
Garg K, Nayar SK (2007) Vision and rain. IJCV 75(1):3–27. https://doi.org/10.1007/s11263-006-0028-6
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: CVPR, pp 7132–7141. https://doi.org/10.1109/CVPR.2018.00745
Huang D, Kang L, Yang M, Lin C, Wang YF (2012) Context-aware single image rain removal. In: ICME, pp 164–169. https://doi.org/10.1109/ICME.2012.92
Jain V, Seung HS (2008) Natural image denoising with convolutional networks. In: NIPS, pp 769–776
Johnson J, Alahi A, Fei-Fei L. (2016) Perceptual losses for real-time style transfer and super-resolution. In: ECCV, pp 694–711. https://doi.org/10.1007/978-3-319-46475-6_43
Kang L, Lin C, Fu Y (2012) Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans Image Process 21(4):1742–1755. https://doi.org/10.1109/TIP.2011.2179057
Kim J, Lee C, Sim J, Kim C (2013) Single-image deraining using an adaptive nonlocal means filter. In: ICIP, pp 914–917. https://doi.org/10.1109/ICIP.2013.6738189
Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980
Li G., He X., Zhang W., Chang H., Dong L., Lin L. (2018) Non-locally enhanced encoder-decoder network for single image de-raining. In: ACM MM, pp 1056–1064. https://doi.org/10.1145/3240508.3240636. http://doi.acm.org/10.1145/3240508.3240636
Li S, Araujo IB, Ren W, Wang Z, Tokuda EK, Junior RH, Cesar-Junior R, Zhang J, Guo X, Cao X (2019) Single image deraining: a comprehensive benchmark analysis CVPR
Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: ECCV, pp 262–277. https://doi.org/10.1007/978-3-030-01234-2_16
Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors. In: CVPR, pp 2736–2744. https://doi.org/10.1109/CVPR.2016.299
Liu Z, Yeh RA, Tang X, Liu Y, Agarwala A (2017) Video frame synthesis using deep voxel flow. In: ICCV, pp 4473–4481. https://doi.org/10.1109/ICCV.2017.478
Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding. In: ICCV, pp 3397–3405. https://doi.org/10.1109/ICCV.2015.388
Mustaniemi J, Kannala J, Särkkä S, Matas J, Heikkilä J (2018) Inertial-aided motion deblurring with deep networks. CoRR arXiv:1810.00986
Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) Progressive image deraining networks: a better and simpler baseline. In: CVPR
Ren W, Liu S, Zhang H, Pan J, Cao X, Yang M (2016) Single image dehazing via multi-scale convolutional neural networks. In: ECCV, pp 154–169. https://doi.org/10.1007/978-3-319-46475-6_10
Sharma V, Diba A, Neven D, Brown MS, Gool LV, Stiefelhagen R (2018) Classification-driven dynamic image enhancement. In: CVPR, pp 4033–4041. https://doi.org/10.1109/CVPR.2018.00424
Su S, Delbracio M, Wang J, Sapiro G, Heidrich W, Wang O (2017) Deep video deblurring for hand-held cameras. In: CVPR, pp 237–246. https://doi.org/10.1109/CVPR.2017.33
Tao X, Gao H, Liao R, Wang J, Jia J (2017) Detail-revealing deep video super-resolution. In: ICCV, pp 4482–4490. https://doi.org/10.1109/ICCV.2017.479
Tripathi AK, Mukhopadhyay S (2014) Removal of rain from videos: a review. SIViP 8(8):1421–1430. https://doi.org/10.1007/s11760-012-0373-6
Wang X, Girshick RB, Gupta A, He K (2018) Non-local neural networks. In: CVPR, pp 7794–7803. https://doi.org/10.1109/CVPR.2018.00813
Xu N, Price BL, Cohen S, Huang TS (2017) Deep image matting. In: CVPR, pp 311–320. https://doi.org/10.1109/CVPR.2017.41
Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: CVPR, pp 1685–1694. https://doi.org/10.1109/CVPR.2017.183
Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: CVPR. https://doi.org/10.1109/CVPR.2018.00079om/content_cvpr_2018/html/Zhang_Density-Aware_Single_Image_, pp 695–704
Zhang H, Sindagi V, Patel VM (2017) Image de-raining using a conditional generative adversarial network. arXiv:1701.05957
Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: ECCV, pp 294–310. https://doi.org/10.1007/978-3-030-01234-2_18
Zhu L, Fu C, Lischinski D, Heng P (2017) Joint bi-layer optimization for single-image rain streak removal. In: ICCV, pp 2545–2553. https://doi.org/10.1109/ICCV.2017.276
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This work was supported by the Natural Science Foundation of China [grant numbers 61572099]; Major National Science and Technology Project of China [grant number 2018ZX04016001-011].
Rights and permissions
About this article
Cite this article
Wang, C., Fan, W., Zhu, H. et al. Single image deraining via nonlocal squeeze-and-excitation enhancing network. Appl Intell 50, 2932–2944 (2020). https://doi.org/10.1007/s10489-020-01693-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-020-01693-5