Advertisement

Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining

  • Xia Li
  • Jianlong Wu
  • Zhouchen Lin
  • Hong LiuEmail author
  • Hongbin Zha
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11211)

Abstract

Rain streaks can severely degrade the visibility, which causes many current computer vision algorithms fail to work. So it is necessary to remove the rain from images. We propose a novel deep network architecture based on deep convolutional and recurrent neural networks for single image deraining. As contextual information is very important for rain removal, we first adopt the dilated convolutional neural network to acquire large receptive field. To better fit the rain removal task, we also modify the network. In heavy rain, rain streaks have various directions and shapes, which can be regarded as the accumulation of multiple rain streak layers. We assign different alpha-values to various rain streak layers according to the intensity and transparency by incorporating the squeeze-and-excitation block. Since rain streak layers overlap with each other, it is not easy to remove the rain in one stage. So we further decompose the rain removal into multiple stages. Recurrent neural network is incorporated to preserve the useful information in previous stages and benefit the rain removal in later stages. We conduct extensive experiments on both synthetic and real-world datasets. Our proposed method outperforms the state-of-the-art approaches under all evaluation metrics. Codes and supplementary material are available at our project webpage: https://xialipku.github.io/RESCAN.

Keywords

Recurrent neural network Squeeze and excitation block Image deraining 

Notes

Acknowledgment

Zhouchen Lin is supported by National Basic Research Program of China (973 Program) (Grant no. 2015CB352502), National Natural Science Foundation (NSF) of China (Grant nos. 61625301 and 61731018), Qualcomm, and Microsoft Research Asia. Hong Liu is supported by National Natural Science Foundation of China (Grant nos. U1613209 and 61673030). Hongbin Zha is supported by Beijing Municipal Natural Science Foundation (Grant no. 4152006).

Supplementary material

474212_1_En_16_MOESM1_ESM.pdf (15.9 mb)
Supplementary material 1 (pdf 16269 KB)

References

  1. 1.
    Zhang, X., Li, H., Qi, Y., Leow, W.K., Ng, T.K.: Rain removal in video by combining temporal and chromatic properties. In: IEEE ICME, pp. 461–464 (2006)Google Scholar
  2. 2.
    Garg, K., Nayar, S.K.: Vision and rain. Int. J. Comput. Vis. 75(1), 3–27 (2007)CrossRefGoogle Scholar
  3. 3.
    Santhaseelan, V., Asari, V.K.: Utilizing local phase information to remove rain from video. Int. J. Comput. Vis. 112(1), 71–89 (2015)CrossRefGoogle Scholar
  4. 4.
    Tripathi, A.K., Mukhopadhyay, S.: Removal of rain from videos: a review. Signal Image Video Process. 8(8), 1421–1430 (2014)CrossRefGoogle Scholar
  5. 5.
    Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: IEEE ICCV, pp. 3397–3405 (2015)Google Scholar
  6. 6.
    Chang, Y., Yan, L., Zhong, S.: Transformed low-rank model for line pattern noise removal. In: IEEE ICCV, pp. 1726–1734 (2017)Google Scholar
  7. 7.
    Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: IEEE CVPR, pp. 2736–2744 (2016)Google Scholar
  8. 8.
    Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: IEEE CVPR, pp. 1715–1723 (2017)Google Scholar
  9. 9.
    Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: IEEE CVPR, pp. 1357–1366 (2017)Google Scholar
  10. 10.
    Huang, D.A., Kang, L.W., Yang, M.C., Lin, C.W., Wang, Y.C.F.: Context-aware single image rain removal. In: IEEE ICME, pp. 164–169 (2012)Google Scholar
  11. 11.
    Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 (2015)
  12. 12.
    Chen, Q., Xu, J., Koltun, V.: Fast image processing with fully-convolutional networks. In: IEEE ICCV, pp. 2516–2525 (2017)Google Scholar
  13. 13.
    Li, R., Cheong, L.F., Tan, R.T.: Single image deraining using scale-aware multi-stage recurrent network. arXiv preprint arXiv:1712.06830 (2017)
  14. 14.
    Barnum, P.C., Narasimhan, S., Kanade, T.: Analysis of rain and snow in frequency space. Int. J. Comput. Vis. 86(2–3), 256 (2010)CrossRefGoogle Scholar
  15. 15.
    Garg, K., Nayar, S.K.: Detection and removal of rain from videos. In: IEEE CVPR, vol. 1, pp. 528–535 (2004)Google Scholar
  16. 16.
    Garg, K., Nayar, S.K.: When does a camera see rain? In: IEEE ICCV, vol. 2, pp. 1067–1074 (2005)Google Scholar
  17. 17.
    Bossu, J., Hautière, N., Tarel, J.P.: Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int. J. Comput. Vis. 93(3), 348–367 (2011)CrossRefGoogle Scholar
  18. 18.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online dictionary learning for sparse coding. In: ICML, pp. 689–696 (2009)Google Scholar
  19. 19.
    Reynolds, D.A., Quatieri, T.F., Dunn, R.B.: Speaker verification using adapted Gaussian mixture models. Digit. Signal Process. 10(1–3), 19–41 (2000)CrossRefGoogle Scholar
  20. 20.
    Liu, G., Lin, Z., Yan, S., Sun, J., Yu, Y., Ma, Y.: Robust recovery of subspace structures by low-rank representation. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 171–184 (2013)CrossRefGoogle Scholar
  21. 21.
    Kang, L.W., Lin, C.W., Fu, Y.H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2012)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Wang, Y., Liu, S., Chen, C., Zeng, B.: A hierarchical approach for rain or snow removing in a single color image. IEEE Trans. Image Process. 26(8), 3936–3950 (2017)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Gu, S., Meng, D., Zuo, W., Zhang, L.: Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: IEEE ICCV, pp. 1717–1725 (2017)Google Scholar
  24. 24.
    Zhu, L., Fu, C.W., Lischinski, D., Heng, P.A.: Joint bi-layer optimization for single-image rain streak removal. In: IEEE CVPR, pp. 2526–2534 (2017)Google Scholar
  25. 25.
    Fu, X., Huang, J., Ding, X., Liao, Y., Paisley, J.: Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans. Image Process. 26(6), 2944–2956 (2017)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Joint rain detection and removal via iterative region dependent multi-task learning. CoRR, abs/1609.07769 (2016)Google Scholar
  27. 27.
    Zhang, H., Sindagi, V., Patel, V.M.: Image de-raining using a conditional generative adversarial network. arXiv preprint arXiv:1701.05957 (2017)
  28. 28.
    Kaushal, H., Jain, V., Kar, S.: Free-space optical channel models. In: Kaushal, H., Jain, V.K., Kar, S. (eds.) Free Space Optical Communication, pp. 41–89. Springer, New Delhi (2017).  https://doi.org/10.1007/978-81-322-3691-7_2CrossRefGoogle Scholar
  29. 29.
    Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. arXiv preprint arXiv:1709.01507 (2017)
  30. 30.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML, pp. 448–456 (2015)Google Scholar
  31. 31.
    Mandic, D.P., Chambers, J.A., et al.: Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability. Wiley Online Library (2001)Google Scholar
  32. 32.
    Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
  33. 33.
    Zaremba, W., Sutskever, I., Vinyals, O.: Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 (2014)
  34. 34.
    Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: ICML, vol. 30, p. 3 (2013)Google Scholar
  35. 35.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  36. 36.
    Huynh-Thu, Q., Ghanbari, M.: Scope of validity of PSNR in image/video quality assessment. IET Electron. Lett. 44(13), 800–801 (2008)CrossRefGoogle Scholar
  37. 37.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Key Laboratory of Machine Perception, Shenzhen Graduate SchoolPeking UniversityBeijingChina
  2. 2.Key Laboratory of Machine Perception (MOE), School of EECSPeking UniversityBeijingChina
  3. 3.Cooperative Medianet Innovation CenterShanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations