Advertisement

Rethinking Image Deraining via Rain Streaks and Vapors

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12362)

Abstract

Single image deraining regards an input image as a fusion of a background image, a transmission map, rain streaks, and atmosphere light. While advanced models are proposed for image restoration (i.e., background image generation), they regard rain streaks with the same properties as background rather than transmission medium. As vapors (i.e., rain streaks accumulation or fog-like rain) are conveyed in the transmission map to model the veiling effect, the fusion of rain streaks and vapors do not naturally reflect the rain image formation. In this work, we reformulate rain streaks as transmission medium together with vapors to model rain imaging. We propose an encoder-decoder CNN named as SNet to learn the transmission map of rain streaks. As rain streaks appear with various shapes and directions, we use ShuffleNet units within SNet to capture their anisotropic representations. As vapors are brought by rain streaks, we propose a VNet containing spatial pyramid pooling (SSP) to predict the transmission map of vapors in multi-scales based on that of rain streaks. Meanwhile, we use an encoder CNN named ANet to estimate atmosphere light. The SNet, VNet, and ANet are jointly trained to predict transmission maps and atmosphere light for rain image restoration. Extensive experiments on the benchmark datasets demonstrate the effectiveness of the proposed visual model to predict rain streaks and vapors. The proposed deraining method performs favorably against state-of-the-art deraining approaches.

Keyword

Deep image deraining 

Notes

Acknowledgement

This work was supported by NSFC (60906119) and Shanghai Pujiang Program.

Supplementary material

Supplementary material 1 (avi 15351 KB)

504472_1_En_22_MOESM2_ESM.pdf (12.1 mb)
Supplementary material 2 (pdf 12434 KB)

References

  1. 1.
    Chen, D., Chen, C.-C., Kang, L.: Visual depth guided color image rain streaks removal using sparse coding. IEEE Trans. Circuits Syst. Video Technol. 24(8), 1430–1455 (2014)CrossRefGoogle Scholar
  2. 2.
    Chen, Y.L., Hsu, C.T.: A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: IEEE International Conference on Computer Vision (ICCV 2013), Sydney, Australia, pp. 1968–1975. IEEE, December 2013Google Scholar
  3. 3.
    Fu, X., Huang, J., Ding, X., Liao, Y., Paisley, J.: Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans. Image Process. 26(6), 2944–2956 (2017)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA, pp. 1715–1723. IEEE, July 2017Google Scholar
  5. 5.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). IEEE, July 2015Google Scholar
  6. 6.
    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: International Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, USA, pp. 6629–6640, December 2017Google Scholar
  7. 7.
    Hu, X., Fu, C.-W., Zhu, L., Heng, P.-A.: Depth-attentional features for single-image rain removal. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019), Long Beach CA, USA. IEEE, June 2019Google Scholar
  8. 8.
    Huang, D.A., Kang, L.W., Wang, Y.C.F., Lin, C.W.: Self-learning based image decomposition with applications to single image denoising. IEEE Trans. Multimedia 16(1), 83–93 (2014)CrossRefGoogle Scholar
  9. 9.
    Huang, D.A., Kang, L., Yang, M.C., Lin, C.W., Wang, Y.C.F.: Context-aware single image rain removal. In: IEEE International Conference on Multimedia and Expo (ICME 2012), Melbourne, Australia, pp. 164–169. IEEE, July 2012Google Scholar
  10. 10.
    Jiang, T.X., Huang, T.Z., Zhao, X.L., Deng, L.J., Wang, Y.: A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA, pp. 2818–2827. IEEE, July 2017Google Scholar
  11. 11.
    Kang, L., Lin, C.-W., Fu, Y.-H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2012)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: the 3rd International Conference for Learning Representations (ICLR 2015), San Diego. IEEE (2015)Google Scholar
  13. 13.
    Li, R., Cheong, L.F., Tan, R.T.: Heavy rain image restoration: integrating physics model and conditional adversarial learning. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019), Long Beach CA, USA. IEEE, June 2019Google Scholar
  14. 14.
    Li, S., Ren, W., Zhang, J., Yu, J., Guo, X.: Fast single image rain removal via a deep decomposition-composition network. Comput. Vis. Image Underst. 186, 48–57 (2018)CrossRefGoogle Scholar
  15. 15.
    Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 262–277. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01234-2_16CrossRefGoogle Scholar
  16. 16.
    Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), Las Vegas, Nevada, USA, pp. 2736–2744, June 2016Google Scholar
  17. 17.
    Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: IEEE International Conference on Computer Vision (ICCV 2015), Boston, MA, USA, pp. 3397–3405, December 2015Google Scholar
  18. 18.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online learning for matrix factorization and sparse coding. J. Mach. Learn. Res. 11, 19–60 (2010)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: a better and simpler baseline. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019), Long Beach CA, USA. IEEE, June 2019Google Scholar
  20. 20.
    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), pp. 2818–2826, Las Vegas, NV, USA, June 2016Google Scholar
  21. 21.
    Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., Lau, R.W.: Spatial attentive single-image deraining with a high quality real rain dataset. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019), Long Beach CA, USA. IEEE, June 2019Google Scholar
  22. 22.
    Wang, Y., Chen, C., Zhu, S., Zeng, B.: A framework of single-image deraining method based on analysis of rain characteristics. In: IEEE International Conference on Image Processing (ICIP 2013), Phoenix, USA, pp. 4087–4091. IEEE, September 2016Google Scholar
  23. 23.
    Wang, Y., Liu, S., Chen, C., Zeng, B.: A hierarchical approach for rain or snow removing in a single color image. IEEE Trans. Image Process. 26(8), 3936–3950 (2017)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  25. 25.
    Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA, pp. 1685–1694. IEEE, July 2017Google Scholar
  26. 26.
    Yang, W., Tan, R.T., Feng, J., Liu, J., Yan, S., Guo, Z.: Joint rain detection and removal from a single image with contextualized deep networks. IEEE Trans. Pattern Anal. Mach. Intell. 42(6), 1377–1393 (2019)CrossRefGoogle Scholar
  27. 27.
    Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, pp. 1685–1694. IEEE, July 2018Google Scholar
  28. 28.
    Zhang, X., Zhou, X., Lin, M., Sun, J.: ShuffleNet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856 (2018) Google Scholar
  29. 29.
    Zhang, X., Li, H., Qi, Y., Leow, W.K., Ng, T.K.: Rain removal in video by combining temporal and chromatic properties. In: IEEE International Conference on Multimedia and Expo (ICME 2006), Toronto, Ontario, Canada, vol. 1, pp. 461–464. IEEE, July 2006Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of Electronic Science and Technology of ChinaChengduChina
  2. 2.Tencent AI LabShenzhenChina
  3. 3.MoE Key Lab of Artificial Intelligence, AI InstituteShanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations