Skip to main content
SpringerLink
Log in

A Modified Syn2Real Network for Nighttime Rainy Image Restoration

  • Conference paper
  • First Online:
Advances in Visual Computing (ISVC 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12510))

Included in the following conference series:

  • 1677 Accesses

Abstract

The restoration or enhancement of rainy images at nighttime is of great significance to outdoor computer vision applications such as self-driving and traffic surveillance. While image deraining has drawn increasingly research attention currently and the majority of deraining methods are able to achieve satisfying performance for daytime image rain removal, there are few related studies for nighttime image deraining, as the conditions of nighttime rainy scenes are more complicated and challenging. To address the nighttime image deraining issues, we designed an improved model based on the Syn2Real network, called NIRR. In order to obtain good rain removal and visual effect under the nighttime rainy scene, we propose a new refined loss function for the supervised learning phase, which combines the perceptual loss and SSIM loss. The qualitative and quantitative experimental results show that our proposed method outperforms the state-of-the-arts whether it is on the synthetic nighttime rainy image or on the real-world nighttime rainy image.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bahnsen, C.H., Moeslund, T.B.: Rain removal in traffic surveillance: does it matter? IEEE Trans. Intell. Transp. Syst. 20(8), 2802–2819 (2018)

    Article  Google Scholar 

  2. Chen, D., et al.: Gated context aggregation network for image dehazing and deraining. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1375–1383. IEEE (2019)

    Google Scholar 

  3. Chen, Y., Li, W., Sakaridis, C., Dai, D., Van Gool, L.: Domain adaptive faster R-CNN for object detection in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3339–3348 (2018)

    Google Scholar 

  4. Deng, L.J., Huang, T.Z., Zhao, X.L., Jiang, T.X.: A directional global sparse model for single image rain removal. Appl. Math. Model. 59, 662–679 (2018)

    Article  MathSciNet  Google Scholar 

  5. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3855–3863 (2017)

    Google Scholar 

  6. Gao, S., Cheng, M.M., Zhao, K., Zhang, X.Y., Yang, M.H., Torr, P.H.: Res2Net: a new multi-scale backbone architecture. IEEE Trans. Pattern Anal. Mach. Intell., 1 (2019)

    Google Scholar 

  7. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    Chapter  Google Scholar 

  8. Kang, L.W., Lin, C.W., Fu, Y.H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2011)

    Article  MathSciNet  Google Scholar 

  9. Li, R., Cheong, L.F., Tan, R.T.: Heavy rain image restoration: integrating physics model and conditional adversarial learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1633–1642 (2019)

    Google Scholar 

  10. Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2736–2744 (2016)

    Google Scholar 

  11. Liang, M., Yang, B., Wang, S., Urtasun, R.: Deep continuous fusion for multi-sensor 3D object detection. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 641–656 (2018)

    Google Scholar 

  12. Loh, Y.P., Chan, C.S.: Getting to know low-light images with the exclusively dark dataset. Comput. Vis. Image Underst. 178, 30–42 (2019)

    Article  Google Scholar 

  13. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3397–3405 (2015)

    Google Scholar 

  14. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8026–8037 (2019)

    Google Scholar 

  15. Perera, A.G., Wei Law, Y., Chahl, J.: UAV-gesture: a dataset for UAV control and gesture recognition. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)

    Google Scholar 

  16. Qian, R., Tan, R.T., Yang, W., Su, J., Liu, J.: Attentive generative adversarial network for raindrop removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2482–2491 (2018)

    Google Scholar 

  17. Rasmussen, C.E.: Gaussian processes in machine learning. In: Bousquet, O., von Luxburg, U., Rätsch, G. (eds.) ML -2003. LNCS (LNAI), vol. 3176, pp. 63–71. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-28650-9_4

    Chapter  Google Scholar 

  18. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)

    Google Scholar 

  19. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  20. Shi, Z., Feng, Y., Zhao, M., He, L.: A joint deep neural networks-based method for single nighttime rainy image enhancement. Neural Comput. Appl., 1–14 (2019)

    Google Scholar 

  21. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  22. Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6479–6488 (2018)

    Google Scholar 

  23. Sun, S.H., Fan, S.P., Wang, Y.C.F.: Exploiting image structural similarity for single image rain removal. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 4482–4486. IEEE (2014)

    Google Scholar 

  24. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  25. Wei, W., Meng, D., Zhao, Q., Xu, Z., Wu, Y.: Semi-supervised transfer learning for image rain removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3877–3886 (2019)

    Google Scholar 

  26. Wu, Y., Lim, J., Yang, M.H.: Online object tracking: a benchmark. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2411–2418 (2013)

    Google Scholar 

  27. Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1357–1366 (2017)

    Google Scholar 

  28. Yang, W., Tan, R.T., Wang, S., Fang, Y., Liu, J.: Single image deraining: from model-based to data-driven and beyond. IEEE Trans. Pattern Anal. Mach. Intell. (2020)

    Google Scholar 

  29. Yasarla, R., Sindagi, V.A., Patel, V.M.: Syn2Real transfer learning for image deraining using gaussian processes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2726–2736 (2020)

    Google Scholar 

  30. Zhang, H., Sindagi, V., Patel, V.M.: Image de-raining using a conditional generative adversarial network. IEEE Trans. Circ. Syst. Video Technol., 3943–3956 (2019)

    Google Scholar 

  31. Zhu, L., Fu, C.W., Lischinski, D., Heng, P.A.: Joint bi-layer optimization for single-image rain streak removal. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2526–2534 (2017)

    Google Scholar 

Download references

Acknowledgment

The authors would like to thank the authors of compared papers, who provided the original codes, and the anonymous reviewers for their insightful comments and valuable suggestions. This work was supported by National Natural Science Foundation of China (No. 51879211), Hunan Provincial Natural Science Foundation of China (No. 2017JJ3053), and Hunan Provincial Science Research Project of China (Nos. 17A051).

Author information

Authors and Affiliations

  1. Hubei Key Laboratory of Broadband Wireless Communication and Sensor Networks, Wuhan University of Technology, Wuhan, China

    Qunfang Tang, Jie Yang & Zhiqiang Guo

  2. School of Electrical Information Engineering, Hunan Institute of Technology, Hengyang, China

    Qunfang Tang & Haibo Liu

Authors
  1. Qunfang Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jie Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haibo Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhiqiang Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Yang .

Editor information

Editors and Affiliations

  1. University of Nevada Reno, Reno, NV, USA

    George Bebis

  2. Stony Brook University, Stony Brook, NY, USA

    Zhaozheng Yin

  3. Drexel University, Philadelphia, PA, USA

    Edward Kim

  4. RWTH Aachen University, Aachen, Germany

    Jan Bender

  5. University of Edinburgh, Edinburgh, UK

    Kartic Subr

  6. IBM Research – Cambridge, Cambridge, MA, USA

    Bum Chul Kwon

  7. University of Waterloo, Waterloo, ON, Canada

    Jian Zhao

  8. Graz University of Technology, Graz, Austria

    Denis Kalkofen

  9. The Hong Kong Polytechnic University, Hong Kong, Hong Kong

    George Baciu

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tang, Q., Yang, J., Liu, H., Guo, Z. (2020). A Modified Syn2Real Network for Nighttime Rainy Image Restoration. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2020. Lecture Notes in Computer Science(), vol 12510. Springer, Cham. https://doi.org/10.1007/978-3-030-64559-5_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-64559-5_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-64558-8

  • Online ISBN: 978-3-030-64559-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation