Skip to main content
SpringerLink
Log in

Deep learning-based image de-raining using discrete Fourier transformation

Analyzing behaviour of deep CNNs towards uncorrelated transformed domain data

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Single image rain streak removal is a well-explored topic in the field of computer vision. The de-raining problem is modeled as an image decomposition task where a rainy image is decomposed into rain-free background image and rain streek map. Unlike most of the existing de-raining methods, this paper attempts to decompose the rainy image in the frequency domain. The idea is inspired by pseudo-periodic characteristics of the noise signal (here the rain streaks) which leave some traces in the frequency domain, and the same can be utilized to predict the noise signal. In this paper, a deep learning-based rain streak prediction model is proposed which learns in discrete Fourier transform Oppenheim and Schafer (Discrete-TimeSignal Processing, Prentice Hall, Upper Saddle River, 1989) domain. To the best of our knowledge, this is the first approach where compressed domain coefficients are directly used as input to a deep convolutional neural network. The proposed model has been tested on publicly available synthetic datasets Fu et al. (in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. https://doi.org/10.1109/CVPR.2017.186, Yang et al. (in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. https://doi.org/10.1109/CVPR.2017.183), Yeh et al. (in: 2015 IEEE International Conference on Consumer Electronics-Taiwan, 2015. https://doi.org/10.1109/ICCE-TW.2015.7216999) and results are found to be comparable with the state of the art methods in the spatial domain. The presented analysis and study have an obvious indication to extend transform domain input to train the deep learning architecture especially image de-noising like problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. DFT : Discrete Fourier Transformation.

  2. Figure 5a, e are first converted into YCbCr color space. Magnitude and phase are then calculated by performing DFT [25] on Y channel for each image. Chrominance values of Fig. 5a has been used to construct Fig. 5b, c. Chrominance values of Fig. 5e has been used to construct Fig. 5d.

  3. The quantitative analysis has been given in Appendix 6.1.

  4. Rain-streaks in this test-set may contradict with the real-rain.

References

  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). URL https://www.tensorflow.org/, software available from tensorflow.org

  2. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011). https://doi.org/10.1109/TPAMI.2010.161

    Article  Google Scholar 

  3. Chang, Y., Yan, L., Zhong, S.: Transformed low-rank model for line pattern noise removal. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp 1735–1743 (2017). https://doi.org/10.1109/ICCV.2017.191

  4. Chen, D.Y., Chen, C.C., Kang, L.W.: Visual depth guided color image rain streaks removal using sparse coding. IEEE Trans. Circuits Syst. Video Technol. 24(8), 1430–1455 (2014). https://doi.org/10.1109/TCSVT.2014.2308627

    Article  Google Scholar 

  5. Chen, Q., Yi, X., Ni, B., Shen, Z., Yang, X.: Rain removal via residual generation cascading. In: 2017 IEEE Visual Communications and Image Processing (VCIP), pp 1–4 (2017). https://doi.org/10.1109/VCIP.2017.8305092

  6. Davis, C.: The norm of the schur product operation. Numer. Math. 4(1), 343–344 (1962). https://doi.org/10.1007/BF01386329

    Article  MathSciNet  MATH  Google Scholar 

  7. Dunham, W.: Euler: the Master of Us All. No. v. 22 in Dolciani Mathematical Expositions, Mathematical Association of America (1999). https://books.google.co.in/books?id=uKOVNvGOkhQC

  8. Fu, X., Huang, J., Ding, X., Liao, Y., Paisley, J.: Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans. Image Process. 26(6), 2944–2956 (2017). https://doi.org/10.1109/TIP.2017.2691802

    Article  MathSciNet  MATH  Google Scholar 

  9. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1715–1723 (2017). https://doi.org/10.1109/CVPR.2017.186

  10. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)

    MATH  Google Scholar 

  11. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems—Volume 2, MIT Press, Cambridge, MA, USA, NIPS’14, pp 2672–2680 (2014). http://dl.acm.org/citation.cfm?id=2969033.2969125

  12. Gu, S., Meng, D., Zuo, W., Zhang, L.: Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp 1717–1725 (2017). https://doi.org/10.1109/ICCV.2017.189

  13. Haar, A.: Zur theorie der orthogonalen funktionensysteme. Math Ann 69(3), 331–371 (1910). https://doi.org/10.1007/BF01456326

    Article  MathSciNet  MATH  Google Scholar 

  14. He, K., Sun, J., Tang, X.: Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2011). https://doi.org/10.1109/TPAMI.2010.168

    Article  Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015). arXiv:1512.03385

  16. Hore, A., Ziou, D.: Image quality metrics: Psnr vs. ssim. In: Proceedings of the 2010 20th International Conference on Pattern Recognition, IEEE Computer Society, Washington, DC, USA, ICPR ’10, pp 2366–2369 (2010). https://doi.org/10.1109/ICPR.2010.579

  17. Huang, D.A., Kang, L.W., Wang, Y.C.F., Lin, C.W.: Self-learning based image decomposition with applications to single image denoising. IEEE Trans. Multimed. 16(1), 83–93 (2014). https://doi.org/10.1109/TMM.2013.2284759

    Article  Google Scholar 

  18. Kang, L.W., Lin, C.W., Fu, Y.H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2012). https://doi.org/10.1109/TIP.2011.2179057

    Article  MathSciNet  MATH  Google Scholar 

  19. Katznelson, Y.: An introduction to harmonic analysis. Cambridge Mathematical Library (1976)

  20. Lee, J.H., Heo, M., Kim, K.R., Kim, C.S.: Single-image depth estimation based on Fourier domain analysis. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

  21. Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 2736–2744 (2016). https://doi.org/10.1109/CVPR.2016.299

  22. Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Single image rain streak decomposition using layer priors. IEEE Trans. Image Process. 26(8), 3874–3885 (2017). https://doi.org/10.1109/TIP.2017.2708841

    Article  MathSciNet  MATH  Google Scholar 

  23. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp 3397–3405 (2015). https://doi.org/10.1109/ICCV.2015.388

  24. Lian, N.-X., Zagorodnov, V., Tan, Y.-P.: Edge-preserving image denoising via optimal color space projection. IEEE Trans. Image Process. 15(9), 2575–2587 (2006)

    Article  Google Scholar 

  25. Oppenheim, A.V., Schafer, R.W.: Discrete-Time Signal Processing. Prentice Hall, Upper Saddle River (1989)

    MATH  Google Scholar 

  26. Park, K., Yu, S., Jeong, J.: A contrast restoration method for effective single image rain removal algorithm. In: 2018 International Workshop on Advanced Image Technology (IWAIT), pp 1–4 (2018). https://doi.org/10.1109/IWAIT.2018.8369644

  27. Pratt, H., Williams, B., Coenen, F., Zheng, Y.: Fcnn: Fourier convolutional neural networks. In: Ceci, M., Hollmén, J., Todorovski, L., Vens, C., Džeroski, S. (eds.) Machine Learning and Knowledge Discovery in Databases, pp. 786–798. Springer, Cham (2017)

    Chapter  Google Scholar 

  28. Ren, W., Tian, J., Han, Z., Chan, A., Tang, Y.: Video desnowing and deraining based on matrix decomposition. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 2838–2847 (2017). https://doi.org/10.1109/CVPR.2017.303

  29. Schaefer, G., Stich, M.: UCID: an uncompressed color image database. In: Yeung, M.M., Lienhart, R.W., Li, C.S. (eds.) Storage and Retrieval Methods and Applications for Multimedia 2004, vol 5307, pp 472–480 (2003). https://doi.org/10.1117/12.525375

  30. Sharma, P.K., Jain, P., Sur, A.: Dual-domain single image de-raining using conditional generative adversarial network. In: 2019 IEEE International Conference on Image Processing (ICIP), pp 2796–2800 (2019). https://doi.org/10.1109/ICIP.2019.8803353

  31. Shen, L., Yue, Z., Chen, Q., Feng, F., Ma, J.: Deep joint rain and haze removal from single images (2018). arXiv:1801.06769

  32. Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., Lau, R.W.: Spatial attentive single-image deraining with a high quality real rain dataset. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

  33. Wang, Y., Chen, C., Zhu, S., Zeng, B.: A framework of single-image deraining method based on analysis of rain characteristics. In: 2016 IEEE International Conference on Image Processing (ICIP), pp 4087–4091 (2016). https://doi.org/10.1109/ICIP.2016.7533128

  34. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. Trans. Image Proc. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  35. Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1685–1694 (2017). https://doi.org/10.1109/CVPR.2017.183

  36. Yeh, C.H., Liu, P.H., Yu, C.E., Lin, C.Y.: Single image rain removal based on part-based model. In: 2015 IEEE International Conference on Consumer Electronics—Taiwan, pp 462–463 (2015). https://doi.org/10.1109/ICCE-TW.2015.7216999

  37. Yu, S., Ou, W., You, X., Mou, Y., Jiang, X., Tang, Y.: Single image rain streaks removal based on self-learning and structured sparse representation. In: 2015 IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP), pp 215–219 (2015). https://doi.org/10.1109/ChinaSIP.2015.7230394

  38. Zhang, H., Patel, V.M.: Convolutional sparse and low-rank coding-based rain streak removal. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp 1259–1267 (2017). https://doi.org/10.1109/WACV.2017.145

  39. Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network (2018). arXiv:1802.07412

  40. Zhu, L., Fu, C.W., Lischinski, D., Heng, P.A.: Joint bi-layer optimization for single-image rain streak removal. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp 2545–2553 (2017). https://doi.org/10.1109/ICCV.2017.276

Download references

Acknowledgements

Authors would like to thank the anonymous reviewers for their insightful comments and suggestions. Authors would also like to acknowledge the funding agency, Ministry of Human Resource Development, Government of India.

Author information

Authors and Affiliations

  1. Multimedia Laboratory, Department of Computer Science and Engineering, Indian Institute of Technology, Guwahati, India

    Prasen Kumar Sharma, Sathisha Basavaraju & Arijit Sur

Authors
  1. Prasen Kumar Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sathisha Basavaraju
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arijit Sur
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Prasen Kumar Sharma.

Ethics declarations

Conflict of interest

Authors of this manuscript declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

1.1 Quantitative analysis of rain-streaks present in Y channel compared to Cb & Cr

Deep learning models for image restoration, in RGB color-space due to its highly correlated nature, may induce white pixel artifacts and color saturation, especially in the case of image de-raining [30]. YCbCr is a more suitable color-space for image restoration when the noise in an image exhibits linear or a pseudo-periodic nature [24, 30]. To quantify the noise present in the Y channel compared to chrominance channels, we have adopted the concept of sparsity and conducted the following experiment on the test-set TD- Zhang et al. that consists of 1200 rainy-clean image pairs.

  1. 1.

    We convert the rainy and clean images into YCbCr color-space.

  2. 2.

    Obtains the pixel-wise difference between corresponding Y Cb Cr channels of rainy and clean images.

  3. 3.

    Measure the sparsity ratio based on the following equation

    $$\begin{aligned} \mathcal {S} = \frac{\text {No. of Zero-pixels}}{\text {Total No. of pixels}} \end{aligned}$$
    (19)
  4. 4.

    High sparsity indicates the low rain-noise present in the channel compared to its clean counterpart. We have obtained the following results. The same has been added in the appendix with qualitative results.

    Metric

    Y

    Cr

    Cb

    \(\mathcal {S}\) (avg.)

    0.0449

    0.4481

    0.4040

  5. 5.

    It can be observed that the Y channel has the lowest sparsity ratio that indicates the highest rain-streak noise present compared to other channels.

  6. 6.

    Although the values are test-set specific, it may obey for any rain-streak removal test-set.

1.2 Run-time comparison

We have implemented the proposed model using Tensorflow framework [1]. It takes \(\sim \) 0.56 s to test an image of size \(512 \times 512\) on an 8 GB GPU. We have also compared the proposed model with existing schemes based on the run-time per image, and results are shown in Table 5.

Table 5 Run-time comparison of the proposed work with existing schemes
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, P.K., Basavaraju, S. & Sur, A. Deep learning-based image de-raining using discrete Fourier transformation. Vis Comput 37, 2083–2096 (2021). https://doi.org/10.1007/s00371-020-01971-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01971-w

Keywords

Search

Navigation