Skip to main content

Advertisement

SpringerLink
Log in

Rain Streaks and Snowflakes Removal for Video Sequences via Motion Compensation and Matrix Completion

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Image and video deraining tasks aim to reconstruct original scenes, from which human vision and computer vision systems can better identify objects and more details present in images and video sequences. This paper proposes a three-step method to detect and remove rain streaks, even snowflakes from great majority video sequences, using motion compensation and low-rank matrix completion method. Firstly, we adopt the optical flow estimation between consecutive frames to detect the motion of rain streaks. We then employ the online dictionary learning for sparse representation technique, and SVM classifier to eliminate parts that are not rain streaks. Finally, we reconstruct the video sequence by using low-rank matrix completion techniques. In particular, by introducing the image dehazing network(GCANet) to our proposed method, the heavy rain caused dense rain accumulation and blurry phenomenon can be worked out well. The experimental results demonstrate the proposed algorithm and perform qualitatively and quantitatively better in several image quality metrics, boosting the best published PSNR metric by 4.47%, 6.05% on two static video sequences and 12.13% on a more challenging dynamic video sequence. In addition, to demonstrate the generality of the proposed method, we further apply it to two challenge tasks, which also achieves state-of-the-art performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Notes

  1. https://github.com/cddlyf/GCANet.

  2. https://github.com/wwzjer/RainRemoval_ICCV2017.

  3. https://github.com/MinghanLi/MS-CSC-Rain-Streak-Removal.

  4. https://github.com/liruoteng/HeavyRainRemoval.

  5. https://github.com/csdwren/PReNet.

  6. https://github.com/rui1996/DeRaindrop.

  7. https://youtu.be/5Jht7tqTbe8

  8. https://cloud.google.com/vision/.

References

  1. Abdel-Hakim AE. A novel approach for rain removal from videos using low-rank recovery. In: 2014 5th international conference on intelligent systems, modelling and simulation, 2014; pp. 351–356. IEEE.

  2. Ancuti CO, Ancuti C, Sbert M, Timofte R. Dense-haze: a benchmark for image dehazing with dense-haze and haze-free images. In: 2019 IEEE international conference on image processing (ICIP), 2019; pp. 1014–1018. IEEE.

  3. Bai X, Yang M, Huang T, Dou Z, Yu R, Xu Y. Deep-person: learning discriminative deep features for person re-identification. Pattern Recogn. 2020;98:107036.

    Article  Google Scholar 

  4. Barnum PC, Narasimhan SG, Kanade T. Analysis of rain and snow in frequency space. Int J Comput Vis. 2008;86:256–74.

    Article  Google Scholar 

  5. Bossu J, Hautière N, Tarel JP. Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int J Comput Vis. 2011;93(3):348–67.

    Article  Google Scholar 

  6. Cai JF, Candès EJ, Shen Z. A singular value thresholding algorithm for matrix completion. SIAM J Optim. 2010;20(4):1956–82.

    Article  MathSciNet  MATH  Google Scholar 

  7. Candès EJ, Recht B. Exact matrix completion via convex optimization. Found Comput Math. 2009;9(6):717.

    Article  MathSciNet  MATH  Google Scholar 

  8. Chen D, He M, Fan Q, Liao J, Zhang L, Hou D, Yuan L, Hua G. Gated context aggregation network for image dehazing and deraining. In: 2019 IEEE winter conference on applications of computer vision (WACV), 2019; pp. 1375–1383. IEEE.

  9. Chen J, Tan CH, Hou J, Chau LP, Li H. Robust video content alignment and compensation for clear vision through the rain. 2018. arXiv:1804.09555.

  10. Comaniciu D, Ramesh V, Meer P. Kernel-based object tracking. IEEE Trans Pattern Anal Mach Intell. 2003;25(5):564–77.

    Article  Google Scholar 

  11. Eigen D, Krishnan D, Fergus R. Restoring an image taken through a window covered with dirt or rain. In: Proceedings of the IEEE international conference on computer vision, 2013. pp. 633–640.

  12. Elad M. Sparse and redundant representations: from theory to applications in signal and image processing. Berlin: Springer Science & Business Media; 2010.

    Book  MATH  Google Scholar 

  13. Elad M, Aharon M. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans Image Process. 2006;15(12):3736–45.

    Article  MathSciNet  Google Scholar 

  14. Fabbri C, Islam MJ, Sattar J. Enhancing underwater imagery using generative adversarial networks. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018;7159–7165. IEEE.

  15. Fadili MJ, Starck JL, Bobin J, Moudden Y. Image decomposition and separation using sparse representations: an overview. Proc IEEE. 2009;98(6):983–94.

    Article  Google Scholar 

  16. Garg K, Nayar SK. Detection and removal of rain from videos. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004. 2004;1:I–I.

  17. Garg K, Nayar SK. When does a camera see rain? In: Tenth IEEE international conference on computer vision (ICCV’05) Volume 1, 2005; vol. 2, pp. 1067–1074.

  18. Garg K, Nayar SK. Vision and rain. Int J Comput Vis. 2006;75:3–27.

    Article  Google Scholar 

  19. George J, Bhavani S, Jaya J. Certain explorations on removal of rain streaks using morphological component analysis. Int J Eng Res Technol (IJERT). 2013;2(2):2278–81.

    Google Scholar 

  20. Gonzalez RC, Woods RE, Eddins SL. Digital image processing using MATLAB. Pearson Education India, 2004.

  21. Horn BK, Schunck BG. Determining optical flow. In: Techniques and Applications of Image Understanding, 1981;281:319–331. International Society for Optics and Photonics

  22. Huynh-Thu Q, Ghanbari M. Scope of validity of psnr in image/video quality assessment. Electron Lett. 2008;44(13):800–1.

    Article  Google Scholar 

  23. Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell. 1998;20(11):1254–9.

    Article  Google Scholar 

  24. Jiang TX, Huang TZ, Zhao XL, Deng LJ, Wang Y. Fastderain: a novel video rain streak removal method using directional gradient priors. IEEE Trans Image Process. 2018;28(4):2089–102.

    Article  MathSciNet  Google Scholar 

  25. Joshi KD, Chauhan V, Surgenor B. A flexible machine vision system for small part inspection based on a hybrid svm/ann approach. J Intell Manuf. 2020;31(1):103–25.

    Article  Google Scholar 

  26. Kang LW, Lin CW, Fu YH. Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans Image Process. 2011;21(4):1742–55.

    Article  MathSciNet  MATH  Google Scholar 

  27. Kim JH, Sim JY, Kim CS. Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Trans Image Process. 2015;24:2658–70.

    Article  MathSciNet  MATH  Google Scholar 

  28. Kuanar S, Rao K, Mahapatra D, Bilas M. Night time haze and glow removal using deep dilated convolutional network, 2019. arXiv:1902.00855.

  29. Li J, Skinner KA, Eustice RM, Johnson-Roberson M. Watergan: unsupervised generative network to enable real-time color correction of monocular underwater images. IEEE Robot Autom Lett. 2017;3(1):387–94.

    Google Scholar 

  30. Li M, Xie Q, Zhao Q, Wei W, Gu S, Tao J, Meng D. Video rain streak removal by multiscale convolutional sparse coding. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 2018; pp. 6644–6653.

  31. Li R, Cheong LF, Tan RT. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019; pp. 1633–1642.

  32. Li Y, Tan RT, Guo X, Lu J, Brown, MS. Rain streak removal using layer priors. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016; pp. 2736–2744.

  33. Li Y, Tan RT, Guo X, Lu J, Brown MS. Single image rain streak decomposition using layer priors. IEEE Trans Image Process. 2017;26(8):3874–85.

    Article  MathSciNet  MATH  Google Scholar 

  34. Liu C, et al. Beyond pixels: exploring new representations and applications for motion analysis. Ph.D. thesis, Massachusetts Institute of Technology, 2009.

  35. Liu J, Yang W, Yang S, Guo Z. D3r-net: dynamic routing residue recurrent network for video rain removal. IEEE Trans Image Process. 2018;28(2):699–712.

    Article  MathSciNet  MATH  Google Scholar 

  36. Liu J, Yang W, Yang S, Guo Z. Erase or fill? deep joint recurrent rain removal and reconstruction in videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018; pp. 3233–3242.

  37. Liu P, Xu J, Liu J, Tang X. Pixel based temporal analysis using chromatic property for removing rain from videos. Comput Inf Sci. 2009;2:53–60.

    Google Scholar 

  38. Luo Y, Xu Y, Ji H. Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE international conference on computer vision, 2015; pp. 3397–3405.

  39. Mittal A, Soundararajan R, Bovik AC. Making a completely blind image quality analyzer. IEEE Signal Process Lett. 2012;20(3):209–12.

    Article  Google Scholar 

  40. Mukhopadhyay S, Tripathi AK. Combating bad weather part i: rain removal from video. Synth Lecture Image Video Multimedia Process. 2014;7(2):1–93.

    Article  Google Scholar 

  41. Narasimhan SG, Nayar SK. Vision and the atmosphere. Int J Comput Vis. 2002;48(3):233–54.

    Article  MATH  Google Scholar 

  42. Qian R, Tan RT, Yang W, Su J, Liu J. Attentive generative adversarial network for raindrop removal from a single image. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 2017; pp. 2482–2491.

  43. Qin X, Wang Z, Bai Y, Xie X, Jia H. Ffa-net: feature fusion attention network for single image dehazing, 2019. arXiv:1911.07559.

  44. Ramya C, Rani SS. Rain removal in image sequence using sparse coding. In: International conference on intelligent robotics, automation, and manufacturing, 2012; pp. 361–370. Springer.

  45. Rao SR, Ni KY, Owechko Y. Video scene analysis system for situational awareness (2020). US Patent 10,528,818.

  46. Ren D, Zuo W, Hu Q, Zhu P, Meng D. Progressive image deraining networks: a better and simpler baseline. In: IEEE Conference on Computer Vision and Pattern Recognition, 2019.

  47. Ren W, Tian J, Han Z, Chan A, Tang Y. Video desnowing and deraining based on matrix decomposition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017; pp. 4210–4219.

  48. Ren W, Zhang J, Xu X, Ma L, Cao X, Meng G, Liu W. Deep video dehazing with semantic segmentation. IEEE Trans Image Process. 2018;28(4):1895–908.

    Article  MathSciNet  Google Scholar 

  49. Srebro N, Jaakkola T. Weighted low-rank approximations. In: Proceedings of the 20th International Conference on Machine Learning (ICML-03), 2003; pp. 720–727.

  50. Tan CH, Chen J, Chau LP. Dynamic scene rain removal for moving cameras. In: 2014 19th International Conference on Digital Signal Processing, 2014; pp. 372–376. IEEE.

  51. Tripathi A, Mukhopadhyay S. Video post processing: low-latency spatiotemporal approach for detection and removal of rain. IET Image Proc. 2012;6(2):181–96.

    Article  MathSciNet  Google Scholar 

  52. Tripathi AK, Mukhopadhyay S. Removal of rain from videos: a review. SIViP. 2014;8:1421–30.

    Article  Google Scholar 

  53. Wang H, Li M, Wu Y, Zhao Q, Meng D. A survey on rain removal from video and single image, 2019. arXiv:1909.08326.

  54. Wang T, Yang X, Xu K, Chen S, Zhang Q, Lau RW. Spatial attentive single-image deraining with a high quality real rain dataset. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

  55. Wang Z, Bovik AC. A universal image quality index. IEEE Signal Process Lett. 2002;9(3):81–4.

    Article  Google Scholar 

  56. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process. 2004;13(4):600–12.

    Article  Google Scholar 

  57. Wei W, Yi L, Xie Q, Zhao Q, Meng D, Xu Z. Should we encode rain streaks in video as deterministic or stochastic? 2017 IEEE international conference on computer vision (ICCV) 2017; pp. 2535–2544.

  58. Xu H, Li B, Ramanishka V, Sigal L, Saenko K. Joint event detection and description in continuous video streams. In: 2019 IEEE winter conference on applications of computer vision (WACV), 2019; pp. 396–405. IEEE.

  59. Zhang J, Cao Y, Fang S, Kang Y, Wen Chen C. Fast haze removal for nighttime image using maximum reflectance prior. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017; pp. 7418–7426

  60. Zhang L, Zhang L, Mou X, Zhang D. Fsim: a feature similarity index for image quality assessment. IEEE Trans Image Process. 2011;20(8):2378–86.

    Article  MathSciNet  MATH  Google Scholar 

  61. Zhang X, Li H, Qi Y, Leow WK, Ng TK. Rain removal in video by combining temporal and chromatic properties. In: 2006 IEEE international conference on multimedia and expo, 2006; pp. 461–464. IEEE.

  62. Zhang Z, Xu Y, Yang J, Li X, Zhang D. A survey of sparse representation: algorithms and applications. IEEE Access. 2015;3:490–530.

    Article  Google Scholar 

  63. Zhou Y, Shimada N. Using motion compensation and matrix completion algorithm to remove rain streaks and snow for video sequences. In: Pattern Recognition, 2020; pp. 91–104. Springer International Publishing, Cham.

Download references

Author information

Authors and Affiliations

  1. College of Information Science and Engineering, Ritsumeikan University, 1-1-1 Noji-higashi, Kusatsu, Shiga, 525-8577, Japan

    Yutong Zhou & Nobutaka Shimada

Authors
  1. Yutong Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nobutaka Shimada
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yutong Zhou.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection “Machine Learning in Pattern Analysis” guest edited by Reinhard Klette, Brendan McCane, Gabriella Sanniti di Baja, Palaiahnakote Shivakumara and Liang Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, Y., Shimada, N. Rain Streaks and Snowflakes Removal for Video Sequences via Motion Compensation and Matrix Completion. SN COMPUT. SCI. 1, 328 (2020). https://doi.org/10.1007/s42979-020-00333-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-020-00333-6

Keywords

Search

Navigation