Abstract
Existing video synthetic models and deraining methods are mostly built on a simplified video rain model assuming that rain streak layers of different video frames are uncorrelated, thereby producing degraded performance on real-world rainy videos. To address this problem, we devise a new video rain sy nthesis model with the concept of rain streak motions to enforce a consistency of rain layers between video frames, thereby generating more realistic rainy video data for network training, and then develop a recurrent disentangled deraining network (RDD-Net) based on our video rain model for boosting video deraining. More specifically, taking adjacent frames of a key frame as the input, our RDD-Net recurrently aggregates each adjacent frame and the key frame by a fusion module, and then devise a disentangle model to decouple the fused features by predicting not only a clean background layer and a rain layer, but also a rain streak motion layer. After that, we develop three attentive recovery modules to combine the decoupled features from different adjacent frames for predicting the final derained result of the key frame. Experiments on three widely-used benchmark datasets and a collected dataset, as well as real-world rainy videos show that our RDD-Net quantitatively and qualitatively outperforms state-of-the-art deraining methods. Our code, our dataset, and our results on four datasets are released at https://github.com/wangshauitj/RDD-Net.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Barnum, P.C., Narasimhan, S., Kanade, T.: Analysis of rain and snow in frequency space. IJCV 86(2–3), 256–274 (2010)
Bossu, J., Hautière, N., Tarel, J.P.: Rain or snow detection in image sequences through use of a histogram of orientation of streaks. IJCV 93(3), 348–367 (2011)
Brewer, N., Liu, N.: Using the shape characteristics of rain to identify and remove rain from video. In: da Vitoria Lobo, N., et al. (eds.) SSPR /SPR 2008. LNCS, vol. 5342, pp. 451–458. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-89689-0_49
Chen, J., Chau, L.P.: A rain pixel recovery algorithm for videos with highly dynamic scenes. IEEE TIP 23(3), 1097–1104 (2013)
Chen, J., Tan, C.H., Hou, J., Chau, L.P., Li, H.: Robust video content alignment and compensation for rain removal in a CNN framework. In: CVPR, pp. 6286–6295 (2018)
Chen, Y.L., Hsu, C.T.: A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: ICCV, pp. 1968–1975 (2013)
Fu, X., Huang, J., Ding, X., Liao, Y., Paisley, J.: Clearing the skies: a deep network architecture for single-image rain removal. IEEE TIP 26(6), 2944–2956 (2017)
Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: CVPR, pp. 3855–3863 (2017)
Garg, K., Nayar, S.K.: Detection and removal of rain from videos. In: CVPR, pp. 528–535 (2004)
Garg, K., Nayar, S.K.: Photorealistic rendering of rain streaks. ACM Trans. Graph. (TOG) 25(3), 996–1002 (2006)
Haris, M., Shakhnarovich, G., Ukita, N.: Recurrent back-projection network for video super-resolution. In: CVPR (2019)
Huang, D.A., Kang, L.W., Wang, Y.C.F., Lin, C.W.: Self-learning based image decomposition with applications to single image denoising. IEEE Trans. Multimed. 16(1), 83–93 (2014)
Jiang, K., et al.: Multi-scale progressive fusion network for single image deraining. In: CVPR, pp. 8346–8355 (2020)
Jiang, T.X., Huang, T.Z., Zhao, X.L., Deng, L.J., Wang, Y.: A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors. In: CVPR, pp. 4057–4066 (2017)
Jiang, T.X., Huang, T.Z., Zhao, X.L., Deng, L.J., Wang, Y.: FastDeRain: a novel video rain streak removal method using directional gradient priors. IEEE TIP 28(4), 2089–2102 (2018)
Kang, L.W., Lin, C.W., Fu, Y.H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE TIP 21(4), 1742–1755 (2012)
Kim, J.H., Sim, J.Y., Kim, C.S.: Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE TIP 24(9), 2658–2670 (2015)
Li, G., He, X., Zhang, W., Chang, H., Dong, L., Lin, L.: Non-locally enhanced encoder-decoder network for single image de-raining. In: ACM Multimedia, pp. 1056–1064 (2018)
Li, M., et al.: Video rain streak removal by multiscale convolutional sparse coding. In: CVPR, pp. 6644–6653 (2018)
Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: ECCV, pp. 254–269 (2018)
Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: CVPR, pp. 2736–2744 (2016)
Liu, J., Yang, W., Yang, S., Guo, Z.: D3R-Net: dynamic routing residue recurrent network for video rain removal. IEEE TIP 28(2), 699–712 (2018)
Liu, J., Yang, W., Yang, S., Guo, Z.: Erase or fill? Deep joint recurrent rain removal and reconstruction in videos. In: CVPR, pp. 3233–3242 (2018)
Liu, P., Xu, J., Liu, J., Tang, X.: Pixel based temporal analysis using chromatic property for removing rain from videos. Comput. Inf. Sci. 2(1), 53–60 (2009)
Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: ICCV, pp. 3397–3405 (2015)
Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: a better and simpler baseline. In: CVPR, pp. 3937–3946 (2019)
Ren, W., Tian, J., Han, Z., Chan, A., Tang, Y.: Video desnowing and deraining based on matrix decomposition. In: CVPR, pp. 4210–4219 (2017)
Santhaseelan, V., Asari, V.K.: Utilizing local phase information to remove rain from video. IJCV 112(1), 71–89 (2015)
Sun, S.H., Fan, S.P., Wang, Y.C.F.: Exploiting image structural similarity for single image rain removal. In: IEEE ICIP, pp. 4482–4486 (2014)
Tripathi, A.K., Mukhopadhyay, S.: A probabilistic approach for detection and removal of rain from videos. IETE J. Res. 57(1), 82–91 (2011)
Tripathi, A., Mukhopadhyay, S.: Video post processing: low-latency spatiotemporal approach for detection and removal of rain. IET Image Proc. 6(2), 181–196 (2012)
Wang, C., Xing, X., Wu, Y., Su, Z., Chen, J.: DCSFN: deep cross-scale fusion network for single image rain removal. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 1643–1651 (2020)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE TIP 13(4), 600–612 (2004)
Wei, W., Yi, L., Xie, Q., Zhao, Q., Meng, D., Xu, Z.: Should we encode rain streaks in video as deterministic or stochastic? In: ICCV, pp. 2516–2525 (2017)
Yan, W., Tan, R.T., Yang, W., Dai, D.: Self-aligned video deraining with transmission-depth consistency. In: CVPR, pp. 11966–11976 (2021)
Yang, W., Liu, J., Feng, J.: Frame-consistent recurrent video deraining with dual-level flow. In: CVPR, pp. 1661–1670 (2019)
Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Joint rain detection and removal from a single image. In: CVPR, pp. 1685–1694 (2017)
Yang, W., Tan, R.T., Wang, S., Liu, J.: Self-learning video rain streak removal: when cyclic consistency meets temporal correspondence. In: CVPR, pp. 1720–1729 (2020)
Yue, Z., Xie, J., Zhao, Q., Meng, D.: Semi-supervised video deraining with dynamical rain generator. In: CVPR, pp. 642–652 (2021)
Zamir, S.W., et al.: Multi-stage progressive image restoration. In: CVPR (2021)
Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: CVPR, pp. 695–704 (2018)
Zhang, X., Li, H., Qi, Y., Leow, W.K., Ng, T.K.: Rain removal in video by combining temporal and chromatic properties. In: IEEE International Conference on Multimedia and Expo, pp. 461–464 (2006)
Zhu, L., et al.: Learning gated non-local residual for single-image rain streak removal. IEEE Trans. Circuits Syst. Video Technol. 31(6), 2147–2159 (2020)
Zhu, L., Fu, C.W., Lischinski, D., Heng, P.A.: Joint bi-layer optimization for single-image rain streak removal. In: ICCV, pp. 2526–2534 (2017)
Acknowledgement
The work is supported by the National Natural Science Foundation of China (Grant No. 61902275, NSFC-U1803264), and a grant of Hong Kong Research Grants Council under General Research Fund (no. 15205919).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, S. et al. (2022). Rethinking Video Rain Streak Removal: A New Synthesis Model and a Deraining Network with Video Rain Prior. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13679. Springer, Cham. https://doi.org/10.1007/978-3-031-19800-7_33
Download citation
DOI: https://doi.org/10.1007/978-3-031-19800-7_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-19799-4
Online ISBN: 978-3-031-19800-7
eBook Packages: Computer ScienceComputer Science (R0)