Skip to main content
SpringerLink
Log in

Rain removal system for dynamic scene in diminished reality

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Visibility in the rain is poor. Here, we develop a rain removal system for dynamic scene in diminished reality. Our system provides a clear and comfortable view. We apply and extend chromatic prior and explore chromatic pair, dark tail removal and spatial chromatic prior. We design an optimized intuitive alignment to solve moving camera problem. We apply anti-flicker to completely remove drizzle and defocused rain. Benefit from parallel pixel-based chromatic prior, our system has high scalability and extensibility. Our experiments reference numerous videos from prior related works, involving realistic and diverse property of rain, motion of the scene and motion of the camera. The experimental results show a satisfactory rain removal quality, especially for drizzle and defocused rain.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

References

  1. Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.: Recent advances in augmented reality. IEEE Comput. Graph. Appl. 21(6), 34–47 (2001)

    Article  Google Scholar 

  2. Herling, J., Broll, W.: Advanced self-contained object removal for realizing real-time diminished reality in unconstrained environments. In: 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 207–212, (2010)

  3. Leao, C.W.M., Lima, J.P., Teichrieb, V., Albuquerque, E.S., Kelner, J.: Altered reality: augmenting and diminishing reality in real time. In: IEEE Virtual Reality Conference (VR), pp. 219–220, (2011)

  4. Zokai, S., Esteve, J., Genc, Y., Navab, N.: Multiview paraperspective projection model for diminished reality. In: The 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 217–226, (2003)

  5. Kawai, N., Sato, T., Yokoya, N.: Diminished reality considering background structures. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 259–260, (2013)

  6. Korkalo, O., Aittala, M., Siltanen, S.: Light-weight marker hiding for augmented reality. In: 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 247–248, (2010)

  7. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch. ACM Trans. Graph. 28(3), 1 (2009)

    Article  Google Scholar 

  8. Bossu, J., Hautière, N., Tarel, J.-P.: Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int. J. Comput. Vis. 93(3), 348–367 (2011)

    Article  Google Scholar 

  9. Barnum, P., Kanade, T., Narasimhan, S.: Spatio-temporal frequency analysis for removing rain and snow from videos. In: Proceedings of the 1st International Workshop on Photometric Analysis for Computer Vision, (2007)

  10. Barnum, P.C., Narasimhan, S., Kanade, T.: Analysis of rain and snow in frequency space. Int. J. Comput. Vis. 86(2–3), 256–274 (2009)

    Google Scholar 

  11. Zhang, X., Li, H., Qi, Y., Leow, W., Ng, T.: Rain removal in video by combining temporal and chromatic properties. In: IEEE International Conference on Multimedia and Expo, pp. 461–464, (2006)

  12. Huang, D.-A., Kang, L.-W., Yang, M.-C., Lin, C.-W., Wang, Y.-C.F.: Context-aware single image rain removal. In: IEEE International Conference on Multimedia and Expo, pp. 164–169, (2012)

  13. Fu, Y.-H., Kang, L.-W., Lin, C.-W., Hsu, C.-T.: Single-frame-based rain removal via image decomposition. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1453–1456, (2011)

  14. Kang, L.-W., Lin, C.-W., Fu, Y.-H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–55 (2012)

    Article  MathSciNet  Google Scholar 

  15. Kang, L.-W., Lin, C.-W., Lin, C.-T., Lin, Y.-C.: Self-learning-based rain streak removal for image/video. In: IEEE International Symposium on Circuits and Systems, pp. 1871–1874, (2012)

  16. Xu, J., Zhao, W., Liu, P., Tang, X.: Removing rain and snow in a single image using guided filter. In: IEEE International Conference on Computer Science and Automation Engineering (CSAE), pp. 304–307, (2012)

  17. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–409 (2013)

    Article  Google Scholar 

  18. Garg, K., Nayar, S.K.: When does a camera see rain? In: 10th IEEE International Conference on Computer Vision (ICCV’05) vol. 1, pp. 1067–1074, (2005)

  19. Garg, K., Nayar, S.K.: Vision and rain. Int. J. Comput. Vis. 75(1), 3–27 (2007)

    Article  Google Scholar 

  20. de Charette, R., Tamburo, R., Barnum, P.C., Rowe, A., Kanade, T., Narasimhan, S.G.: Fast reactive control for illumination through rain and snow. In: IEEE International Conference on Computational Photography, pp. 1–10, (2012)

  21. Qian, R., Tan, R.T., Yang, W., Su, J., Liu, J.: Attentive generative adversarial network for raindrop removal from a single image, In: IEEE Conference on Computer Vision and Pattern Recognition, (2018)

  22. Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: IEEE Conference on Computer Vision and Pattern Recognition, (2018)

  23. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: IEEE Conference on Computer Vision and Pattern Recognition, (2017)

  24. Jarusirisawad, S., Saito, H.: Diminished reality via multiple hand-held CAMERAS. In: 1st ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC), pp. 251–258, (2007)

  25. Seo, B.-K., Lee, M.-H., Park, H., Park, J.-I.: Projection-based diminished reality system. In: International Symposium on Ubiquitous Virtual Reality, pp. 25–28, (2008)

  26. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  27. “Speed of a Falling Raindrop.” http://hypertextbook.com/facts/2007/EvanKaplan.shtml. Accessed 20 Sep 2018

  28. Mohler, B.J., Thompson, W.B., Creem-Regehr, S.H., Pick, H.L., Warren, W.H.: Visual flow influences gait transition speed and preferred walking speed. Exp. Brain Res. 181(2), 221–8 (2007)

    Article  Google Scholar 

  29. Garg, K., Nayar, S.K.: Photorealistic rendering of rain streaks. In: ACM SIGGRAPH, vol. 25, no. 3, p. 996, (2006)

  30. Gonda, F.: “Rain Simulator.” http://rain.felixgonda.com/. Accessed 20 Sep 2018

  31. Rousseau, P., Jolivet, V., Ghazanfarpour, D.: Realistic real-time rain rendering. Comput. Graph. 30(4), 507–518 (2006)

    Article  Google Scholar 

  32. “Valve.” http://www.valvesoftware.com/. Accessed 20 Sep 2018

Download references

Acknowledgements

This work of Professor Damon Shing-Min Liu was supported in part by the Ministry of Science and Technology of Taiwan under Grant Number NSC102-2221-E-194-048.

Author information

Authors and Affiliations

  1. Computer Science Department, National Chung Cheng University, 168 University Road, Chiayi, Taiwan

    Damon Shing-Min Liu & Yi-Jhih Chen

Authors
  1. Damon Shing-Min Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yi-Jhih Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Damon Shing-Min Liu.

Ethics declarations

Conflict of interest

Mr. Yi-Jhih Chen has no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, D.SM., Chen, YJ. Rain removal system for dynamic scene in diminished reality. SIViP 14, 945–953 (2020). https://doi.org/10.1007/s11760-019-01626-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-019-01626-x

Keywords

Search

Navigation