Analysis of Rain and Snow in Frequency Space

  • Peter C. Barnum
  • Srinivasa Narasimhan
  • Takeo Kanade
Article

Abstract

Dynamic weather such as rain and snow causes complex spatio-temporal intensity fluctuations in videos. Such fluctuations can adversely impact vision systems that rely on small image features for tracking, object detection and recognition. While these effects appear to be chaotic in space and time, we show that dynamic weather has a predictable global effect in frequency space. For this, we first develop a model of the shape and appearance of a single rain or snow streak in image space. Detecting individual streaks is difficult even with an accurate appearance model, so we combine the streak model with the statistical characteristics of rain and snow to create a model of the overall effect of dynamic weather in frequency space. Our model is then fit to a video and is used to detect rain or snow streaks first in frequency space, and the detection result is then transferred to image space. Once detected, the amount of rain or snow can be reduced or increased. We demonstrate that our frequency analysis allows for greater accuracy in the removal of dynamic weather and in the performance of feature extraction than previous pixel-based or patch-based methods. We also show that unlike previous techniques, our approach is effective for videos with both scene and camera motions.

Keywords

De-weathering Image enhancement Noise removal 

References

  1. Auer, A. H. Jr. (1972). Distribution of graupel and hail with size. Monthly Weather Review, 100(5), 325–328. Google Scholar
  2. Auer, A. H. Jr., & Veal, D. L. (1970). The dimension of ice crystals in natural clouds. Journal of the Atmospheric Sciences, 27(6), 919–926. CrossRefGoogle Scholar
  3. Barnum, P., Kanade, T., & Narasimhan, S. (2007). Spatio-temporal frequency analysis for removing rain and snow from videos. In Workshop on photometric analysis for computer vision, in conjunction with international conference on computer vision. Google Scholar
  4. Böhm, H. P. (1989). A general equation for the terminal fall speed of solid hydrometeors. Journal of the Atmospheric Sciences, 46, 2419–27. CrossRefGoogle Scholar
  5. Bouguet, J.-Y. (2000). Pyramidal implementation of the Lucas Kanade feature tracker. Intel Corporation, Microprocessor Research Labs. Google Scholar
  6. Cozman, F., & Krotkov, E. (1997). Depth from scattering. In International conference on computer vision. Google Scholar
  7. de la Torre, F., & Black, M. L. (2001). Robust principal component analysis for computer vision. In International conference on computer vision. Google Scholar
  8. Desaulniers-Soucy, N. (1999). Empirical test of the multifractal continuum limit in rain. PhD thesis, Mcgill. Google Scholar
  9. Desaulniers-Soucy, N., Lovejoy, S., & Schertzer, D. (2001). The HYDROP experiment: an empirical method for the determination of the continuum limit in rain. Atmospheric Research, 59–60, 163–197. CrossRefGoogle Scholar
  10. Elgammal, A., Harwood, D., & Davis, L. (2000). Non-parametric model for background subtraction. In European conference on computer vision. Google Scholar
  11. Feingold, G., & Levin, Z. (1986). The lognormal fit to raindrop spectra from frontal convective clouds in Israel. Journal of Climate and Applied Meteorology, 25, 1346–63. CrossRefGoogle Scholar
  12. Fishler, M., & Bolles, R. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. In Communications of the ACM. Google Scholar
  13. Foote, G. B., & duToit, P. S. (1969). Terminal velocity of raindrops aloft. Journal of Applied Meteorology, 8(2), 249–53. CrossRefGoogle Scholar
  14. Garg, K., & Nayar, S. K. (2004). Detection and removal of rain from videos. In Computer vision and pattern recognition. Google Scholar
  15. Garg, K., & Nayar, S. K. (2005). When does a camera see rain? In International conference on computer vision. Google Scholar
  16. Garg, K., & Nayar, S. K. (2006). Photorealistic rendering of rain streaks. In SIGGRAPH. Google Scholar
  17. Gonzalez, R. C., & Woods, R. E. (2002). Digital image processing (2nd ed.). New York: Prentice Hall. Google Scholar
  18. Gunn, K., & Marshall, J. (1958). The distribution with size of aggregate snowflakes. Journal of Meteorology, 15, 452–461. Google Scholar
  19. Hase, H., Miyake, K., & Yoneda, M. (1999). Real-time snowfall noise elimination. Google Scholar
  20. Heeger, D. (1987). Optical flow from spatiotemporal filters. In International conference on computer vision. Google Scholar
  21. Jameson, A. R., & Kostinski, A. B. (2001). What is a raindrop size distribution? Bulletin of the American Meteorological Society, 8(6), 1169–1177. CrossRefGoogle Scholar
  22. Jameson, A., & Kostinski, A. (2002). When is rain steady? Journal of Applied Meteorology, 41(1), 83–90. CrossRefGoogle Scholar
  23. Ke, Q., & Kanade, T. (2002). A robust subspace approach to layer extraction. In IEEE workshop on motion and video computing. Google Scholar
  24. Kubesh, R. J., & Beard, K. (1993). Laboratory measurements of spontaneous oscillations of moderate-size raindrops. Journal of the Atmospheric Sciences, 50, 1089–1098. CrossRefGoogle Scholar
  25. Langer, M., & Mann, R. (2003). Optical snow. International Journal of Computer Vision, 55(1), 55–71. CrossRefGoogle Scholar
  26. Langer, M., & Zhang, Q. (2003). Rendering falling snow using and inverse Fourier transform. In ACM SIGGRAPH technical sketches program. Google Scholar
  27. Langer, M. S., Zhang, L., Klein, A., Bhatia, A., Pereira, J., & Rekhi, D. (2004). A spectral-particle hybrid method for rendering falling snow. In Eurographics symposium on rendering. Google Scholar
  28. Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 20, 91–110. CrossRefGoogle Scholar
  29. Magono, C., & Nakamura, T. (1965). Aerodynamic studies of falling snowflakes. Journal of the Meteorological Society of Japan, 43, 139–147. Google Scholar
  30. Marshall, J., & Palmer, W. (1948). The distribution of raindrops with size. Journal of Meteorology, 5, 165–166. Google Scholar
  31. Mikolajczyk, K., & Schmid, C. (2001). Indexing based on scale invariant interest points. In International conference on computer vision. Google Scholar
  32. Narasimhan, S. G., & Nayar, S. K. (2002). Vision and the atmosphere. International Journal of Computer Vision, 48(3), 233–254. MATHCrossRefGoogle Scholar
  33. Nayar, S. K., & Narasimhan, S. G. (1999). Vision in bad weather. In International conference on computer vision. Google Scholar
  34. Ohtake, T. (1965). Preliminary observations on size distribution of snowflakes and raindrops at just above and below the melting layer. In International conference on cloud physics. Google Scholar
  35. Pruppacher, H. R., & Klett, J. D. (1997). Microphysics of clouds and precipitation. Amsterdam: Kluwer Academic. Second revised and enlarged edition. Google Scholar
  36. Reeves, W. T. (1983). Particle systems—a technique for modeling a class of fuzzy objects. ACM Transactions on Graphics, 2(2), 91–108. CrossRefGoogle Scholar
  37. Sand, P., & Teller, S. (2006). Particle video: long-range motion estimation using point trajectories. In Computer vision and pattern recognition. Google Scholar
  38. Shi, J., & Tomasi, C. (1994). Good features to track. In Computer vision and pattern recognition. Google Scholar
  39. Starik, S., & Werman, M. (2003). Simulation of rain in videos. In International workshop on texture analysis and synthesis. Google Scholar
  40. Stauffer, C., & Grimson, W. (1998). Adaptive background mixture models for real-time tracking. In Computer vision and pattern recognition. Google Scholar
  41. Tariq, S. (2007). Rain (Technical report). NVIDIA. Google Scholar
  42. Tatarchuk, N., & Isidoro, J. (2006). Artist-directable real-time rain rendering in city environments. In Eurographics workshop on natural phenomena. Google Scholar
  43. Tokay, A., & Beard, K. (1996). A field study of raindrop oscillations. Part I: Observation of size spectra and evaluation of oscillation causes. Journal of Applied Meteorology, 35, 1671–1687. CrossRefGoogle Scholar
  44. Torr, P. H. S., Szeliski, R., & Anandan, P. (1999). An integrated Bayesian approach to layer extraction from image sequences. In International conference on computer vision. Google Scholar
  45. Ulbrich, C. W. (1983). Natural variations in the analytical form of the raindrop size distribution. Journal of Applied Meteorology, 22(10), 1764–1775. CrossRefGoogle Scholar
  46. Van de Hulst, H. (1957). Light scattering by small particles. New York: Wiley. Google Scholar
  47. Zelnik-Manor, L., Machline, M., & Irani, M. (2006). Multi-body factorization with uncertainty: Revisiting motion consistency. International Journal of Computer Vision, 68(1), 27–41. CrossRefGoogle Scholar
  48. Zhang, X., Li, H., Qi, Y., Kheng, W., & Ng, T. K. (2006). Rain removal in video by combining temporal and chromatic properties. In International conference on multimedia and expo. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Peter C. Barnum
    • 1
  • Srinivasa Narasimhan
    • 1
  • Takeo Kanade
    • 1
  1. 1.Carnegie Mellon UniversityPittsburghUSA

Personalised recommendations