International Journal of Computer Vision

, Volume 76, Issue 2, pp 123–139 | Cite as

Nonlocal Image and Movie Denoising

Article

Abstract

Neighborhood filters are nonlocal image and movie filters which reduce the noise by averaging similar pixels. The first object of the paper is to present a unified theory of these filters and reliable criteria to compare them to other filter classes. A CCD noise model will be presented justifying the involvement of neighborhood filters. A classification of neighborhood filters will be proposed, including classical image and movie denoising methods and discussing further a recently introduced neighborhood filter, NL-means. In order to compare denoising methods three principles will be discussed. The first principle, “method noise”, specifies that only noise must be removed from an image. A second principle will be introduced, “noise to noise”, according to which a denoising method must transform a white noise into a white noise. Contrarily to “method noise”, this principle, which characterizes artifact-free methods, eliminates any subjectivity and can be checked by mathematical arguments and Fourier analysis. “Noise to noise” will be proven to rule out most denoising methods, with the exception of neighborhood filters. This is why a third and new comparison principle, the “statistical optimality”, is needed and will be introduced to compare the performance of all neighborhood filters.

The three principles will be applied to compare ten different image and movie denoising methods. It will be first shown that only wavelet thresholding methods and NL-means give an acceptable method noise. Second, that neighborhood filters are the only ones to satisfy the “noise to noise” principle. Third, that among them NL-means is closest to statistical optimality. A particular attention will be paid to the application of the statistical optimality criterion for movie denoising methods. It will be pointed out that current movie denoising methods are motion compensated neighborhood filters. This amounts to say that they are neighborhood filters and that the ideal neighborhood of a pixel is its trajectory. Unfortunately the aperture problem makes it impossible to estimate ground true trajectories. It will be demonstrated that computing trajectories and restricting the neighborhood to them is harmful for denoising purposes and that space-time NL-means preserves more movie details.

Keywords

Image denoising Movie denoising Motion estimation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alvarez, L., Guichard, F., Lions, P. L., & Morel, J. M. (1993). Axioms and fundamental equations of image processing. Archive for Rational Mechanics and Analysis, 123, 199–257. MATHCrossRefMathSciNetGoogle Scholar
  2. Attneave, F. (1954). Some informational aspects of visual perception. Psychological Review, 61, 183–193. CrossRefGoogle Scholar
  3. Awate, S. P. & Whitaker, R. T. (2005). Higher-order image statistics for unsupervised, information-theoretic, adaptive, image filtering. In Proceedings of the 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05) (Vol. 2, pp. 44–51). Google Scholar
  4. Azzabou, N., Paragios, N., & Guichard, F. (2006). Random walks, constrained multiple hypothesis testing and image enhancement. In ECCV (Vol. 1, pp. 379–390). Google Scholar
  5. Boulanger, J., Kervrann, C., & Bouthemy, P. (2006). Adaptive space-time patch-based method for image sequence restoration. In Proceedings of the ECCV’06 workshop on statistical methods in multi-image and video processing (SMVP’06), Graz, Austria, May 2006. Google Scholar
  6. Brailean, J. C., Kleihorst, R. P., Efsratiadis, S., Katsaggelos, A. K., & Lagendijk, R. L. (1995). Noise reduction filters for dynamic image sequences: a review. Proceedings of the IEEE, 83, 1272–1292. CrossRefGoogle Scholar
  7. Buades, A., Coll, B., & Morel, J. M. (2005a). A review of image denoising methods, with a new one. Multiscale Modeling and Simulation, 4(2), 490–530. MATHCrossRefMathSciNetGoogle Scholar
  8. Buades, A., Coll, B., & Morel, J. M. (2005b). A non-local algorithm for image denoising, In IEEE international conference on computer vision and pattern recognition. Google Scholar
  9. Buades, A., Coll, B., & Morel, J. M. (2005c). Denoising image sequences does not require motion estimation. Preprint, CMLA, N 2005-18, May 2005. http://www.cmla.ens-cachan.fr/Cmla/.
  10. Colleen Gino, M. (2004). “Noise, noise, noise”. http://www.astrophys-assist.com/educate/noise/noise.htm.
  11. Cremers, D., & Grady, L. (2006). Statistical priors for efficient combinatorial optimization via graph cuts. In European conference on computer vision. Google Scholar
  12. Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2006, submitted). Image denoising by sparse 3D transform-domain collaborative filtering. IEEE Transactions on Image Processing. Google Scholar
  13. Donoho, D., & Johnstone, I. (1994). Ideal spatial adaptation via wavelet shrinkage. Biometrika, 81, 425–455. MATHCrossRefMathSciNetGoogle Scholar
  14. Efros, A., & Leung, T. (1999). Texture synthesis by nonparametric sampling. In Proceedings of the international conference on computer vision (ICCV 99) (Vol. 2, pp. 1033–1038). Google Scholar
  15. Gilboa, G., & Osher, S. (2006). Nonlocal linear image regularization and supervised segmentation. UCLA CAM Report 06-47. Google Scholar
  16. Gilboa, G., Darbon, J., Osher, S., & Chan, T. F. (2006). Nonlocal convex functionals for image regularization. UCLA CAM Report 06-57. Google Scholar
  17. Gonzalez, R. C., & Woods, R. E. (2002). Digital image processing (2nd ed.). New York: Prentice Hall. Google Scholar
  18. Horn, B., & Schunck, B. (1981). Determining optical flow. Artificial Intelligence, 17, 185–203. CrossRefGoogle Scholar
  19. Howell, S. B. (2000). Handbook of CCD astronomy. Cambridge: Cambridge University Press. Google Scholar
  20. Huang, T. (1981). Image sequence analysis. Berlin: Springer. MATHGoogle Scholar
  21. Keeling, S. L., & Stollberger, R. (2002). Nonlinear anisotropic diffusion filtering for multiscale edge enhancement. Inverse Problems, 18, 175–190. MATHCrossRefMathSciNetGoogle Scholar
  22. Kervrann, C., & Boulanger, J. (2006). Unsupervised patch-based image regularization and representation. In Proceedings of the European conference on computer vision (ECCV’06), Graz, Austria, May 2006. Google Scholar
  23. Kindermann, S., Osher, S., & Jones, P. W. (2005). Deblurring and denoising of images by nonlocal functionals. Multiscale Modeling and Simulation, 4(4), 1091–1115. MATHCrossRefMathSciNetGoogle Scholar
  24. Kokaram, A. C. (1993). Motion picture restoration. PhD thesis, Cambridge University. Google Scholar
  25. Lee, J. S. (1980). Digital image enhancement and noise filtering by use of local statistics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2, 165–168. CrossRefGoogle Scholar
  26. Lee, J. S. (1983). Digital image smoothing and the sigma filter. Computer Vision, Graphics and Image Processing, 24, 255–269. CrossRefGoogle Scholar
  27. Liu, C., Freeman, W. T., Szeliski, R., & Kang, S. B. (2006). Noise estimation from a single image. In CVPR. Google Scholar
  28. Mahmoudi, M., & Sapiro, G. (2005). Fast image and video denoising via non-local means of similar neighborhoods. IEEE Signal Processing Letters, 12(12), 839–842. CrossRefGoogle Scholar
  29. Martinez, D. M. (1986). Model-based motion estimation and its application to restoration and interpolation of motion pictures. PhD thesis, Massachusetts Institute of Technology. Google Scholar
  30. Merriman, B., Bence, J., & Osher, S. (1992). Diffusion generated motion by mean curvature. In Proceedings of the geometry center workshop. Google Scholar
  31. Meyer, Y. (2002). Oscillating patterns in image processing and nonlinear evolution equations. In AMS university lecture series (Vol. 22). Google Scholar
  32. Nagel, H. H. (1983). Constraints for the estimation of displacement vector fields from image sequences. In Proceedings of the eighth international joint conference on artificial intelligence (IJCAI ’83) (pp. 945–951). Google Scholar
  33. Osher, S., Burger, M., Goldfarb, D., Xu, J., & Yin, W. (2005). An iterative regularization method for total variation based image restoration. Multiscale Modelling and Simulation, 4, 460–489. MATHCrossRefMathSciNetGoogle Scholar
  34. Ozkan, M. K., Sezan, M. I., & Tekalp, A. M. (1993). Adaptive motion compensated filtering of noisy image sequences. IEEE Transactions on Circuits and Systems for Video Technology, 3, 277–290. CrossRefGoogle Scholar
  35. Papenberg, N., Bruhn, A., Brox, T., Didas, S., & Weickert, J. (2006). Highly accurate optic flow computation with theoretically justified warping. International Journal of Computer Vision, 67(2), 141–158. CrossRefGoogle Scholar
  36. Rudin, L., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D, 60, 259–268. MATHCrossRefGoogle Scholar
  37. Samy, R. (1985). An adaptive image sequence filtering scheme based on motion detection. SPIE, 596, 135–144. Google Scholar
  38. Sezan, M. I., Ozkan, M. K., & Fogel, S. V. (1991). Temporally adaptive filtering of noisy sequences using a robust motion estimation algorithm. In Proceedings of the international conference on acoustics, speech, signal processing (Vol. 91, pp. 2429–2432). Google Scholar
  39. Smith, S. M., & Brady, J. M. (1997). Susan—new approach to low level image processing. International Journal of Computer Vision, 23(1), 45–78. CrossRefGoogle Scholar
  40. Tadmor, E., Nezzar, S., & Vese, L. (2004). A multiscale image representation using hierarchical (BV,L 2) decompositions. Multiscale Modeling and Simulation, 2, 554–579. MATHCrossRefMathSciNetGoogle Scholar
  41. Tomasi, C., & Manduchi, R. (1998). Bilateral filtering for gray and color images. In Sixth international conference on computer vision (pp. 839–846). Google Scholar
  42. Tukey, J. (1977). Exploratory data analysis. Reading: Addison-Wesley. MATHGoogle Scholar
  43. Weickert, J. (1998). On discontinuity-preserving optic flow. In Proceedings of the computer vision and mobile robotics workshop (pp. 115–122). Google Scholar
  44. Weickert, J., & Schnörr, C. (2001). Variational optic flow computation with a spatio-temporal smoothness constraint. Journal of Mathematical Imaging and Vision, 14, 245–255. MATHCrossRefGoogle Scholar
  45. Yaroslavsky, L. P. (1985). Digital picture processing—an introduction. Berlin: Springer. MATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  • Antoni Buades
    • 1
  • Bartomeu Coll
    • 1
  • Jean-Michel Morel
    • 1
  1. 1.CMLA (Mathematics)Ecole Normale Supérieure, de CachanCachanFrance

Personalised recommendations