Abstract
We review the evolution of the nonparametric regression modeling in imaging from the local Nadaraya-Watson kernel estimate to the nonlocal means and further to transform-domain filtering based on nonlocal block-matching. The considered methods are classified mainly according to two main features: local/nonlocal and pointwise/multipoint. Here nonlocal is an alternative to local, and multipoint is an alternative to pointwise. These alternatives, though obvious simplifications, allow to impose a fruitful and transparent classification of the basic ideas in the advanced techniques. Within this framework, we introduce a novel single- and multiple-model transform domain nonlocal approach. The Block Matching and 3-D Filtering (BM3D) algorithm, which is currently one of the best performing denoising algorithms, is treated as a special case of the latter approach.
Similar content being viewed by others
References
Abramovich, F., & Benjamini, Y. (1996). Adaptive thresholding of wavelet coefficients. Computational Statistics and Data Analysis, 22, 351–361.
Abramovich, F., Benjamini, Y., Donoho, D., & Johnstone, I. (2006). Adapting to unknown sparsity by controlling the false discovery rate. The Annals of Statistics, 34(2), 584–653.
Aharon, M., Elad, M., & Bruckstein, A. M. (2006). The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing, 54(11), 4311–4322.
Alexander, S., Kovacic, S., & Vrscay, E. (2007). A simple model for image self-similarity and the possible use of mutual information. In Proc. 15th Eur. signal process. conf., EUSIPCO 2007, Poznan, Poland, 2007.
Astola, J., & Yaroslavsky, L. (Eds.) (2007). EURASIP book series on signal processing and communications : Vol. 7. Advances in signal transforms: theory and applications. Cairo: Hindawi Publishing.
Barash, D. (2002). A fundamental relationship between bilateral filtering, adaptive smoothing, and the nonlinear diffusion equation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(6), 844–847.
Benjamini, Y., & Liu, W. (1999). A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. Journal of Statistical Planning and Inference, 82(1–2), 163–170.
Birge, L. (2006). Model selection via testing: an alternative to (penalized) maximum likelihood estimators. Annales de l’Institut Henri Poincaré (B) Probabilités et Statistiques, 40, 273–325.
Brown, R. (1963). Smoothing, forecasting and prediction of discrete time series. Prentice-Hall: Englewood Cliffs.
Buades, A., Coll, B., & Morel, J. M. (2005). A review of image denoising algorithms, with a new one. SIAM Multiscale Modeling and Simulation, 4, 490–530.
Buades, A., Coll, B., & Morel, J. M. (2006a). The staircasing effect in neighborhood filters and its solution. IEEE Transactions on Image Processing, 15, 1499–1505.
Buades, A., Coll, B., & Morel, J. M. (2006b). Neighborhood filters and PDE’s. Numerische Mathematik, 105(1), 1–34.
Buades, A., Coll, B., & Morel, J. M. (2008). Nonlocal image and movie denoising. International Journal of Computer Vision, 76(2), 123–139.
Bunea, F., Tsybakov, A., & Wegkamp, M. (2007). Aggregation for regression learning. Annals of Statistics, 35, 1674–1697.
Chan, T., & Shen, J. (2005). Image processing and analysis: variational, PDE, wavelet, and stochastic methods. Philadelphia: SIAM.
Chatterjee, P., & Milanfar, P. (2008). A generalization of non-local means via kernel regression. In Proc. SPIE conf. on computational imaging, San Jose, January 2008.
Cleveland, W. S., & Devlin, S. J. (1988). Locally weighted regression: an approach to regression analysis by local fitting. Journal of American Statistical Association, 83, 596–610.
Coifman, R., & Donoho, D. (1995). Translation-invariant de-noising. In A. Antoniadis & G. Oppenheim (Eds.), Lecture notes in statistics. Wavelets and statistics (pp. 125–150). Berlin: Springer.
Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8), 2080–2095.
Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2008). A nonlocal and shape-adaptive transform-domain collaborative filtering. In Proc. 2008 int. workshop on local and non-local approximation in image processing, LNLA 2008, Lausanne, Switzerland.
Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2009). BM3D image denoising with shape-adaptive principal component analysis. In Proc. workshop on signal processing with adaptive sparse structured representations (SPARS’09), Saint-Malo, France, April 2009.
Donoho, D., & Johnstone, I. (1995). Adapting to unknown smoothness via wavelet shrinkage. Journal of American Statistical Association, 90(432), 1200–1224.
Egiazarian, K., Katkovnik, V., & Astola, J. (2001). Local transform-based image de-noising with adaptive window size selection. In Proc. SPIE image and signal processing for remote sensing VI (Vol. 4170, p. 4170-4), January 2001.
Elad, M. (2002). On the origin of the bilateral filter and ways to improve it. IEEE Transactions on Image Processing, 10(10), 1141–1151.
Elad, M. (2006). Why shrinkage is still relevant for redundant representations? IEEE Transactions on Information Theory, 52(12), 5559–5569.
Elad, M., & Aharon, M. (2006). Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, 15(12), 3736–3745.
Elad, M., & Yavneh, I. (2009). A plurality of sparse representations is better than the sparsest one alone. IEEE Transactions on Information Theory (to appear).
Elmoataz, A., Lezoray, O., Bougleux, S., & Ta, V. T. (2008). Unifying local and nonlocal processing with partial difference operators on weighted graphs. In Proc. 2008 int. workshop on local and non-local approximation in image processing, LNLA 2008, Lausanne, Switzerland, August 2008.
Ercole, C., Foi, A., Katkovnik, V., & Egiazarian, K. (2005). Spatio-temporal pointwise adaptive denoising of video: 3D non-parametric approach. In Proc. of the 1st international workshop on video processing and quality metrics for consumer electronics, VPQM2005, Scottsdale, AZ, January 2005.
Fan, J., & Gijbels, I. (1996). Local polynomial modeling and its application. London: Chapman and Hall.
Foi, A. (2005). Anisotropic nonparametric image processing: theory, algorithms and applications. Ph.D. Thesis, Dip. di Matematica, Politecnico di Milano, ERLTDD-D01290.
Foi, A. (2007). Pointwise shape-adaptive DCT image filtering and signal-dependent noise estimation. D.Sc. Tech. Thesis, Institute of Signal Processing, Tampere University of Technology, Publication 710, December 2007.
Foi, A., Katkovnik, V., Egiazarian, K., & Astola, J. (2004a). A novel anisotropic local polynomial estimator based on directional multiscale optimizations. In Proc. 6th IMA int. conf. math. in signal processing (pp. 79–82), Cirencester (UK).
Foi, A., Katkovnik, V., Egiazarian, K., & Astola, J. (2004b). Inverse halftoning based on the anisotropic LPA-ICI deconvolution. In Proc. int. TICSP workshop spectral meth. multirate signal proc., SMMSP 2004 (pp. 49–56), Vienna, September 2004.
Foi, A., Alenius, S., Trimeche, M., Katkovnik, V., & Egiazarian, K. (2005a). A spatially adaptive Poissonian image deblurring. In IEEE 2005 int. conf. image processing, ICIP 2005, September 2005.
Foi, A., Bilcu, R., Katkovnik, V., & Egiazarian, K. (2005b). Anisotropic local approximations for pointwise adaptive signal-dependent noise removal. In XIII European signal proc. conf., EUSIPCO 2005.
Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Pointwise shape-adaptive DCT for high-quality denoising and deblocking of grayscale and color images. IEEE Transactions on Image Processing, 16(5), 1395–1411.
Freeman, W. T., & Adelson, E. H. (1991). The design and use of steerable filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(9), 891–906.
Gilboa, G., & Osher, S. (2007a). Nonlocal linear image regularization and supervised segmentation. SIAM Multiscale Modeling and Simulation, 6(2), 595–630.
Gilboa, G., & Osher, S. (2007b). Nonlocal operators with applications to image processing, UCLA Computational and Applied Mathematics. Reports cam (07-23), July 2007. http://www.math.ucla.edu/applied/cam.
Goldenshluger, A. (2009). A universal procedure for aggregating estimators. Annals of Statistics, 37(1), 542–568.
Goldenshluger, A., & Nemirovski, A. (1997). On spatial adaptive estimation of nonparametric regression. Mathematical Methods of Statistics, 6, 135–170.
Guleryuz, O. (2007). Weighted averaging for denoising with overcomplete dictionaries. IEEE Transactions on Image Processing, 16(12), 3020–3034.
Hammond, D. K., & Simoncelli, E. P. (2008). Image modeling and denoising with orientation-adapted Gaussian scale mixtures. IEEE Transactions on Image Processing, 17(11), 2089–2101.
Hastie, T. J., & Loader, C. (1993). Local regression: automatic kernel carpentry (with discussion). Statistical Science, 8(2), 120–143.
Hel-Or, Y., & Shaked, D. (2008). A discriminative approach for wavelet denoising. IEEE Transactions on Image Processing, 17(4), 443–457.
Hirakawa, K., & Parks, T. W. (2006). Image denoising using total least squares. IEEE Transactions on Image Processing, 15(9), 2730–2742.
Hua, G., & Orchard, M. T. (2003). A new interpretation of translation invariant image denoising. In Proc. asilomar conf. signals syst. comput. (pp. 332–336), Pacific Grove, CA.
Hurvich, C. M., Simonoff, J. S., & Tsai, C.-L. (1998). Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society, Series B, 60, 271–293.
Hyvärinen, A., Karhunen, J., & Oja, E. (2001). Independent component analysis. New York: Wiley.
Jolliffe, I. T. (2002). Principal component analysis (2nd edn.). Berlin: Springer.
Katkovnik, V. (1976). Linear estimation and stochastic optimization problems. Moscow: Nauka (in Russian).
Katkovnik, V. (1985). Nonparametric identification and smoothing of data (local approximation method). Moscow: Nauka (in Russian).
Katkovnik, V. (1999). A new method for varying adaptive bandwidth selection. IEEE Transactions on Signal Processing, 47(9), 2567–2571.
Katkovnik, V., & Spokoiny, V. (2008). Spatially adaptive estimation via fitted local likelihood techniques. IEEE Transactions on Image Processing, 56(3), 873–886.
Katkovnik, V., Foi, A., Egiazarian, K., & Astola, J. (2004). Directional varying scale approximations for anisotropic signal processing. In Proc. XII European signal proc. conf., EUSIPCO 2004 (pp. 101–104), Vienna, September 2004.
Katkovnik, V., Foi, A., Egiazarian, K., & Astola, J. (2005). Anisotropic local likelihood approximations. In Proc. of electronic imaging 2005 (p. 5672-19), January 2005.
Katkovnik, V., Egiazarian, K., & Astola, J. (2006). Local approximation techniques in signal and image processing. Bellingham: SPIE Press.
Katkovnik, V., Foi, A., & Egiazarian, K. (2007). Mix-distribution modeling for overcomplete denoising. In Proc. 9th workshop on adaptation and learning in control and signal processing (ALCOSP’07), St. Petersburg, Russia, 29–31 August 2007.
Katkovnik, V., Foi, A., Egiazarian, K., & Astola, J. (2008). Nonparametric regression in imaging: from local kernel to multiple-model nonlocal collaborative filtering. In Proc. 2008 int. workshop on local and non-local approximation in image processing, LNLA 2008. Lausanne, Switzerland, August 2008.
Kervrann, C., & Boulanger, J. (2005). Local adaptivity to variable smoothness for exemplar-based image denoising and representation (Research Report INRIA, RR-5624).
Kervrann, C., & Boulanger, J. (2006). Unsupervised patch-based image regularization and representation. In LNCS : Vol. 3954. ECCV 2006, Part IV (pp. 555–567). Berlin: Springer.
Kervrann, C., & Boulanger, J. (2008). Local adaptivity to variable smoothness for exemplar-based image regularization and representation. International Journal of Computer Vision, 79, 45–69.
Kindermann, S., Osher, S., & Jones, P. W. (2005). Deblurring and denoising of images by nonlocal functionals. SIAM Multiscale Modeling & Simulation, 4(4), 1091–1115.
Kingsbury, N. G. (2001). Complex wavelets for shift invariant analysis and filtering of signals. Journal of Applied and Computational Harmonic Analysis, 10(3), 234–253.
Lansel, S., Donoho, D., & Weissman, T. (2009). DenoiseLab: a standard test set and evaluation method to compare denoising algorithms. http://www.stanford.edu/~slansel/DenoiseLab/.
Lee, J. S. (1983). Digital image smoothing and the sigma filter. Computer Vision, Graphics, and Image Processing, 24, 255–269.
Lepski, O. (1990). One problem of adaptive estimation in Gaussian white noise. Theory of Probability and Its Applications, 35(3), 459–470.
Lepski, O., Mammen, E., & Spokoiny, V. (1997). Ideal spatial adaptation to inhomogeneous smoothness: an approach based on kernel estimates with variable bandwidth selection. The Annals of Statistics, 25(3), 929–947.
Lindeberg, T. (1998). Edge detection and ridge detection with automatic scale selection. International Journal of Computer Vision, 30(2), 117–154.
Lindeberg, T. (2009). Scale-space. In B. Wah (Ed.), Encyclopedia of computer science and engineering (Vol. IV, pp. 2495–2504). Hoboken: Wiley.
Loader, C. (1999). Series Statistics and Computing. Local regression and likelihood. New York: Springer.
Lou, Y., Favaro, P., & Soatto, S. (2008a). Nonlocal similarity image filtering. In UCLA computational and applied mathematics. Reports cam (8-26), April 2008. http://www.math.ucla.edu/applied/cam.
Lou, Y., Zhang, X., Osher, S., & Bertozzi, A. (2008b). Image recovery via nonlocal operators. Reports cam (8-35), May 2008. http://www.math.ucla.edu/applied/cam.
Mairal, J., Sapiro, G., & Elad, M. (2008). Learning multiscale sparse representations for image and video restoration. SIAM Multiscale Modeling Simulation, 7(1), 214–241.
Mallat, S. (1999). A wavelet tour of signal processing. San Diego: Academic Press.
Meyer, Y. (2001). Univ. lecture ser. 22. Oscillating patterns in image processing and nonlinear evolution equations. Providence: AMS.
Muresan, D., & Parks, T. (2003). Adaptive principal components and image denoising. In Proc. 2003 IEEE int. conf. image process., ICIP 2003 (pp. 101–104), September 2003.
Nadaraya, E. A. (1964). On estimating regression. Theory of Probability and Its Applications, 9, 141–142.
Öktem, H., Katkovnik, V., Egiazarian, K., & Astola, J. (2001). Local adaptive transform based image de-noising with varying window size. In Proc. IEEE int. conf. image process., ICIP 2001 (pp. 273–276), Thessaloniki, Greece.
Öktem, R., Yaroslavsky, L., Egiazarian, K., & Astola, J. (1999). Transform based denoising algorithms: comparative study. Tampere: Tampere University of Technology.
Osher, S., Sole, A., & Vese, L. (2003). Image decomposition and restoration using total variation minimization and the H−1 norm. SIAM Multiscale Modelling & Simulation, 1, 349–370.
Perona, P., & Malik, J. (1990). Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12, 629–639.
Polzehl, J., & Spokoiny, V. (2000). Adaptive weights smoothing with applications to image restoration. Journal of Royal Statistical Society, Series B, 62, 335–354.
Polzehl, J., & Spokoiny, V. (2003). Image denoising: pointwise adaptive approach. The Annals of Statistics, 31(1), 30–57.
Polzehl, J., & Spokoiny, V. (2005). Propagation-separation approach for local likelihood estimation. Probability Theory and Related Fields, 135(3), 335–362.
Portilla, J., Strela, V., Wainwright, M. J., & Simoncelli, E. P. (2003). Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Transactions on Image Processing, 12(11), 1338–1351.
Roth, S., & Black, M. J. (2005). Fields of experts: A framework for learning image priors. In Proceedings of the IEEE international conference on computer vision and pattern recognition (CVPR) (pp. 860–867), San Diego, CA.
Roth, S., & Black, M. J. (2009). Fields of experts. International Journal of Computer Vision, 82(2), 205–229.
Rudin, L., Osher, S., & Fatemi, E. (1992). Nonlinear algorithms. Physica D, 60, 259–268.
Rudin, L., Osher, S., & Fatemi, E. (1993). Nonlinear total variation based noise removal algorithms. Physica D, 60(2), 259–268.
Ruppert, D. (1997). Empirical-bias bandwidths for local polynomial nonparametric regression and density estimation. Journal of American Statistical Association, 92(439), 1049–1062.
Savitzky, A., & Golay, M. (1964). Smoothing and differentiation of data by simplified least squares procedures. Analytical Chemistry, 36, 1627–1639.
Schucany, W. R. (1995). Adaptive bandwidth choice for kernel regression. Journal of American Statistical Association, 90(430), 535–540.
Selesnick, I. W., Baraniuk, R. G., & Kingsbury, N. G. (2005). The dual-tree complex wavelet transform. IEEE Signal Processing Magazine, 22(6), 123–151.
Seuhling, M., Arigovindan, M., Hunziker, P., & Unser, M. (2004). Multiresolution moment filters: theory and applications. IEEE Transactions on Image Processing, 13(4), 484–495.
Simoncelli, E. P., Freeman, W. T., Adelson, E. H., & Heeger, D. J. (1992). Shiftable multi-scale transforms. IEEE Transactions on Information Theory, 38, 587–607.
Simonoff, J. S. (1998). Smoothing methods in statistics. New York: Springer.
Smith, S. M., & Brady, J. M. (1997). SUSAN—a new approach to low level image processing. International Journal of Computer Vision, 23(1), 45–78.
Spokoiny, V. (1998). Estimation of a function with discontinuities via local polynomial fit with an adaptive window choice. Annals of Statistics, 26, 1356–1378.
Spokoiny, V. (2009). Local parametric methods in nonparametric estimation. Berlin: Springer (to appear). http://www.wias-berlin.de/people/spokoiny/adabook/ada.pdf.
Steidl, G., Weickert, J., Brox, T., Mrazek, P., & Welk, M. (2004). On the equivalence of soft wavelet shrinkage, total variation diffusion, regularization and SIDEs. SIAM Journal of Numerical Analysis, 42(2), 686–713.
Takeda, H., Farsiu, S., & Milanfar, P. (2007a). Higher order bilateral filters and their properties. In Proc. of the SPIE conf. on computational imaging, San Jose, January 2007.
Takeda, H., Farsiu, S., & Milanfar, P. (2007b). Kernel regression for image processing and reconstruction. IEEE Transactions on Image Processing, 16(2), 349–366.
Tomasi, C., & Manduchi, R. (1998). Bilateral filtering for gray and color images. In Proc. of the sixth int. conf. on computer vision (pp. 839–846), 1998.
Tschumperlé, D., & Brun, L. (2008a). Defining some variational methods on the space of patches: application to multi-valued image denoising and registration (Research Report: Les cahiers du GREYC, no. 08-01). Caen, France.
Tschumperlé, D., & Brun, L. (2008b). Image denoising and registration by PDE’s on the space of patches. In Proc. int. workshop on local and non-local approximation in image processing (LNLA’08), Lausanne, Switzerland, August 2008.
Vansteenkiste, E., Van der Weken, D., Philips, W., & Kerre, E. (2006). Perceived image quality measurement of state-of-the-art noise reduction schemes. In LNCS : Vol. 4179. ACIVS 2006 (pp. 114–124). Berlin: Springer.
Vese, L., & Osher, S. (2003). Modeling textures with total variation minimization and oscillating patterns in image processing. Journal of Scientific Computing, 19, 553–572.
Wand, M. P., & Jones, M. C. (1995). Monographs on statistics and applied probability : Vol. 60. Kernel smoothing. Boca Raton: Hartman&Hall/CRC.
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.
Watson, G. S. (1964). Smooth regression analysis. Sankhya, Series A, 26, 359–372.
Wei, J. (2005). Lebesgue anisotropic image denoising. International Journal of Imaging Systems and Technology, 15(1), 64–73.
Weickert, J. (1996). Theoretical foundations of anisotropic diffusion in image processing. Computing, Suppl. 11, 221–236.
Weickert, J. (1998). European consortium for mathematics in industry. Anisotropic diffusion in image processing. Stuttgart: B.G. Teubner.
Yaroslavsky, L. (1985). Digital picture processing—an introduction. New York: Springer.
Yaroslavsky, L. (1996). Local adaptive image restoration and enhancement with the use of DFT and DCT in a running window. In Proc. SPIE wavelet applications in signal and image process. IV (Vol. 2825, pp. 1–13), 1996.
Yaroslavsky, L., & Eden, M. (1996). Fundamentals of digital optics. Boston: Birkhäuser Boston.
Yaroslavsky, L., Egiazarian, K., & Astola, J. (2001). Transform domain image restoration methods: review, comparison and interpretation. In Proc. SPIE, nonlinear image process. pattern anal. XII (Vol. 4304, pp. 155–169). San Jose, CA, 2001.
Zimmer, S., Didas, S., & Weickert, J. (2008). A rotationally invariant block matching strategy improving image denoising with non-local means. In Proc. 2008 int. workshop on local and non-local approximation in image processing, LNLA 2008, Lausanne, Switzerland, August 2008.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Katkovnik, V., Foi, A., Egiazarian, K. et al. From Local Kernel to Nonlocal Multiple-Model Image Denoising. Int J Comput Vis 86, 1–32 (2010). https://doi.org/10.1007/s11263-009-0272-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-009-0272-7