Statistics and Computing

, Volume 16, Issue 1, pp 37–55 | Cite as

Wavelet kernel penalized estimation for non-equispaced design regression

  • Umberto Amato
  • Anestis Antoniadis
  • Marianna PenskyEmail author


The paper considers regression problems with univariate design points. The design points are irregular and no assumptions on their distribution are imposed. The regression function is retrieved by a wavelet based reproducing kernel Hilbert space (RKHS) technique with the penalty equal to the sum of blockwise RKHS norms. In order to simplify numerical optimization, the problem is replaced by an equivalent quadratic minimization problem with an additional penalty term. The computational algorithm is described in detail and is implemented with both the sets of simulated and real data. Comparison with existing methods showed that the technique suggested in the paper does not oversmooth the function and is superior in terms of the mean squared error. It is also demonstrated that under additional assumptions on design points the method achieves asymptotic optimality in a wide range of Besov spaces.


Reproducing kernel Wavelet decomposition Penalization Besov spaces Smoothing splines ANOVA Entropy 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Abramovich F., Bailey T. and Sapatinas T. 2000. Wavelet analysis and its statistical applications. The Statistician—Journal of the Royal Statistical Society, Ser. D 49: 1–29.CrossRefGoogle Scholar
  2. Amato, U. and Vuza, D.T. 1997. Wavelet approximation of a function from samples affected by noise. Rev. Roumaine Math. Pures Appl. 42: 481–493.zbMATHMathSciNetGoogle Scholar
  3. Antoniadis, A. 1996. Smoothing noisy data with tapered coiflets series. Scandinavian Journal of Statistics, 23: 313–330.zbMATHMathSciNetGoogle Scholar
  4. Antoniadis, A., Bigot, J. and Sapatinas, T. 2001. Wavelet estimators in nonparametric regression: a comparative simulation study. Journal of Statistical Software 6.Google Scholar
  5. Antoniadis, A. and Fan, J. 2001. Regularization by Wavelet Approximations. J. Amer. Statist. Assoc. 96: 939–967.zbMATHMathSciNetCrossRefGoogle Scholar
  6. Aronszajn, N. 1950. Theory of reproducing kernels. Trans. Am. Math. Soc. 68: 337–404.zbMATHMathSciNetCrossRefGoogle Scholar
  7. Birgé, L. and Massart, P. 2000. An adaptive compression algorithm in Besov spaces, Journal of Constructive Approximation 16: 1–36.zbMATHCrossRefGoogle Scholar
  8. Birman, M.S. and Solomjak, M.Z. 1967. Piecewise-polynomial approximation of functions of the classes W p. Mat. Sbornik. 73: 295–317.MathSciNetCrossRefGoogle Scholar
  9. Brinkman, N. 1981. Ethanol fuel - a single-cylinder engine study of efficiency and exhaust emissions. SAE Transactions 90: 1414–1424.Google Scholar
  10. Cai, T. 1999. Adaptive wavelet estimation: A block thresholding and oracle inequality approach. Ann. Statist. 27: 898–924.zbMATHMathSciNetCrossRefGoogle Scholar
  11. Cai, T. 2001. Discussion of “Regularization of Wavelets Approximations” by A. Antoniadis and J. Fan. J. American Statistical Association 96: 960–962.Google Scholar
  12. Cai, T. and Brown, L. D. 1998. Wavelet Shrinkage for nonequispaced samples, The Annals of Statistics 26: 1783–1799.zbMATHMathSciNetCrossRefGoogle Scholar
  13. Cai, T. and Silverman, B.W. 2001. Incorporating information on neighboring coefficients into wavelet estimation. Sankhya 63: 127–148.MathSciNetGoogle Scholar
  14. Canu, S., Mary, X., and Rakotomamonjy, A. 2003. Functional learning through kernel, in Advances in Learning Theory: Methods, Models and Applications, NATO Science Series III: Computer and Systems Sciences, Suykens, J. et al., Eds. IOS Press, Amsterdam 90: 89–110.Google Scholar
  15. Craven, P. and Wahba, G. 1979. Smoothing noisy data with spline functions. Numer. Math. 31: 377–403.zbMATHMathSciNetCrossRefGoogle Scholar
  16. Daubechies, I. 1992. Ten Lectures on Wavelets. Philadelphia: SIAM.zbMATHGoogle Scholar
  17. Delouille, V., Franke, J. and Von Sachs, R. 2001. Nonparametric stochastic regression with design-adapted wavelets. Sankhya, Series A, Vol 63 (3), pp. 328–366.Google Scholar
  18. DeVore, R.A. and Popov, V. 1988. Interpolation of Besov Spaces. Transactions of the American Mathematical Society 305: 397–414.zbMATHMathSciNetCrossRefGoogle Scholar
  19. Donoho D.L., Elad, M. and Temlyakov, V. 2004. Stable Recovery of Sparse Overcomplete Representations in the Presence of Noise. Technical report, Stanford University.Google Scholar
  20. Donoho, D.L., Johnstone, I.M., Kerkyacharian, G. and Picard, D. 1995. Wavelet shrinkage: asymptopia? (with discussion). Journal of the Royal Statistical Society, Series B 57: 301–337.Google Scholar
  21. Eubank, R.L. 1988 Spline Smoothing and Nonparametric Regression, New York: Marcel Dekker, Inc.zbMATHGoogle Scholar
  22. van de Geer, S. 2000. Empirical Processes in M-Estimation. Cambridge University Press.Google Scholar
  23. Gradshtein, I.S., and Ryzhik, I.M. 1980 Tables of Integrals, Series, and Products. Academic Press, New York.Google Scholar
  24. Green, P.J. and Silverman, B.W. 1994. Nonparametric Regression and Generalised Linear Models. London: Chapman and Hall.Google Scholar
  25. Gunn, S.R. and Kandola, J.S. 2002. Structural modeling with sparse kernels. Mach. Learning 48: 115–136.CrossRefGoogle Scholar
  26. Hall, P., Kerkyacharian, G. and Picard, D. 1999. On the minimax optimality of block thresholded wavelet estimators. Statist. Sinica 9: 33–50.zbMATHMathSciNetGoogle Scholar
  27. Hall, P. and Turlach, B.A. 1997 Interpolation methods for nonlinear wavelet regression with ir- regularly spaced design. Ann. Statist. 25: 1912–1925.zbMATHMathSciNetCrossRefGoogle Scholar
  28. Härdle, W., Kerkyacharian, G., Picard, D. and Tsybakov, A. 1998. Wavelets, Approximation, and Statistical Applications, Lecture Notes in Statistics, 129: Springer-Verlag, New-York.Google Scholar
  29. Karlovitz, L.A. 1970. Construction of nearest points in the l p, p even and l 1 norms, Journal of Approximation Theory 3: 123–127.zbMATHMathSciNetCrossRefGoogle Scholar
  30. Kerkyacharian, G. and Picard, D. 2003. Replicant compression coding in Besov spaces, ESAIM: P and S 7: 239–250.zbMATHMathSciNetCrossRefGoogle Scholar
  31. Kimeldorf G., and Wahba, G.1971. Some results on Tchebycheffian spline functions. J. Math. Anal. Applic. 33: 82–95.zbMATHMathSciNetCrossRefGoogle Scholar
  32. Kohler, M. 2003 Nonlinear orthogonal series estimation for random design regression. J. Stat. Plan. Infer. 115: 491–520.zbMATHMathSciNetCrossRefGoogle Scholar
  33. Kovac, A. and Silverman, B. 2000. Extending the scope of wavelet regression methods by coefficient-dependent thresholding. J. Am. Stat. Assoc. 95: 172–183.CrossRefGoogle Scholar
  34. Lin, Y. and Zhang, H.H. 2003. Component Selection and Smoothing in Smoothing Spline Analysis of Variance Models, Technical report, University of Wisconsin—Madison.Google Scholar
  35. Lin, X., Wahba, G., Xiang, D., Gao, F., Klein, R. and Klein, B. 2000. Smoothing spline ANOVA models for large data sets with Bernoulli observations and the randomized GACV. Ann. Statist. 28: 1570–1600.zbMATHMathSciNetCrossRefGoogle Scholar
  36. Loubes, J.-M. and van de Geer, S. 2002, Adaptive estimation with soft thresholding penalties, Statistica Neerlandica 56: 454–479.zbMATHMathSciNetCrossRefGoogle Scholar
  37. Mallat, S.G. 1999. A Wavelet Tour of Signal Processing. 2nd ed. San Diego: Academic Press.zbMATHGoogle Scholar
  38. Meyer, Y. 1992. Wavelets and Operators. Cambridge: Cambridge University Press.Google Scholar
  39. Nason, G. 1998. WaveThresh3 Software. Department of Mathematics, University of Bristol, Bristol, UK.Google Scholar
  40. Nason, G. 2002 Choice of wavelet smoothness, primary resolution and threshold in wavelet shrinkage. Statistics and Computing 12: 219–227.MathSciNetCrossRefGoogle Scholar
  41. Sardy, S., Percival, D.B., Bruce A., G., Gao, H.-Y. and Stuelzle, W. 1999. Wavelet shrinkage for unequally spaced data, Statistics and Computing 9: 65–75.CrossRefGoogle Scholar
  42. Silverman, B.W. 1985 Some aspects of the spline smoothing approach to non-parametric curve fitting. Journal of the Royal Statistical Society series B. 47: 1–52.zbMATHGoogle Scholar
  43. Tapia, R. and Thompson, J. 1978. Nonparametric Probability Density Estimation. Baltimore, MD, Johns Hopkins University Press.zbMATHGoogle Scholar
  44. Tibshirani, R. J. 1996. Regression shrinkage and selection via the lasso. Journal of Royal Statistical Society, B 58: 267–288.zbMATHMathSciNetGoogle Scholar
  45. Triebel, H. 1983. Theory of Function Spaces. Birkhäuser Verlag, Basel.Google Scholar
  46. Vidakovic, B. 1999. Statistical Modeling by Wavelets. New York: John Wiley and Sons.zbMATHGoogle Scholar
  47. Wahba, G. 1990. Spline Models for Observational Data, SIAM. CBMS-NSF Regional Conference Series in Applied Mathematics, 59.Google Scholar
  48. Wahba, G., Wang, Y., Gu, C., Klein, R. and Klein, B. 1995 Smoothing spline ANOVA for exponential families, with application to the Wisconsin Epidemiological Study of Diabetic Retinopathy. Ann. Statist. 23: 1865–1895.zbMATHMathSciNetCrossRefGoogle Scholar
  49. Zhang, H., Wahba, G., Lin, Y., Voelker, M., Ferris, M., Klein, R. and Klein, B. 2002. Variable selection and model building via likelihood basis pursuit. Technical report, University of Wisconsin—Madison.Google Scholar

Copyright information

© Springer Science + Business Media, Inc. 2006

Authors and Affiliations

  • Umberto Amato
    • 1
  • Anestis Antoniadis
    • 2
  • Marianna Pensky
    • 3
    Email author
  1. 1.Istituto per le Applicazioni del Calcolo ‘M. Picone’ CNR - Sezione di NapoliNapoliItaly
  2. 2.Laboratoire IMAG-LMCUniversity Joseph FourierGrenoble Cedex 9France
  3. 3.Department of StatisticsUniversity of Central FloridaOrlandoUSA

Personalised recommendations