Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Iterative bias reduction: a comparative study

Abstract

Multivariate nonparametric smoothers, such as kernel based smoothers and thin plate splines smoothers, are adversely impacted by the sparseness of data in high dimension, also known as the curse of dimensionality. Adaptive smoothers, that can exploit the underlying smoothness of the regression function, may partially mitigate this effect. This paper presents a comparative simulation study of a novel adaptive smoother (IBR) with competing multivariate smoothers available as package or function within the R language and environment for statistical computing. Comparison between the methods are made on simulated datasets of moderate size, from 50 to 200 observations, with two, five or 10 potential explanatory variables, and on a real dataset. The results show that the good asymptotic properties of IBR are complemented by a very good behavior on moderate sized datasets, results which are similar to those obtained with Duchon low rank splines.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. Akaike, H.: Information theory and an extension of the maximum likelihood principle. In: Petrov, B.N., Csaki, B.F. (eds.) Second International Symposium on Information Theory, pp. 267–281. Academiai Kiado, Budapest (1973)

  2. Breiman, L.: Bagging predictors. Mach. Learn. 24, 123–140 (1996)

  3. Breiman, L.: Using adaptive bagging to Debais regressions. Tech. Rep. 547, Department of Statistics, UC Berkeley (1999)

  4. Breiman, L., Freiman, J., Olshen, R., Stone, C.: Classification and Regression Trees, 4th edn. CRC Press, Boca Raton (1984)

  5. Bühlmann, P., Yu, B.: Boosting with the l 2 loss: regression and classification. J. Am. Stat. Assoc. 98, 324–339 (2003)

  6. Bühlmann, P., Yu, B.: Sparse boosting. J. Mach. Learn. Res. 7, 1001–1024 (2006)

  7. Buja, A., Hastie, T., Tibshirani, R.: Linear smoothers and additive models. Ann. Stat. 17, 453–510 (1989)

  8. Cornillon, P.A., Hengartner, N., Matzner-Løber, E.: Iterative bias reduction multivariate smoothing in R: the IBR package (2011a). arXiv:1105.3605v1

  9. Cornillon, P.A., Hengartner, N., Matzner-Løber, E.: Recursive bias estimation for multivariate regression (2011b). arXiv:1105.3430v2

  10. Craven, P., Wahba, G.: Smoothing noisy data with spline functions: estimating the correct degree of smoothing by the method of generalized cross-validation. Numer. Math. 31, 377–403 (1979)

  11. Di Marzio, M., Taylor, C.: On boosting kernel regression. J. Stat. Plan. Inference 138, 2483–2498 (2008)

  12. Duchon, J.: Splines minimizing rotation-invariant semi-norms in Sobolev spaces. In: Shemp, W., Zeller, K. (eds.) Construction Theory of Functions of Several Variables, pp. 85–100. Springer, Berlin (1977)

  13. Eubank, R.: Spline Smoothing and Nonparametric Regression. Marcel Dekker, New York (1988)

  14. Fan, J., Gijbels, I.: Local Polynomial Modeling and Its Application, Theory and Methodologies. Chapman & Hall, New York (1996)

  15. Friedman, J.: Multivariate adaptive regression splines. Ann. Stat. 19, 337–407 (1991)

  16. Friedman, J.: Greedy function approximation: A gradient boosting machine. Ann. Stat. 28, 1189–1232 (2001)

  17. Friedman, J., Stuetzle, W.: Projection pursuit regression. J. Am. Stat. Assoc. 76, 817–823 (1981)

  18. Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Ann. Stat. 28, 337–407 (2000)

  19. Gu, C.: Smoothing Spline ANOVA Models. Springer, Berlin (2002)

  20. Gyorfi, L., Kohler, M., Krzyzak, A., Walk, H.: A Distribution-Free Theory of Nonparametric Regression. Springer, Berlin (2002)

  21. Hastie, T.J., Tibshirani, R.J.: Generalized Additive Models. Chapman & Hall, New York (1995)

  22. Hurvich, C., Simonoff, G., Tsai, C.L.: Smoothing parameter selection in nonparametric regression using and improved Akaike information criterion. J. R. Stat. Soc. B 60, 271–294 (1998)

  23. Lepski, O.: Asymptotically minimax adaptive estimation. I: Upper bounds. Opitmally adaptive estimates. Theory Probab. Appl. 37, 682–697 (1991)

  24. Li, K.C.: Asymptotic optimality for C p , C L , cross-validation and generalized cross-validation: discrete index set. Ann. Stat. 15, 958–975 (1987)

  25. Ridgeway, G.: Additive logistic regression: a statistical view of boosting: discussion. Ann. Stat. 28, 393–400 (2000)

  26. Schwarz, G.: Estimating the dimension of a model. Ann. Stat. 6, 461–464 (1978)

  27. Simonoff, J.S.: Smoothing Methods in Statistics. Springer, New York (1996)

  28. Tsybakov, A.: Introduction to Nonparametric Estimation. Springer, Berlin (2009)

  29. Tukey, J.W.: Explanatory Data Analysis. Addison-Wesley, Reading (1977)

  30. Wood, S.N.: Thin plate regression splines. J. R. Stat. Soc. B 65, 95–114 (2003)

  31. Wood, S.N.: Stable and efficient multiple smoothing parameter estimation for generalized additive models. J. Am. Stat. Assoc. 99, 673–686 (2004)

  32. Yang, Y.: Combining different procedures for adaptive regression. J. Multivar. Anal. 74, 135–161 (2000)

Download references

Acknowledgements

We would like to thank the associate editor and the referees for very valuable remarks and for pointing out to us the work of Duchon (1977).

Author information

Correspondence to E. Matzner-Løber.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Cornillon, P., Hengartner, N., Jegou, N. et al. Iterative bias reduction: a comparative study. Stat Comput 23, 777–791 (2013). https://doi.org/10.1007/s11222-012-9346-4

Download citation

Keywords

  • Multivariate smoothing
  • Thin-plate splines
  • Duchon splines
  • Kernel regression
  • Iterative bias reduction