Statistics and Computing

, Volume 26, Issue 1–2, pp 93–105 | Cite as

Wavelet-based gradient boosting

  • E. Dubossarsky
  • J. H. Friedman
  • J. T. Ormerod
  • M. P. Wand


A new data science tool named wavelet-based gradient boosting is proposed and tested. The approach is special case of componentwise linear least squares gradient boosting, and involves wavelet functions of the original predictors. Wavelet-based gradient boosting takes advantages of the approximate \(\ell _1\) penalization induced by gradient boosting to give appropriate penalized additive fits. The method is readily implemented in R and produces parsimonious and interpretable regression fits and classifiers.


Classification Data science Generalized additive models Nonparametric regression 



We are grateful to Andrew Chernih for his provision of the Sydney residential property price data and to Peter Green for his comments on aspects of this research. Partial support was provided by Australian Research Council Discovery Project DP0877055. Assistance from the University of Technology, Sydney’s Distinguished Visitor programme is gratefully acknowledged.


  1. Binder, H., Tutz, G.: A comparison of methods for the fitting of generalized additive models. Stat. Comput. 18, 87–99 (2008)MathSciNetCrossRefGoogle Scholar
  2. Bühlmann, P.: Boosting for high-dimensional linear models. Ann. Stat. 34, 559–583 (2006)zbMATHCrossRefGoogle Scholar
  3. Bühlmann, P., Yu, B.: Sparse boosting. J. Mach. Learn. Res. 7, 1001–1024 (2006)zbMATHMathSciNetGoogle Scholar
  4. Bühlmann, P., Hothorn, T.: Boosting algorithms: regularization, prediction and model fitting (with discussion). Stat. Sci. 22, 477–522 (2007)zbMATHCrossRefGoogle Scholar
  5. Donoho, D.L.: De-noising by soft-thresholding. IEEE Trans. Inf. Theor. 41, 613–627 (1995)zbMATHMathSciNetCrossRefGoogle Scholar
  6. Donoho, D.L., Johnstone, I.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81, 425–456 (1994)zbMATHMathSciNetCrossRefGoogle Scholar
  7. Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Ann. Stat. 32, 407–451 (2004)zbMATHMathSciNetCrossRefGoogle Scholar
  8. Friedman, J.: Greedy function approximation: a gradient boosting machine. Ann. Stat. 29, 1189–1232 (2001)zbMATHCrossRefGoogle Scholar
  9. Hansen, M.H., Yu, B.: Model selection and the principle of minimum description length. J. Am. Stati. Assoc. 96, 746–774 (2001)zbMATHMathSciNetCrossRefGoogle Scholar
  10. Hastie, T.: Comment on paper by Bühlmann & Hothorn. Stat. Sci. 22, 513–515 (2007)zbMATHMathSciNetCrossRefGoogle Scholar
  11. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning, 2nd edn. Springer, New York (2009)zbMATHCrossRefGoogle Scholar
  12. Hothorn, T., Bühlmann, P., Kneib, T., Schmid, M. & Hofner, B.: mboost 2.2. Model-based boosting. R package.(2011)
  13. Hurvich, C.M., Simonoff, J.S., Tsai, C.: Smoothing parameter selection in nonparametric regression using an improved A kaike information criterion. J. R. Stat. Soc. B 60, 271–293 (1998)zbMATHMathSciNetCrossRefGoogle Scholar
  14. Hyndman, R.J.: hdrcde 2.15. Highest density regions and conditional density estimation. R package. (2010)
  15. Leitenstorfer, F., Tutz, G.: Knot selection by boosting techniques. Comput. Stat. Data Anal. 51, 4605–4621 (2007)zbMATHMathSciNetCrossRefGoogle Scholar
  16. Nason, G.P.: Wavelet Methods in Statistics with R. Springer, New York (2008)zbMATHCrossRefGoogle Scholar
  17. Nason, G.P.: wavethresh 4.5. Wavelets statistics and transforms. R package. (2010)
  18. R Development Core Team R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, (2012)
  19. Ridgeway G.: gbm 1.6. Generalized boosted regression models. R package. (2012)
  20. Samworth, R.J., Wand, M.P.: Asymptotics and optimal bandwidth selection for highest density region estimation. Ann. Stat. 38, 1767–1792 (2010)zbMATHMathSciNetCrossRefGoogle Scholar
  21. Vidakovic, B.: Statistical Modeling by Wavelets. Wiley, New York (1999)zbMATHCrossRefGoogle Scholar
  22. Wand, M.P., Jones, M.C.: Kernel Smoothing. Chapman and Hall, London (1995)zbMATHCrossRefGoogle Scholar
  23. Wand, M.P., Ormerod, J.T.: Penalized wavelets: embedding wavelets into semiparametric regression. Electron. J. Stat. 5, 1654–1717 (2011)zbMATHMathSciNetCrossRefGoogle Scholar
  24. Zou, H., Hastie, T., Tibshirani, R.: On the “degrees of freedom” of the lasso. Ann. Stat. 5, 2173–2192 (2007)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • E. Dubossarsky
    • 1
  • J. H. Friedman
    • 2
  • J. T. Ormerod
    • 3
  • M. P. Wand
    • 4
  1. 1.Presciient Pty. LtdEppingAustralia
  2. 2.Department of StatisticsStanford UniversityStanfordUSA
  3. 3.School of Mathematics and StatisticsUniversity of SydneySydneyAustralia
  4. 4.School of Mathematical SciencesUniversity of Technology, SydneyUltimoAustralia

Personalised recommendations