Statistics and Computing

, Volume 26, Issue 1–2, pp 93–105 | Cite as

Wavelet-based gradient boosting

  • E. Dubossarsky
  • J. H. Friedman
  • J. T. Ormerod
  • M. P. Wand


A new data science tool named wavelet-based gradient boosting is proposed and tested. The approach is special case of componentwise linear least squares gradient boosting, and involves wavelet functions of the original predictors. Wavelet-based gradient boosting takes advantages of the approximate \(\ell _1\) penalization induced by gradient boosting to give appropriate penalized additive fits. The method is readily implemented in R and produces parsimonious and interpretable regression fits and classifiers.


Classification Data science Generalized additive models Nonparametric regression 



We are grateful to Andrew Chernih for his provision of the Sydney residential property price data and to Peter Green for his comments on aspects of this research. Partial support was provided by Australian Research Council Discovery Project DP0877055. Assistance from the University of Technology, Sydney’s Distinguished Visitor programme is gratefully acknowledged.


  1. Binder, H., Tutz, G.: A comparison of methods for the fitting of generalized additive models. Stat. Comput. 18, 87–99 (2008)MathSciNetCrossRefGoogle Scholar
  2. Bühlmann, P.: Boosting for high-dimensional linear models. Ann. Stat. 34, 559–583 (2006)MATHCrossRefGoogle Scholar
  3. Bühlmann, P., Yu, B.: Sparse boosting. J. Mach. Learn. Res. 7, 1001–1024 (2006)MATHMathSciNetGoogle Scholar
  4. Bühlmann, P., Hothorn, T.: Boosting algorithms: regularization, prediction and model fitting (with discussion). Stat. Sci. 22, 477–522 (2007)MATHCrossRefGoogle Scholar
  5. Donoho, D.L.: De-noising by soft-thresholding. IEEE Trans. Inf. Theor. 41, 613–627 (1995)MATHMathSciNetCrossRefGoogle Scholar
  6. Donoho, D.L., Johnstone, I.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81, 425–456 (1994)MATHMathSciNetCrossRefGoogle Scholar
  7. Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Ann. Stat. 32, 407–451 (2004)MATHMathSciNetCrossRefGoogle Scholar
  8. Friedman, J.: Greedy function approximation: a gradient boosting machine. Ann. Stat. 29, 1189–1232 (2001)MATHCrossRefGoogle Scholar
  9. Hansen, M.H., Yu, B.: Model selection and the principle of minimum description length. J. Am. Stati. Assoc. 96, 746–774 (2001)MATHMathSciNetCrossRefGoogle Scholar
  10. Hastie, T.: Comment on paper by Bühlmann & Hothorn. Stat. Sci. 22, 513–515 (2007)MATHMathSciNetCrossRefGoogle Scholar
  11. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning, 2nd edn. Springer, New York (2009)MATHCrossRefGoogle Scholar
  12. Hothorn, T., Bühlmann, P., Kneib, T., Schmid, M. & Hofner, B.: mboost 2.2. Model-based boosting. R package.(2011)
  13. Hurvich, C.M., Simonoff, J.S., Tsai, C.: Smoothing parameter selection in nonparametric regression using an improved A kaike information criterion. J. R. Stat. Soc. B 60, 271–293 (1998)MATHMathSciNetCrossRefGoogle Scholar
  14. Hyndman, R.J.: hdrcde 2.15. Highest density regions and conditional density estimation. R package. (2010)
  15. Leitenstorfer, F., Tutz, G.: Knot selection by boosting techniques. Comput. Stat. Data Anal. 51, 4605–4621 (2007)MATHMathSciNetCrossRefGoogle Scholar
  16. Nason, G.P.: Wavelet Methods in Statistics with R. Springer, New York (2008)MATHCrossRefGoogle Scholar
  17. Nason, G.P.: wavethresh 4.5. Wavelets statistics and transforms. R package. (2010)
  18. R Development Core Team R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, (2012)
  19. Ridgeway G.: gbm 1.6. Generalized boosted regression models. R package. (2012)
  20. Samworth, R.J., Wand, M.P.: Asymptotics and optimal bandwidth selection for highest density region estimation. Ann. Stat. 38, 1767–1792 (2010)MATHMathSciNetCrossRefGoogle Scholar
  21. Vidakovic, B.: Statistical Modeling by Wavelets. Wiley, New York (1999)MATHCrossRefGoogle Scholar
  22. Wand, M.P., Jones, M.C.: Kernel Smoothing. Chapman and Hall, London (1995)MATHCrossRefGoogle Scholar
  23. Wand, M.P., Ormerod, J.T.: Penalized wavelets: embedding wavelets into semiparametric regression. Electron. J. Stat. 5, 1654–1717 (2011)MATHMathSciNetCrossRefGoogle Scholar
  24. Zou, H., Hastie, T., Tibshirani, R.: On the “degrees of freedom” of the lasso. Ann. Stat. 5, 2173–2192 (2007)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • E. Dubossarsky
    • 1
  • J. H. Friedman
    • 2
  • J. T. Ormerod
    • 3
  • M. P. Wand
    • 4
  1. 1.Presciient Pty. LtdEppingAustralia
  2. 2.Department of StatisticsStanford UniversityStanfordUSA
  3. 3.School of Mathematics and StatisticsUniversity of SydneySydneyAustralia
  4. 4.School of Mathematical SciencesUniversity of Technology, SydneyUltimoAustralia

Personalised recommendations