Abstract
This paper tackles the problem of model complexity in the context of additive models. Several methods have been proposed to estimate smoothing parameters, as well as to perform variable selection. However, these procedures are inefficient or computationally expensive in high dimension. To answer this problem, the lasso technique has been adapted to additive models, but its experimental performance has not been analyzed.
We propose a modified lasso for additive models, performing variable selection. A benchmark is developed to examine its practical behavior, comparing it with forward selection. Our simulation studies suggest ability to carry out model selection of the proposed method. The lasso technique shows up better than forward selection in the most complex situations. The computing time of modified lasso is considerably smaller since it does not depend on the number of relevant variables.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bakin, S.: Adaptive Regression and Model Selection in Data Mining Problems. PhD thesis, School of Mathematical Sciences, The Australian National University, Canberra (1999)
Breiman, L.: Heuristics of instability and stabilization in model selection. The Annals of Statistics 24(6), 2350–2383 (1996)
Brumback, B.A., Ruppert, D., Wand, M.P.: Comment on Variable selection and function estimation in additive nonparametric regression using a data–based prior by T. S. Shively, R. Khon and S. Wood. Journal of the American Statistical Association 94(447), 794–797 (1999)
Cantoni, E., Hastie, T.J.: Degrees of freedom tests for smoothing splines. Biometrika 89, 251–263 (2002)
Chen, Z.: Fitting multivariate regression functions by interaction spline models. J. R. Statist. Soc. B 55(2), 473–491 (1993)
Grandvalet, Y.: Least absolute shrinkage is equivalent to quadratic penalization. In: Niklasson, L., Bodén, M., Ziemske, T. (eds.) GI 1973. Perspectives in Neural Computing, vol. 1, pp. 201–206. Springer, Heidelberg (1998)
Grandvalet, Y., Canu, S.: Outcomes of the equivalence of adaptive ridge with least absolute shrinkage. In: Kearns, M.S., Solla, S.A., Cohn, D.A. (eds.) Advances in Neural Information Processing Systems, vol. 11, pp. 445–451. MIT Press, Cambridge (1998)
Gu, C., Wahba, G.: Minimizing gcv/gml scores with multiple smoothing parameters via the newton method. SIAM J. Sci. Statist. Comput. 12, 383–398 (1991)
Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. Journal of Machine Learning Research, Special Issue on Variable/Feature Selection 3, 1157–1182 (2003)
Hastie, T.J., Tibshirani, R.J.: Generalized Additive Models. Monographs on Statistics and Applied Probability, vol. 43. Chapman & Hall, Boca Raton (1990)
Opsomer, J.D., Ruppert, D.: A fully automated bandwidth selection method for fitting additive models. J. Multivariate Analysis 73, 166–179 (1998)
Osborne, M.R., Presnell, B., Turlach, B.A.: On the lasso and its dual. Journal of Computational and Graphical Statistics 9(2), 319–337 (2000)
Shively, T.S., Khon, R., Wood, S.: Variable selection and function estimation in additive nonparametric regression using a data–based prior. Journal of the American Statistical Association 94(447), 777–806 (1999)
Tibshirani, R.J.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, B 58(1), 267–288 (1995)
Tibshirani, R.J., Knight, K.: The covariance inflation criterion for adaptive model selection. Journal of the Royal Statistical Society, B 61(3), 529–546 (1999)
Wood, S.N.: Modelling and smoothing parameter estimation with multiple quadratic penalties. J. R. Statist. Soc. B 62(2), 413–428 (2000)
Ye, J.: On measuring and correcting the effects of data mining and model selection. Journal of the American Statistical Association 93, 120–131 (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Avalos, M., Grandvalet, Y., Ambroise, C. (2003). Regularization Methods for Additive Models. In: R. Berthold, M., Lenz, HJ., Bradley, E., Kruse, R., Borgelt, C. (eds) Advances in Intelligent Data Analysis V. IDA 2003. Lecture Notes in Computer Science, vol 2810. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-45231-7_47
Download citation
DOI: https://doi.org/10.1007/978-3-540-45231-7_47
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40813-0
Online ISBN: 978-3-540-45231-7
eBook Packages: Springer Book Archive