Abstract
Adaptive ridge is a special form of ridge regression, balancing the quadratic penalization on each parameter of the model. This paper shows the equivalence between adaptive ridge and lasso (least absolute shrinkage and selection operator). This equivalence states that both procedures produce the same estimate. Least absolute shrinkage can thus be viewed as a particular quadratic penalization.
From this observation, we derive an EM algorithm to compute the lasso solution. We finally present a series of applications of this type of algorithm in regression problems: kernel regression, additive modeling and neural net training.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
L. Breiman. Heuristics of instability and stabilization in model selection. The Annals of Statistics 1996, 24(6):2350–2383.
R.J. Tibshirani. Regression shrinkage and selection via the lasso. Technical report, University of Toronto, June 1994.
Y. Grandvalet and S. Canu. Adaptive noise injection for input variables relevance determination. In: ICANN’97, Springer-Verlag 1997, pp 463–468.
S.J. Nowlan and G.E. Hinton. Simplifying neural networks by soft weight-sharing. Neural Computation 1992, 4(4):473–493.
D.J.C. MacKay. A practical Bayesian framework for backprop networks. Neural Computation 1992, 4(3):448–472.
R. M. Neal. Bayesian Learning for Neural Networks. Lecture Notes in Statistics. Springer-Verlag, New York, 1996.
W. Härdie. Applied Nonparametric Regression, volume 19 of Economic Society Monographs. Cambridge University Press, New York, 1990.
F. Girosi. An equivalence between sparse approximation and support vector machines. Technical Report 1606, M.I.T. AI Laboratory, Cambridge, MA., 1997.
V.N. Vapnik. The Nature of Statistical Learning Theory. Springer Series in Statistics. Springer-Verlag, New York, 1995.
T.J. Hastie and R.J. Tibshirani. Generalized Additive Models, volume 43 of Monographs on Statistics and Applied Probability. Chapman & Hall, New York, 1990.
S. Canu, Y. Grandvalet, and M.-H. Masson. Black-box software sensor design for environmental monitoring. In: ICANN’98, Springer-Verlag 1998.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1998 Springer-Verlag London
About this paper
Cite this paper
Grandvalet, Y. (1998). Least Absolute Shrinkage is Equivalent to Quadratic Penalization. In: Niklasson, L., Bodén, M., Ziemke, T. (eds) ICANN 98. ICANN 1998. Perspectives in Neural Computing. Springer, London. https://doi.org/10.1007/978-1-4471-1599-1_27
Download citation
DOI: https://doi.org/10.1007/978-1-4471-1599-1_27
Published:
Publisher Name: Springer, London
Print ISBN: 978-3-540-76263-8
Online ISBN: 978-1-4471-1599-1
eBook Packages: Springer Book Archive