Abstract
A new regularization method for regression models is proposed. The criterion to be minimized contains a penalty term which explicitly links strength of penalization to the correlation between predictors. Like the elastic net, the method encourages a grouping effect where strongly correlated predictors tend to be in or out of the model together. A boosted version of the penalized estimator, which is based on a new boosting method, allows to select variables. Real world data and simulations show that the method compares well to competing regularization techniques. In settings where the number of predictors is smaller than the number of observations it frequently performs better than competitors, in high dimensional settings prediction measures favor the elastic net while accuracy of estimation and stability of variable selection favors the newly proposed method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bühlmann, P.: Boosting for high-dimensional linear models. Ann. Stat. 34, 559–583 (2006)
Bühlmann, P., Yu, B.: Boosting with the L2 loss: Regression and classification. J. Am. Stat. Assoc. 98, 324–339 (2003)
Donoho, D.L., Johnstone, I.M.: Adapting to unknown smoothness via wavelet shrinkage. J. Am. Stat. Assoc. 90, 1200–1224 (1995)
Frank, I.E., Friedman, J.H.: A statistical view of some chemometrics regression tools (with discussion). Technometrics 35, 109–148 (1993)
Friedman, J.H., Hastie, T., Tibshirani, R.: Additive logistic regression: A statistical view of boosting. Ann. Stat. 28, 337–407 (1999)
Fu, W.J.: Penalized regression: the bridge versus the lasso. J. Comput. Graph. Stat. 7, 397–416 (1998)
Hastie, T., Tibshirani, R., Friedman, J.H.: The Elements of Statistical learning. Springer, New York (2001)
Hoerl, A.E., Kennard, R.W.: Ridge regression: Bias estimation for nonorthogonal problems. Technometrics 12, 55–67 (1970)
Hurvich, C.M., Simonoff, J.S., Tsai, C.: Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. J. R. Stat. Soc. B 60, 271–293 (1998)
Klinger, A.: Hochdimensionale Generalisierte Lineare Modelle. Ph.D. Thesis, LMU München. Shaker Verlag, Aachen (1998)
Penrose, K.W., Nelson, A.G., Fisher, A.G.: Generalized body composition prediction equation for men using simple measurement techniques. Med. Sci. Sports Exerc. 17, 189 (1985)
Siri, W.B.: The gross composition of the body. In Tobias, C.A., Lawrence, J.H. (eds.) Advances in Biological and Medical Physics, vol. 4, pp. 239–280. Academic Press, San Diego (1956)
Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B 58, 267–288 (1996)
Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. B 67, 301–320 (2005)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Tutz, G., Ulbricht, J. Penalized regression with correlation-based penalty. Stat Comput 19, 239–253 (2009). https://doi.org/10.1007/s11222-008-9088-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11222-008-9088-5