Advertisement

Statistics and Computing

, Volume 19, Issue 3, pp 239–253 | Cite as

Penalized regression with correlation-based penalty

  • Gerhard Tutz
  • Jan UlbrichtEmail author
Article

Abstract

A new regularization method for regression models is proposed. The criterion to be minimized contains a penalty term which explicitly links strength of penalization to the correlation between predictors. Like the elastic net, the method encourages a grouping effect where strongly correlated predictors tend to be in or out of the model together. A boosted version of the penalized estimator, which is based on a new boosting method, allows to select variables. Real world data and simulations show that the method compares well to competing regularization techniques. In settings where the number of predictors is smaller than the number of observations it frequently performs better than competitors, in high dimensional settings prediction measures favor the elastic net while accuracy of estimation and stability of variable selection favors the newly proposed method.

Keywords

Correlation-based estimator Boosting Variable selection Elastic net Lasso Penalization 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bühlmann, P.: Boosting for high-dimensional linear models. Ann. Stat. 34, 559–583 (2006) zbMATHCrossRefGoogle Scholar
  2. Bühlmann, P., Yu, B.: Boosting with the L2 loss: Regression and classification. J. Am. Stat. Assoc. 98, 324–339 (2003) zbMATHCrossRefGoogle Scholar
  3. Donoho, D.L., Johnstone, I.M.: Adapting to unknown smoothness via wavelet shrinkage. J. Am. Stat. Assoc. 90, 1200–1224 (1995) zbMATHCrossRefMathSciNetGoogle Scholar
  4. Frank, I.E., Friedman, J.H.: A statistical view of some chemometrics regression tools (with discussion). Technometrics 35, 109–148 (1993) zbMATHCrossRefGoogle Scholar
  5. Friedman, J.H., Hastie, T., Tibshirani, R.: Additive logistic regression: A statistical view of boosting. Ann. Stat. 28, 337–407 (1999) CrossRefMathSciNetGoogle Scholar
  6. Fu, W.J.: Penalized regression: the bridge versus the lasso. J. Comput. Graph. Stat. 7, 397–416 (1998) CrossRefGoogle Scholar
  7. Hastie, T., Tibshirani, R., Friedman, J.H.: The Elements of Statistical learning. Springer, New York (2001) zbMATHGoogle Scholar
  8. Hoerl, A.E., Kennard, R.W.: Ridge regression: Bias estimation for nonorthogonal problems. Technometrics 12, 55–67 (1970) zbMATHCrossRefGoogle Scholar
  9. Hurvich, C.M., Simonoff, J.S., Tsai, C.: Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. J. R. Stat. Soc. B 60, 271–293 (1998) zbMATHCrossRefMathSciNetGoogle Scholar
  10. Klinger, A.: Hochdimensionale Generalisierte Lineare Modelle. Ph.D. Thesis, LMU München. Shaker Verlag, Aachen (1998) Google Scholar
  11. Penrose, K.W., Nelson, A.G., Fisher, A.G.: Generalized body composition prediction equation for men using simple measurement techniques. Med. Sci. Sports Exerc. 17, 189 (1985) Google Scholar
  12. Siri, W.B.: The gross composition of the body. In Tobias, C.A., Lawrence, J.H. (eds.) Advances in Biological and Medical Physics, vol. 4, pp. 239–280. Academic Press, San Diego (1956) Google Scholar
  13. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B 58, 267–288 (1996) zbMATHMathSciNetGoogle Scholar
  14. Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. B 67, 301–320 (2005) zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  1. 1.Ludwig-Maximilians-Universität MünchenMünchenGermany

Personalised recommendations