Model selection via standard error adjusted adaptive lasso

Article

Abstract

The adaptive lasso is a model selection method shown to be both consistent in variable selection and asymptotically normal in coefficient estimation. The actual variable selection performance of the adaptive lasso depends on the weight used. It turns out that the weight assignment using the OLS estimate (OLS-adaptive lasso) can result in very poor performance when collinearity of the model matrix is a concern. To achieve better variable selection results, we take into account the standard errors of the OLS estimate for weight calculation, and propose two different versions of the adaptive lasso denoted by SEA-lasso and NSEA-lasso. We show through numerical studies that when the predictors are highly correlated, SEA-lasso and NSEA-lasso can outperform OLS-adaptive lasso under a variety of linear regression settings while maintaining the same theoretical properties of the adaptive lasso.

Keywords

BIC Model selection consistency Solution path Variable selection 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Belsley D. A., Kuh E., Welsch R. E. (1980) Regression diagnostics: Identifying influential data and sources of collinearity. Wiley, New YorkMATHCrossRefGoogle Scholar
  2. Efron B., Hastie T., Johnstone I., Tibshirani R. (2004) Least angle regression. The Annals of Statistics 32: 407–499MathSciNetMATHCrossRefGoogle Scholar
  3. Fan J., Li R. (2001) Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association 96: 1348–1360MathSciNetMATHCrossRefGoogle Scholar
  4. Friedman J., Hastie T., Tibshirani R. (2010) Regularization paths for generalized linear models via coordinate descent.. Journal of Statistical Software 33: 1–22Google Scholar
  5. Harrison D., Rubinfeld D. L. (1978) Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management 5: 81–102MATHCrossRefGoogle Scholar
  6. Härdle, W., Simar, L. (2007). Applied multivariate statistical analysis (2nd ed.). New York: Springer.Google Scholar
  7. Huang, J., Ma, S., Zhang, C.-H. (2008). Adaptive Lasso for sparse high-dimensional regression models. Statistica Sinica, 18, 1603–1618.Google Scholar
  8. Meinshausen N., Bühlmann P. (2006) High-dimensional graphs and variable selection with the lasso. The Annals of Statistics 34: 1436–1462MathSciNetMATHCrossRefGoogle Scholar
  9. Osborne, M., Presnell, B., Turlach, B. (2000). A new approach to variable selection in least squares problems. IMA Journal of Numerical Analysis, 20, 389–404.Google Scholar
  10. Shao J. (1993) Linear model selection by cross-validation. Journal of the American Statistical Association 88: 486–494MATHGoogle Scholar
  11. Tibshirani R. (1996) Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B: Methodological 58: 267–288MathSciNetMATHGoogle Scholar
  12. Wang H., Leng C. (2007) Unified LASSO estimation via least squares approximation. Journal of the American Statistical Association 102: 1039–1048MathSciNetMATHCrossRefGoogle Scholar
  13. Wang H., Li R., Tsai C.-L. (2007) Tuning parameter selectors for the smoothly clipped absolute deviation method.. Biometrika 94: 553–568MathSciNetMATHCrossRefGoogle Scholar
  14. Yang Y. (2007) Consistency of cross validation for comparing regression procedures. The Annals of Statistics 35: 2450–2473MathSciNetMATHCrossRefGoogle Scholar
  15. Zhang C.-H. (2010) Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics 38: 894–942MathSciNetMATHCrossRefGoogle Scholar
  16. Zhang T. (2011a) Adaptive forward-backward greedy algorithm for learning sparse representations. IEEE Transactions on Information Theory 57: 4689–4708CrossRefGoogle Scholar
  17. Zhang, T. (2011b). Multi-stage convex relaxation for feature selection. arXiv:1106.0565.Google Scholar
  18. Zhao P., Yu B. (2006) On model selection consistency of lasso. Journal of Machine Learning Research 7: 2541–2567MathSciNetMATHGoogle Scholar
  19. Zou H. (2006) The adaptive lasso and its oracle properties. Journal of the American Statistical Association 101: 1418–1429MATHGoogle Scholar
  20. Zou H., Hastie T., Tibshirani R. (2007) On the degrees of freedom of the lasso. The Annals of Statistics 35: 2173–2192MathSciNetMATHCrossRefGoogle Scholar
  21. Zou H., Zhang H. H. (2009) On the adaptive elastic-net with a diverging number of parameters. The Annals of Statistics 37: 1733–1751MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© The Institute of Statistical Mathematics, Tokyo 2012

Authors and Affiliations

  1. 1.School of StatisticsThe University of MinnesotaMinneapolisUSA

Personalised recommendations