Shrinkage and Sparse Estimation for High-Dimensional Linear Models

Abstract

In this paper, the high-dimensional sparse linear regression model  is considered, where the overall number of variables is larger than the number of observations. Many penalized regularization approaches including LASSO, group LASSO, and Elastic-Net, typically focus on selecting variables with strong effects. This may result in biased prediction, especially when weak signals outnumber strong signals. To solve this problem, we incorporate weak signals in variable selection and estimation. We propose a two-stage procedure, consisting of variable selection and post-selection estimation. The variable selection is done using the LASSO and Elastic-Net penalties to detect weak signals, whereas the post-selection estimation involves by shrinking a post-selection weighted ridge estimator in the direction of a selected candidate subset from the first stage. Monte-Carlo simulation experiment is conducted to evaluate the performance of each estimator in terms of the relative mean squared error. As a particular example, we apply the proposed method to analyze the GDP growth data.