Model Selection for High-Dimensional Problems

Abstract

High-dimensional data analysis is becoming more and more important to both academics and practitioners in finance and economics but is also very challenging because the number of variables or parameters in connection with such data can be larger than the sample size. Recently, several variable selection approaches have been developed and used to help us select significant variables and construct a parsimonious model simultaneously. In this chapter, we first provide an overview of model selection approaches in the context of penalized least squares. We then review independence screening, a recently developed method for analyzing ultrahigh-dimensional data where the number of variables or parameters can be exponentially larger than the sample size. Finally, we discuss and advocate multistage procedures that combine independence screening and variable selection and that may be especially suitable for analyzing high-frequency financial data.

Penalized least squares seek to keep important predictors in a model while penalizing coefficients associated with irrelevant predictors. As such, under certain conditions, penalized least squares can lead to a sparse solution for linear models and achieve asymptotic consistency in separating relevant variables from irrelevant ones. Independence screening selects relevant variables based on certain measures of marginal correlations between candidate variables and the response.

Appendix 1: One-Step Sparse Estimates

This appendix outlines the local linear approximation (LLA) algorithm, discussed in Sect. 77.2.3, for finding a solution of penalized least squares for a broad class of penalty functions. For illustration, we focus on the nondifferentiable, nonconvex L _0.5 and SCAD penalties introduced in Sect. 77.2.2, as both are ideally suited to the LLA algorithm (Zou and Li 2008). Recall that these two penalty functions are given by Eqs. (77.18) and (77.23), respectively. We note that L _0.5 and SCAD apparently lead to concave objective functions that are singular at the origin. We illustrate later that in linear regressions this optimization problem can be reduced into solving penalized least squares with an L ₁ penalty.

We begin with the following one-step LLA estimator:

$$ \widehat{\beta}= \arg\ \underset{\beta }{ \min}\left\{\frac{1}{2}{\left\Vert \mathrm{y}-\mathbf{X}\beta \right\Vert}^2+n{\displaystyle \sum_{j=1}^pp{\prime}_{\lambda }}\left(\left|{\tilde{\beta}}_j\right|\right)\left|{\beta}_j\right|\right\}, $$

(77.40)

where the initial estimate $\tilde{\beta}$ is usually represented by the ordinary least squares estimator in practice if n > p. Theoretically, $\tilde{\beta}$ could be any root-n-consistent estimator for β. In the case where n < p, the plain vanilla L ₁ estimator (LASSO) would be employed instead.

Following Zou and Li (2008), we discuss separately algorithms for the objective function Q(β) with two different types of penalty functions, which correspond to L _0.5 and SCAD, respectively. Nonetheless, the idea underlying the two algorithms is the same – namely, using a data transform to simplify the penalized least squares to a standard L ₁ regularization problem, for which we can take advantage of the related efficient algorithms. In particular, the LARS algorithm (Efron et al. 2004) has been widely applied to obtain the entire solution path of LASSO and forward stage-wise regressions. The relevant R package can be downloaded from http://www.stanford.edu/~hastie/swData.htm#SvmP.

Consider first type one of penalty functions.

Type 1. The tuning parameter could be disentangled from the penalty function, i.e., p _λ (θ) satisfies that p _λ (θ) = λp _λ (θ) and p ^′ _λ (θ) > 0. For example, bridge penalties p _λ (|θ|) = λ|θ|^q for 0 < q < 1 and the logarithm penalty p _λ (|θ|) = λ log|θ|.

Algorithm 1

• Step 1. Create working data by ${\mathrm{x}}_j^{*}={\mathrm{x}}_j/p{\prime}_{\lambda}\left(\left|{\tilde{\beta}}_j\right|\right)$, for j = 1 , … , p.

• Step 2. Apply the LARS algorithm to solve the penalized L1 regression:

  $$ {\widehat{\beta}}^{*}= arg\underset{\beta }{ \min}\left\{\frac{1}{2}{\left\Vert \mathrm{y}-{\mathbf{X}}^{*}\beta \right\Vert}^2+ n\lambda {\displaystyle \sum_{j=1}^p\left|{\beta}_j\right|}\right\}. $$

• Step 3. Let the final one-step estimator be ${\widehat{\beta}}_j={\widehat{\beta}}_j^{*}/p{\prime}_{\lambda}\left(\left|{\tilde{\beta}}_j\right|\right)$ for j =1 , … , p.

As we can see, through the one-step sparse estimator, LLA for the L ₀. ₅ penalty is equivalent to the adaptive LASSO, with ${\widehat{w}}_j={\left|{\tilde{\beta}}_j\right|}^{-0.5}$ (see Eq. 4 in Zou 2006). In computing the LARS estimates, tuning parameter λ can be chosen using the cv.lars routine embedded in the “lars” package. For example, the following command

$$ cv. lars\left( xstar,y,K=5, plot. it= TRUE, se= TRUE, type=`` lasso"\right) $$

instructs the R to plot fivefold cross-validated mean squared prediction error (MSE) for different values of λ. Then we are able to pick the λ with the smallest MSE and find out the step in LARS corresponding to that λ value. The main function of the package

$$ lars\left( xstar,y, type=`` lasso"\right) $$

provides the entire sequence of coefficients and fits, starting from zero to the least squares fit.

Next, consider type 2 of penalty functions.

Type 2. p _λ (θ) satisfies that the derivative p ^′ _λ (θ) is zero for some values. In addition, the regularization parameter λ cannot be separated from p _λ (θ). For example, the SCAD penalty with the first derivative

$$ p{\prime}_{\lambda}\left(\theta \right)=\lambda \left\{I\left(\theta \le \lambda \right)+\frac{{\left( a\lambda -\theta \right)}_{+}}{\left(a-1\right)\lambda }I\left(\theta>\lambda \right)\right\}, $$

(77.41)

where θ > 0, p _λ (0) = 0, and a > 2.

We define U = {j : p ^′ _λ (θ) = 0} and V = {j : p ^′ _λ (θ) > 0}. Accordingly, we write X = [X _U , X _V ] and $\widehat{\beta}={\left({\widehat{\beta}}_U^{\mathrm{T}},{\widehat{\beta}}_V^{\mathrm{T}}\right)}^{\mathrm{T}}$.

Algorithm 2

• Step 1. Create working data by ${\mathrm{x}}_j^{*}={\mathrm{x}}_j\lambda /p{\prime}_{\lambda}\left(\left|{\tilde{\beta}}_j\right|\right)$, for j ∈ V. Let H _U be the projection matrix in the space of {x ^* _j , j ∈ U}. Compute $\tilde{\mathrm{y}}=\left(I-{H}_U\right)\mathrm{y}$ and ${\tilde{\mathrm{X}}}_V^{*}=\left(I-H\right){\mathrm{X}}_V^{*}$.

• Step 2. Apply the LARS algorithm to solve the penalized L ₁ regression:

  $$ {\widehat{\beta}}_V^{*}= arg\ \underset{\beta }{ \min}\left\{\frac{1}{2}{\left\Vert \tilde{\mathrm{y}}-{\tilde{\mathrm{X}}}_V^{*}\beta \right\Vert}^2+ n\lambda {\displaystyle \sum_{j=1}^p\left|{\beta}_j\right|}\right\}. $$

• Step 3. Compute ${\widehat{\beta}}_U^{*}={\left({\mathbf{X}}_U^{*T}{\mathbf{X}}_U^{*}\right)}^{-1}\;{\mathbf{X}}_U^{*T}\left(\tilde{\mathrm{y}}-{\mathbf{X}}_V^{*}{\widehat{\beta}}_V^{*}\right)$. Then, the final one-step estimator $\widehat{\beta}$ is obtained by

$$ {\widehat{\beta}}_U={\widehat{\beta}}_U^{*}\kern0.5em and\kern0.5em {\widehat{\beta}}_j={\widehat{\beta}}_j^{*}\lambda /p{\prime}_{\lambda}\left(\left|{\tilde{\beta}}_j\right|\right)\kern0.62em for\kern0.5em j\in V. $$

We note that cross-validation routine is not applicable in this situation, as the sets U and V could change as λ varies. In other words, different values of the tuning parameter lead to different transforms of observations. As a result, the SCAD-type penalty requires reexecuting the cross-validation and solving the one-step estimator for each fixed λ. Alternatively, we can determine λ by a non-data-driven approach such as the BIC-based tuning parameter selector. Indeed, Wang et al. (2007) show that the commonly used generalized cross-validation tends to overshoot the correct number of nonzero coefficients. Instead, BIC can be used to consistently identify the true model. For practitioners, it would be more convenient to directly call the procedure getfinalSCADcoef included in the “SIS” package (Fan et al. 2010). The option tune.method = c (“AIC”, “BIC”) is used to specify the selection criterion.

Finally, we use a simple numerical example to illustrate the performance of one-step sparse estimates. In this example, simulation data is generated by executing the following command in the R 2.15.0 program:

$$ \begin{array}{cc}\hfill \begin{array}{l} set. seed(0)\\ {}b<-\kern0.5em c\left(4,4,4,-6* sqrt(2)\right)\\ {}n=150\\ {}p=300\\ {}x= matrix\left( rnorm\left(n*p, mean=0, sd=1\right),n,p\right)\\ {}y<-x\left[,1:4\right]\%*\% b+ rnorm(150)\end{array}\hfill & \hfill .\hfill \end{array} $$

The one-step sparse estimates with the L ₀.₅ and SCAD penalty are summarized as follows. Both selection methods include all four significant variables, and the L _0.5 penalty falsely selects one noise variable. The estimated nonzero coefficients under these two methods are reported in the following:

$$ \begin{array}{l}{b}_{scad}=\left[3.986,3.937,3.944,-8.369\right],\\ {}{b}_{L05}=\left[3.942,3.964,3.997,-8.388,0.237\right].\end{array} $$

Note that this example alone does not mean that the SCAD outperforms the L _0.5 penalty, because a valid Monte Carlo simulation study usually requires at least 1,000 simulated data sets.