LAD, LASSO and Related Strategies in Regression Models

Abstract

In the context of linear regression models, it is well-known that the ordinary least squares estimator is very sensitive to outliers whereas the least absolute deviations (LAD) is an alternative method to estimate the known regression coefficients. Selecting significant variables is very important; however, by choosing these variables some information may be sacrificed. To prevent this, in our proposal, we can combine the full model estimates toward the candidate sub-model, resulting in improved estimators in risk sense. In this article, we consider shrinkage estimators in a sparse linear regression model and study their relative asymptotic properties. Advantages of the proposed estimators over the usual LAD estimator are demonstrated through a Monte Carlo simulation as well as a real data example.

Appendix

Lemma 1

[6] Whenever the assumptions (A) and (B) hold, $\sqrt{n}(\hat{\varvec{\beta }}^{F}-\varvec{\beta })$ convergences in distribution to a p-dimensional normal distribution with mean $\varvec{0}$ and covariance matrix $\tau ^2 \varvec{C}^{-1}$, where $\tau ^2$ is the asymptotic variance of the sample median of symmetric F distribution with median 0, i.e. $\tau ^2=[2f(0)]^{-2}$. On the other hand,

$$\begin{aligned} \sqrt{n}\left( \hat{\varvec{\beta }}^{F}-\varvec{\beta }\right) \overset{d}{\rightarrow }\mathscr {N}_p \left( \varvec{0}, \tau ^2 \varvec{C}^{-1}\right) . \end{aligned}$$

(17)

Corollary 1

Assume (A) and (B) are held, thus

$$\begin{aligned} \hat{\varvec{\beta }}_1^{F}\sim & {} \mathscr {N}_{p_1}\left( \varvec{\beta }_1,\tau ^2 \varvec{C}_{11.2}^{-1} \right) ,\end{aligned}$$

(18)

$$\begin{aligned} \hat{\varvec{\beta }}_2^{F}\sim & {} \mathscr {N}_{q} \left( \varvec{\beta }_2, \tau ^2 \varvec{C}_{22.1}^{-1}\right) . \end{aligned}$$

(19)

Proof

Suppose that the matrix $\varvec{x}$ is partitioned as $\varvec{X}_1$ and $\varvec{X}_2$ with sizes $n\times p_1$ and $n\times p_2$, respectively. Accordingly, $\varvec{\mu }$ and $\varvec{\Sigma }$ partitioned as follows:

$$\begin{aligned} \varvec{\mu }=\begin{bmatrix} \varvec{\mu }_1 \\ \varvec{\mu }_2 \end{bmatrix}\quad \text {with dimension} \begin{bmatrix} p_1\times 1 \\ p_2 \times 1 \end{bmatrix}, \end{aligned}$$

and

$$\begin{aligned} \varvec{\Sigma }=\begin{bmatrix} \varvec{\Sigma }_{11}&\varvec{\Sigma }_{12}\\ \varvec{\Sigma }_{21}&\varvec{\Sigma }_{22} \end{bmatrix} \quad \text {with dimension} \quad \begin{bmatrix} p_1\times p_1&p_1\times p_2\\ p_2 \times p_1&p_2 \times p_2 \end{bmatrix}. \end{aligned}$$

Now, based on assumption (A) consider the covariance matrix as $\tau ^2 \varvec{C}^{-1}$. Using Schure complement, the inverted blockwise is as below

$$\begin{aligned} \varvec{\Sigma }^{-1} = \tau ^2\begin{bmatrix} \varvec{C}_{11.2}^{-1}&-\varvec{C}_{11.2}^{-1}\varvec{C}_{12}\varvec{C}_{22}^{-1}\\ -\varvec{C}_{22.1}^{-1} \varvec{C}_{21} \varvec{C}_{11}^{-1}&\varvec{C}_{22.1}^{-1} \end{bmatrix}. \end{aligned}$$

(20)

Base on Lemma 1, the result is concluded.

Proof

(Theorem 1). Based on Eq. (19), we can construct Hotelling’s T-squared statistic as

$$\begin{aligned} \frac{\left( \hat{\varvec{\beta }}_2^{F}-\varvec{\beta }_2\right) ^T \varvec{C}_{22.1} \left( \hat{\varvec{\beta }}_2^{F}-\varvec{\beta }_2\right) }{\tau ^2}, \end{aligned}$$

(21)

which follows of a $\chi ^2$-distribution with $p_2$ d.f. and non-centrality parameter $\varDelta$. Under the null hypothesis, the test statistic, $\mathscr {L}_n$, is a central $\chi ^2$ distribution with $p_2$ d.f.

Proof

(Theorem 2). The first equation is derived from (18). To obtain the asymptotic distribution of the restricted LAD estimator and complete the proof, we must use the conditional normal $\left( \hat{\varvec{\beta }}^{F}_1|\hat{\varvec{\beta }}_2^{F}=\varvec{\beta }_2=\varvec{0}\right)$.

The distribution $X_1|X_2=x_2$ is

$$\begin{aligned} \mathscr {N}_{p_1-p_2}\left( \varvec{\mu }_1+\varvec{C}_{12}\varvec{C}_{22}^{-1}(x_2-\varvec{\mu }_2), \varvec{C}_{11}-\varvec{C}_{12}\varvec{C}_{22}^{-1}\varvec{C}_{21} \right) . \end{aligned}$$

(22)

Now, with substituting the Eqs. (18)–(20) into (22) the result is obtained.

The mean vector is as

$$\begin{aligned} \varvec{\beta }_1+\left( -\varvec{C}_{11.2}^{-1}\varvec{C}_{12}\varvec{C}_{22}^{-1}\right) \left( \varvec{C}_{22.1}^{-1}\right) ^{-1}(\varvec{\beta }_2 -\varvec{\beta }_2)=\varvec{\beta }_1. \end{aligned}$$

(23)

Also, the covariance matrix has the form

$$\begin{aligned} \mathrm{Cov}\left( \hat{\varvec{\beta }}_1^{F}|\hat{\varvec{\beta }}_2^{F}=\varvec{\beta }_2 = \varvec{0}\right)= & {} \varvec{C}_{11.2}^{-1}-\left( -\varvec{C}_{11.2}^{-1}\varvec{C}_{12}\varvec{C}_{22}^{-1}\right) \varvec{C}_{22.1}\left( -\varvec{C}_{22.1}^{-1}\varvec{C}_{21}\varvec{C}_{11}^{-1}\right) \\= & {} \varvec{C}_{11.2}^{-1}-\varvec{C}_{11.2}^{-1}\varvec{C}_{12}\varvec{C}_{22}^{-1}\varvec{C}_{21}\varvec{C}_{11}^{-1}. \end{aligned}$$

(24)

Using this fact that $\varvec{C}_{12}\varvec{C}_{22}^{-1}\varvec{C}_{21}=\varvec{C}_{11}-\varvec{C}_{11.2}$, the Eq. (24) equals to

$$\begin{aligned} \varvec{C}_{11.2}^{-1}-\varvec{C}_{11.2}^{-1}\left( \varvec{C}_{11}-\varvec{C}_{11.2}\right) \varvec{C}_{11}^{-1}= & {} \varvec{C}_{11.2}^{-1}-\varvec{C}_{11.2}^{-1}+\varvec{C}_{11}^{-1}.\\= & {} \varvec{C}_{11}^{-1}. \end{aligned}$$

(25)

The proof is complete.