Asymptotic bias of the $\ell _2$-regularized error variance estimator

Abstract

This study considers the $\ell _2$-regularized error variance estimator, an effective tool for non-sparse linear models. Specifically, this study investigates previously unexplored theoretical properties of this estimator, particularly its asymptotic bias in high dimensional settings where the number of variables increases with increase in the number of observations. This study proves that the estimator is asymptotically unbiased in large-sample settings ($\lim p/n <1$), while it is asymptotically biased in general ultra-high dimensional settings ($\lim p/n >1$). Specifically, the asymptotic bias of the $\ell _2$-regularized estimator in ultra-high dimensional settings is accurately derived where the covariates are independent.

Appendices

Appendix 1: Proof for Theorem 3

Proof

We begin by proving the first theoretical result with general $\Sigma$. For ease of notation, we use $\lambda = \lambda _{n}$ as a shorthand. Recall that the bias of the estimator (6) is expressed as follows:

$$\begin{aligned} \frac{{\textrm{E}}\left( {\hat{\sigma }}_\lambda ^2\right) - \sigma ^2}{\Vert {{\varvec{\beta }}}\Vert _2^2} = \frac{\lambda }{1-n^{-1}{\text{ tr }}\left( H_n^{\lambda _n} \right) } \cdot \frac{1}{\Vert {{\varvec{\beta }}}\Vert _2^2} {{\varvec{\beta }}}^T S_n (S_n+\lambda I_p)^{-1} {{\varvec{\beta }}}. \end{aligned}$$

(8)

Simple algebra yields that the denominator of the first term in Eq. (8) is as follows:

$$\begin{aligned} 1- \frac{{\text {tr}}(H_n^{\lambda _n})}{n} = 1- \frac{{\text {tr}}\left( \frac{1}{n}X(S_n + \lambda I_p)^{-1}X^T\right) }{n} = 1-\frac{p}{n} + \frac{p}{n} \cdot \frac{\lambda }{p} {\text {tr}}\bigg (\Big (S_n + \lambda I_p\Big )^{-1}\bigg ). \end{aligned}$$

Meanwhile, for any fixed $\lambda _0 \in {\mathbb {R}}^+$, the equations (2) and (3) in Dobriban et al. (2018) imply that

$$\begin{aligned} \lim _{n \rightarrow \infty } \frac{\lambda _0}{p} {\text{ tr }}\bigg (\Big (S_n + \lambda _0 I_p\Big )^{-1}\bigg ) = \frac{1}{\tau } \lambda _0 v(-\lambda _0) - \frac{1}{\tau } + 1 \qquad a.s. \end{aligned}$$

(9)

For ease of notation, let $f_n(\lambda ) := \frac{1}{\lambda }\left\{ \frac{n}{p} - 1 + \frac{\lambda }{p} {\text{ tr }}\left( \left( S_n + \lambda I_p\right) ^{-1}\right) \right\}$. Then, it is noted that $\lim _{n\rightarrow \infty } f_n(\lambda _0) = \frac{v(-\lambda _0)}{\tau }$ for each $\lambda _0 \in {\mathbb {R}}^+$. For the existence of $\lim _{n\rightarrow \infty } f_n(\lambda _n)$, it suffices to show that the sequence $f_n$, $n=1,2,\ldots$ is uniformly convergent on ${\mathbb {R}}^+$ by the Moore–Osgood theorem.

Since the nonzero eigenvalues of $S_n$ and $\frac{1}{n}XX^T$ are equal where $S_n$ has 0 as an additional eigenvalue with multiplicity $(p-n)$, we have

$$\begin{aligned} {\text{ tr }}\left( \left( S_n + \lambda I_p\right) ^{-1}\right) = \frac{p-n}{\lambda } + {\text{ tr }}\left( \left( \frac{1}{n}XX^T + \lambda I_n\right) ^{-1}\right) . \end{aligned}$$

Simple algebra yields that

$$\begin{aligned} f_n(\lambda ) = \frac{1}{\lambda }\left\{ \frac{n}{p} - 1 + \frac{\lambda }{p} {\text{ tr }}\left( \left( S_n + \lambda I_p\right) ^{-1}\right) \right\} = \frac{1}{p}{\text{ tr }}\left( \left( \frac{1}{n}XX^T + \lambda I_n\right) ^{-1}\right) . \end{aligned}$$

Hence, we have

$$\begin{aligned} |f_n(\lambda _0)|\le \left|\frac{n}{p\cdot \Lambda _{\min }^+(S_n)}\right|\le \left|\frac{n}{p\cdot \Lambda _{\min }^+ (Z^TZ/n) \Lambda _{\min }(\Sigma )}\right|\le \frac{1}{2\tau }\cdot \frac{2}{\rho _{\min }(1-\sqrt{\tau })^2} \end{aligned}$$

for any fixed $\lambda _0 \in {\mathbb {R}}^+$ and sufficiently all large n, where $\Lambda _{\min }^+(\cdot )$ denotes the smallest nonzero eigenvalue. Here, the last inequality follows from the fact that $\lim _{n\rightarrow \infty }\Lambda _{\min }^+ (Z^TZ/n) = (1-\sqrt{\tau })^2$ by Theorem 2 in Bai and Yin (1993).

Similarly, for the derivative $f_n'(\lambda )=\partial f_n(\lambda )/\partial \lambda$, it is also observed that

$$\begin{aligned} |f_n'(\lambda _0) |=\left|\frac{1}{p}{\text{ tr }}\left( \left( \frac{1}{n}XX^T + \lambda _0 I_n\right) ^{-2}\right) \right|\le \left|\frac{n}{p\cdot (\Lambda _{\min }^+ (S_n))^2}\right|\le \frac{1}{2\tau }\cdot \frac{4}{\rho _{\min }^2(1-\sqrt{\tau })^4}, \end{aligned}$$

for any fixed $\lambda _0 \in {\mathbb {R}}^+$ and sufficiently large n.

Hence, $f_n$ and its derivative are uniformly bounded, and hence the sequence $f_n$, $n=1,2,\ldots$ is uniformly convergent by the Arzela–Ascoli theorem. Consequently, it concludes that $\lim _{n\rightarrow \infty } f_n(\lambda _n)$ exists. In addition, according to Lemma 2.3 in Dobriban et al. (2018), $\lim _{n\rightarrow \infty } v(-\lambda _n) = v(-c)$ for $\lim \lambda _n = c$, and hence, we have

$$\begin{aligned} \lim _{n \rightarrow \infty }\frac{1}{\lambda _n}\bigg \{\frac{n}{p} - 1 + \frac{\lambda _n}{p} {\text{ tr }}\bigg (\Big (S_n + \lambda _n I_p\Big )^{-1}\bigg )\bigg \} = \frac{v(-c)}{\tau } \qquad a.s. \end{aligned}$$

for all $c \in [0,\infty )$.

Multiplying $\tau$ on both sides, the limit of the first term of the bias is as follows:

$$\begin{aligned} \lim _{n\rightarrow \infty } \frac{\lambda _n}{1-n^{-1}{\text{ tr }}\left( H_n^{\lambda _n}\right) } = \frac{1}{v(-c)} \qquad a.s. \end{aligned}$$

(10)

Now, it is focused on the second part of the bias, $\frac{1}{\Vert {{\varvec{\beta }}}\Vert _2^2} {{\varvec{\beta }}}^T S_n (S_n+\lambda _n I_p)^{-1} {{\varvec{\beta }}}$. As shown above, $S_n$ has 0 as an eigenvalue with multiplicity $(p-n)$ and $\Lambda _{\min }^+(S_n) \ge (1-\sqrt{\tau })^2\rho _{\min }/2$ for sufficiently all large n. Define $\{e_1,\ldots ,e_{n}\}$ as the eigenvectors of $S_n$ corresponding to the nonzero eigenvalues and $\{e_{n+1},\ldots ,e_{p}\}$ as the eigenvectors of $S_n$ corresponding to the eigenvalue 0. Then, ${{\varvec{\beta }}}$ is decomposed as ${{\varvec{\beta }}}= {{\varvec{\beta }}}_{E_1} + {{\varvec{\beta }}}_{E_2}$ where $E_1$ and $E_2$ are subspaces of ${\mathbb {R}}^p$ spanned by $\{e_1,\ldots ,e_{n}\}$ and $\{e_{n+1},\ldots ,e_{p}\}$, respectively. Hence, we have

$$\begin{aligned} \frac{1}{\Vert {{\varvec{\beta }}}\Vert _2^2}{{\varvec{\beta }}}^T S_n (S_n+\lambda _n I_p)^{-1} {{\varvec{\beta }}}&= \frac{1}{\Vert {{\varvec{\beta }}}\Vert _2^2}{{\varvec{\beta }}}_{E_1}^T S_n (S_n+\lambda _n I_p)^{-1} {{\varvec{\beta }}}_{E_1} \\&\ge \frac{\Vert {{\varvec{\beta }}}_{E_1}\Vert _2^2}{\Vert {{\varvec{\beta }}}\Vert _2^2} \cdot \frac{(1-\sqrt{\tau })^2\rho _{\min }/2}{\lambda _n+(1-\sqrt{\tau })^2\rho _{\min }/2} \qquad a.s. \end{aligned}$$

for all sufficiently large n.

Applying Eqs. (8) and (10), we obtain

$$\begin{aligned} \liminf _{n\rightarrow \infty } \frac{\left[ {\textrm{E}}\left( {\hat{\sigma }}_{\lambda _n}^2\right) - \sigma ^2\right] }{\Vert {{\varvec{\beta }}}\Vert _2^2} \ge \frac{1}{v(-c)} \cdot \frac{(1-\sqrt{\tau })^2\rho _{\min }/2}{c+(1-\sqrt{\tau })^2\rho _{\min }/2} \cdot \liminf _{n\rightarrow \infty } \frac{\Vert {{\varvec{\beta }}}_{E_1}\Vert _2^2}{\Vert {{\varvec{\beta }}}\Vert _2^2} \qquad a.s. \end{aligned}$$

Since $E_1$ is determined by the design matrix $X = Z\Sigma ^{1/2}$ under the setting where Z is randomly generated and $\Sigma$ has full rank, it holds that $\liminf _{n\rightarrow \infty } \frac{\Vert {{\varvec{\beta }}}_{E_1}\Vert _2^2}{\Vert {{\varvec{\beta }}}\Vert _2^2} > 0$ almost surely. This completes the proof of the theoretical result with general $\Sigma$. $\square$

Appendix 2: Proof for Theorem 4

Proof

Now, we prove the second theoretical result with $\Sigma = I_p$. Eq. (10) also holds for $\Sigma =I_p$, and hence it suffices to focus on the term $\frac{1}{\Vert {{\varvec{\beta }}}\Vert _2^2} {{\varvec{\beta }}}^T S_n (S_n+\lambda _n I_p)^{-1} {{\varvec{\beta }}}= 1 - \frac{\lambda _n}{\Vert {{\varvec{\beta }}}\Vert _2^2} {{\varvec{\beta }}}^T (S_n+\lambda _n I_p)^{-1} {{\varvec{\beta }}}$. First, we use Theorem 1 in Rubio and Mestre (2011) with $\Theta = I_p/p$ that implies

$$\begin{aligned} \lim _{n \rightarrow \infty } \left[ \frac{1}{p}{\text{ tr }}\left( (S_n+\lambda _0 I_p)^{-1} \right) - \frac{1}{p}{\text{ tr }}\left( (x_n(\lambda _0) I_p)^{-1})\right) \right] = 0 \qquad a.s., \end{aligned}$$

for any fixed $\lambda _0 \in {\mathbb {R}}^+$ where $x_n(\lambda _0)>0$ is the deterministic sequence satisfying a certain fixed-point equation for each n.

By Eq. (9), the following holds that

$$\begin{aligned} \lim _{n\rightarrow \infty } \lambda _0\left( x_n(\lambda _0)\right) ^{-1} = \lim _{n\rightarrow \infty }\frac{\lambda _0}{p}{\text{ tr }}\left( (S_n + \lambda _0 I_p)^{-1}\right) = \frac{1}{\tau } \lambda _0 v(-\lambda _0) - \frac{1}{\tau } + 1, \qquad {a.s.} \end{aligned}$$

for any fixed $\lambda _0 \in {\mathbb {R}}^+$, and hence we can define $x(\lambda _0): = \lim _{n\rightarrow \infty } x_n(\lambda _0)$.

Furthermore, by Theorem 1 in Rubio and Mestre (2011) with $\Theta = {{\varvec{\beta }}}{{\varvec{\beta }}}^T/\Vert {{\varvec{\beta }}}\Vert _2^2$,

$$\begin{aligned} \lim _{n\rightarrow \infty } \frac{1}{\Vert {{\varvec{\beta }}}\Vert _2^2}{{\varvec{\beta }}}^T(S_n+\lambda _0 I_p)^{-1}{{\varvec{\beta }}}- \frac{1}{\Vert {{\varvec{\beta }}}\Vert _2^2}{{\varvec{\beta }}}^T\left( x_n(\lambda _0) I_p\right) ^{-1}{{\varvec{\beta }}}= 0 \qquad a.s. \end{aligned}$$

for any fixed $\lambda _0 \in {\mathbb {R}}^+$, and it implies that

$$\begin{aligned} \lim _{n\rightarrow \infty } \frac{\lambda _0}{\Vert {{\varvec{\beta }}}\Vert _2^2}{{\varvec{\beta }}}^T(S_n+\lambda _0 I_p)^{-1}{{\varvec{\beta }}}= \lambda _0 x(\lambda _0)^{-1} \qquad a.s. \end{aligned}$$

for any fixed $\lambda _0 \in {\mathbb {R}}^+$. Following the same argument for Eq. (10), the existence of the limit of $g_n(\lambda _n) := \frac{\lambda _n}{\Vert {{\varvec{\beta }}}\Vert _2^2}{{\varvec{\beta }}}^T(S_n+\lambda _n I_p)^{-1}{{\varvec{\beta }}}=1-\frac{1}{\Vert {{\varvec{\beta }}}\Vert _2^2}{{\varvec{\beta }}}^TS_n(S_n+\lambda _n I_p)^{-1}{{\varvec{\beta }}}$ is confirmed by the following inequalities:

$$\begin{aligned} |g_n(\lambda _0)|&\le \left|\lambda _0 \cdot \Lambda _{\max } \left( (S_n+\lambda _0 I_p)^{-1}\right) \right|\le 1,\\ |g_n'(\lambda _0)|&= \left|\frac{1}{\Vert {{\varvec{\beta }}}\Vert _2^2}{{\varvec{\beta }}}^T S_n(S_n+\lambda _0 I_p)^{-2} {{\varvec{\beta }}}\right|\le \left|\Lambda _{\max } \left( S_n(S_n+\lambda _0 I_p)^{-2}\right) \right|\\&\le \frac{2}{\rho _{\min }(1-\sqrt{\tau })^2}, \end{aligned}$$

for any fixed $\lambda _0 \in {\mathbb {R}}^+$ and sufficiently all large n.

Then, it holds that

$$\begin{aligned} \lim _{n\rightarrow \infty } \frac{\lambda _n }{\Vert {{\varvec{\beta }}}\Vert _2^2}{{\varvec{\beta }}}^T(S_n+\lambda _n I_p)^{-1}{{\varvec{\beta }}}&= \lim _{n\rightarrow \infty } \lambda _n x(\lambda _n)^{-1} \\&= \lim _{n\rightarrow \infty } \left[ \frac{1}{\tau } \lambda _n v(-\lambda _n) - \frac{1}{\tau } + 1 \right] \\&= \frac{cv(-c)}{\tau }-\frac{1}{\tau } + 1 \qquad a.s. \end{aligned}$$

Hence, it is concluded that

$$\begin{aligned} \frac{\left[ {\textrm{E}}\left( {\hat{\sigma }}_{\lambda }^2\right) - \sigma ^2 \right] }{\Vert {{\varvec{\beta }}}\Vert _2^2}&= \frac{\lambda }{1-n^{-1}{\text{ tr }}\left( H_n^{\lambda _n} \right) } \cdot \frac{1}{\Vert {{\varvec{\beta }}}\Vert _2^2}{{\varvec{\beta }}}^T S_n (S_n+\lambda I_p)^{-1} {{\varvec{\beta }}}\\ {}&= \frac{\lambda }{1-n^{-1}{\text{ tr }}\left( H_n^{\lambda _n} \right) } \cdot \left[ \frac{\Vert {{\varvec{\beta }}}\Vert _2^2}{\Vert {{\varvec{\beta }}}\Vert _2^2} - \frac{\lambda }{\Vert {{\varvec{\beta }}}\Vert _2^2} {{\varvec{\beta }}}^T(S_n+\lambda I_p)^{-1}{{\varvec{\beta }}}\right] \\&\rightarrow \frac{1}{v(-c)} \cdot \left[ 1- \left( \frac{cv(-c)}{\tau }-\frac{1}{\tau } + 1\right) \right] = \frac{1}{\tau } \left( \frac{1}{v(-c)}-c\right) \qquad a.s. \end{aligned}$$

as $n\rightarrow \infty$. $\square$