Prediction error bounds for linear regression with the TREX

Abstract

The TREX is a recently introduced approach to sparse linear regression. In contrast to most well-known approaches to penalized regression, the TREX can be formulated without the use of tuning parameters. In this paper, we establish the first known prediction error bounds for the TREX. Additionally, we introduce extensions of the TREX to a more general class of penalties, and we provide a bound on the prediction error in this generalized setting. These results deepen the understanding of the TREX from a theoretical perspective and provide new insights into penalized regression in general.

Appendices

Appendix A: proof of a generalization of Theorem 1

We first consider a generalization of Assumption 1.

Assumption 4

The signal $\beta ^*$ is sufficiently small such that for some $\kappa _{1}>1$ and $\kappa _{2}>2$ with $1/\kappa _1 + 2/\kappa _2 <1$,

$$\begin{aligned} \Vert \beta ^*\Vert _1\le \frac{1}{4}\left( 1-\frac{1}{\kappa _{1}}-\frac{2}{\kappa _{2}}\right) \frac{\Vert \varepsilon \Vert _2^2}{\Vert X^{\top }\varepsilon \Vert _{\infty }}. \end{aligned}$$

As a first step toward the proof of Theorem 1, we show that any TREX solution has larger $\ell _{1}$-norm than any LASSO solution with tuning parameter $\lambda =\hat{u}$.

Lemma 3

Any TREX solution (4) satisfies

$$\begin{aligned} \Vert \hat{\beta }\Vert _{1}\ge \Vert \tilde{\beta }(\hat{u})\Vert _{1}, \end{aligned}$$

where $\tilde{\beta }(\hat{u})$ is any LASSO solution as in (2) with tuning parameter $\lambda =\hat{u}$.

Proof (of Lemma 3)

If $\tilde{\beta }(\hat{u})=0$, the statement holds trivially. Now for $\tilde{\beta }(\hat{u})\ne 0$, the KKT conditions for LASSO imply that

$$\begin{aligned} \Vert X^{\top }(Y-X\tilde{\beta }(\hat{u}))\Vert _{\infty }=\hat{u}. \end{aligned}$$

Together with the definition of $\hat{\beta }$, this yields

$$\begin{aligned} \Vert Y-X\hat{\beta }\Vert ^{2}_{2}+c\hat{u}\Vert \hat{\beta }\Vert _{1}\le \Vert Y-X\tilde{\beta }(\hat{u})\Vert ^{2}_{2}+c\hat{u}\Vert \tilde{\beta }(\hat{u})\Vert _{1}. \end{aligned}$$

On the other hand, the definition of the LASSO implies

$$\begin{aligned} \Vert Y-X\tilde{\beta }(\hat{u})\Vert ^{2}_{2}+2\hat{u}\Vert \tilde{\beta }(\hat{u})\Vert _{1}\le \Vert Y-X\hat{\beta }\Vert ^{2}_{2}+2\hat{u}\Vert \hat{\beta }\Vert _{1}. \end{aligned}$$

Combining these two displays gives us

$$\begin{aligned} (c-2)\hat{u}\Vert \hat{\beta }\Vert _{1}\le (c-2)\hat{u}\Vert \tilde{\beta }(\hat{u})\Vert _{1}. \end{aligned}$$

The claim follows now from $c< 2$.

We are now ready to prove a generalization of Theorem 1.

Theorem 4

Let Assumption 4 be fulfilled, and let $\tilde{\lambda }:=\max \{\kappa _{1} \hat{u},\frac{\kappa _{2}}{c}\Vert X^\top \varepsilon \Vert _\infty \}$. Then, for any $\hat{u}\le \Vert X^\top Y\Vert _\infty /\kappa _1$, the prediction loss of the TREX satisfies

$$\begin{aligned} \Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\le & {} \left( \frac{1}{\kappa _{1}}+\frac{2}{\kappa _{2}}\right) \Vert X\tilde{\beta }(\tilde{\lambda })-X\beta ^*\Vert ^{2}_{2} \\&+\left( 2+\frac{2}{\kappa _{1}}+\frac{4}{\kappa _{2}}\right) \Vert X^\top \varepsilon \Vert _\infty \Vert \tilde{\beta }(\tilde{\lambda })-\beta ^*\Vert _1. \end{aligned}$$

Theorem 1 follows from Theorem 4 by setting $\kappa _{1}=2$, and $\kappa _{2}=8$.

Proof (of Theorem 4)

Assume first $\tilde{\beta }(\tilde{\lambda })=0$. Then, since $\hat{u}\le \Vert X^\top Y\Vert _\infty /\kappa _1$, the definition of the TREX implies

$$\begin{aligned} \Vert Y-X\hat{\beta }\Vert ^{2}_{2}+c\hat{u}\Vert \hat{\beta }\Vert _{1}\le \Vert Y\Vert _2^2/\kappa _1 =\Vert Y-X\tilde{\beta }(\tilde{\lambda })\Vert ^{2}_{2}/\kappa _1\,. \end{aligned}$$

Assume now $\tilde{\beta }(\tilde{\lambda })\ne 0$. In view of the KKT conditions for LASSO, $\tilde{\beta }(\tilde{\lambda })$ fulfills

$$\begin{aligned} \Vert X^\top (Y-X\tilde{\beta }(\tilde{\lambda }))\Vert _\infty =\tilde{\lambda }=\max \{\kappa _{1} \hat{u},\frac{\kappa _{2}}{c}\Vert X^\top \varepsilon \Vert _\infty \}\ge \kappa _{1}\hat{u}. \end{aligned}$$

The definition of the TREX therefore yields

$$\begin{aligned} \Vert Y-X\hat{\beta }\Vert ^{2}_{2}+c\hat{u}\Vert \hat{\beta }\Vert _{1}\le \frac{\Vert Y-X\tilde{\beta }(\tilde{\lambda })\Vert ^{2}_{2}}{{\kappa _{1}}}+c\hat{u}\Vert \tilde{\beta }(\tilde{\lambda })\Vert _1. \end{aligned}$$

We now observe that since $\kappa _1> 1$ by assumption, we have $\tilde{\lambda }\ge \hat{u}$, and one can verify easily that this implies $\Vert \tilde{\beta }(\hat{u})\Vert _{1}\ge \Vert \tilde{\beta }(\tilde{\lambda })\Vert _1$. At the same time, Lemma 3 ensures $\Vert \hat{\beta }\Vert _1\ge \Vert \tilde{\beta }^{\hat{u}}\Vert _1$. Thus, $c\hat{u}\Vert \hat{\beta }\Vert _{1}\ge c\hat{u}\Vert \tilde{\beta }(\tilde{\lambda })\Vert _1$ and, therefore, we find again

$$\begin{aligned} \Vert Y-X\hat{\beta }\Vert ^{2}_{2}\le \frac{\Vert Y-X\tilde{\beta }(\tilde{\lambda })\Vert ^{2}_{2}}{\kappa _{1}}. \end{aligned}$$

Invoking the model, this results in

$$\begin{aligned}&\kappa _1\Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\\&\quad \le \left( 1-\kappa _{1}\right) \Vert \varepsilon \Vert ^{2}_{2}+{2\varepsilon ^\top (X\beta ^*- X\tilde{\beta }(\tilde{\lambda }))}+2\varepsilon ^\top (X\hat{\beta }- X\beta ^*)+{\Vert X\tilde{\beta }(\tilde{\lambda })-X\beta ^*\Vert ^{2}_{2}}. \end{aligned}$$

We can now use Hölder’s inequality and the triangle inequality to deduce

$$\begin{aligned}&\Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\\&\quad \le \left( \frac{1-\kappa _{1}}{\kappa _{1}}\right) \Vert \varepsilon \Vert _2^2+\frac{2\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*-\tilde{\beta }(\tilde{\lambda })\Vert _1}{\kappa _{1}}\\&\qquad + 2\Vert X^\top \varepsilon \Vert _\infty \Vert \hat{\beta }\Vert _1+2\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*\Vert _1+ \frac{\Vert X\tilde{\beta }(\tilde{\lambda })-X\beta ^*\Vert ^{2}_{2}}{\kappa _1} \\&\quad \le \left( \frac{1-\kappa _{1}}{\kappa _{1}}\right) \Vert \varepsilon \Vert _2^2+\frac{2\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*-\tilde{\beta }(\tilde{\lambda })\Vert _1}{\kappa _{1}}\\&\qquad + 2\Vert X^\top \varepsilon \Vert _\infty \left( \frac{\Vert Y-X\hat{\beta }\Vert _2^2}{c\hat{u}}+ \Vert \hat{\beta }\Vert _1\right) +2\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*\Vert _1+ \frac{\Vert X\tilde{\beta }(\tilde{\lambda })-X\beta ^*\Vert ^{2}_{2}}{\kappa _1}. \end{aligned}$$

Next, we observe that by the definition of our estimator $\hat{\beta }$ and of $\tilde{\lambda }$,

$$\begin{aligned}&\frac{\Vert Y-X\hat{\beta }\Vert _2^2}{c\hat{u}}+ \Vert \hat{\beta }\Vert _1\le \frac{\Vert Y-X\tilde{\beta }(\tilde{\lambda })\Vert _2^2}{c\tilde{\lambda }}+ \Vert \tilde{\beta }(\tilde{\lambda })\Vert _1\le \frac{\Vert Y-X\tilde{\beta }(\tilde{\lambda })\Vert _2^2}{\kappa _2\Vert X^\top \varepsilon \Vert _\infty }+ \Vert \tilde{\beta }(\tilde{\lambda })\Vert _1. \end{aligned}$$

Combining these two displays and using the model assumption then gives

$$\begin{aligned}&\Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\\&\quad \le \left( \frac{1-\kappa _{1}}{\kappa _{1}}\right) \Vert \varepsilon \Vert _2^2+\frac{2\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*-\tilde{\beta }(\tilde{\lambda })\Vert _1}{\kappa _{1}}\\&\qquad + 2\Vert X^\top \varepsilon \Vert _\infty \left( \frac{\Vert Y-X\tilde{\beta }(\tilde{\lambda })\Vert _2^2}{\kappa _2\Vert X^\top \varepsilon \Vert _\infty }+ \Vert \tilde{\beta }(\tilde{\lambda })\Vert _1\right) +2\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*\Vert _1 + \frac{\Vert X\tilde{\beta }(\tilde{\lambda })-X\beta ^*\Vert ^{2}_{2}}{\kappa _1}\\&\quad = \left( \frac{1-\kappa _{1}}{\kappa _{1}}+\frac{2}{\kappa _2}\right) \Vert \varepsilon \Vert _2^2+\frac{2\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*-\tilde{\beta }(\tilde{\lambda })\Vert _1}{\kappa _{1}}\\&\qquad + 2\Vert X^\top \varepsilon \Vert _\infty \left( \frac{2\varepsilon ^\top X(\beta ^*-\tilde{\beta }(\tilde{\lambda }))}{\kappa _2\Vert X^\top \varepsilon \Vert _\infty }+ \Vert \tilde{\beta }(\tilde{\lambda })\Vert _1\right) +2\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*\Vert _1\\&\qquad +\left( \frac{1}{\kappa _1}+\frac{2}{\kappa _2}\right) {\Vert X\tilde{\beta }(\tilde{\lambda })-X\beta ^*\Vert ^{2}_{2}}. \end{aligned}$$

We can now use Hölder’s inequality and rearrange the terms to get

$$\begin{aligned}&\Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\\&\quad \le \left( \frac{1-\kappa _{1}}{\kappa _{1}}+\frac{2}{\kappa _2}\right) \Vert \varepsilon \Vert _2^2+\left( \frac{1}{\kappa _1}+\frac{2}{\kappa _2}\right) 2\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*-\tilde{\beta }(\tilde{\lambda })\Vert _1\\&\qquad + 2\Vert X^\top \varepsilon \Vert _\infty \Vert \tilde{\beta }(\tilde{\lambda })\Vert _1+2\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*\Vert _1+\left( \frac{1}{\kappa _1}+\frac{2}{\kappa _2}\right) {\Vert X\tilde{\beta }(\tilde{\lambda })-X\beta ^*\Vert ^{2}_{2}}. \end{aligned}$$

The last step is to use Assumption 4, which ensures

$$\begin{aligned} \Vert \beta ^*\Vert _1\le \left( \frac{1}{4}-\frac{1}{4\kappa _{1}}-\frac{1}{2\kappa _{2}}\right) \frac{\Vert \varepsilon \Vert _2^2}{\Vert X^{\top }\varepsilon \Vert _{\infty }} \end{aligned}$$

and, therefore,

$$\begin{aligned} 4\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*\Vert _1\le \left( \frac{\kappa _{1}-1}{\kappa _{1}}-\frac{2}{\kappa _{2}}\right) \Vert \varepsilon \Vert _2^2. \end{aligned}$$

The above display therefore yields

$$\begin{aligned} \Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}&\le -\,4\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*\Vert _1 +\left( \frac{1}{\kappa _1}+\frac{2}{\kappa _2}\right) 2\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*-\tilde{\beta }(\tilde{\lambda })\Vert _1\\&\quad + 2\Vert X^\top \varepsilon \Vert _\infty \Vert \tilde{\beta }(\tilde{\lambda })\Vert _1+2\Vert X^\top \varepsilon \Vert _\infty \Vert \beta ^*\Vert _1\\&\quad +\left( \frac{1}{\kappa _1} +\frac{2}{\kappa _2}\right) {\Vert X\tilde{\beta }(\tilde{\lambda })-X\beta ^*\Vert ^{2}_{2}}. \end{aligned}$$

Using the triangle inequality, we finally obtain

$$\begin{aligned} \Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\le & {} \left( 2+\frac{2}{\kappa _1}+\frac{4}{\kappa _2}\right) \Vert X^\top \varepsilon \Vert _\infty \Vert \tilde{\beta }(\tilde{\lambda })-\beta ^*\Vert _1\\&+\left( \frac{1}{\kappa _1}+\frac{2}{\kappa _2}\right) {\Vert X\tilde{\beta }(\tilde{\lambda })-X\beta ^*\Vert ^{2}_{2}} \end{aligned}$$

as desired.

Corollary 4

Let Assumption 4 be fulfilled, and let $\tilde{\lambda }:=\max \{\kappa _{1} \hat{u},\frac{\kappa _{2}}{c}\Vert X^\top \varepsilon \Vert _\infty \}$. Furthermore, let $\kappa _1,\kappa _2>0$ be such that

$$\begin{aligned} \frac{1}{\kappa _2}+\frac{\kappa _1}{\kappa _2 + 2\kappa _1} \le \frac{1}{c}. \end{aligned}$$

Then for any $\hat{u}\le \Vert X^\top Y\Vert _\infty /\kappa _1$, the prediction loss of TREX satisfies

$$\begin{aligned} \Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\le \left( \frac{1}{\kappa _1}+\frac{2}{\kappa _2}\right) \frac{16s\tilde{\lambda }^2}{\nu ^2 n}, \end{aligned}$$

where $\nu$ is the compatibility constant defined in (5).

Corollary 3 follows from Corollary 4 by setting $\kappa _{1}=2$, and $\kappa _{2}=8$, which satisfy the requirement for any $c\in (0,2)$.

Proof (of Corollary 4)

Using Theorem 4 and the definition of $\tilde{\lambda }$, the TREX prediction loss satisfies

$$\begin{aligned} \Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}&\le \left( \frac{1}{\kappa _1}+\frac{2}{\kappa _2}\right) {\Vert X\tilde{\beta }(\tilde{\lambda })-X\beta ^*\Vert ^{2}_{2}}\\&\quad + \left( 2+\frac{2}{\kappa _1}+\frac{4}{\kappa _2}\right) \Vert X^\top \varepsilon \Vert _\infty \Vert \tilde{\beta }(\tilde{\lambda })-\beta ^*\Vert _1\\&\le \left( \frac{1}{\kappa _1}+\frac{2}{\kappa _2}\right) \left[ \Vert X\tilde{\beta }(\tilde{\lambda })-X\beta ^*\Vert ^{2}_{2} + \left( 2+\frac{2\kappa _1\kappa _2}{\kappa _2 + 2\kappa _1}\right) \frac{c}{{\kappa _2 }}\tilde{\lambda }\Vert \tilde{\beta }(\tilde{\lambda })-\beta ^*\Vert _1\right] . \end{aligned}$$

On the other hand, the LASSO estimator $\tilde{\beta }(\tilde{\lambda })$ satisfies (Bühlmann and van de Geer 2011, Theorem 6.1)

$$\begin{aligned} \Vert X\tilde{\beta }(\tilde{\lambda })-X\beta ^*\Vert ^{2}_{2} +2\tilde{\lambda }\Vert \tilde{\beta }(\tilde{\lambda })-\beta ^*\Vert _1 \le \frac{16s\tilde{\lambda }^2}{\nu ^2n}. \end{aligned}$$

Since by assumption

$$\begin{aligned} \left( 2 + \frac{2\kappa _1\kappa _2}{\kappa _2 + 2\kappa _1}\right) \frac{c}{{\kappa _2} }\le 2, \end{aligned}$$

the LASSO bound implies

$$\begin{aligned} \Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}&\le \left( \frac{1}{\kappa _1}+\frac{2}{\kappa _2}\right) \frac{16s\tilde{\lambda }^2}{\nu ^2 n} \end{aligned}$$

as desired.

Appendix B: proof of a generalization of Theorem 2

We consider a generalization of the TREX according to

$$\begin{aligned} \hat{\beta }\in \mathop {\mathrm {arg\,min}}\left\{ \frac{\Vert Y-X\beta \Vert ^2_2}{c\Omega ^*(X^\top (Y-X\beta ))}+\Omega (\beta )\right\} , \end{aligned}$$

where $\Omega$ is a norm on $\mathbb {R}^p$, $\Omega ^{*}(\eta ):=\sup \{\eta ^{\top }\beta :\Omega (\beta )\le 1\}$ is the dual of $\Omega$, $0<c<2$, and the minimum is taken over all $\beta \in \mathbb {R}^p$. We also set $\hat{u}:=\Omega ^{*}(X^{\top }(Y-X\hat{\beta }))$ with some abuse of notation. The corresponding generalization of Assumption 2 then reads as follows.

Assumption 5

The regression vector $\beta ^*$ is sufficiently large such that

$$\begin{aligned} \Omega ^{*}(X^{\top }X\beta ^*)\ge \left( 1+\frac{2}{c}\right) \Omega ^{*}(X^{\top }\varepsilon ). \end{aligned}$$

We now prove a generalization of Theorem 2.

Theorem 5

Let Assumption 5 be fulfilled. If $\hat{u}\le \Omega ^{*}(X^{\top }Y)$, then the prediction loss of TREX satisfies

$$\begin{aligned} \frac{\Vert X\hat{\beta }-X\beta ^*\Vert _{2}^{2}}{n}\le \frac{\left( 2\Omega ^{*}(X^{\top }\varepsilon )+\max \left\{ \hat{u},\frac{2}{c}\Omega ^{*}(X^{\top }\varepsilon )\right\} \right) \Omega (\beta ^*)}{n}. \end{aligned}$$

Theorem 2 follows from Theorem 5 by setting $\Omega (\cdot ):=\Vert \cdot \Vert _{1}.$

Proof (of Theorem 5)

The definition of the estimator implies

$$\begin{aligned} \frac{\Vert Y-X\hat{\beta }\Vert ^2_2}{c\cdot \hat{u}}+\Omega (\hat{\beta })\le \frac{\Vert Y\Vert ^2_2}{c\cdot \Omega ^*(X^\top Y)}, \end{aligned}$$

which yields together with the model assumptions

$$\begin{aligned}&\Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}+\Vert \varepsilon \Vert ^{2}_{2}+2\varepsilon ^{\top }(X\beta ^*-X\hat{\beta })+c\hat{u}\Omega (\hat{\beta })\\&\quad \le \frac{\hat{u}}{\Omega ^{*}(X^{\top }Y)}\left( \Vert \varepsilon \Vert ^{2}_{2}+\Vert X\beta ^*\Vert _{2}^{2}+2\varepsilon ^{\top }X\beta ^*\right) . \end{aligned}$$

Rearranging the terms and Hölder’s inequality in the form of $2\varepsilon ^\top X\hat{\beta }\le 2\Omega ^*(X^\top \varepsilon )\Omega (\hat{\beta })$ and $\Vert X\beta ^*\Vert ^{2}_{2}=\beta ^*{}^{\top }X^{\top }X\beta ^*\le \Omega ^{*}(X^{\top }X\beta ^*)\Omega (\beta ^*)$ then gives

$$\begin{aligned} \begin{aligned} \Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\le&\left( \frac{\hat{u}}{\Omega ^{*}(X^{\top }Y)}-1\right) \Vert \varepsilon \Vert ^{2}_{2}+(2\Omega ^{*}(X^{\top }\varepsilon )-c\hat{u})\Omega (\hat{\beta })\\&+\frac{\hat{u}\Omega ^{*}(X^{\top }X\beta ^*)\Omega (\beta ^*)}{\Omega ^{*}(X^{\top }Y)}+2\left( \frac{\hat{u}}{\Omega ^{*}(X^{\top }Y)}-1\right) 2\varepsilon ^{\top }X\beta ^*. \end{aligned} \end{aligned}$$

(9)

Case 1: We first consider the case $2\Omega ^{*}(X^{\top }\varepsilon )\le c\hat{u}$.

For this, we first note that $\hat{u} \le \Omega ^{*}(X^{\top }Y)$ by assumption. Using this and $2\Omega ^{*}(X^{\top }\varepsilon )\le c\hat{u}$ allows us to remove the first two terms on the right-hand side of Inequality (9) so that

$$\begin{aligned} \Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\le \frac{\hat{u}\Omega ^{*}(X^{\top }X\beta ^*)\Omega (\beta ^*)}{\Omega ^{*}(X^{\top }Y)}+\left( \frac{\hat{u}}{\Omega ^{*}(X^{\top }Y)}-1\right) 2\varepsilon ^{\top }X\beta ^*. \end{aligned}$$

Since $2\varepsilon ^\top X\beta ^*\le 2\Omega ^{*}(X^{\top }\varepsilon )\Omega (\beta ^*)$ due to Hölder’s Inequality, and since $\hat{u}\le \Omega ^*(X^\top Y)$ by definition of our estimator, we therefore obtain from the above display and the model assumptions

$$\begin{aligned} \Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}&\le \frac{\hat{u}\Omega ^{*}(X^{\top }X\beta ^*)\Omega (\beta ^*)}{\Omega ^{*}(X^{\top }Y)}+\left( 1-\frac{\hat{u}}{\Omega ^{*}(X^{\top }Y)}\right) 2\Omega ^{*}(X^{\top }\varepsilon )\Omega (\beta ^*)\\&=2\Omega ^{*}(X^{\top }\varepsilon )\Omega (\beta ^*)+\hat{u}\left( \frac{\Omega ^{*}(X^{\top }X\beta ^*)-2\Omega ^{*}(X^{\top }\varepsilon )}{\Omega ^{*}(X^{\top }Y)}\right) \Omega (\beta ^*)\\&=2\Omega ^{*}(X^{\top }\varepsilon )\Omega (\beta ^*)+\hat{u}\left( \frac{\Omega ^{*}(X^{\top }X\beta ^*)-2\Omega ^{*}(X^{\top }\varepsilon )}{\Omega ^{*}(X^{\top }X\beta ^*+X^{\top }\varepsilon )}\right) \Omega (\beta ^*). \end{aligned}$$

Next, we note that the triangle inequality gives

$$\begin{aligned} \Omega ^{*}(X^{\top }X\beta ^*+X^{\top }\varepsilon )\ge \Omega ^{*}(X^{\top }X\beta ^*)-\Omega ^{*}(X^{\top }\varepsilon ). \end{aligned}$$

Plugging this in the previous display finally yields

$$\begin{aligned} \Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\le 2\Omega ^{*}(X^{\top }\varepsilon )\Omega (\beta ^*)+\hat{u}\Omega (\beta ^*), \end{aligned}$$

which concludes the proof for Case 1.

Case 2: We now consider the case $2\Omega ^{*}(X^{\top }\varepsilon )\ge c\hat{u}$.

Similarly as before, we start with the definition of the estimator, which yields in particular

$$\begin{aligned} \Omega (\hat{\beta })\le \frac{\Vert Y\Vert ^2_2}{c\cdot \Omega ^*(X^\top Y)}. \end{aligned}$$

Invoking the model assumptions and Hölder’s inequality then gives

$$\begin{aligned} \Omega (\hat{\beta })&\le \frac{1}{c\Omega ^{*}(X^{\top }Y)}\left( \Vert \varepsilon \Vert ^{2}_{2}+\Vert X\beta ^*\Vert ^{2}_{2}+2\varepsilon ^{\top }X\beta ^*\right) \\&\le \frac{1}{c\Omega ^{*}(X^{\top }Y)}\left( \Vert \varepsilon \Vert ^{2}_{2}+\Omega ^{*}(X^{\top }X\beta ^*)\Omega (\beta ^*)+2\varepsilon ^\top X\beta ^*\right) . \end{aligned}$$

We can now plug this into Inequality (9) to obtain

$$\begin{aligned}&\Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\le \left( \frac{\hat{u}}{\Omega ^{*}(X^{\top }Y)}-1\right) \Vert \varepsilon \Vert ^{2}_{2}\\&\quad +(2\Omega ^*(X^\top \varepsilon )-c\hat{u})\frac{1}{c\Omega ^{*}(X^{\top }Y)}\left( \Vert \varepsilon \Vert ^{2}_{2}+\Omega ^{*}(X^{\top }X\beta ^*)\Omega (\beta ^*)+2\varepsilon ^{\top }X\beta ^*\right) \\&\quad +\frac{\hat{u}\Omega ^{*}(X^{\top }X\beta ^*)\Omega (\beta ^*)}{\Omega ^{*}(X^{\top }Y)}+\left( \frac{\hat{u}}{\Omega ^{*}(X^{\top }Y)}-1\right) 2\varepsilon ^{\top }X\beta ^*. \end{aligned}$$

We can now rearrange the terms to get

$$\begin{aligned} \Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\le&\left( \frac{2\Omega ^{*}(X^{\top }\varepsilon )}{c\Omega ^{*}(X^{\top }Y)}-1\right) \Vert \varepsilon \Vert ^{2}_{2}+\left( \frac{2\Omega ^{*}(X^{\top }\varepsilon )}{c\Omega ^{*}(X^{\top }Y)}-1\right) 2\varepsilon ^{\top }X\beta ^*\\&+\frac{2\Omega ^{*}(X^{\top }\varepsilon )}{c\Omega ^{*}(X^{\top }Y)}\Omega ^{*}(X^{\top }X\beta ^*)\Omega (\beta ^*). \end{aligned}$$

We now observe that Assumption 5 implies via the triangle inequality and the model assumptions that

$$\begin{aligned} \Omega ^{*}(X^{\top }Y)\ge \frac{2\Omega ^{*}(X^{\top }\varepsilon )}{c}. \end{aligned}$$

Using this, Hölder’s inequality, and the triangle inequality, we then find

$$\begin{aligned}&\Vert X\hat{\beta }-X\beta ^*\Vert ^{2}_{2}\\&\quad \le \left( 1-\frac{2\Omega ^{*}(X^{\top }\varepsilon )}{c\Omega ^{*}(X^{\top }Y)}\right) 2\Omega ^*(X^{\top }\varepsilon )\Omega (\beta ^*)\\&\qquad +2\Omega ^*(X^\top \varepsilon )\Omega (\beta ^*)/c+\frac{2\Omega ^*(X^\top \varepsilon )}{c\Omega ^*(X^\top Y)}\Omega ^*(X^\top \varepsilon )\Omega (\beta ^*)\\&\quad = 2\Omega ^*(X^{\top }\varepsilon )\Omega (\beta ^*)+2\Omega ^*(X^\top \varepsilon )\Omega (\beta ^*)/c-\frac{2\Omega ^*(X^\top \varepsilon )}{c\Omega ^*(X^\top Y)}\Omega ^*(X^\top \varepsilon )\Omega (\beta ^*)\\&\quad \le (2+2/c)\Omega ^*(X^{\top }\varepsilon )\Omega (\beta ^*). \end{aligned}$$

This concludes the proof for Case 2 and, therefore, the proof of Theorem 5.

Appendix C: Prediction error results for $p=128$

See Fig. 2.

Fig. 2

Prediction error versus relative regularization path $\rho$ for $p=128$ for $N=51$ repetitions. LASSO prediction errors for $\lambda \ge \lambda _\text {lb}$ are shown in solid blue and TREX prediction errors for $c \ge c_\text {lb}$ in solid red. The blue and red intervals at the bottom mark the range where $\lambda _\text {lb}$ and $c_\text {lb}$ values fall across N repetitions. Dashed transparent lines show prediction errors without theoretical guarantees. For the LASSO, we show the locations of the regularization parameters selected with cross-validation and the MSE (orange) and 1SE (purple) rule as well as with BIC (green) (color figure online)