On the Asymptotic Properties of SLOPE

Abstract

Sorted L-One Penalized Estimator (SLOPE) is a relatively new convex optimization procedure for selecting predictors in high dimensional regression analyses. SLOPE extends LASSO by replacing the L₁ penalty norm with a Sorted L₁ norm, based on the non-increasing sequence of tuning parameters. This allows SLOPE to adapt to unknown sparsity and achieve an asymptotic minimax convergency rate under a wide range of high dimensional generalized linear models. Additionally, in the case when the design matrix is orthogonal, SLOPE with the sequence of tuning parameters λ^BH corresponding to the sequence of decaying thresholds for the Benjamini-Hochberg multiple testing correction provably controls the False Discovery Rate (FDR) in the multiple regression model. In this article we provide new asymptotic results on the properties of SLOPE when the elements of the design matrix are iid random variables from the Gaussian distribution. Specifically, we provide conditions under which the asymptotic FDR of SLOPE based on the sequence λ^BH converges to zero and the power converges to 1. We illustrate our theoretical asymptotic results with an extensive simulation study. We also provide precise formulas describing FDR of SLOPE under different loss functions, which sets the stage for future investigation on the model selection properties of SLOPE and its extensions.

1 Introduction

In this article we consider the classical problem of identifying important predictors in the multiple regression model

$$ Y=X b^{0}+\epsilon , $$

(1.1)

where X is the design matrix of dimension n × p and $\epsilon \sim N(0,\sigma ^{2} I_{n\times n})$ is the noise vector. In the case when p > n, the vector of parameters b⁰ is not identifiable and can be uniquely estimated only under certain additional assumptions concerning e.g. its sparsity (i.e. the number of non-zero elements). One of the most popular and computationally tractable methods for estimating b⁰ in the case when p > n is the Least Absolute Shrinkage and Selection Operator (LASSO, Tibshirani (1996)), firstly introduced in the context of signal processing as the Basis Pursuit Denoising (BPDN, Chen and Donoho. D. (1994)). LASSO estimator is defined as

$$ \hat b^{L}=argmin_{b} \left\{ \frac{1}{2} || Y-Xb||_{2}^{2}+\lambda |b|_{1}\right\} , $$

(1.2)

where ||⋅||₂ denotes the regular Euclidean norm in Rⁿ and $|b|_{1}={\sum }_{j=1}^{p} |b_{j}|$ is the L₁ norm of b. If $X^{\prime }X=I$ then LASSO is reduced to the simple shrinkage operator imposed on the elements of the vector $\tilde Y = X^{\prime }Y$,

$$ {b^{L}_{i}}=\eta_{\lambda}(\tilde Y_{i}) , $$

where

$$ \eta_{\lambda}(x)=\left\{\begin{array}{ccc} x-\lambda&\text{if}&x>\lambda\\ 0&\text{if}&|x|<\lambda\\ x+\lambda&\text{if}&x<-\lambda \end{array} \right. . $$

In this case the choice of the tuning parameter

$$ \lambda=\lambda^{Bon}=\sigma {\Phi}^{-1}\left( 1-\frac{\alpha}{2p}\right)=\sigma \sqrt{2\log p}(1+o_{p}) , $$

allows for the control of the probability of at least one false discovery (Family Wise Error Rate, FWER) at level α.

In the context of high dimensional multiple testing, the Bonferroni correction is often replaced by the Benjamini and Hochberg (1995) multiple testing procedure aimed at the control of the False Discovery Rate (FDR). Apart from FDR control, this procedure also has appealing properties in the context of the estimation of the vector of means for multivariate normal distribution with independent entries (Abramovich et al. 2006) or in the context of minimizing the Bayes Risk related to 0-1 loss (Bogdan et al. 2011; Neuvial and Roquain, 2012; Frommlet and Bogdan, 2013). In the context of the multiple regression with an orthogonal design, the Benjamini-Hochberg procedure works as follows:

• (1) Fix q ∈ (0, 1) and sort $\tilde Y$ such that

  $$ |\tilde Y |_{(1)} \ge |\tilde Y|_{(2)} \ge {\ldots} \ge |\tilde Y|_{(p)} , $$

• (2) Identify the largest j such that

  $$ |\tilde Y|_{(j)} \ge \lambda^{BH}_{j}= \sigma {\Phi}^{-1}\left( 1- \frac{jq}{2p}\right), $$

  (1.3)

  Call this index j_BH.

• (3) Reject H_(j) for every j ≤ j_BH.

Thus, in BH, the fixed threshold of the Bonferroni correction, λ^Bon, is replaced with the sequence λ^BH of ‘sloped’ thresholds for the sorted test statistics (see Fig. 1). This allows for a substantial increase of power and for an improvement of prediction properties in the case when some of the predictors are relatively weak.

Figure 1

Bonferroni and Benjamini-Hochberg procedures for multiple testing

The idea of using a decreasing sequence of thresholds was subsequently used in the Sorted L-One Penalized Estimator (SLOPE, Bogdan et al. (2013) and Bogdan et al. (2015)) for the estimation of coefficients in the multiple regression model:

$$ b^{SL}=argmin_{b} \left\{\frac 12\left\|y-Xb\right\|_{2}^{2}+\sum\limits_{i=1}^{p}\lambda_{i}\left|b\right|_{(i)}\right\}, $$

(1.4)

where $\left |b\right |_{(1)}\geq \ldots \geq \left |b\right |_{(p)}$ are ordered magnitudes of elements of b and λ = (λ₁, … , λ_p) is a non-zero, non-increasing and non-negative sequence of tuning parameters. As noted in Bogdan et al. (2013) and Bogdan et al. (2015), the function $J_{\lambda }(b)={\sum }_{i=1}^{p}\lambda _{i}\left |b\right |_{(i)}$ is a norm. To see this, observe that:

• for any constant a ∈ R and a sequence b ∈ R^p, J_λ(ab) = |a|J_λ(b)

• J_λ(b) = 0 if and only if b = 0 ∈ R^p.

To show the triangular inequality:

• J_λ(x + y) ≤ J_λ(x) + J_λ(y)

let us denote by ρ the permutation of the set {1, … , p} such that

$$ \left|x_{\rho(1)}+y_{\rho(1)}\right|\geq \left|x_{\rho(2)}+y_{\rho(2)}\right|\geq ... \geq \left|x_{\rho(p)}+y_{\rho(p)}\right| .$$

Then

$$ J_{\lambda}(x+y) = \sum\limits_{i=1}^{p}\lambda_{i}\left|x_{\rho(i)}+y_{\rho(i)}\right| \leqslant \sum\limits_{i=1}^{p}\lambda_{i}\left|x_{\rho(i)}\right|+\sum\limits_{i=1}^{p}\lambda_{i}\left|y_{\rho(i)}\right| $$

and the result can be derived from the well known rearrangement inequality, according to which for any permutation ρ and any sequence x ∈ R^p:

$$ \sum\limits_{i=1}^{p}\lambda_{i}\left|x_{\rho(i)}\right| \leqslant \sum\limits_{i=1}^{p} \lambda_{i} |x|_{(i)} . $$

Thus SLOPE is a convex optimization procedure which can be efficiently solved using classical optimization tools. It is also easy to observe that in the case λ₁ = … = λ_p, SLOPE is reduced to LASSO, while in the case λ₁ > λ₂ = … = λ_p = 0, the Sorted L-One norm J_λ is reduced to the $L_{\infty }$ norm.

Figure 2 illustrates different shapes of the unit spheres corresponding to different versions of the Sorted L-One Norm. Since the solutions of SLOPE tend to occur on the edges of respective spheres, Fig. 2 demonstrates the large flexibility of SLOPE with respect to the dimensionality reduction. In the case when λ₁ = … = λ_p, SLOPE reduces dimensionality by shrinking the coefficients to zero. In the case when λ₁ > λ₂ = … = λ_p = 0, the reduction of dimensionality is performed by shrinking the coefficients towards each other (since the edges of the $l_{\infty }$ sphere correspond to vectors b such that at least two coefficients are equal to each other). In the case when the sequence of thresholding parameters is monotonically decreasing, SLOPE reduces the dimensionality both ways: it shrinks them towards zero and towards each other. Thus it returns sparse and stable estimators, which have recently been proved to achieve minimax rate of convergence in the context of sparse high dimensional regression and logistic regression (Su and Candès, 2016; Bellec et al. 2018; Abramovich and Grinshtein, 2017).

Figure 2

Shapes of different SLOPE spheres

From the perspective of model selection it has been proved in Bogdan et al. (2013) and Bogdan et al. (2015) that SLOPE with the vector of tuning parameters λ^BH (1.3) controls FDR at level q under the orthogonal design. This is no longer true if the inner products between columns of the design matrix are different from zero, which almost always occurs if the predictors are random variables. Similar problems with the control of the number of False Discoveries occur for LASSO. Specifically, in Bogdan et al. (2013) it is shown that in the case of the Gaussian design with independent predictors, LASSO with a fixed parameter λ can control FDR only if the true parameter vector is sufficiently sparse. The natural question is whether there exists a bound on the sparsity under which SLOPE can control FDR ? In this article we address this question and report a theoretical result on the asymptotic control of FDR by SLOPE under the Gaussian design. Our main theoretical result states that by multiplying sequence λ^BH by a constant larger than 1, one can achieve the full asymptotic power and FDR converging to 0 if the number k(n) of nonzero elements in the true vector of the regression coefficients satisfies $k=o\left (\sqrt {\frac {n}{\log p}}\right )$ and the values of these non-zero elements are sufficiently large. We also report results of a simulation study which suggest that the assumption on the signal sparsity is necessary when using λ^BH sequence but is unnecessarily strong when using the heuristic adjustment of this sequence, proposed in Bogdan et al. (2015). Simulations also suggest that the asymptotic FDR control is guaranteed independently of the magnitude of the non-zero regression coefficients.

2 Asymptotic Properties of SLOPE

2.1 False Discovery Rate and Power

Let us consider the multiple regression model (1.1) and let $\hat b$ be some sparse estimator of b⁰. The numbers of false, true and all rejections, and the number of non-zero elements in b⁰ (respectively: V, TR, R, k), are defined as follows

$$ V = \#\{j: {b_{j}^{0}}=0 ~\text{and}~ \hat b_{j} \neq 0\}, \ \ \ \ \ \ \ R = \#\{j:\hat b_{j} \neq 0\} , $$

(2.1)

$$ TR = \#\{j: {b_{j}^{0}} \neq 0 ~\text{and}~ \hat b_{j} \neq 0\}, \ \ \ \ \ \ \ k = \#\{j: {b^{0}_{j}}\neq 0\} . $$

(2.2)

The False Discovery Rate is defined as:

$$ FDR := \mathbb{E}\left( \frac{V}{R \vee 1}\right) $$

(2.3)

and the Power as:

$$ {\Pi} := \frac{\mathbb{E} \left( TR\right)}{k}. $$

(2.4)

2.2 Asymptotic FDR and Power

We formulate our asymptotic results under the setup where n and p diverge to infinity and p can grow faster than n. Similarly as in the case of the support recovery results for LASSO, we need to impose a constraint on the sparsity of b⁰, which is measured by the number of truly important predictors $k = \#\{i: {b^{0}_{i}} \neq 0\}$. Thus we consider the sequence of linear models of the form (1.1) indexed by n and with their ”dimensions” characterized by the triplets: (n,p_n,k_n). For the sake of clarity, further in the text we skip the subscripts by p and k.

The main result of this article is as follows:

Theorem 2.1.

Consider the linear model of the form (1.1) and assume that all elements of the design matrix $X \in \mathbb {R}^{n \times p}$ are i.i.d. random variables from the normal N(0,1/n) distribution. Moreover, suppose there exists δ > 0 such that

$$ \min_{{b^{0}_{j}} \neq 0} \vert {b^{0}_{j}} \vert \geqslant 2\sigma (1+\delta)\sqrt{2 \log p} $$

(2.5)

and suppose

$$ p\rightarrow \infty, \frac{k}{p} \rightarrow 0 ~\text{and}~ \frac{k^{2} \log p}{n} \rightarrow 0. $$

(2.6)

Then for any q ∈ (0,1), the SLOPE procedure with the sequence of tuning parameters

$$ \lambda_{i}(q,\delta) = \sigma (1+\delta) {\Phi}^{-1}\left( 1- \frac{qi}{2p}\right) $$

(2.7)

has the following properties

$$ FDR \rightarrow 0 , {\Pi} \rightarrow 1.$$

Proof

The proof of Theorem 2.1 makes extensive use of the results on the asymptotic properties of SLOPE reported in Su and Candès (2016). The roadmap of this proof is provided in Section 3, while the proof details can be found in the Appendix and in supplementary materials. □

Remark 2.2 (The assumption on the design matrix).

The assumption that the elements of X are i.i.d. random variables from the normal N(0,1/n) distribution is technical. We expect that the results can be generalized to the case where the predictors are independent, sub-gaussian random variables. The assumption that the variance is equal to 1/n can be satisfied by an appropriate scaling of such a design matrix. As compared to the classical standardization towards unit variance, our scaling allows for the control of FDR with the sequence of the tuning parameters λ, which does not depend on the number of observations n. If the data are standardized such that $X_{ij}\sim N(0,1)$, then Theorem 2.1 holds when the sequence of tuning parameters (2.7) and the lower bound on the signal magnitude (2.5) are both multiplied by n^− 0.5.

Remark 2.3 (The assumption on the signal strength).

Our assumption on the signal strength is not very restrictive. When using the classical standardization of the explanatory variables (i.e. assuming that $X_{ij}\sim N(0,1)$), this assumption allows for the magnitude of the signal to converge to zero at such a rate that

$$ \min_{{b^{0}_{j}} \neq 0} \vert {b^{0}_{j}} \vert \geqslant 2\sigma (1+\delta)\sqrt{\frac{2 \log p}{n}} . $$

This assumption is needed to obtain the power that converges to 1. The proof of Theorem 2.1 implies that this assumption is not needed for the asymptotic FDR control if k is bounded by some constant. Moreover, our simulations suggest that the asymptotic FDR control holds independently of the signal strength if only k satisfies the assumption (2.6). The proof of this conjecture would require a substantial refinement of the proof techniques and remains an interesting topic for future work.

2.3 Simulations

In this section we present results of the simulation study. The data are generated according to the linear model:

$$ Y = Xb^{0} +\epsilon , $$

where elements of the design matrix X are i.i.d. random variables from the normal N(0,1/n) distribution and 𝜖 is independent of X and comes from the standard multivariate normal distribution N(0,I). The parameter vector b⁰ has k non-zero elements and p − k zeroes.

We present the comparison of the three methods - two versions of SLOPE and LASSO:

• 1. SLOPE with the sequence of the tuning parameters provided by the formula (2.7), denoted by ”SLOPE”.

• 2. SLOPE with the sequence:

  $$ \lambda_{i}(q) = \left\{\begin{array}{ccc} \sigma {\Phi}^{-1}(1-q/2p) & if & i=1\\ \min\left( \lambda_{i-1}, \sigma{\Phi}^{-1}(1-qi/2p)\sqrt{1+\frac{{\sum}_{j<i}{\lambda_{j}^{2}}}{n-i-2}}\right) & if & i>1 . \end{array}\right. $$

  This sequence was proposed in Bogdan et al. (2015) as a heuristic correction which takes into account the influence of the cross products between the columns of the design matrix X. We refer to this procedure as heuristic SLOPE (”SLOPE_heur”).

• 3. LASSO with the tuning parameter:

  $$ \lambda= \sigma (1+\delta) {\Phi}^{-1}\left( 1- \frac{q}{2p}\right) . $$

  (2.8)

The tuning parameter for LASSO is equal to the first element of the tuning sequence of SLOPE, so FDR of LASSO and SLOPE are approximately equal when k = 0.

Figure 3 presents FDR and Power of different procedures. First, let us concentrate on the behavior of SLOPE when the sequence of tuning parameters is defined as in Theorem 2.1 (green line in each sub-plot). The green rectangle contains plots where the sequences of tuning parameters and the signal sparsity k(n) satisfy the assumptions of Theorem 2.1. It is noticeable that in this area FDR of SLOPE slowly converges to zero and the Power converges to 1. Moreover, FDR is close to or below the nominal level q = 0.2 for the whole range of considered values of n. It is also clear that larger values of δ lead to the more conservative versions of SLOPE.

Figure 3

FDR and Power of different procedures as functions of n and the parameters δ (see (2.7) and (2.8)) and α, for p = 0.05n^1.5, k = round(n^α), ${b^{0}_{1}}=\ldots ={b^{0}_{k}}=2(1+\delta )\sqrt {2 \log p}$ and q = 0.2. The results presented in a green rectangle correspond to the values of parameters which meet assumptions of Theorem 2.1. The numbers above green lines correspond to actual values of the parameter k. Each point was obtained by averaging false and true positive proportions over at least 500 independent replicates

In the red area the assumptions of Theorem 2.1 are violated. Here we can see that when δ > 0 and α = 0.5, FDR is still a decreasing function of n but the rate of this decrease is slow and FDR is substantially above level q = 0.2 even for n = 2000. In the case when δ = 0 (i.e. when the original λ^BH sequence is used), FDR stabilizes at the value which exceeds the nominal level.

Let us now turn our attention to other methods. We can observe that LASSO is the most conservative procedure and that, as expected, its FDR converges to 0 when k increases. Since the simulated signals are strong, this does not lead to a substantial decrease of Power as compared to SLOPE. Interestingly, SLOPE with a heuristic choice of tuning parameters seems to provide a stable FDR control over the whole range of considered parameter values. This suggests that the upper bound on k provided in assumption (2.6) could be relaxed when working with this heuristic sequence. The proof of this claim remains an interesting topic for further research.

Figure 4 presents simulations for the case when ${b^{0}_{1}}=\ldots ={b^{0}_{k}}=0.9\sqrt {2 \log p}$, i.e. when the signal magnitude does not satisfy the assumption (2.5). Here FDR of SLOPE behaves similarly as in the case of strong signals. These results suggest that the assumption on the signal strength might not be necessary in the context of FDR control. Figure 4 also illustrates a strikingly good control of FDR by the SLOPE with the heuristically adjusted sequence of tuning parameters. LASSO is substantially more conservative than both versions of SLOPE. Its FDR converges to zero, which in the case of such moderate signals leads to a substantial decrease of Power as compared to SLOPE.

Figure 4

FDR and Power of different procedures as functions of n and the parameters δ (see (2.7) and (2.8)) and α, for p = 0.05n^1.5, k = round(n^α), ${b^{0}_{1}}=\ldots ={b^{0}_{k}}=0.9\sqrt {2 \log p}$ and q = 0.2. The results presented in a green rectangle correspond to the values of parameters which meet assumptions of Theorem 2.1, except the condition on signal strength. The numbers above green lines correspond to actual values of the parameter k. Each point was obtained by averaging false and true positive proportions over at least 500 independent replicates

3 Roadmap of the Proof

In the first part of this section we characterize the support of the SLOPE estimator. The proofs of the Theorems presented in this part rely only on differentiability and convexity of the loss function. Therefore we decided to present them in a general form, will be useful for a further work on extensions of SLOPE for Generalized Linear Models or Gaussian Graphical Models.

3.1 Support of the SLOPE estimator under the general loss function

Let us consider the following generalization of SLOPE:

$$ \hat b=argmin_{b} \left\{l(b)+\sum\limits_{i=1}^{p}\lambda_{i}\left|b\right|_{(i)}\right\}, $$

(3.1)

where l(b) is a convex and a differentiable loss function (e.g. $0.5\Vert Y-X^{\prime }b{\Vert ^{2}_{2}}$ for the multiple linear regression). Let R denote the number of non-zero elements of $\hat b$.

The following Theorems 3.1 and 3.2 characterize events $\{\hat {b}_{i} \neq 0\}$ and {R = r} by using the gradient U(b) of the negative loss function − l(b);

$$ U(b)=(U_{1}(b), ... , U_{p}(b))^{\prime}= -\triangledown (l(b)) . $$

(3.2)

Additionally, for a > 0 we define the vector T(a) as

$$ T(a) = U(\hat{b})+ a\hat{b} . $$

Thus, $T_{i}(a)=U_{i}(\hat {b})$ if $\hat {b}_{i}=0$. Also, the additional term $a\hat {b}_{i}$ has the same sign as $U_{i}(\hat {b})$, so $|T_{i}(a)|>|U_{i}(\hat b)|$ if $\hat {b}_{i}\neq 0$. By calculating the subgradient of the objective function of the LASSO estimator, it is possible to check that LASSO selects these variables for which the respective coordinates of |T(a)| exceed the value of the tuning parameter λ. In Bogdan et al. (2015) the support of $\hat b$ for SLOPE under the orthogonal design is provided. It is shown that, similarly as in the case of the Benjamini-Hochberg correction for multiple testing, it is not sufficient to compare the ordered coordinates of |T(a)| to the respective values of the decaying sequence of tuning parameters. It can happen that while performing this simple operation one could eliminate regressors with the value of |T(a)| larger than for some of the regressors retained in the model. The SLOPE estimator preserves the ordering of |T(a)|. Thus, identification of the SLOPE support is more involved and requires introduction of the following sets H_r: for r ∈{1, … , p} we define

$$ H_{r} = \left\{ w \in \mathbb{R}^{p}: \forall_{j \leqslant r} \sum\limits_{i=j}^{r} \lambda_{i} < \sum\limits_{i=j}^{r} \vert w \vert_{(i)} ~and~ \forall_{j \geqslant r+1} \sum\limits_{i=r+1}^{j}\lambda_{i} \geqslant \sum\limits_{i=r+1}^{j} \vert w\vert_{(i)} \right\}. $$

(3.3)

Theorem 3.1.

Consider the optimization problem 3.1 with an arbitrary sequence $\lambda _{1}\geqslant \lambda _{2} \geqslant ... \geqslant \lambda _{p} \geqslant 0$. Assume that l(b) is a convex and a differentiable function. Then for any a > 0,

$$ R = r \Longleftrightarrow T(a) \in H_{r} . $$

Theorem 3.2.

Consider the optimization problem 3.1 with an arbitrary sequence $\lambda _{1}\geqslant \lambda _{2} \geqslant ... \geqslant \lambda _{p} \geqslant 0$. Assume that l(b) is a convex and a differentiable function and R = r. Then for any a > 0 it holds:

$$ \left( \hat{b}_{i} \neq 0 \right) \Leftrightarrow \left( \vert T_{i}(a) \vert >\lambda_{r}\right) $$

and

$$ \left( \hat{b}_{i} \neq 0 \right) \Rightarrow \left( \vert U_{i}(\hat b) \vert \geqslant\lambda_{r}\right). $$

Moreover if we assume that $\lambda _{1}>\lambda _{2}>...>\lambda _{p}\geqslant 0$, then:

$$ \left( \hat{b}_{i} \neq 0 \right) \Leftarrow \left( \vert U_{i}(\hat b) \vert \geqslant \lambda_{r} \right). $$

The proofs of Theorems 3.1 and 3.2 are provided in the supplementary materials.

3.2 FDR of SLOPE for the general loss function

Corollary 3.3.

Consider the optimization problem (3.1) with an arbitrary sequence $\lambda _{1}\geqslant \lambda _{2} \geqslant ... \geqslant \lambda _{p} \geqslant 0$. Assume that l(b) is a convex and a differentiable function. Then for any a > 0, FDR of SLOPE is equal to:

$$ FDR = \sum\limits_{r=1}^{p}\frac{1}{r}\sum\limits_{i \in S^{c}}P\left( |T_{i}(a)|>\lambda_{r},T(a) \in H_{r} \right). $$

(3.4)

Proof

Let us denote the support of the true parameter vector b⁰ by:

$$ S = Supp(b^{0}) $$

(3.5)

and a set that is the complement of S in {1, ... , p} by:

$$ S^{c} = \{1, ... , p\}\setminus S . $$

(3.6)

Directly from the definition we obtain:

(3.7)

and Corollary 3.3 is a direct consequence of Theorems 3.1 and 3.2. □

3.3 Proof of Theorem 2.1

We now focus on the multiple regression model (1.1). Elementary calculations show that in this case the vector U (for def. see (3.2)) takes the form

$$ U(b) = X^{\prime}(Y-Xb) = X^{\prime}\epsilon + X^{\prime}X(b^{0}-b). $$

Let us denote for simplicity

$$ T = T(1) = \hat{b}+U(\hat{b}) = X^{\prime}\epsilon + b^{0} + (X^{\prime}X - \mathbb{I})(b^{0}-\hat{b}) $$

and introduce the following notation for the components of T:

$$ M = X^{\prime}\epsilon + b^{0} $$

(3.8)

and

$$ {\Gamma} = (\mathbb{I}-X^{\prime}X)(\hat{b}-b^{0}) . $$

(3.9)

Naturally, T = M + Γ. Due to (3.4), we can express FDR for linear regression in the following way:

$$ FDR = \sum\limits_{r=1}^{p}\frac{1}{r}\sum\limits_{i \in S^{c}}P(T \in H_{r},\vert T_{i} \vert > \lambda_{r}) . $$

(3.10)

Deeper analysis shows that under the assumptions of Theorem 2.1, the FDR expression (3.4) can be simplified. Corollary 3.5, stated below, follows directly from Lemma 4.4 in Su and Candès (2016) (see the supplementary materials) and shows that, with a large probability, only the first elements of the summation over r are different from zero. Furthermore, the following Lemma 3.6 shows that elements of the vector Γ are sufficiently small, so we can focus on the properties of the vector M.

Definition 3.4 (Resolvent set, Su and Candès (2016)).

Fix S = supp(b⁰) of cardinality k, and an integer $\tilde k^{*}$ obeying $k < \tilde k^{*} < p$. The set $\tilde S^{*} = \tilde S^{*}(S, \tilde k^{*})$ is said to be a resolvent set if it is the union of S and the $\tilde k^{*}-k$ indexes with the largest values of $\vert X_{i}^{\prime } \epsilon \vert$ among all i ∈{1, ... , p}∖ S.

Let us introduce the following notation on a sequence of events when the union of supports of b⁰ and $\hat b$ is contained in $\tilde S^{*}$

$$ Q_{1}(n,\tilde k^{*}) = \{Supp(b^{0})\cup Supp(\hat{b})\subseteq \tilde S^{*}\}. $$

(3.11)

Corollary 3.5.

Suppose the assumptions of Theorem 2.1 hold. Then there exists a deterministic sequence k^∗ such that $k^{*}/p \rightarrow 0$, $((k^{*})^{2} \log p)/n \rightarrow 0$ and:

$$ P(Q_{1}(n, k^{*})) \rightarrow 1. $$

(3.12)

Corollary 3.5 follows directly from Lemma 4.4 in Su and Candès (2016) (see Lemma S.2.2 in the supplementary materials and the discussion below). From now on k^∗ will denote the sequence satisfying Corollary 3.5.

Lemma 3.6.

Let us denote by $Q_{2}(n, \tilde {\gamma }(n))$ a sequence of events when the $l_{\infty }$ norm of vector Γ is smaller than $\tilde \gamma (n)$:

$$ Q_{2}(n,\tilde \gamma (n)) = \{\max_{i}\vert {\Gamma}_{i} \vert \leqslant \tilde\gamma(n) \} . $$

(3.13)

If the assumptions of Theorem 2.1 hold then there exists a constant C_q, dependent only on q, such that the sequence $\gamma (n) = C_{q} \sqrt {\frac {(k^{*})^{2} \log p}{n}} \lambda ^{BH}_{k^{*}}$ satisfies:

$$ P\left( Q_{2}(n,\gamma(n)) \right) \rightarrow 1 . $$

(3.14)

The proof of Lemma 3.6 is provided in the Appendix.

Let us denote by Q₃(n,u) a sequence of events such that the l₂ norm of the vector 𝜖 divided by $\sigma \sqrt {n}$ is smaller than 1 + 1/u:

$$ Q_{3}(n,u) = \left\{ \frac{\Vert \epsilon \Vert_{2}}{\sigma\sqrt{n}} \leqslant 1+1/u \right\}. $$

(3.15)

The following Corollary 3.7 is a consequence of the well known results on the concentration of the Gaussian measure (see Theorem S.2.4 in the supplementary materials).

Corollary 3.7.

Let k^∗ = k^∗(n) be the sequence satisfying Corollary 3.5. Then

$$ P\left( Q_{3}(n,k^{*}) \right)\geqslant 1 - e^{- \frac{n}{2(k^{*})^{2}}} \rightarrow 1. $$

(3.16)

From now on, for simplicity, we shall denote by Q₁,Q₂ and Q₃ the sequences Q₁(n,k^∗),Q₂(n,γ) and Q₃(n,k^∗) respectively. Moreover, let us introduce the following notation on the intersection of Q₁,Q₂ and Q₃:

$$ Q = Q(n) = Q_{1}(n,k^{*}) \cap Q_{2}(n,\gamma) \cap Q_{3}(n,k^{*}). $$

(3.17)

By using an event Q, we can bound FDR in the following way:

(3.18)

The first equality follows from the fact that . The inequality uses the fact that $\frac {V}{R\vee 1} \leqslant 1$. The second equality is a consequence of the formula (3.7) applied to the second term. Naturally, due to conditions (3.12), (3.14) and (3.16), we obtain $P(Q^{c})\rightarrow 0$. Therefore, we can focus on the properties of the second term in (3.18). Notice that Q₁ implies that $R\leqslant k^{*}$ ($supp(\hat b) \subset S^{*}$), therefore we can limit the summation over r to the first k^∗ elements:

$$ \sum\limits_{r=1}^{p}\frac{1}{r}\sum\limits_{i \in S^{c}} P(R=r,\hat{b}_{i} \neq 0,Q)= \sum\limits_{r=1}^{k^{*}}\frac{1}{r}\sum\limits_{i \in S^{c}}P(R=r,\hat{b}_{i} \neq 0,Q) . $$

(3.19)

Furthermore, according to Theorems 3.1 and 3.2:

$$ \sum\limits_{r=1}^{k^{*}}\frac{1}{r}\sum\limits_{i \in S^{c}} P(R=r,\hat{b}_{i} \neq 0,Q)= \sum\limits_{r=1}^{k^{*}}\frac{1}{r}\sum\limits_{i \in S^{c}}P(T \in H_{r},\vert T_{i} \vert> \lambda_{r},Q) . $$

(3.20)

We now introduce the useful notation:

• a vector $M^{(i)} = (M^{(i)}_{1}, ... , M^{(i)}_{p})^{\prime }$ is the following modification of M

  $$ M^{(i)}_{j}:= \left\lbrace\begin{array}{l} M_{j}~\text{for}~ j\neq i \\ \infty~\text{for}~ j=i\end{array}\right. $$

• a set $H_{r}^{\gamma }$, which is a generalization of the set H_r (3.3),

  $$ \begin{array}{@{}rcl@{}} H_{r}^{\gamma} &=& \{ w \in \mathbb{R}^{p}: \forall_{j \leqslant r} \sum\limits_{i=j}^{r} (\lambda_{i}- \gamma)\\ &<& \sum\limits_{i=j}^{r} \vert w \vert_{(i)} ~and~ \forall_{j \geqslant r+1} \sum\limits_{i=r+1}^{j}(\lambda_{i}+ \gamma) \geqslant \sum\limits_{i=r+1}^{j} \vert w\vert_{(i)} \}. \end{array} $$

Lemma 3.8 (see the Appendix for the proof) allows for the replacement of an event {T ∈ H_r,|T_i| > λ_r,Q₂} by an event which depends only on M⁽ⁱ⁾.

Lemma 3.8.

If T ∈ H_r, |T_i| > λ_r and Q₂ occurs, then $M^{(i)} \in H_{r}^{\gamma }$.

Lemma 3.8, together with a fact that under Q₂ |T_i| > λ_r, allows for the conclusion that |M_i| > λ_r − γ and therefore

$$ \sum\limits_{r=1}^{k^{*}}\frac{1}{r}\sum\limits_{i \in S^{c}}\! P(T \!\in\! H_{r},\vert T_{i} \vert\!>\! \lambda_{r},Q) \!\leqslant\! \sum\limits_{r=1}^{k^{*}}\frac{1}{r}\sum\limits_{i \in S^{c}}P\!\left( M^{(i)} \!\in\! H_{r}^{\gamma},\vert M_{i} \vert\!>\! \lambda_{r} - \gamma,Q_{3}\right)\! . $$

(3.21)

The following Lemma 3.9 (see the Appendix for the proof ) provides asymptotic behavior of the right-hand side of (3.21):

Lemma 3.9.

Under the assumptions of Theorem 2.1, it holds:

$$ \sum\limits_{r=1}^{k^{*}}\frac{1}{r}\sum\limits_{i \in S^{c}}P\left( M^{(i)} \in H_{r}^{\gamma},\vert M_{i} \vert> \lambda_{r}-\gamma,Q_{3}\right) \rightarrow 0. $$

(3.22)

The proof of Lemma 3.9 is based on several properties. First,

$$ P\left( M^{(i)} \in H_{r}^{\gamma},\vert M_{i} \vert> \lambda_{r}-\gamma,Q_{3}\right) $$

can be well approximated by

$$ P\left( M^{(i)} \in H_{r}^{\gamma},Q_{3}\right) P\left( \vert M_{i} \vert> \lambda_{r}-\gamma,Q_{3}\right) $$

This approximation is a consequence of the fact that conditionally on 𝜖, M⁽ⁱ⁾ and M_i are independent. Second, for i ∈ S^c:

$$ P\left( \vert M_i \vert> \lambda_r-\gamma,Q_3\right) $$

can be well approximated by

$$ 2(1-{\Phi}((1+\delta)\lambda_r^{BH})) \approx \left( \frac{rq}{p} \right)^{(1+\delta)^2} , $$

where Φ(⋅) is the cumulative distribution function of the standard normal distribution. The first approximation is a consequence of the fact that for i ∈ S^c, M_i = X_i𝜖. Thus, conditionally on 𝜖, M_i has a normal distribution with the mean equal to 0 and the variance equal to $\frac {||\epsilon ||^2}{n}$, which is close to σ² (see Corollary 3.7). The second approximation relies on the well known formula

$$ 1-{\Phi}(c)=\frac{\phi(c)}{c}(1+o(c)) , $$

(3.23)

where ϕ(⋅) is the density of the standard normal distribution and o(c) converges to zero as c diverges to infinity.

Lastly

$$ \sum\limits_{r=1}^{k^{*}}P\left( M^{(i)} \in H_r^{\gamma}\right) $$

equals approximately to

$$ \sum\limits_{r=1}^{k^{*}}P\left( M^{(i)} \in H_r\right) \leqslant 1. $$

This approximation is a consequence of the fact that Hrγ does not differ much from H_r and the family of the sets H_r with r ∈{1, … , p} is disjoint. By applying the above approximations we obtain:

$$ \sum\limits_{r=1}^{k^{*}}\frac{1}{r}\sum\limits_{i \in S^{c}}P\left( M^{(i)} \in H_r^{\gamma},\vert M_i \vert> \lambda_r-\gamma,Q_3\right) \approx \sum\limits_{i \in S^{c}}\sum\limits_{r=1}^{k^{*}}\frac{1}{r} \left( \frac{rq}{p} \right)^{(1+\delta)^2}P\left( M^{(i)} \in H_r^{\gamma}\right) \leqslant $$

$$ \left( \frac{q}{p} \right)\left( \frac{k^{*}q}{p} \right)^{(1+\delta)^2-1}\underbrace{\sum\limits_{i \in S^{c}}\sum\limits_{r=1}^{k^{*}} P\left( M^{(i)} \in H_r^{\gamma}\right)}_{\approx p-k} \approx \left( \frac{(p-k)q}{p} \right)\left( \frac{k^{*}q}{p} \right)^{(1+\delta)^2-1} \rightarrow 0 $$

Lemma 3.9, together with the fact that under assumptions of Theorem 2.1 $P(Q^c)\rightarrow 0$, provides $FDR \rightarrow 0$. One can notice that $\left (\frac {k^{*}q}{p} \right )^{(1+\delta )^2-1}$ is the factor responsible for the convergence of FDR to 0. It remains an open question if in the definition of the λ sequence (2.7) a constant δ can be replaced by a sequence converging to 0 at such a rate that the asymptotic FDR is exactly equal to the nominal level q. The proof of this assertion would require a refinement of the bounds provided in Su and Candès (2016) and we leave this as a topic for future research.

Now, we will argue that under our assumptions the Power of SLOPE converges to 1 . Recall that $TR = \#\{j: b_j^0 \neq 0 \text {and} \hat b_j \neq 0\}$ denotes the number of true rejections. Observe that

$$ {\Pi} = \frac{1}{k}\mathbb{E}\left( TR \right)= \frac{1}{k} \sum\limits_{j = 1}^k jP(TR=j) \geqslant P(TR = k) = P\left( \bigcap_{i \in S}\{\hat b_i \neq 0 \}\right). $$

Naturally ${\Pi } \leqslant 1$, therefore by showing that $P\left (\bigcap _{i \in S}\{\hat b_i \neq 0 \}\right ) \rightarrow 1$ we obtain the thesis.

Lemma 3.10.

Under the assumptions of Theorem 2.1, it holds:

$$ P\left( \bigcap_{i \in S}\{\hat b_i \neq 0 \}\right) \rightarrow 1. $$

(3.24)

Proof

The proof of Lemma 3.10 is provided in the Appendix. It is based on the sequence of the following inequalities:

$$ P\left( \bigcap_{i \in S}\{\hat b_i \neq 0 \}\right)\geq P\left( \bigcap_{i \in S}\{|T_i|>\lambda_1\}\right)\geq P\left( \bigcap_{i \in S}\{|M_i|>\lambda_1+\gamma \}\right) $$

$$ \geq P\left( \bigcap_{i \in S}\{ X_i^{\prime} \epsilon> \lambda_1+\gamma-|b_i^0| \}\right)\geq P\left( \bigcap_{i \in S}\left\{ X_i^{\prime} \epsilon> -\sigma\left( 1+\frac{\delta}{2}\right)\sqrt{2\log p}\right.\right) . $$

The result follows by the conditional independence of $X_i^{\prime }\epsilon$ (given 𝜖) and the classical approximation (3.23). □

4 Discussion

In this article we provide new asymptotic results on the model selection properties of SLOPE in the case when the elements of the design matrix X come from the normal distribution. Specifically, we provide conditions on the sparsity and the magnitude of true signals such that FDR of SLOPE based on the sequence λ^BH, corresponding to the thresholds of the Benjamini-Hochberg correction for multiple testing, converges to 0 and the Power converges to 1. We believe these results can be extended to the sub-gaussian design matrices with independent columns, which is the topic of ongoing research. Additionally, the general results on the support of SLOPE open the way for investigation of the properties of SLOPE under arbitrary convex and differentiable loss functions.

In simulations we compared SLOPE based on the sequence λ^BH with SLOPE based on the heuristic sequence proposed in Bogdan et al. (2015) and with LASSO with the tuning parameter λ adjusted to the first value of the tuning sequence for SLOPE. When regressors are independent and the vector of true regression coefficients is relatively sparse then the comparison between SLOPE and LASSO bears similarity to the comparison between the Bonferroni and the Benjamini-Hochberg corrections for multiple testing. When k = 0 both methods perform similarly and control FDR (which for k = 0 is equal to the Family Wise Error Rate) close to the nominal level. When k increases, FDR of LASSO converges to zero, which however comes at the prize of a substantial loss of Power for moderately strong signals. Concerning the two versions of SLOPE, our simulations suggest that the heuristic sequence allows for a more accurate FDR control over a wide range of sparsity values. We believe the techniques developed in this article form a good foundation for the analysis of the statistical properties of the heuristic version of SLOPE, which we consider an interesting topic for further research.

Our assumptions on the independence of predictors and the sparsity of the vector of true regression coefficients are restrictive, which is related to the well known problems with FDR control by LASSO (Bogdan et al. 2013; Su et al. 2017). In the case of LASSO, these problems can be solved by using adaptive or reweighted LASSO (Zou, 2006; Candès et al. 2008), which allow for the consistent model selection under much weaker assumptions. In these modifications the values of the tuning parameters corresponding to predictors which are deemed important (based on the values of initial estimates) are reduced, which substantially reduces the bias due to shrinkage and improves model selection accuracy. Recently (Jiang et al. 2019) developed the Adaptive Bayesian version of SLOPE (ABSLOPE). According to the results of simulations presented in (Jiang et al. 2019), ABSLOPE controls FDR under a much wider set of scenarios than the regular SLOPE, including examples with strongly correlated predictors. We believe our proof techniques can be extended to derive asymptotic FDR control for ABSLOPE, which we leave as an interesting topic for future research.

Appendix

Proof

The proof of Lemma 3.6

We prove a stronger condition: $P(Q_1 \cap Q_2) \rightarrow 1$.

Denote by $X_I,\hat {b}_I,b^0_I$ a submatrix (subvector) of $X ,\hat {b},b^0$ consisting of columns (vector elements) with indexes in a set I.

Observe that when $Q_1 = \{Supp(b^0)\cup Supp(\hat {b})\subseteq S^{*}\}$, we can express $\max \limits _i\vert {\Gamma }_i \vert$ in the following way:

$$ \max_i\vert {\Gamma}_i \vert = \max ([\max_{i}\vert [X_{{(S^{*})}^c}^{\prime}X_{S^{*}}(\hat{b}_{S^{*}}-b^0_{S^{*}})]_{i} \vert , \max_{i }\!\vert [(X_{S^{*}}^{\prime}X_{S^{*}}-\mathbb{I}\!)\!(\hat{b}_{S^{*}}-b^0_{S^{*}})]_{i} \vert ) $$

We show that both elements are bounded by $C_q \sqrt {\frac {(k^{*})^2 \log p}{n}} \lambda ^{BH}_{k^{*}}$ with the probability tending to 1. The bound on the first component is a direct corollary from Lemma A.12 proved in Su and Candès (2016).

Corollary 5.1.

Under the assumptions of Theorem 2.1, there exists a constant C_q only depending on q such that:

$$ \max_{i \in {(S^{*})}^c}\vert [X_{{(S^{*})}^c}^{\prime}X_{S^{*}}(\hat{b}_{S^{*}}-b^0_{S^{*}})]_{i} \vert \leqslant C_q \sqrt{\frac{k^{*} \log p}{n}} \lambda^{BH}_{k^{*} +1} $$

(5.1)

with the probability tending to 1.

Lemma A.12 of Su and Candès (2016) and the proof of Corollary 5.1 can be found in the supplementary materials (see Lemma S.2.5 and the discussion below).

It remains to be proven that the second component is also bounded by $C_q \sqrt {\frac {(k^{*})^2 \log p}{n}} \lambda ^{BH}_{k^{*}}$ with the probability tending to 1. To do this, we use Lemma A.11 proved in Su and Candès (2016) which provides bounds on the largest and the smallest singular values of $X_{S^{*}}$ (for details see Lemma S.2.6 in the supplementary materials).

Let $X_{S^{*}} = G{\Sigma } V^{\prime }$ be the Singular Value Decomposition (SVD) of the matrix $X_{S^{*}}$, where $G \in \mathbb {M}_{n \times n}$ is a unitary matrix, ${\Sigma } \in \mathbb {M}_{n \times k^{*}}$ is a diagonal matrix with non-negative real numbers on the diagonal and $V \in \mathbb {M}_{k^{*} \times k^{*}}$ is a unitary matrix. Moreover, let us denote by σ_i,σ_min and σ_max the i-th, the smallest and the largest singular value.

Assume that u is an arbitrary unit vector. Due to the relation between the $l_{\infty }$ and the l₂ vector norms, and additionally between the l₂ vector norm and the ∥⋅∥₂ matrix norm, it holds:

$$ \max_{i}\vert [(X_{S^{*}}^{\prime}X_{S^{*}}-\mathbb{I})u]_{i} \vert \leqslant \Vert (X_{S^{*}}^{\prime}X_{S^{*}}-\mathbb{I})u \Vert_2 \leqslant \Vert X_{S^{*}}^{\prime}X_{S^{*}}-\mathbb{I} \Vert_2 . $$

Using the SVD of the matrix $X_{S^{*}}$ and the sub-multiplicity of the ∥⋅∥₂ matrix norm we continue:

$$ \Vert V({\Sigma}^{\prime}{\Sigma}-\mathbb{I})V^{\prime} \Vert_2 \leqslant \Vert V \Vert_2 \Vert {\Sigma}^{\prime}{\Sigma}-\mathbb{I} \Vert_2 \Vert V \Vert_2 = \Vert {\Sigma}^{\prime}{\Sigma}-\mathbb{I} \Vert_2 = \sigma_{max}({\Sigma}^{\prime}{\Sigma}-\mathbb{I}). $$

By definition the maximal singular value of the matrix A equals to the square root of the largest eigenvalue of the positive-semidefinite matrix $A^{\prime }A$. Therefore in our case it is obvious that:

$$ \sigma_{max}({\Sigma}^{\prime}{\Sigma}-\mathbb{I}) = \max_i(|\sigma_i^2 - 1|) = \max(|\sigma_{min}^2 - 1|,|\sigma_{max}^2 - 1|) , $$

where σ_i is the i-th singular value of $X_{S^{*}}$. Due to Lemma A.11 of Su and Candès (2016) (see Lemma S.2.6 in the supplementary materials) ,we obtain that for some constants C₁ and C₂

$$ \max(|\sigma_{min}^2 - 1|,|\sigma_{max}^2 - 1|) \leqslant C_1 \max(|\sigma_{min} - 1|,|\sigma_{max} - 1|) \leqslant C_2\sqrt{k^{*} \log(p/k^{*})/n} $$

and in consequence for an arbitrary unit vector u

$$ \max_{i}\vert [(X_{S^{*}}^{\prime}X_{S^{*}}-\mathbb{I})u]_{i} \vert \leqslant C_2\sqrt{k^{*} \log(p/k^{*})/n} $$

(5.2)

with the probability at least $1 - 2e^{-k^{*} \log (p/k^{*})/2} - (\sqrt {2}ek^{*}/p)^{k^{*}}$.

On the other hand, due to Theorem 1.2 in Su and Candès (2016), for any constant δ₁ > 0

$$ P\left( \Vert \hat{b}-b^0 \Vert_2<(1+\delta_1)\sigma \sqrt{2 k \log\left( p/k\right)}\right) \rightarrow 1 . $$

(5.3)

Finally, observe that

$$ \max_{i }\vert [(X_{S^{*}}^{\prime}X_{S^{*}}-\mathbb{I})(\hat{b}_{S^{*}}-b^0_{S^{*}})]_{i} \vert = \max_{i }\vert [(X_{S^{*}}^{\prime}X_{S^{*}}-\mathbb{I})\frac{\hat{b}_{S^{*}}-b^0_{S^{*}}}{\Vert \hat{b}_{S^{*}}-b^0_{S^{*}} \Vert_2} ]_{i} \vert \Vert \hat{b}_{S^{*}}-b^0_{S^{*}} \Vert_2 .$$

Thus, the relations (5.2) and (5.3) imply that for certain constants C₁ and C₂:

$$ \max_{i }\vert [(X_{S^{*}}^{\prime}X_{S^{*}}-\mathbb{I})(\hat{b}_{S^{*}}-b^0_{S^{*}})]_{i} \vert \leqslant C_1\sqrt{\frac{k^{*}k \log(p/k)}{n}}\sqrt{\log(p/k^{*})} \leqslant C_2\sqrt{\frac{(k^{*})^2 \log(p)}{n}}\lambda^{BH}_{k^{*}} $$

(5.4)

with the probability tending to 1.

The inequality (5.4), together with (5.1), provide the thesis of the Lemma. □

Proof

The proof of Lemma 3.8

In order to prove Lemma 3.8, we have to introduce the modification of a vector T similar to that of vector M. Denote $T^{(i)} =(T^{(i)}_1, ... , T^{(i)}_p)^{\prime }$:

$$ T^{(i)}_j:= \left\lbrace\begin{array}{l} T_j i\neq j \\ \infty i=j\end{array}\right. . $$

In the first step we show that:

Proposition 5.2.

If T ∈ H_r and |T_i| > λ_r occurs, then T⁽ⁱ⁾ ∈ H_r

Proof

Let us recall the definition of a set H_r:

$$ H_r = \{ w \in \mathbb{R}^p: \forall_{j \leqslant r} \sum\limits_{l=j}^r \lambda_l < \sum\limits_{l=j}^r \vert w \vert_{(l)} and \forall_{j \geqslant r+1} \sum\limits_{l=r+1}^j\lambda_l \geqslant \sum\limits_{l=r+1}^j \vert w\vert_{(l)} \}. $$

On the one hand, we know that |T_i| > λ_r. On the other hand, T ∈ H_r implies that $\vert T \vert _{(r+1)} \leqslant \lambda _{r+1}$ (the second condition in the definition of H_r for j = r + 1). These inequalities imply together that $\vert T_i \vert \geqslant \vert T \vert _{(r)}$. Hence we only have to show that:

$$ (\forall_{j \leqslant r} \sum\limits_{l=j}^r \lambda_l < \sum\limits_{l=j}^r \vert T \vert_{(l)}) \Rightarrow (\forall_{j \leqslant r} \sum\limits_{l=j}^r \lambda_l < \sum\limits_{l=j}^r \vert T^{(i)} \vert_{(l)}). $$

(5.5)

To see that the above is true we have to consider the relations between the r largest elements of vectors |T| and |T⁽ⁱ⁾|. Let us assume that |T_i| = |T|_(k) for some $k \leqslant r$. By the definition of the vector |T⁽ⁱ⁾| we know that $\vert T^{(i)} \vert _{(1)} = \vert T^{(i)}_i \vert = \infty$ and that the other ordered statistics of |T⁽ⁱ⁾| are related to the ordered statistics of |T| in the following way:

$$ \forall_{2 \leqslant s \leqslant k} \vert T^{(i)} \vert_{(s)} = \vert T \vert_{(s-1)} $$

$$ \forall_{k+1 \leqslant s \leqslant r} \vert T^{(i)} \vert_{(s)} = \vert T \vert_{(s)}. $$

In consequence we obtain that:

$$ \forall_{1 \leqslant s \leqslant r} \vert T^{(i)} \vert_{(s)} \geqslant \vert T \vert_{(s)} $$

and this implies (5.5). □

In the second step of the proof of Lemma 3.8 we show that:

$$ T^{(i)} \in H_r \land Q_2 \Rightarrow M^{(i)} \in H^{\gamma}_r . $$

In order to prove this, we use the following Proposition:

Proposition 5.3.

Let us assume we have three vectors $A,B,C \in \mathbb {R}^p$ and that A = B + C. Furthermore, let us assume that the vector A is ordered ($\vert A_1 \vert \geqslant ...\geqslant \vert A_p\vert$) and define $d = \sup _i \vert C_i \vert$. Under the above assumptions:

$$ \left\vert \vert A_i \vert - \vert B \vert_{(i)} \right\vert \leqslant d $$

for all i = 1, ... , p.

Proof

Let

$$ e = \sup_i \left\vert \vert A_i \vert - \vert B_i \vert \right\vert . $$

From the triangular inequality, we have:

$$ \vert C_i \vert = \vert A_i-B_i \vert\geqslant \left\vert \vert A_i \vert - \vert B_i \vert \right\vert \text{for} i = 1,2, ... , p . $$

In consequence, $d \geqslant e$. Thus, to prove the Proposition 5.3 it is sufficient to prove that

$$ \left\vert \vert A_i \vert - \vert B \vert_{(i)} \right\vert \leqslant e \text{for} i = 1,2, ... , p . $$

For this aim, let i₀ be the index such that:

$$ \left\vert \vert A_{i_0} \vert - \vert B \vert_{(i_0)} \right\vert = \sup_i \left\vert \vert A_i \vert - \vert B \vert_{(i)} \right\vert = f . $$

When $\vert A_{i_0} \vert = \vert B \vert _{(i_0)}$, then f = 0 and the thesis is obtained immediately. Let us consider the case when $\vert A_{i_0} \vert > \vert B \vert _{(i_0)}$. If $\vert B_{i_0} \vert \leqslant \vert B \vert _{(i_0)}$, then:

$$ f = \left\vert \vert A_{i_0} \vert - \vert B \vert_{(i_0)} \right\vert = \vert A_{i_0} \vert - \vert B \vert_{(i_0)} \leqslant \vert A_{i_0} \vert - \vert B_{i_0} \vert\leqslant e $$

and in consequence we obtain the thesis. If $\vert B_{i_0} \vert \ > \vert B \vert _{(i_0)}$, then:

$$ \vert B_{i_0} \vert \in \mathbb{B} = \left\{\vert B \vert_{(1)},\vert B \vert_{(2)}, ... , \vert B \vert_{(i_0-1)} \right\} . $$

On the other hand we know that:

$$ \vert A_{i_0}\vert \in \mathbb{A} = \left\{ \vert A_1\vert, \vert A_2\vert, ... , \vert A_{i_0}\vert \right\} . $$

Therefore, the set $\mathbb {A}$ has one more element than $\mathbb {B}$ does. Hence, there exists an element $|A_{i_1}| \in \mathbb {A}$, which is associated with an element $|B_{i_1}|$ from a complement of the set $\mathbb {B}$:

$$ |B_{i_1}| \in \mathbb{B}^c = \left\{\vert B \vert_{(i_0)},\vert B \vert_{(i_0+1)}, ... , \vert B \vert_{(p)} \right\} . $$

In consequence, due to facts $|A_{i_0}| \leqslant |A_{i_1}|$ and $|B_{i_1}| \leqslant \vert B \vert _{(i_0)}$ we obtain:

$$ f= |A_{i_0}| - \vert B \vert_{(i_0)} \leqslant |A_{i_1}| - |B_{i_1}|\leqslant e , $$

which completes the proof.

The proof for the case when $\vert A_{i_0} \vert < \vert B \vert _{(i_0)}$ is analogous. □

Now let us recall the relation between T⁽ⁱ⁾ and M⁽ⁱ⁾:

$$ T^{(i)}_j:= \left\lbrace\begin{array}{l} M^{(i)}_j + {\Gamma}_j i\neq j \\ M^{(i)}_j i=j\end{array}\right. . $$

Furthermore recall that $Q_2 = \{\max \limits _i\vert {\Gamma }_i \vert \leqslant \gamma \}$. When we assume that Q₂ occurs and apply Proposition 5.3 to a vector $(\vert T^{(i)}\vert _{(1)}, ... , \vert T^{(i)}\vert _{(p)})^{\prime }$, we obtain that:

$$ \vert M^{(i)} \vert_{(j)} -\gamma \leqslant \vert T^{(i)} \vert_{(j)} \leqslant \vert M^{(i)} \vert_{(j)} +\gamma $$

(5.6)

for all j = 1, ... , p. Let us recall the definitions of H_r and $H^{\gamma }_r$:

$$ H_r = \{ w \in \mathbb{R}^p: \forall_{j \leqslant r} \sum\limits_{l=j}^r \lambda_l < \sum\limits_{l=j}^r \vert w \vert_{(l)} and \forall_{j \geqslant r+1} \sum\limits_{l=r+1}^j\lambda_l \geqslant \sum\limits_{l=r+1}^j \vert w\vert_{(l)} \} $$

$$ H_r^{\gamma} = \{ w \in \mathbb{R}^p: \forall_{j \leqslant r} \sum\limits_{i=j}^r (\lambda_i- \gamma) < \sum\limits_{i=j}^r \vert w \vert_{(i)} and \forall_{j \geqslant r+1} \sum\limits_{i=r+1}^j(\lambda_i+ \gamma) \geqslant \sum\limits_{i=r+1}^j \vert w\vert_{(i)} . \} $$

It can be noticed that due to condition (5.6) we have:

$$ T^{(i)} \in H_r \land Q_2 \Rightarrow M^{(i)} \in H^{\gamma}_r $$

and the proof of Lemma 3.8 is completed. □

Proof

The proof of Lemma 3.9

Without the loss of generality, let us assume that 1 ∈ S^c. Observe that conditionally on 𝜖, the vector M⁽ⁱ⁾ and the variable M_i are independent. Recall that $Q_3 = \{ \frac {\Vert \epsilon \Vert _2}{\sigma \sqrt {n}} \leqslant 1+1/k^{*} \}$ depends only on 𝜖. Therefore:

$$ P\left( M^{(i)} \in H_r^{\gamma},\vert M_i \vert> \lambda_r-\gamma,Q_3\right) = E \left( P\left( M^{(i)} \in H_r^{\gamma},Q_3| \epsilon) P(\vert M_i \vert> \lambda_r-\gamma,Q_3| \epsilon \right) \right) . $$

(5.7)

Now for i ∈ S^c:

$$ P\left( \vert M_i \vert> \lambda_r-\gamma,Q_3| \epsilon\right) = P\left( \frac{\vert X_i^{\prime}\epsilon\vert}{\frac{\Vert \epsilon \Vert_2}{\sigma\sqrt{n}}}>\frac{\lambda_r-\gamma}{\frac{\Vert \epsilon \Vert_2}{\sigma\sqrt{n}}},Q_3|\epsilon\right) \leqslant $$

$$ \leqslant P\left( \sqrt{n}\frac{\vert X_i^{\prime}\epsilon\vert}{\Vert \epsilon \Vert_2}>\frac{\lambda_r-\gamma}{\sigma(1+1/k^{*})}| \epsilon\right) = 2 \left( 1-{\Phi}\left( \frac{\lambda_r-\gamma}{\sigma(1+1/k^{*})}\right) \right) , $$

(5.8)

where the last equality is a consequence of the fact that conditionally on 𝜖, $\sqrt {n}\frac {X_i^{\prime }\epsilon }{\Vert \epsilon \Vert _2}$ has a standard normal distribution. Furthermore, from the definition of λ_r and γ we know that for a large enough n

$$ \frac{\lambda_r-\gamma}{\sigma(1+1/k^{*})} = \frac{ 1+\delta- \sqrt{\frac{(k^{*})^2 \log p}{n}}C_q\frac{\lambda^{BH}_{k^{*}}}{\lambda^{BH}_r}}{1+1/k^{*}}\lambda^{BH}_r \geqslant \left( 1+3\delta/4\right)\lambda^{BH}_r \geqslant (1+\delta/2)\sqrt{2 \log(p/qr)} , $$

where in the last inequality we used the fact that $\lambda ^{BH}_r/\sqrt {2 \log (p/qr)} \rightarrow 1$. Let us denote

$$ s = (1+\delta/2)\sqrt{2 \log(p/qr)} . $$

By applying the above inequality to (5.8) we obtain for a large enough n:

$$ P\left( \vert M_i \vert> \lambda_r-\gamma,Q_3| \epsilon\right) \leqslant 2 \left( 1-{\Phi}(s) \right)\leqslant \exp(-s^2/2) = \left( \frac{qr}{p} \right)^{(1+\delta/2)^2} , $$

(5.9)

where in the second inequality we used the classical approximation to the tail probability of the standard normal distribution (3.23).

By applying (5.9) and (5.7) to

$$ \sum\limits_{r=1}^{k^{*}}\frac{1}{r}\sum\limits_{i \in S^{c}}P\left( M^{(i)} \in H_r^{\gamma},\vert M_i \vert> \lambda_r-\gamma,Q_3\right) , $$

we obtain for a large enough n:

$$ \sum\limits_{r=1}^{k^{*}}\frac{1}{r}\sum\limits_{i \in S^{c}}P\left( M^{(i)} \!\in\! H_r^{\gamma},\vert M_i \vert\!>\! \lambda_r-\gamma,Q_3\right) \leqslant \sum\limits_{r=1}^{k^{*}}\frac{1}{r}\sum\limits_{i \in S^{c}}P\left( M^{(i)} \in H_r^{\gamma},Q_3\right) \left( \frac{qr}{p} \right)^{(1+\delta/2)^2} \leqslant $$

$$ \leqslant \frac{q}{p}\left( \frac{k^{*}}{p} \right)^{\delta} \sum\limits_{r=1}^{k^{*}}\sum\limits_{i \in S^{c}}P\left( M^{(i)} \in H_r^{\gamma},Q_3\right) = \frac{(p-k)q}{p}\left( \frac{k^{*}}{p} \right)^{\delta} \sum\limits_{r=1}^{k^{*}}P\left( M^{(1)} \in H_r^{\gamma},Q_3\right) , $$

where in the last equality we used the assumption that 1 ∈ S^c, the fact that for i ∈ S^c all M⁽ⁱ⁾ have the same distribution and the fact that there are p − k elements in S^c.

Due to the fact that $(k^{*}/p)^{\delta } \rightarrow 0$, in order to prove the Lemma it remains to be shown that

$$ \sum\limits_{r=1}^{k^{*}}P\left( M^{(1)} \in H_r^{\gamma},Q_3\right) \text{ is bounded.} $$

(5.10)

We prove this by showing that

$$ \sum\limits_{r=1}^{k}P(M^{(1)} \in H^{\gamma}_r,Q_3) \text{ is bounded} $$

(5.11)

and

$$ \sum\limits_{r=k+2}^{k^{*}}P(M^{(1)} \in H^{\gamma}_r,Q_3) \rightarrow 0 . $$

(5.12)

Naturally, $P(M^{(1)} \in H^{\gamma }_{k+1}) \leqslant 1$ which, together with (5.11) and (5.12) would provide (5.10) and in consequence the thesis.

We begin by proving (5.11). Let us denote by $\tilde {W}$ a vector of elements of M⁽¹⁾ with indices in S (corresponding to non-zero elements in $b^0_i$). Directly from the definition of the set $H_r^{\gamma }$, we have for $r \leqslant k$:

$$ P(M^{(1)} \in H^{\gamma}_r,Q_3) \leqslant P\left( \vert M^{(1)} \vert_{(r+1)}\leqslant \lambda_{r+1}+\gamma,Q_3\right) . $$

Furthermore

$$ P\left( \vert M^{(1)} \vert_{(r+1)}\leqslant \lambda_{r+1}+\gamma,Q_3\right) \leqslant P\left( \vert \tilde{W} \vert_{(r)}\leqslant \lambda_{r+1}+\gamma,Q_3\right) , $$

which is a consequence of the fact that $\tilde {W}$ is a subvector of M⁽¹⁾ and that $M^{(1)}_1= \vert M^{(1)} \vert _{(1)} = \infty$ (1 ∈ S^c). Therefore, it is obvious that $\vert \tilde {W} \vert _{(r)} \leqslant \vert M^{(1)} \vert _{(r+1)}$.Moreover, let W be a modification of $\tilde {W}$, where each $b^0_i$ is replaced by $2\sigma (1+\delta )\sqrt {2 \log p}$ and the resulting vector is multiplied by a function:

$$ g(\epsilon) = \left\lbrace\begin{array}{cc} 1 & \text{if} Q_3 \text{holds} \\ \infty & \text{otherwise} .\end{array}\right. $$

Therefore, $W_i = (X_j^{\prime }\epsilon + 2\sigma (1+\delta )\sqrt {2 \log p})g(\epsilon )$ for the corresponding j ∈ S. Naturally, the elements of W, conditionally on 𝜖, are independent and identically distributed. Furthermore, due to the fact that $|X_i^{\prime }\epsilon +b^0_i| =^D|X_i^{\prime }\epsilon +|b^0_i||$, the fact that the density of $X^{\prime }_i \epsilon +|b^0_i|$ has a mode at $|b^0_i|$, the assumption $\min \limits _{i \in S} \vert b^0_i \vert > 2\sigma (1+\delta )\sqrt {2 \log p}$ and the fact that $2\sigma (1+\delta )\sqrt {2 \log p} \geqslant \lambda _{r+1}+\gamma$ for a large enough n, it holds that:

$$ P\left( \vert \tilde{W} \vert_{(r)}\leqslant \lambda_{r+1}+\gamma,Q_3\right) \leqslant P\left( \vert W \vert_{(r)}\leqslant \lambda_{r+1}+\gamma\right) . $$

Now,

$$ P\left( \vert W \vert_{(r)}\!\leqslant\! \lambda_{r+1} + \gamma\right) = \sum\limits_{i = k-r+1}^k P\left( exactly i of \vert W \vert^{\prime}s are \leqslant \lambda_{r+1}+\gamma \right) = $$

$$ \sum\limits_{i = k-r+1}^k E\left[ P\left( exactly i of \vert W \vert^{\prime}s are \leqslant \lambda_{r+1}+\gamma | \epsilon\right)\right] = $$

$$ = \sum\limits_{i = k-r+1}^k {k \choose i} E\left[ P\left( \vert W_1 \vert \leqslant \lambda_{r+1} +\gamma| \epsilon\right)^i \left( 1-P\left( \vert W_1 \vert \leqslant \lambda_{r+1} +\gamma| \epsilon\right)\right)^{k-i}\right]\leqslant $$

(5.13)

$$ \leqslant \sum\limits_{i = k-r+1}^k {k \choose i} E\left[ P\left( \vert W_1 \vert \leqslant \lambda_{r+1} +\gamma| \epsilon\right)^i \right] , $$

where in the equality between the second and the third line we used the standard combinatorial arguments for calculating the cumulative distribution function of the r-th order statistic and the fact that elements of W are independent conditionally on 𝜖. In the inequality we used the fact that $P\left (\vert W_1 \vert \leqslant \lambda _{r+1} +\gamma | \epsilon \right ) \leqslant 1$.

Now, for some j ∈ S (corresponding to W₁) and a large enough n we have

$$ P\left( \vert W_1 \vert \leqslant \lambda_{r+1} +\gamma| \epsilon\right) = P\left( \vert (X_j^{\prime}\epsilon+ 2\sigma(1+\delta)\sqrt{2 \log p})g(\epsilon) \vert \leqslant \lambda_{r+1} +\gamma| \epsilon\right) = $$

$$ =P\left( \vert (X_j^{\prime}\epsilon+ 2\sigma(1+\delta)\sqrt{2 \log p}) \vert \leqslant \lambda_{r+1} +\gamma, Q_3| \epsilon\right) \leqslant P\left( X_j^{\prime}\epsilon \leqslant -\sigma(1+\delta/2)\sqrt{2 \log p}, Q_3| \epsilon\right) , $$

where in the second line we first use the definition of the function g(𝜖) and in the inequality we skip the absolute value and use the fact that for a large enough n, $\lambda _{r+1} +\gamma \leqslant \sigma (1+1.5\delta )\sqrt {2 \log p}$. Now, due to the fact that conditionally on 𝜖, $X_j^{\prime } \epsilon$ is normal with a mean 0 and a standard deviation $\Vert \epsilon \Vert _2/\sqrt {n}$, and the fact that Q₃ provides the upper bound on this standard deviation, for a large enough n we obtain:

$$ P\left( X_j^{\prime}\epsilon \leqslant -\sigma(1+\delta/2)\sqrt{2 \log p}, Q_3| \epsilon\right) = $$

$$ = P\left( \frac{X_j^{\prime}\epsilon}{\Vert \epsilon \Vert_2/\sqrt{n}} \leqslant -\sigma \frac{(1+\delta/2)\sqrt{2 \log p}}{\Vert \epsilon \Vert_2/\sqrt{n}}, \{ \frac{\Vert \epsilon \Vert_2}{\sigma\sqrt{n}} \leqslant 1+1/k^{*} \}| \epsilon\right) \leqslant $$

$$ \leqslant P\left( \frac{X_j^{\prime}\epsilon}{\Vert \epsilon \Vert_2/\sqrt{n}} \leqslant -\frac{(1+\delta/2)\sqrt{2 \log p}}{1+1/k^{*}}| \epsilon\right) = $$

$$ = 1 - {\Phi} \left( \frac{(1+\delta/2)\sqrt{2 \log p}}{(1 +1/k^{*})} \right) \leqslant \left( \frac{1}{p}\right)^{(1+\delta/4)^2} . $$

In consequence we can limit (5.11) from the above

$$ \sum\limits_{r=1}^{k}P(M^{(1)} \in H^{\gamma}_r) \leqslant \sum\limits_{r=1}^{k} \sum\limits_{i = k-r+1}^k {k \choose i} \left( \left( \frac{1}{p}\right)^{(1+\delta/4)^2} \right)^i = $$

$$ =\sum\limits_{r=1}^{k} r {k \choose r} \left( \left( \frac{1}{p}\right)^{(1+\delta/4)^2}\right)^r \leqslant \sum\limits_{r=1}^{k} \left( \frac{1}{r}\right)^{(r-1)} \left( \frac{ek}{p}\left( \frac{1}{p}\right)^{(1+\delta/4)^2-1} \right)^r \rightarrow 0 $$

for $n \rightarrow \infty$. To see the equality between the first and the second line observe that only index i depends on r. Therefore, we sum multiple times the same elements and for a given i, the summation element occurs i times. The inequality uses the upper bound on the Newton symbol ${k \choose r} \leqslant \left (\frac {ke}{r}\right )^r$. Naturally, this result is much stronger than the relation (5.11).

Remark 5.4.

The proof of the relation (5.11) is the only place where we use the assumption on the signal strength in the context of FDR control. Naturally, when k is bounded we obtain (5.11) immediately:

$$ \sum\limits_{r=1}^{k}P(M^{(1)} \in H^{\gamma}_r,Q_3) \leqslant k $$

and the condition on the signal strength is redundant.

Now, we turn our attention to (5.12). From now on we assume that $r\geqslant k+2$. Let us denote by V a subvector of M⁽¹⁾ consisting of elements with indices in S^c ∖{1} (corresponding to elements of $b^0_i$ equal to 0, except $M^{(1)}_1$). Notice that $\vert M^{(1)} \vert _{(k+2)} \leqslant \vert V \vert _{(1)}$. This is a consequence of the fact that |V |₍₁₎ is the largest element in |V | and is equal or larger than p − k − 1 elements in |M⁽¹⁾|. Similarly, we can show that:

$$ P(M^{(1)} \in H^{\gamma}_r,Q_3) \leqslant P\left( \vert M^{(1)} \vert_{(r)}\geqslant \lambda_{r}-\gamma,Q_3\right) \leqslant P\left( \vert V \vert_{(r-k-1)}\geqslant \lambda_{r}-\gamma,Q_3\right) . $$

(5.14)

Conditionally on 𝜖, elements of V are independent and identically distributed. Therefore we have:

$$ V =^{D}\frac{\Vert \epsilon \Vert_2}{\sqrt{n}}(Z_1, ... , Z_{p-k-1})^{\prime} , $$

where the random vector $Z = (Z_1, ... , Z_{p-k-1})^{\prime } \sim N(0,\mathbb {I}_{p-k-1})$ is independent of ∥𝜖∥₂. Notice that the probability in (5.14) is maximized for the largest possible standard deviation of V (maximal $\frac {\Vert \epsilon \Vert _2}{\sqrt {n}}$). Therefore, when restricting to Q₃, we obtain:

$$ P\left( \vert V \vert_{(r-k-1)}\geqslant \lambda_{r}-\gamma, Q_3\right)= P\left( \frac{\Vert \epsilon \Vert_2}{\sqrt{n}}\vert Z \vert_{(r-k-1)}\geqslant \lambda_{r}-\gamma,Q_3\right) \leqslant $$

$$ \leqslant P\left( \vert Z \vert_{(r-k-1)}\geqslant \frac{\lambda_{r}-\gamma}{\sigma(1+1/k^{*})}\right). $$

Moreover for a large enough n we have

$$ (1+\delta/2)\sqrt{2 \log p} \leqslant \frac{\lambda_{r}-\gamma}{\sigma(1+1/k^{*})} , $$

which follows directly from the fact that for a large enough n:

$$ \frac{(\lambda_{r}-\gamma)/(1+1/k^{*})}{\sigma\sqrt{2 \log(p/r)}} \geqslant (1+3\delta/4) $$

and from the fact that:

$$ \frac{\log(p/r)}{ \log p} = 1- \frac{\log r}{ \log p}\rightarrow 1 $$

for $r \leqslant k^{*}$.In consequence we obtain for a large enough n

$$ P\left( \vert Z \vert_{(r-k-1)}\geqslant \frac{\lambda_{r}-\gamma}{\sigma(1+1/k^{*})}\right) \leqslant P\left( \vert Z \vert_{(r-k-1)}\geqslant s\right) $$

where $s = (1+\delta /2)\sqrt {2 \log p}$.

Let us denote by u₁, ... , u_p−k− 1 i.i.d. random variables from the uniform distribution U[0,1] and by $u_{[1]} \leqslant ... \leqslant u_{[p-k-1]}$, the corresponding order statistics. We know that

$$ \vert Z \vert_{(r-k-1)} =^D {\Phi}^{-1}(1 - u_{[r-k-1]}/2) . $$

Therefore, by using the classical upper bound

$$ \text{for} s>0 , 1-{\Phi}(s)\leq \frac{\phi(s)}{s} $$

(5.15)

we obtain for a large enough n:

$$ P\left( \vert Z \vert_{(r-k-1)}\geqslant s\right) = P\left( 2(1-{\Phi}(s))\geqslant u_{[r-k-1]} \right) \leqslant P\left( \left( \frac{1}{p} \right)^{(1+\delta/2)^2}\geqslant u_{[r-k-1]} \right) . $$

We also know that the i-th order statistic of the uniform distribution is a beta-distributed random variable:

$$ u_{[i]} \sim Beta(i,p-k-i) . $$

On the other hand, a well known fact is that if $A_1 \sim Gamma(i,\theta )$ and $A_2 \sim Gamma(p-k-i,\theta )$ are independent, then $A_1/(A_1+A_2) \sim Beta(i,p-k-i)$. In consequence, when E₁, ... , E_p−k are i.i.d. random variables from the exponential distribution with a mean equal to 1 it holds:

$$ u_{[i]} =^D \frac{{\sum}_{j = 1}^i E_j}{{\sum}_{j = 1}^{p-k} E_j} . $$

Therefore

$$ P\left( \left( \frac{1}{p} \right)^{(1+\delta/2)^2}\geqslant u_{[r-k-1]} \right) =P\left( (p-k)\left( \frac{1}{p} \right)^{(1+\delta/2)^2}\geqslant \frac{{\sum}_{j = 1}^{r-k-1} E_j}{\frac{1}{p-k}{\sum}_{j = 1}^{p-k} E_j} \right) $$

and

$$ P\left( (p-k)\left( \frac{1}{p} \right)^{(1+\delta/2)^2}\geqslant \frac{{\sum}_{j = 1}^{r-k-1} E_j}{\frac{1}{p-k}{\sum}_{j = 1}^{p-k} E_j} \right) \leqslant P\left( 2\left( \frac{1}{p} \right)^{\delta}\geqslant \sum\limits_{j = 1}^{r-k-1} E_j \right) +P\left( \frac{1}{p-k}\sum\limits_{j = 1}^{p-k} E_j>2\right) $$

$$ =F_E(2 p^{-\delta}, r-k-1,1) +P\left( \frac{1}{p-k}\sum\limits_{j = 1}^{p-k} E_j>2\right) , $$

where F_E is the Erlang cumulative distribution function. Therefore we obtain that

$$ P(M^{(1)} \in H^{\gamma}_r) \leqslant F_E(2 p^{-\delta}, r-k-1,1) +P\left( \frac{1}{p-k}\sum\limits_{j = 1}^{p-k} E_j>2\right) . $$

To prove (5.12) it remains to be shown that:

$$ \sum\limits_{r = k+2}^{k^{*}}F_E(2 p^{-\delta}, r-k-1,1) \rightarrow 0 $$

(5.16)

and

$$ \sum\limits_{r = k+2}^{k^{*}}P\left( \frac{1}{p-k}\sum\limits_{j = 1}^{p-k} E_j>2\right) = (k^{*}-k-1)P\left( \frac{1}{p-k}\sum\limits_{j = 1}^{p-k} E_j>2\right) \rightarrow 0 . $$

(5.17)

The first relation follows directly from the properties of the Erlang cumulative distribution function. The second relation is a consequence of Chebyshev’s inequality (for details see the supplementary materials). This ends the proof. □

Proof

Proof of Lemma 3.10

Recall that we want to show

$$ P\left( \bigcap_{i \in S}\{\hat b_i \neq 0 \}\right) \rightarrow 1. $$

We can bound the considered probability in the following way:

$$ P\left( \bigcap_{i \in S}\{\hat b_i \neq 0 \}\right) \!= \sum\limits_{r = 1}^p P\left( \bigcap_{i \in S}(\{\hat b_i \neq 0 \}\cap \{R = r\})\right) \!= \sum\limits_{r = 1}^p P\left( \bigcap_{i \in S}(\{|T_i|> \lambda_r \}\cap \{R = r\})\right) \geqslant $$

$$ \geqslant \sum\limits_{r = 1}^p P\left( \bigcap_{i \in S}(\{|T_i|> \lambda_1 \}\cap \{R = r\})\right) = P\left( \bigcap_{i \in S}\{|T_i|> \lambda_1 \} \right) , $$

where in the first and the last equation we used the law of total probability ($\{R = r\}_{r = 1}^p$ is a partition of a sample space). The second equation is a consequence of Theorem 3.2 and the inequality comes from the fact that λ is a decreasing sequence. Now, observe that:

$$ P\left( \bigcap_{i \in S}\{|T_i|> \lambda_1 \} \right) \geqslant P\left( \bigcap_{i \in S}\{|T_i|> \lambda_1 \} \cap Q_2 \right) $$

and recall that T_i = M_i + Γ_i and $Q_2 = \{\max \limits _i |{\Gamma }_i|\leqslant \gamma \}$. Therefore, due to the triangle inequality we have:

$$ P\left( \bigcap_{i \in S}\{|T_i|> \lambda_1 \} \cap Q_2 \right) \geqslant P\left( \bigcap_{i \in S}\{|M_i|> \lambda_1 + \gamma \} \cap Q_2 \right) $$

Now, because $P(Q_2) \rightarrow 1$, we only have to show that

$$ P\left( \bigcap_{i \in S}\{|M_i|> \lambda_1 + \gamma \} \right) \rightarrow 1 . $$

Let us consider the properties of $M_i = X_i^{\prime }\epsilon +b^0_i$. Notice that due to the symmetry of $X_i^{\prime }\epsilon$ distribution, we have $|X_i^{\prime }\epsilon +b^0_i| =^D |X_i^{\prime }\epsilon +|b^0_i||$, therefore

$$ P\left( \bigcap_{i \in S}\{|M_i|> \lambda_1 + \gamma \} \right) = P\left( \bigcap_{i \in S}\{|X_i^{\prime}\epsilon +|b^0_i||> \lambda_1 + \gamma \} \right) \geqslant P\left( \bigcap_{i \in S}\{X_i^{\prime}\epsilon > \lambda_1 + \gamma -|b^0_i| \} \right) , $$

where in the last inequality we omit the absolute value in $|X_i^{\prime }\epsilon +|b^0_i||$ and subtract $|b^0_i|$. Now, due to the assumption on the signal strength, we have for a large enough n:

$$ P\left( \bigcap_{i \in S}\{X_i^{\prime}\epsilon > \lambda_1 + \gamma -|b^0_i| \} \right) \geqslant P\left( \bigcap_{i \in S}\{X_i^{\prime}\epsilon > -\sigma(1+\delta/2)\sqrt{2 \log p} \} \right) . $$

This is a consequence of the fact that for a large enough n:

$$ \lambda_1 + \gamma \leqslant\sigma(1+1.5\delta)\sqrt{2 \log p} . $$

Moreover, we know that conditionally on 𝜖, $X^{\prime }_i\epsilon$ are independent from the normal distribution $N(0, \Vert \epsilon \Vert _2^2/n)$. Therefore we have:

$$ P\left( \bigcap_{i \in S}\{X_i^{\prime}\epsilon > -\sigma(1+\delta/2)\sqrt{2 \log p} \} \right) \geqslant P\left( \bigcap_{i \in S}\{X_i^{\prime}\epsilon > -\sigma(1+\delta/2)\sqrt{2 \log p} \}\cap Q_3 \right) = $$

$$ = P\left( \bigcap_{i \in S}\{\sqrt{n}\frac{X_i^{\prime}\epsilon}{\Vert \epsilon \Vert_2} > -\sigma\sqrt{n}\frac{(1+\delta/2)\sqrt{2 \log p}}{\Vert \epsilon \Vert_2} \}\cap Q_3 \right)\geqslant $$

$$ \geqslant P\left( \bigcap_{i \in S}\{\sqrt{n}\frac{X_i^{\prime}\epsilon}{\Vert \epsilon \Vert_2} > -\frac{(1+\delta/2)}{(1+1/k^{*})}\sqrt{2 \log p} \}\cap Q_3 \right) , $$

where in the last inequality we have used the condition defining Q₃. Again, due to the fact that $P(Q_3) \rightarrow 1$, we only have to show that

$$ P\left( \bigcap_{i \in S}\{\sqrt{n}\frac{X_i^{\prime}\epsilon}{\Vert \epsilon \Vert_2} > -a\sqrt{2 \log p} \} \right) \rightarrow 1 , $$

where $a = \frac {(1+\delta /2)}{(1+1/k^{*})}$. Now, using the fact that conditionally on 𝜖, $X^{\prime }_i\epsilon$ are independent random variables from the normal distribution $N(0, \Vert \epsilon \Vert _2^2/n)$, we obtain

$$ P\left( \bigcap_{i \in S}\{\sqrt{n}\frac{X_i^{\prime}\epsilon}{\Vert \epsilon \Vert_2} > -a\sqrt{2 \log p} \} \right) = E\left( P\left( \bigcap_{i \in S}\{\sqrt{n}\frac{X_i^{\prime}\epsilon}{\Vert \epsilon \Vert_2} > -a\sqrt{2 \log p} \}| \epsilon \right) \right) = $$

$$ =E\left( {\prod}_{i \in S}P\left( \{\sqrt{n}\frac{X_i^{\prime}\epsilon}{\Vert \epsilon \Vert_2} > -a\sqrt{2 \log p} \}| \epsilon \right) \right) = E\left[{\prod}_{i \in S}\left( 1-{\Phi} \left( -a\sqrt{2 \log p} \right) \right) \right] = $$

$$ =\left[1-\left( 1 -{\Phi} \left( a\sqrt{2 \log p} \right) \right)\right]^k \geqslant \left[1-exp\left( -0.5 (a\sqrt{2 \log p})^2\right) \right]^k = \left[1-\left( \frac{1}{p}\right)^{a^2} \right]^k \geqslant $$

$$ \geqslant 1 - k \left( \frac{1}{p}\right)^{a^2} \rightarrow 1 , $$

where in the above inequalities we used the bound (5.15) and Bernoulli’s inequality. The convergence is a consequence of the fact that for a large enough n, $a \geqslant 1$ and the assumption that $k/p \rightarrow 0$. This ends the proof of ${\Pi } \rightarrow 1$. □