Comments on `High-dimensional simultaneous inference with the bootstrap'

We provide some comments on the article `High-dimensional simultaneous inference with the bootstrap' by Ruben Dezeure, Peter Buhlmann and Cun-Hui Zhang.


Introduction
We congratulate the authors on their stimulating contribution to the burgeoning highdimensional inference literature. The bootstrap offers such an attractive methodology in these settings, but it is well-known that its naive application in the context of shrinkage/superefficiency is fraught with danger (e.g. Samworth, 2003;Chatterjee and Lahiri, 2011). The authors show how these perils can be elegantly sidestepped by working with de-biased, or de-sparsified, versions of estimators.
In this discussion, we consider alternative approaches to individual and simultaneous inference in high-dimensional linear models, and retain the notation of the paper.

Why penalise coefficients of variables of interest?
Suppose that for some, presumably small, set G ⊆ {1, . . . , p}, we want a confidence set for β 0 G . Much of the recent literature, including the paper under discussion, proceeds by constructing an initial estimator, such as the Lasso estimatorβ, and then attempting to debias it. Our starting point is the following provocative question: since we know in advance the set of variables we are interested in, why would we want to penalise these coefficients in the first place? Of course, it is standard practice not to penalise the intercept term in highdimensional linear models, to preserve location equivariance, but we now consider taking this one stage further. More precisely, consider the linear model where the columns of X have Euclidean length n 1/2 , where X T G X G is positive definite, and where, for simplicity, we assume that ǫ ∼ N n (0, σ 2 I). We further assume that the set S := {j : β 0 j = 0} of signal variables has cardinality s, and let N := {1, . . . , p} \ S. For λ > 0, let where we emphasise that β G 1 is unpenalised. For fixed β −G ∈ R p−|G| , the solution in the first argument is given by ordinary least squares: . We therefore find that denotes the matrix representing an orthogonal projection onto the column space of X G . In other words,β −G is simply the Lasso solution with response and design matrix pre-multiplied by (I − P G ). Moreover, . For our theoretical analysis ofβ G , we will require the following compatibility condition: The theorem below is only a small modification of existing results in the literature (e.g. Bickel, Ritov and Tsybakov, 2009), but for completeness we provide a proof in the Appendix.
Theorem 1. Assume (A1), and let λ := Aσ log p n . Then with probability at least 1 − Theorem 1 allows us to show that if, in addition to (A1), the columns of X G and those of X −G satisfy a strong lack of correlation condition, thenβ G can be used for asymptotically valid inference for β G . To formalise this latter condition, it is convenient to let Θ denote the Consider an asymptotic framework in which s = s n ≥ 1 and p = p n → ∞ as n → ∞, but σ 2 > 0 and G are constant. Assume (A1) holds for sufficiently large n (with φ 0 not depending on n), and also that Θ ∞ = o(s −1 log −1/2 p). If we choose λ := Aσ log p n in the above procedure with constant A > 2 √ 2, then Proof. We can write Moreover, from the proof of Theorem 1, on Ω 0 := X T −G (I − P G )ǫ ∞ /n ≤ λ/2}, Since P(Ω 0 ) → 1, the conclusion follows.
We remark that for j ∈ G c , Θ j is the coefficient in the ordinary least squares regression of X j on X G . Even though the condition on Θ ∞ is strong, it may well be reasonable to suppose that, having pre-specified the index set G of variables that we are interested in, we should avoid including in our model other variables that have significant correlation with X G .

More complicated settings
Without this strong orthogonality condition we might instead consider adjustingβ G by debiasing or de-sparsifyingβ −G . Following van de Geer et al. (2014) for some matrix M ∈ R (p−|G|)×(p−|G|) . This yields the debiased estimator where R is the |G| × (p − |G|) matrix given by Under our Gaussian errors assumption, (X T G X G ) −1 X T G ǫ and n −1 ΘMX T −G (I − P G )ǫ are independent centred Gaussian random vectors; thus if the remainder term R(β −G − β 0 −G ) is of smaller order, we see that our estimateb G is approximately centred Gaussian. The techniques of van de Geer et al. (2014) or Javanmard and Montanari (2014) might then be used to give asymptotic justifications for Gaussian confidence sets and hypothesis tests concerning β 0 G . But another very interesting direction would be to adapt the bootstrap approaches proposed in the current paper to the estimateb G . As in van de Geer et al. (2014) we should choose M depending on X to control Note that we may write the matrix R in terms of the sample covariance matrix of the covariatesΣ := X T X/n (using obvious notation for the partitioning) as can be thought of as an approximation to (Σ −1 ) −G,−G (even thoughΣ is not invertible when p > n). In general, we might use concentration inequalities for entries inΣ to control R ∞ ; if we think of |G| as small, then we only have O(p) entries to control, rather than O(p 2 ) as is more typical in these debiasing problems. We hope to pursue these ideas elsewhere.