Abstract
Let \( X_{1},\ldots ,X_{n} \) be i.i.d. sample in \( \mathbb {R}^{p} \) with zero mean and the covariance matrix \( \varvec{\Sigma }\). The problem of recovering the projector onto an eigenspace of \( \varvec{\Sigma }\) from these observations naturally arises in many applications. Recent technique from Koltchinskii and Lounici (Ann Stat 45(1):121–157, 2017) helps to study the asymptotic distribution of the distance in the Frobenius norm \( \left\| \mathbf {P}_{r} - \widehat{\mathbf {P}}_{r} \right\| _{2} \) between the true projector \( \mathbf {P}_{r} \) on the subspace of the rth eigenvalue and its empirical counterpart \( \widehat{\mathbf {P}}_{r} \) in terms of the effective rank of \( \varvec{\Sigma }\). This paper offers a bootstrap procedure for building sharp confidence sets for the true projector \( \mathbf {P}_{r} \) from the given data. This procedure does not rely on the asymptotic distribution of \( \left\| \mathbf {P}_{r} - \widehat{\mathbf {P}}_{r} \right\| _{2} \) and its moments. It could be applied for small or moderate sample size n and large dimension p. The main result states the validity of the proposed procedure for Gaussian samples with an explicit error bound for the error of bootstrap approximation. This bound involves some new sharp results on Gaussian comparison and Gaussian anti-concentration in high-dimensional spaces. Numeric results confirm a good performance of the method in realistic examples.
Similar content being viewed by others
References
Adamczak, R., Litvak, A., Pajor, A., Tomczak-Jaegermann, N.: Restricted isometry property of matrices with independent columns and neighborly polytopes by random sampling. Constr. Approx. 34(1), 61–88 (2011)
Bengio, I., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)
Bentkus, V.: A Lyapunov-type bound in \(R^d\). Probab. Theory Appl. 49(2), 311–323 (2005)
Chernozhukov, V., Chetverikov, D., Kato, K.: Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors. Ann. Stat. 41(6), 2786–2819 (2013)
Chernozhukov, V., Chetverikov, D., Kato, K.: Comparison and anti-concentration bounds for maxima of Gaussian random vectors. Probab. Theory Relat. Fields 162(1–2), 47–70 (2015)
Chernozhukov, V., Chetverikov, D., Kato, K.: Central limit theorems and bootstrap in high dimensions. Ann. Probab. 45(4), 2309–2352 (2017)
Chung, K.: A Course in Probability Theory, 3rd edn. Academic Press Inc., San Diego (2001)
Götze, F., Ulyanov, V.: Uniform approximation in the CLT for balls in Euclidean spaces. Preprint 00-0034, SFB 343, Univ. Bielefeld (2000)
Götze F, Naumov A, Spokoiny V, Ulyanov V (2019) Large ball probabilities, Gaussian comparison and anti-concentration. Bernoulli 25. arXiv:1708.08663v2
Holtz, M.: Sparse Grid Quadrature in High Dimensions with Applications in Finance and Insurance, Lecture Notes in Computational Science and Engineering, vol. 77. Springer, Berlin (2010)
Koltchinskii, V., Lounici, K.: Concentration inequalities and moment bounds for sample covariance operators. Bernoulli 23(1), 110–133 (2017)
Koltchinskii, V., Lounici, K.: Normal approximation and concentration of spectral projectors of sample covariance. Ann. Stat. 45(1), 121–157 (2017)
Li, Y., Hsing, T.: Deciding the dimension of effective dimension reduction space for functional and high-dimensional data. Ann. Stat. 38(5), 3028–3062 (2010)
Marčenko, V., Pastur, L.: Distribution of eigenvalues in certain sets of random matrices. Mat. Sb. (N.S.) 72(114), 507–536 (1967)
Markus, A.: The eigen- and singular values of the sum and product of linear operators. Russ. Math. Surv. 19(4), 91–120 (1964)
Prokhorov, Y., Ulyanov, V.: Some approximation problems in statistics and probability. In: Limit Theorems in Probability, Statistics and Number Theory, Springer Proc. Math. Stat., vol. 42, pp. 235–249. Springer, Heidelberg (2013)
Rosenthal, H.: On the subspaces of \(L^{p}\) \((p>2)\) spanned by sequences of independent random variables. Isr. J. Math. 8, 273–303 (1970)
Shao, J.: Mathematical Statistics. Springer, Berlin (2003)
Spokoiny, V., Zhilova, M.: Bootstrap confidence sets under model misspecification. Ann. Stat. 43(6), 2653–2675 (2015)
Tropp, J.: User-friendly tail bounds for sums of random matrices. Found. Comput. Math. 12(4), 389–434 (2012)
Tsybakov, A.: Introduction to Nonparametric Estimation. Springer, New York (2008)
van Handel, R.: Structured random matrices. In: Convexity and Concentration, vol. 161, pp. 107–165, IMA. Springer (2017)
Vershynin, R.: Introduction to the non-asymptotic analysis of random matrices. In: Eldar, Y.C., Kutyniok, G. (eds.) Compressed Sensing, pp. 210–268. Cambridge University Press, Cambridge (2012)
Wang, X., Sloan, I.: Why are high-dimensional finance problems often of low effective dimension? SIAM J. Sci. Comput. 27(1), 159–183 (2005)
Acknowledgements
The authors are grateful to the Associate Editor and the Reviewers for the careful reading of the manuscript and pertinent comments. Their constructive feedback helped to improve the quality of this work and shape its final form. This work has been funded by the Russian Academic Excellence Project ‘5-100’. Results of Section 5 have been obtained under support of the RSF Grant No. 18-11-00132.
Author information
Authors and Affiliations
Corresponding author
Auxiliary results
Auxiliary results
1.1 Concentration inequalities for sample covariances and spectral projectors in \(\mathbf {X}\)-world
In this section we present concentration inequalities for sample covariance matrices and spectral projectors in \(\mathbf {X}\)-world.
Theorem 6
Let \(X, X_{1}, \ldots , X_{n}\) be i.i.d. centered Gaussian random vectors in \(\mathbb {R}^{p}\) with covariance \(\varvec{\Sigma }= {{\mathrm{\mathbb {E}}}}(X X^{\mathsf {T}})\). Then
Moreover, for all \(t \ge 1\) with probability \(1 - e^{-t}\)
Proof
See [11, Theorem 6, Corollary 2]. \(\square \)
To deal with spectral projectors we need the following result which was proved in [12]. Let us introduce additional notations. We denote by \(\widetilde{\varvec{\Sigma }}\) an arbitrary perturbation of \(\varvec{\Sigma }\) and \(\widetilde{\mathbf {E}}{\mathop {=}\limits ^{{\textsf {def}}}}\widetilde{\varvec{\Sigma }}- \varvec{\Sigma }\). Recall that
Lemma 4
Let \(\widetilde{\varvec{\Sigma }}\) be an arbitrary perturbation of \(\varvec{\Sigma }\) and let \( \widetilde{\mathbf {P}}_{r} \) be the corresponding projector. The following bound holds:
Moreover, \(\widetilde{\mathbf {P}}_{r} - \mathbf {P}_{r} = L_{r}(\widetilde{\mathbf {E}}) + S_{r}(\widetilde{\mathbf {E}})\), where \(L_{r}(\widetilde{\mathbf {E}}) {\mathop {=}\limits ^{{\textsf {def}}}}\mathbf {C}_{r} \widetilde{\mathbf {E}}\mathbf {P}_{r} + \mathbf {P}_{r} \widetilde{\mathbf {E}}\mathbf {C}_{r}\) and
Proof
See [12, Lemma 1]. \(\square \)
Theorem 7
(Concentration results in \(\mathbf {X}\)-world) Assume that the conditions of Theorem 1 hold. Then for all \(t: 1 \le t \le n^{1/4}\) and
the following bound holds with probability at least \(1 - e^{-t}\)
Proof
The proof follows from [12, Theorems 3, 5]. \(\square \)
1.2 Concentration inequalities for sums of random variables and random matrices
In what follows for a vector \(a = (a_{1}, \ldots , a_{n})\) we denote \(\Vert a\Vert _{s} {\mathop {=}\limits ^{{\textsf {def}}}}\big (\sum _{k=1}^{n} |a_{k}|^{s}\big )^{1/s}\). For a random variable X and \(r > 0\) we define the \(\psi _{r}\)-norm by
If a random variable X is such that for any \(p \ge 1, {{\mathrm{\mathbb {E}}}}^{1/p} |X|^{p} \le p^{1/r} K\), for some \(K > 0\), then \(\Vert X\Vert _{\psi _{r}} \le c K\) where \(c > 0\) is a numerical constant.
Lemma 5
Let \(X, X_{i}, i = 1, \ldots , n\) be i.i.d. random variables with \({{\mathrm{\mathbb {E}}}}X = 0\) and \(\Vert X\Vert _{\psi _{r}} \le 1, 1 \le r \le 2\). Then there exists some absolute constant \(C>0\) such that for all \(p \ge 1\)
where \(a = (a_{1}, \ldots , a_{n})\) and \(1/r + 1/r_{*} = 1\).
Proof
See [1, Lemma 3.6]. \(\square \)
Lemma 6
If \(0< s < 1\) and \(X_{1},\ldots , X_{n}\) are independent random variables satisfying \(\Vert X\Vert _{\psi _{s}} \le 1\), then for all \(a = (a_{1}, \ldots , a_{n}) \in \mathbb {R}^{n}\) and \(p \ge 2\)
Moreover, for \(s \ge 1/2\), \(C_{s}\) is bounded by some absolute constant.
Proof
See [1, Lemma 3.7]. \(\square \)
Lemma 7
Let \(\eta _{1}, \ldots , \eta _{n}\) be i.i.d. standard normal random variables. For all \(t\ge 1\)
Moreover, if \(\overline{\eta }_{1}, \ldots , \overline{\eta }_{n}\) are i.i.d. standard normal random variables and independent of \(\eta _{1}, \ldots , \eta _{n}\) then
Proof
We prove (48) only. The proof of (47) is similar. Let \(\epsilon _{i}, i = 1, \ldots , n\), be i.i.d. Rademacher r.v. Denote \(\xi _{i} {\mathop {=}\limits ^{{\textsf {def}}}}\eta _{i}^{2} \overline{\eta }_{i}^{2} - 1, i = 1, \ldots , n\). Applying Lemma 6 with \(s = 1/2\) we write
From Markov’s inequality
Taking \(p = t/(Ce)^{1/2}\) we finish the proof of the lemma. \(\square \)
Lemma 8
(Matrix Gaussian series) Consider a finite sequence \(\{\mathbf {A}_{k}\}\) of fixed, self-adjoint matrices with dimension d, and let \(\{\xi _{k}\}\) be a finite sequence of independent standard normal random variables. Compute the variance parameter
Then, for all \(t \ge 0\),
Proof
See in [20, Theorem 4.1]. \(\square \)
Lemma 9
(Matrix Bernstein inequality) Consider a finite sequence \({\mathbf {X}_{k}}\) of independent, random, self-adjoint matrices with dimension d. Assume that \({{\mathrm{\mathbb {E}}}}\mathbf {X}_{k} = 0\) and \(\lambda _{\max }(\mathbf {X}_{k}) \le R\) almost surely. Compute the norm of the total variance,
Then the following inequalities hold for all \(t \ge 0\):
Moreover, if \({{\mathrm{\mathbb {E}}}}\mathbf {X}_{k} = 0\) and \({{\mathrm{\mathbb {E}}}}\mathbf {X}_{k}^{p} \preceq \frac{p!}{2} R^{p-2} \mathbf {A}_{k}^{2}\) then the following inequalities hold for all \(t \ge 0\):
where
Proof
See in [20, Theorem 6.1]. \(\square \)
1.3 Auxiliary lemma
Lemma 10
Assume that \(Z_1, Z_2\) be i.i.d. and \({{\mathrm{\mathcal {N}}}}(0,1)\). Let \( \lambda _1, \lambda _2 \) be any positive numbers and \(b \ne 0\). There exists an absolute constant c such that
Proof
Denote the l.h.s. of (49) by \(I'\). Using Euler’s formula for complex exponential function we get for positive g and any \(d \in \mathbb {R}\)
Hence, by (34) we get
where \(\phi _{k} {\mathop {=}\limits ^{{\textsf {def}}}}\phi _{k}(t) {\mathop {=}\limits ^{{\textsf {def}}}}\arcsin \big (2 \lambda _{k} t/(1 + 4 t^{2} \lambda _{k}^{2})^{\frac{1}{2}}\big )\). Since \(\prod _{k=1}^{2}\left( 1+4t^{2}\lambda ^{2}_{k}\right) ^{-1/4}\) is even function and \(\phi _{k}(t), k = 1,2\), is odd function of t, we may rewrite \(I'\) as follows
We note that
Hence, to prove (49) it is enough to show that
We may rewrite \(I''\) as follows
where
The bound \(I_{1}'' \le c\) is true since for any positive A and B we have
To estimate \(I_{2}''\) we shall use the following inequalities
Applying these inequalities we get that
where \(c'\) is some absolute constant. Hence,
The estimates for \(I_3''\) and \(I_4''\) are similar. For simplicity we estimate \(I_3''\) only. Applying the following inequality
we obtain that
where \(c''\) is some absolute constant. \(\square \)
Rights and permissions
About this article
Cite this article
Naumov, A., Spokoiny, V. & Ulyanov, V. Bootstrap confidence sets for spectral projectors of sample covariance. Probab. Theory Relat. Fields 174, 1091–1132 (2019). https://doi.org/10.1007/s00440-018-0877-2
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00440-018-0877-2
Keywords
- Sample covariance matrices
- Spectral projectors
- Multiplier bootstrap
- Gaussian comparison and anti-concentration inequalities
- Effective rank