Covariance matrix testing in high dimension using random projections

Abstract

Estimation and hypothesis tests for the covariance matrix in high dimensions is a challenging problem as the traditional multivariate asymptotic theory is no longer valid. When the dimension is larger than or increasing with the sample size, standard likelihood based tests for the covariance matrix have poor performance. Existing high dimensional tests are either computationally expensive or have very weak control of type I error. In this paper, we propose a test procedure, CRAMP (covariance testing using random matrix projections), for testing hypotheses involving one or more covariance matrices using random projections. Projecting the high dimensional data randomly into lower dimensional subspaces alleviates of the curse of dimensionality, allowing for the use of traditional multivariate tests. An extensive simulation study is performed to compare CRAMP against asymptotics-based high dimensional test procedures. An application of the proposed method to two gene expression data sets is presented.

Appendix

Proof of Theorem 1

The proof of Theorem 1 is along the same lines as the proof of Theorem 2 in Srivastava et al. (2014). To show that the distribution of $\overline{\pi }_U$ is independent of $\sigma$, define $\mathbf {X}^*_{m; i} = \mathcal {R}_m \mathbf {X}_i, i = 1, \ldots , n, m = 1, \ldots , M$ as the projection of the $i^\mathrm{th}$ observation using the $m^\mathrm{th}$ random projection matrix. Then we have

$$\begin{aligned} \mathrm{var} \left( \mathbf {X}_{m;1}, \ldots , \mathbf {X}_{m; n} \right) = \mathcal {S}^*_m = \mathcal {R}_m \mathcal {S} \mathcal {R}_m^{\top }, \end{aligned}$$

where $\mathcal {S}$ and $\mathcal {S}_m^*$ are the sample covariance matrices of the original and projected observations respectively. From equation (17), the p-values based on M i.i.d. random projection matrices are

$$\begin{aligned} \pi _m = 1 - \chi ^2_{\nu } \left( \frac{1}{k} \mathrm{tr}\left\{ \frac{\mathcal {S}_m^*}{ \mathrm{tr}\mathcal {S}_m^* / k} - \mathcal {I}_k \right\} ^2 \right) . \end{aligned}$$

Firstly since the random matrices are independent, conditional on the data $\mathcal {X} = \{\mathbf {X}_1, \ldots , \mathbf {X}_n\}$ and $\mathcal {Y} = \{\mathbf {Y}_1, \ldots , \mathbf {Y}_m \}$, the p-values $\pi _1, \ldots , \pi _M$ are independent and identically distributed. This is because of the orthogonality of the projection matrices which preserves the covariance matrix structure ($\mathcal {R} \left( \sigma ^2 \mathcal {I}_p \right) \mathcal {R}^{\top } = \sigma ^2 \mathcal {I}_k$). Additionally, we can write

$$\begin{aligned} P \left[ \overline{\pi }< u \right] = \mathbb {E}_{\mathcal {X}, \mathcal {Y}} \left\{ P_{\mathcal {R}} \left[ \overline{\pi } < u | \mathcal {X}, \mathcal {Y} \right] \right\} , \end{aligned}$$

(A.20)

where the expected value is with respect to the distribution of the observations and the probability is with respect to the randomness of the projection matrix.

By the conditional independence of $\pi _1, \ldots , \pi _M$ and the central limit theorem, we have a normal approximation to the probability in (A.20)

$$\begin{aligned} \lim \limits _{M \rightarrow \infty } \left| P \left[ \overline{\pi } < u \right] - \Phi \left( \frac{u - \mathbb {E}_{\mathcal {R}} \left[ u | \mathcal {X}, \mathcal {Y} \right] }{ \mathrm{var}_{\mathcal {R}} \left[ u | \mathcal {X}, \mathcal {Y} \right] } \right) \right| = 0. \end{aligned}$$

(A.21)

Hence the probability $P \left[ \overline{\pi } < u \right]$ can be approximated only using the moments of $U | \mathcal {X}, \mathcal {Y}$. Under the null hypothesis $H_{0S}$, the variable $U | \mathcal {X}, \mathcal {Y}$ is defined as

$$\begin{aligned} U | \mathcal {X}, \mathcal {Y}&= 1 - \chi ^2_{\nu } \left( U | \mathcal {X}, \mathcal {Y} \right) = 1 - F_{\chi ^2_{\nu }} \left( \mathrm{tr}\left\{ \frac{\mathcal {S}_m^*}{ \mathrm{tr}\mathcal {S}_m^* / k} - \mathcal {I}_k \right\} ^2 | \mathcal {X}, \mathcal {Y} \right) \nonumber \\&\sim \mathrm{Unif}(0,1). \end{aligned}$$

(A.22)

The uniform distribution is from the standard property of p-value under the null hypothesis, which is independent of $\sigma ^2$. Using this property, we shall show that the distribution of $E_{\mathcal {R}} \left[ U | \mathcal {X}, \mathcal {Y} \right]$ and $\mathrm{var}_{\mathcal {R}} \left[ U | \mathcal {X}, \mathcal {Y} \right]$ with respect to $\mathcal {X}, \mathcal {Y}$ are also independent of $\sigma ^2$.

Let W denote the expected value of $U | \mathcal {X}, \mathcal {Y}$ with respect to $\mathcal {R}$,

$$\begin{aligned} W = \mathbb {E}_{\mathcal {R}} \left[ U | \mathcal {X}, \mathcal {Y} \right]&= \int u \, dP_{\mathcal {R}} \nonumber \\&= \int \left[ 1 - F_{\chi ^2_{\nu }} \left( \mathrm{tr}\left\{ \frac{\mathcal {S}_m^*}{ \mathrm{tr}\mathcal {S}_m^* / k} - \mathcal {I}_k \right\} ^2 | \mathcal {X}, \mathcal {Y} \right) \right] \, dP_{\mathcal {R}} \end{aligned}$$

(A.23)

where the integral is with respect to the distribution of the random projection matrix $\mathcal {R}$. While the exact integral is not of importance, it should be noted that from equation (A.22), the integrand is independent of $\sigma ^2$. As the random projection matrices are generated independent of the distribution of the observations, we can conclude that the variable W is independent of $\sigma ^2$. For any $m \ge 1$, the $\mathrm{m}\mathrm{th}$ moment of W is given by

$$\begin{aligned} \mathbb {E}_{\mathcal {X}, \mathcal {Y}} \left[ W^m \right] = \int W^m \, dF_{\mathcal {X}, \mathcal {Y}}&= \int \mathbb {E}_{\mathcal {R}} \left[ U| \mathcal {X}, \mathcal {Y} \right] ^m \, dF_{\mathcal {X}, \mathcal {Y}} \\&= \int \mathbb {E}_{\mathcal {R}} \left[ U| \mathcal {X}, \mathcal {Y} \right] \times \cdots \times \mathbb {E}_{\mathcal {R}} \left[ U| \mathcal {X}, \mathcal {Y} \right] \, dF_{\mathcal {X}, \mathcal {Y}} \\&= \int \left\{ \int U_{\mathcal {R}_1} \, dP_{\mathcal {R}_1} \right\} \cdots \left\{ \int U_{\mathcal {R}_m} \, dP_{\mathcal {R}_m} \right\} \, dF_{\mathcal {X}, \mathcal {Y}} \end{aligned}$$

Interchanging the integrals by Fubini’s theorem, we have

$$\begin{aligned} \mathbb {E}_{\mathcal {X}, \mathcal {Y}} \left[ W^m \right] = \int \cdots \int \left\{ \int U_{\mathcal {R}_1} \ldots U_{\mathcal {R}_m} \, dF_{\mathcal {X}, \mathcal {Y}} \right\} \, dP_{\mathcal {R}_1} \cdots dP_{\mathcal {R}_m} \end{aligned}$$

(A.24)

By the construction of U in equation (A.22), the integral $\left\{ \int U_{\mathcal {R}_1} \ldots U_{\mathcal {R}_m} \, dF_{\mathcal {X}, \mathcal {Y}} \right\}$ is independent of $\sigma ^2$. Therefore, all moments of W are independent of $\sigma ^2$ which implies that the distribution of W is independent of $\sigma ^2$.

Similarly, it can be shown that the distribution of $\mathrm{var}_{\mathcal {R}} \left( U| \mathcal {X}, \mathcal {Y} \right)$ is also independent of $\sigma ^2$. From the independence of the mean and variance, we have the distributions of

$$\begin{aligned} \Phi \left[ \frac{ u - \mathbb {E}_{\mathcal {R}} \left\{ U | \mathcal {X}, \mathcal {Y} \right\} }{ \mathrm{var}_{\mathcal {R}} \left\{ U | \mathcal {X}, \mathcal {Y} \right\} } \right] \text{ and } \quad \mathbb {E}_{\mathcal {X}, \mathcal {Y}} \left\{ \Phi \left[ \frac{ u - \mathbb {E}_{\mathcal {R}} \left\{ U | \mathcal {X}, \mathcal {Y} \right\} }{ \mathrm{var}_{\mathcal {R}} \left\{ U | \mathcal {X}, \mathcal {Y} \right\} } \right] \right\} \end{aligned}$$

(A.25)

are independent of $\sigma ^2$. Finally, combining this independence with equation (A.21), we have

$$\begin{aligned}&\lim \limits _{M \rightarrow \infty } P_\mathcal {R} \left\{ \overline{\pi } | \mathcal {X}, \mathcal {Y} \right\} = \Phi \left[ \frac{ u - \mathbb {E}_{\mathcal {R}} \left\{ U | \mathcal {X}, \mathcal {Y} \right\} }{ \mathrm{var}_{\mathcal {R}} \left\{ U | \mathcal {X}, \mathcal {Y} \right\} } \right] , \end{aligned}$$

with the right hand side independent of $\sigma ^2$. Taking expected values with respect to $\mathcal {X}$ and $\mathcal {Y}$, we have

$$\begin{aligned} \lim \limits _{M \rightarrow \infty } P \left[ \overline{\pi } < u \right] = \mathbb {E}_{\mathcal {X}, \mathcal {Y}} \left\{ \Phi \left[ \frac{ u - \mathbb {E}_{\mathcal {R}} \left\{ U | \mathcal {X}, \mathcal {Y} \right\} }{ \mathrm{var}_{\mathcal {R}} \left\{ U | \mathcal {X}, \mathcal {Y} \right\} } \right] \right\} . \end{aligned}$$

(A.26)

By equation (A.25), the right hand side in (A.26) is also independent of $\sigma ^2$, completing the proof. $\square$

Proof of Theorem 2

Invariance of the distribution of the two-sample test statistic can be shown similar to the above proof. Besides computation of the test statistic, rest of the argument remains the same since the Box M test statistic also follows a standard uniform distribution under the null hypothesis. Hence in Algorithm 2, $\pi _m \sim \mathrm{Unif}(0, 1)$ under $H_0$, which is independent of the choice of $\Sigma$. $\square$