New multiple testing method under no dependency assumption, with application to multiple comparisons problem

Abstract

Traditional multiple hypotheses testing mainly focuses on constructing stepwise procedures under some error rate control, such as familywise error rate (FWER), false discovery rate, and so forth. However, most of these procedures are obtained in independent case, and when there exists correlation across tests, the dependency may increase or decrease the chance of false rejections. In this paper, a totally different testing method is proposed, which doesn’t focus on specific error control, but pays attention to the overall performance of the collection of hypotheses and the structure utilization among hypotheses. Since the main purpose of multiple testing is to pick out the false ones from the whole hypotheses and present a rejection set, motivated by the principle of simple hypothesis testing, we give the final testing result based on the estimation of the set of all the true null hypotheses. Our method can be applied in any dependent case provided that a reasonable $p$-value can be obtained for each intersection hypothesis. We illustrate the new procedures with application to multiple comparisons problems. Theoretical results show the consistency of our method, and investigate their FWER behavior. Simulation results suggest that our procedures have a better overall performance than some existing procedures in dependent cases, especially in the total number of type I and type II errors.

Appendix

Proof of Lemma 1

(i) Since for $J \subseteq I_0$, $p(J)\sim U(0,1)$, the result is obtained immediately.

(ii) $\forall J\nsubseteq I_0$,

case 1. $\sigma ^2$ is known

Denote

$$\begin{aligned} Y_n=\frac{n}{\sigma ^2}{\bar{\mathbf {X}}}^T{L_J}^T\left( L_J{L_J}^T\right) ^{-1}L_J\bar{\mathbf {X}}, \text {and} \Sigma =\left( L_J{L_J}^T\right) ^{-1}=RR^T, \end{aligned}$$

where $R$ is positive definite. Then

$$\begin{aligned} Y_n&= \frac{n}{\sigma ^2}\left( L_J\bar{\mathbf {X}}-L_J\varvec{\theta }\right) ^T\Sigma \left( L_J\bar{\mathbf {X}}-L_J\varvec{\theta }\right) +\frac{2n}{\sigma ^2}\left( L_J\varvec{\theta }\right) ^T\Sigma \left( L_J\bar{\mathbf {X}}-L_J\varvec{\theta }\right) \\&+\frac{n}{\sigma ^2}\left( L_J\varvec{\theta }\right) ^T\Sigma \left( L_J\varvec{\theta }\right) \\&\triangleq Y_{1n}+Y_{2n}+Y_{3n}. \end{aligned}$$

Notice that

$$\begin{aligned} Y_{1n}\sim \chi ^2_{r_J}, \frac{\sqrt{n}}{\sigma }R^T\left( L_J\bar{\mathbf {X}}-L_J\varvec{\theta }\right) \sim N(\mathbf {0},\mathbf {I}_{r_J}), \end{aligned}$$

then $Y_{1n}=O_p(1)$, $Y_{2n}=O_p(\sqrt{n})$. Additionally, $Y_{3n}\asymp n$, so $Y_n\asymp _p n$, that is to say, $Y_n$ is of the same order as $n$ in probability.

Since $\chi ^{-2}_m(1-\alpha (n))=o(n)$, $\chi ^{-2}_{r_J}(1-\alpha (n))=o(n)$. So

$$\begin{aligned} \lim _{n\rightarrow \infty }\mathrm{Pr}\left( Y_n\ge \chi ^{-2}_{r_J}(1-\alpha (n)) \right) = 1, \text {i.e.} \lim _{n\rightarrow \infty }\mathrm{Pr}\left( p(L_J)\le \alpha \right) = 1. \end{aligned}$$

case 2. $\sigma ^2$ is unknown

Similarly as before, let

$$\begin{aligned} W_n=\frac{n{\bar{\mathbf {X}}}^T{L_J}^T\left( L_J{L_J}^T\right) ^{-1}L_J\bar{\mathbf {X}}}{r_JS^2}=\frac{Y_n}{r_J}/\frac{S^2}{\sigma ^2}. \end{aligned}$$

Since $S^2$ converges to $\sigma ^2$ in probability, the remaining is as similar as that of case 1. $\square$

Proof of Lemma 2

For any $\alpha$ satisfying (C0), from Lemma 1, we have

$$\begin{aligned}&\mathrm{Pr}\left( p(K)>p(J)\right) \\&\quad =\mathrm{Pr}\left( p(K)> p(J),p(K)\le \alpha \right) +\mathrm{Pr}\left( p(K)> p(J),p(K)>\alpha \right) \\&\quad \le \mathrm{Pr}\left( p(J)\le \alpha \right) +\mathrm{Pr}\left( p(K)>\alpha \right) \\&\quad \rightarrow 0, \text {as}\,\, n\rightarrow \infty . \end{aligned}$$

$\square$

Proof of Theorem 1

(i) For All-sets-down procedure,

$$\begin{aligned} \{J_0\ne I_0\}\!=\!\left\{ \bigcup _{|J|>|I_0|}\left\{ p(J)>\alpha \right\} \right\} \!\cup \!\left\{ \bigcup _{|J|=|I_0|,J\ne I_0}\left\{ p(I_0)<p(J)\right\} \right\} \cup \{p(I_0)\le \alpha \}. \end{aligned}$$

From Lemma 1 and Lemma 2,

$$\begin{aligned} \mathrm{Pr}\left( \bigcup _{|J|>|I_0|}\left\{ p(J)>\alpha \right\} \right)&\le \sum _{|J|>|I_0|}\mathrm{Pr}\left( p(J)>\alpha \right) \rightarrow 0,\\&\text {as } n\rightarrow \infty ,\\ \mathrm{Pr}\left( \bigcup _{|J|=|I_0|,J\ne I_0}\left\{ p(I_0)<p(J)\right\} \right)&\le \sum _{|J|=|I_0|,J\ne I_0}\mathrm{Pr}\left( p(I_0)<p(J)\right) \rightarrow 0,\\&\text {as } n\rightarrow \infty , \end{aligned}$$

then

$$\begin{aligned} \lim _{n\rightarrow \infty }\mathrm{Pr}\left( J_0=I_0\right) =1. \end{aligned}$$

(ii) For All-sets-up procedure,

$$\begin{aligned}&\left\{ J_0\ne I_0\right\} \\&\quad =\left\{ \bigcup _{d=1}^{|I_0|-1}\bigcap _{|J|=d}\{p(J)\le \alpha \}\right\} \cup \left\{ \bigcup _{|J|=|I_0|,J\ne I_0}\{p(J)>p(I_0)\}\right\} \cup \{p(I_0)\le \alpha \}\\&\quad \cup \left\{ \bigcup _{|J|=|I_0|+1}\{p(J)>\alpha \}\right\} . \end{aligned}$$

Since

$$\begin{aligned}&\mathrm{Pr}\left( \bigcup _{d=1}^{|I_0|-1}\bigcap _{|J|=d}\{p(J)\!\le \! \alpha \}\right) \le \sum _{d=1}^{|I_0|-1}\mathrm{Pr}\left( \bigcap _{|J|=d,J\subseteq I_0}\{p(J)\le \alpha \}\right) \rightarrow 0, \text {as } n\rightarrow \infty ,\\&\mathrm{Pr}\left( \bigcup _{|J|=|I_0|,J\ne I_0}\{p(J)\!>\!p(I_0)\}\right) \!\le \! \sum _{|J|=|I_0|,J\ne I_0}\mathrm{Pr}\left( p(J)\!>\!p(I_0)\right) \rightarrow 0, \text {as } n\rightarrow \infty ,\\&\mathrm{Pr}\left( \bigcup _{|J|=|I_0|+1}\{p(J)>\alpha \}\right) \le \sum _{|J|=|I_0|+1}\mathrm{Pr}\left( p(J)>\alpha \right) \rightarrow 0, \text {as } n\rightarrow \infty , \end{aligned}$$

and $\lim \limits _{n\rightarrow \infty }\mathrm{Pr}(p(I_0)\le \alpha )=0$, we immediately get

$$\begin{aligned} \lim _{n\rightarrow \infty }\mathrm{Pr}(J_0=I_0)=1. \end{aligned}$$

(iii) For Nested-up procedure, let

$$\begin{aligned} D_n=\{\text {Accept all true null hypotheses without any false null accepted}\}. \end{aligned}$$

From Lemma 1 and Lemma 2, we have $\lim \limits _{n\rightarrow \infty }\mathrm{Pr}(D_n)=1$. Therefore, as long as there exists at least one true null hypothesis in that step, a true null will be accepted in probability. Since

$$\begin{aligned} \{J_0=I_0\}=D_n\cap \left\{ \bigcap _{|J|=|I_0|+1, J\supset I_0}\{p(J)\le \alpha \}\right\} , \end{aligned}$$

and $\lim \limits _{n\rightarrow \infty }\mathrm{Pr}(p(J)\le \alpha )=1,J\supset I_0$, we have

$$\begin{aligned} \lim _{n\rightarrow \infty }\mathrm{Pr}\left( J_0=I_0\right) =1. \end{aligned}$$

$\square$

Proof of Proposition 1

When all the null hypotheses are true, for All-sets-down and Nested-down procedures,

$$\begin{aligned} \mathrm{FWER}=\mathrm{Pr}(p(I)\ge \alpha )=\alpha . \end{aligned}$$

$\square$

Proof of Proposition 2

Without loss of generality, suppose there exists at least one false null hypotheses. Denote $V_n$ be the number of false rejections, so $\mathrm{FWER}=\mathrm{Pr}(V_n\ge 1)$. For All-sets-down procedure,

$$\begin{aligned} \{V_n\ge 1\}=E_n\cup F_n\cup G_n, \end{aligned}$$

where

$$\begin{aligned} E_n&= \bigcup _{d>|I_0|}\left\{ \left\{ \bigcap _{|J|=d+1}\{p(J)\le \alpha \}\right\} \right. \\&\left. \cap \left\{ \bigcup _{|K|=d,I_0\nsubseteq K} \left\{ p(K)>\alpha ,p(K)=\max _{|W|=|K|}p(W)\right\} \right\} \right\} ,\\ F_n&= \left\{ \bigcap _{|J|>|I_0|}\{p(J)\le \alpha \}\right\} \cap \left\{ p(I_0)<\max _{K\ne I_0,|K|=|I_0|}p(K)\right\} ,\\ G_n&= \left\{ \bigcap _{|J|>|I_0|}\{p(J)\le \alpha \}\right\} \cap \left\{ p(I_0)=\max _{|K|=|I_0|}p(K)\right\} \cap \left\{ p(I_0)\le \alpha \right\} . \end{aligned}$$

Since $H_K$ is false, with $|K|>|I_0|$, for any fixed $\alpha$, $\alpha \in (0,1)$,

$$\begin{aligned} \lim _{n\rightarrow \infty }\mathrm{Pr}(p(K)>\alpha )= 0, \text {and} \lim _{n\rightarrow \infty }\mathrm{Pr}(E_n)= 0. \end{aligned}$$

Additionally, from Lemma 2,

$$\begin{aligned} \mathrm{Pr}\left( p(I_0)<\max _{K\ne I_0,|K|=|I_0|}p(K)\right) \rightarrow 0, \end{aligned}$$

so $\lim _{n\rightarrow \infty }\mathrm{Pr}(F_n)=0.$

Furthermore, since $\mathrm{Pr}\left( p(I_0)\le \alpha \right) =\alpha$, and

$$\begin{aligned} \lim _{n\rightarrow \infty }\mathrm{Pr} \left( \bigcap _{|J|>|I_0|}\{p(J)\le \alpha \}\right) = 1, \lim _{n\rightarrow \infty }\mathrm{Pr}\left( p(I_0)=\max _{|K|=|I_0|}p(K)\right) = 1, \end{aligned}$$

we get $\lim _{n\rightarrow \infty }\mathrm{Pr}(G_n)=\alpha .$

It’s easy to see

$$\begin{aligned} \mathrm{Pr}(G_n)\le \mathrm{Pr}(V_n\ge 1)\le \mathrm{Pr}(E_n)+\mathrm{Pr}(F_n)+\mathrm{Pr}(G_n). \end{aligned}$$

Since

$$\begin{aligned} \lim _{n\rightarrow \infty }\mathrm{Pr}(E_n)=0, \lim _{n\rightarrow \infty }\mathrm{Pr}(F_n)= 0, \lim _{n\rightarrow \infty }\mathrm{Pr}(G_n)=\alpha , \end{aligned}$$

we have

$$\begin{aligned} \lim _{n\rightarrow \infty }\mathrm{Pr}(V_n\ge 1)= \alpha . \end{aligned}$$

$\square$