Skip to main content

Frequentist Properties of Bayesian Multiplicity Control for Multiple Testing of Normal Means


We consider the standard problem of multiple testing of normal means, obtaining Bayesian multiplicity control by assuming that the prior inclusion probability (the assumed equal prior probability that each mean is nonzero) is unknown and assigned a prior distribution. The asymptotic frequentist behavior of the Bayesian procedure is studied, as the number of tests grows. Studied quantities include the false positive probability, which is shown to go to zero asymptotically. The asymptotics of a Bayesian decision-theoretic approach are also presented.

This is a preview of subscription content, access via your institution.

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5


  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Series B (Methodol.), 289–300.

  • Berger, J.O., Wang, X. and Shen, L. (2014). A Bayesian approach to subgroup identification. J. Biopharmaceut. Stat. 24, 110–129.

    MathSciNet  Article  Google Scholar 

  • Bogdan, M., Ghosh, J.K., Ochman, A. and Tokdar, S.T. (2007). On the empirical Bayes approach to the problem of multiple testing. Qual. Reliab. Eng. Int. 23, 727–739.

    Article  Google Scholar 

  • Bogdan, M., Ghosh, J.K., Tokdar, S.T. et al. (2008a). A comparison of the Benjamini-Hochberg procedure with some Bayesian rules for multiple testing, in Beyond parametrics in interdisciplinary research: Festschrift in honor of Professor Pranab K. Sen, pp. 211–230, Institute of Mathematical Statistics.

  • Bogdan, M., Ghosh, J.K. and Żak-Szatkowska, M. (2008b). Selecting explanatory variables with the modified version of the Bayesian information criterion. Qual. Reliab. Eng. Int. 24, 627–641.

  • Bogdan, M., Chakrabarti, A., Frommlet, F., Ghosh, J.K. et al. (2011). Asymptotic Bayes-optimality under sparsity of some multiple testing procedures. Ann. Stat. 39, 1551–1579.

    MathSciNet  Article  Google Scholar 

  • Chang, S. and Berger, J.O. (2016). Comparison of Bayesian and Frequentist Multiplicity Correction For Testing Mutually Exclusive Hypotheses Under Data Dependence. arXiv:

  • Cui, W. and George, E.I. (2008). Empirical Bayes vs. fully Bayes variable selection. J. Statist. Plan. Inference 138, 888–900.

    MathSciNet  Article  Google Scholar 

  • Dmitrienko, A., Tamhane, A.C. and Bretz, F (2009). Multiple Testing Problems in Pharmaceutical Statistics. CRC Press.

  • Durrett, R. (2010). Probability: Theory and Examples vol. 3. Cambridge University Press.

  • Efron, B. (2004). Large-scale simultaneous hypothesis testing. J. Am. Stat. Assoc., 99.

  • Efron, B., Tibshirani, R., Storey, J.D. and Tusher, V. (2001). Empirical Bayes analysis of a microarray experiment. J. Am. Stat. Assoc. 96, 1151–1160.

    MathSciNet  Article  Google Scholar 

  • Ferguson, T.S. (2003). U-statistics. Notes for Statistics.

  • Guindani, M., Müller, P. and Zhang, S. (2009). A Bayesian discovery procedure. J. R. Stat. Soc.: Series B (Stat. Methodol.) 71, 905–925.

    MathSciNet  Article  Google Scholar 

  • Scott, J.G. and Berger, J.O. (2006). An exploration of aspects of Bayesian multiple testing. J. Stat. Plan. Inference 136, 2144–2162.

    MathSciNet  Article  Google Scholar 

  • Scott, J. G. and Berger, J.O. (2010). Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem. Ann. Stat. 38, 2587–2619.

    MathSciNet  Article  Google Scholar 

  • Storey, J.D. (2003). The positive false discovery rate: A Bayesian interpretation and the q-value. Ann. Stat., 2013–2035.

  • Storey, J.D. and Tibshirani, R. (2003). Statistical significance for genomewide studies. Proc. Natl. Acad. Sci. 100, 9440–9445.

    MathSciNet  Article  Google Scholar 

Download references


Supported in part by NSF grants DMS-1007773 and DMS-1407775

Author information

Authors and Affiliations


Corresponding author

Correspondence to Sean Chang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.



Fact 6.1 (Normal tail probability).

Let Φ(t) be the cumulative distribution function of a standard normal random variable. Then as \(t \rightarrow \infty \),

$$ \frac{t\frac{1}{\sqrt{2\pi}} e^{-t^{2}/2} }{t^{2}+1}\leq\ 1-{\Phi}(t)\leq \frac{\frac{1}{\sqrt{2\pi}} e^{-t^{2}/2} }{t} , $$
$$ 1-{\Phi}(t) =\frac{\frac{1}{\sqrt{2\pi}} e^{-t^{2}/2}}{t}(1+o_{p}(1)) . $$

The proof can be found in Durrett (2010).

Fact 6.2 (Weak law for triangular arrays).

For each m, let Xm,k, 1 ≤ km be independent random variables. Let βm > 0 with \(\beta _{m} \rightarrow \infty \) and let \(\bar {X}_{m,k}=X_{m,k}1_{\{|X_{m,k}| \leq \beta _{m} \}}\). Suppose that as \(m\rightarrow \infty \)

$$\sum \limits^{m}_{k=1} P(|X_{m,k}|>\beta_{m}) \rightarrow 0 \quad \text{and} \quad \frac{1}{{\beta_{m}^{2}}}\sum \limits^{m}_{k=1} E\bar{X}_{m,k}^{2} \rightarrow 0 .$$

Defining Sm = Xm,1 + ... + Xm,m and \(\alpha _{m}=\sum \limits ^{m}_{k=1} E\bar {X}_{m,k}\), it follows that

$$ \frac{(S_{m}-\alpha_{m})}{\beta_{m}} \rightarrow 0 \quad \text{in probability} . $$

See Durrett (2010) for a proof.

Lemma 6.3.

If \(Y_{j}=\frac {1}{\sqrt {1+\tau ^{2}}}\exp \{\frac {\tau ^{2}}{2(1+\tau ^{2})}{X_{j}^{2}}\}\) and τ2 > 1/3, then

$$ m^{-2\tau^{2} /(1+\tau^{2})}\sum\limits_{j=1}^{m} {Y_{j}^{2}} \rightarrow 0 \quad \text{ when } m\rightarrow \infty . $$


First we need the following order estimate before applying Fact 6.2: If \(\lim _{m\rightarrow \infty } a_{m} \in (0,\infty ]\), \(\lim _{m\rightarrow \infty }b_{m}= \infty \), then

$$ {\int}^{b_{m}}_{0} e^{\frac{a_{m} x^{2}}{2}}dx = O\bigg(\frac{e^{a_{m}{b_{m}^{2}}/2}}{a_{m}b_{m}}\bigg) . $$

To show (6.1) is true, notice that \({\int \limits }^{b_{m}}_{0} e^{\frac {a_{m}x^{2}}{2}}dx = \frac {1}{\sqrt {a_{m}}}{\int \limits }_{0}^{\sqrt {a_{m}}b_{m}} e^{\frac {x^{2}}{2}}dx\) and by L’Hôpital’s rule:

$$\begin{array}{llll} &\lim\limits_{m\rightarrow \infty} \frac{\frac{1}{\sqrt{a_{m}}}{\int}^{\sqrt{a_{m}}b_{m}}_{0} e^{\frac{x^{2}}{2}}dx} {e^{\frac{a_{m}{b_{m}^{2}}}{2}}/(a_{m} b_{m})} = \lim\limits_{m\rightarrow \infty} \frac{{\int}^{\sqrt{a_{m}}b_{m}}_{0} e^{\frac{x^{2}}{2}}dx} {e^{\frac{a_{m}{b_{m}^{2}}}{2}}/(\sqrt{a_{m}} b_{m})}\\ &= \lim\limits_{v\rightarrow \infty}\frac{{{\int}^{v}_{0}} e^{\frac{x^{2}}{2}}dx}{e^{\frac{v^{2}}{2}}/v} \quad \text{ where } v = \sqrt{a_{m}}b_{m}\\ &= \lim\limits_{v\rightarrow \infty} \frac{\frac{d}{dv}{{\int}^{v}_{0}} e^{x^{2}/2}dx}{\frac{d}{dv}\frac{e^{v^{2}/2}}{v}} = \lim\limits_{v \rightarrow \infty} \frac{1}{1-\frac{1}{v^{2}}} =1 . \end{array}$$

Take \(X_{m,j}= \exp \{\frac {\tau ^{2}}{1+\tau ^{2}} {x_{j}^{2}}\}\) and \(\beta _{m} = m^{2\tau ^{2}/(1+\tau ^{2})}\) in Fact 6.2. Both assumptions in Fact 6.2 hold since

$$\sum \limits_{j=1}^{m} P(|X_{m,j}|>\beta_{m}) = O\bigg(\frac{1}{\sqrt{log m}}\bigg) \rightarrow 0 \qquad \text{ by Fact } 6.1 ,$$


$$\begin{array}{llll} \frac{1}{{\beta_{m}^{2}}} \sum \limits^{m}_{j=1} E\bar{X}_{m,j}^{2} &=\frac{m}{m^{4\tau^{2}/(1+\tau^{2})}} \int\limits_{|x|\leq \sqrt{ 2 \log m} }\frac{1}{\sqrt{2\pi}}e^{(\frac{2\tau^{2}}{1+\tau^{2}} -1) \frac{x^{2}}{2}}dx\\ &= \left\{\begin{array}{llll} O\big(m^{-\frac{2\tau^{2}}{1+\tau^{2}}} \frac{1}{\sqrt{log m }}\big) &\text{ if } \tau^{2}>1, \text{ by } (6.1) \\ O\big(m^{\frac{1-3\tau^{2}}{1+\tau^{2}}}\big) &\text{ else,} \end{array}\right. \\&= o(1) \qquad \text{ if } \tau^{2}>1/3 . \end{array} $$

The limit is

$$ \begin{array}{llll} \frac{1}{\beta_{m}} \sum \limits^{m}_{j=1} E\bar{X}_{m,j} &= m^{-2\tau^{2}/(1+\tau^{2})} \frac{m}{\sqrt{2\pi}}\int\limits_{|x|<\sqrt{2 \log m}} \exp\left\{ \frac{x^{2}}{2}(\frac{2\tau^{2}}{1+\tau^{2}}-1) \right\} \\ &= m^{(1-\tau^{2})/(1+\tau^{2})} O\bigg(m^{(\tau^{2}-1)/(1+\tau^{2})} \frac{1}{\sqrt{log m}} \bigg) \text{ by } (6.1)\\ &= O\bigg(\frac{1}{\sqrt{log m}}\bigg) \rightarrow 0 . \end{array} $$

Fact 6.4 (Marcinkiewicz and Zygmund).

Let X1,X2,... be i.i.d. with EX1 = 0 and \(E|X_{1}|^{p} < \infty \) where 1 < p < 2. If Sm = X1 + ... + Xm then

$$\frac{S_{m}}{m^{1/p}} \rightarrow 0 \text{ a.s. } $$

The proof can be found in Durrett (2010).

Lemma 6.5.

If \(Y_{j}=\frac {1}{\sqrt {1+\tau ^{2}}}\exp \{\frac {\tau ^{2}}{2(1+\tau ^{2})}{X_{j}^{2}}\}\) and k = O(mr), with \( r\in [0, \frac {1}{1+\max \limits \{1,\tau ^{2}\}})\), then

$$ \bigg(\frac{1}{m} \sum\limits_{j=1}^{m} Y_{j} \bigg)^{k} = 1+o(1) . $$


If τ2 < 1, then E[Yj] = 1 and Yj has finite variance, so

$$ \bigg(\frac{1}{m} \sum\limits_{j=1}^{m} Y_{j} \bigg)^{k} = (1+O(m^{-1/2}))^{k} = 1+ O(m^{(r-1/2)}) $$

and the result holds.

If τ2 ≥ 1, set Xi = Yi − 1 and p = 1 + τ− 2𝜖. Fact 6.4 then implies that

$$\frac{{{\sum}_{1}^{m}} Y_{j} - m} {m^{1/p}} =\bigg(\frac{{{\sum}_{1}^{m}} Y_{j}} {m^{1/p}} - m^{1-1/p} \bigg)=o(1) ,$$

so that

$$ \frac{{{\sum}^{m}_{1}} Y_{j}}{m} -1 = \frac{1}{m^{1-1/p}}\bigg(\frac{\sum Y_{j}}{m^{1/p}} -m^{1-1/p} \bigg) = o\bigg(\frac{1}{m^{1-1/p}}\bigg) . $$


$$ \left( \frac{{{\sum}^{m}_{1}} Y_{j}}{m} \right)^{k} = (1+o(m^{-(1-1/p)}))^{k} = (1+o(m^{(r-1+1/p)})) . $$

Computation shows that r − 1 + 1/p < 0 for small enough 𝜖, proving the result. □

Proof Proof of Lemma 3.1.

Let \(Y_{j} = \frac {1}{\sqrt {1+\tau ^{2}}} e^{{X_{j}^{2}}\tau ^{2}/2(1+\tau ^{2})}\) and expand \((\frac {1}{m}{{\sum }_{1}^{m}} Y_{j})^{k}\):

$$ \begin{array}{llll} \bigg[\frac{{{\sum}_{1}^{m}} Y_{j}}{m} \bigg]^{k} &= \sum \limits_{k_{1}+...+k_{m} = k} \binom{k}{k_{1},k_{2},...,k_{m}}\prod\limits_{1\leq j \leq m} \frac{Y_{j}^{k_{j}}}{m^{k_{j}}} \\ &= \underbrace{\sum \limits_{\begin{array}{llll}k_{1}+...+k_{m} = k \\ max|k_.|=1 \end{array}} \frac{k!}{m^{k}} \prod\limits_{1\leq j \leq m} Y_{j}^{k_{j}}}_{I} + \underbrace{\sum \limits_{\begin{array}{llll}k_{1}+...+k_{m} = k \\ max|k_.|\geq 2 \end{array}} \frac{k!}{k_{1}!...k_{m}!} \prod\limits_{1\leq j \leq m} \frac{Y_{j}^{k_{j}}}{m^{k_{j}}}}_{II} . \end{array} $$

Notice that

$$ \begin{array}{llll} I &= \frac{(m-1)...(m-k+1)}{m^{k-1}}\frac{1}{\binom{m}{k}} \sum \limits_{\begin{array}{llll}k_{1}+...+k_{m} = k \\ max|k_.|=1 \end{array}} \prod\limits_{1\leq j \leq m} \exp \left\{ \frac{\tau^{2} k_{j}}{2(1+\tau^{2})} {X_{j}^{2}}\right\}\\ &= \frac{(m-1)...(m-k+1)}{m^{k-1}} \frac{1}{\binom{m}{k}} \sum\limits_{|\boldsymbol{\gamma}| =k} B_{\boldsymbol{\gamma}0}\\ &= \bigg[\frac{1}{\binom{m}{k}}\sum\limits_{|\boldsymbol{\gamma}| =k} B_{\boldsymbol{\gamma}0}(\boldsymbol{X})\bigg](1+O(k^{2}/m))= \bigg[\frac{1}{\binom{m}{k}}\sum\limits_{|\boldsymbol{\gamma}| =k} B_{\boldsymbol{\gamma}0}(\boldsymbol{X})\bigg](1+o(1)) , \end{array} $$

where the order of the O term can be bounded by a constant independent of k, as established by a detailed Stirling’s approximation argument.

$$ \begin{array}{llll} II &= \frac{1}{m^{k}}\sum \limits_{\begin{array}{llll}k_{1}+...+k_{m} = k \\ max|k_.|\geq 2 \end{array}} \frac{k!}{k_{1}!...k_{m}!} \prod\limits_{1\leq j \leq m} Y_{j}^{k_{j}} \\ &\leq \frac{k(k-1)}{m^{k}}\bigg({\sum\limits_{1}^{m}} {Y_{i}^{2}}\bigg) \sum\limits_{\tilde{k}_{1}+...+\tilde{k}_{m}=k-2} \frac{(k-2)!}{\tilde{k}_{1}!...\tilde{k}_{m}!} Y_{1}^{\tilde{k}_{1}}...Y_{m}^{\tilde{k}_{m}} \\ &= \bigg(\frac{k(k-1)}{m^{2/(1+\tau^{2})}}\bigg) \bigg(\frac{{{\sum}_{1}^{m}} {Y_{i}^{2}}}{m^{2\tau^{2}/(1+\tau^{2})}}\bigg) \bigg(\frac{{{\sum}_{1}^{m}} Y_{i}}{m}\bigg)^{k-2} . \end{array} $$

To conclude that \(II\rightarrow 0\) when \(m\rightarrow \infty \), note that \(\frac {k(k-1)}{m^{2/(1+\tau ^{2})}}\) converges to 0 because of the k = O(mr) assumption;

By Lemma 6.3, when τ2 > 1/3, \(\frac {{{\sum }_{1}^{m}} {Y_{i}^{2}}}{m^{2\tau ^{2}/(1+\tau ^{2})}}=o_{p}(1)\);

By Lemma 6.5, \(\big (\frac {{{\sum }_{1}^{m}} Y_{i}}{m}\big )^{k-2}= 1+o_{p}(1)\).

Hence, the right hand side of Eq. 6.4 converges to \(1/\binom {m}{k}{\sum }_{|\boldsymbol {\gamma }| =k}B_{\boldsymbol {\gamma }0}(1+o(1))\). By Lemma 6.5, the left hand side goes to 1. Hence,

$$ \lim \limits_{m\rightarrow \infty} \frac{1}{\binom{m}{k} }\sum\limits_{|\boldsymbol{\gamma}| =k}B_{\boldsymbol{\gamma} 0} = 1 . $$

Lemma 6.6.

$$\frac{1}{\binom{m}{k}} \sum \limits_{|\boldsymbol{\gamma}| =k} B_{\boldsymbol{\gamma}0}(x) \text{ is a U-statistic} . $$


By definition,

$$\begin{array}{llll} B_{\boldsymbol{\gamma}0}(x_{1},x_{2},...,x_{m}) = \frac{1}{(1+\tau^{2})^{\frac{|\boldsymbol{\gamma}| }{2}}} \exp \left\{ \frac{\tau^{2}}{2(1+\tau^{2})} \sum \limits_{i:\gamma_{i}\neq 0 } {x_{i}^{2}} \right\} \end{array}$$
$$h^{k}(x_{1},x_{2},...,x_{k})= \frac{1}{(1+\tau^{2})^{\frac{k}{2}}} \exp \left\{ \frac{\tau^{2}}{2(1+\tau^{2})} \sum \limits_{i=1}^{k} {x_{i}^{2}} \right\} . $$

Notice that an iteration over models is the same as an iteration through inputs:

$$\sum \limits_{\gamma:|\boldsymbol{\gamma}| =k} B_{\boldsymbol{\gamma}0}(x) = \sum \limits_{{C^{m}_{k}}} h^{k}(x_{i_{1}},x_{i_{2}},...,x_{i_{k}}) .$$


$$\frac{1}{\binom{m}{k}} \sum \limits_{|\boldsymbol{\gamma}| =k} B_{\boldsymbol{\gamma}0}(x) \text{ is a U-statistics with kernel } h^{k} .$$

Definition 6.7.

For c ∈{0,1,...,k},

$$\begin{array}{llll} {h^{k}_{c}}(x_{1},...,x_{c})&=\mathbbm{E}(h^{k}(x_{1},x_{2},...,x_{c},X_{c+1},...,X_{k}))\\ ({\sigma_{c}^{k}})^{2} &= Var({h^{k}_{c}}(X_{1},...,X_{c})) . \end{array}$$

Note that \({h^{k}_{k}} = h^{k}\), \(({\sigma ^{k}_{k}})^{2} =Var(h^{k})={\sigma _{k}^{2}}\).

Fact 6.8.

Let U be the U-statistics with kernel hk. If \({\sigma _{k}^{2}}<\infty \), then

$$\sqrt{m}\big[U-\mathbbm{E}(h^{k}(X_{1},...,X_{k}))\big] \rightarrow N(0,k^{2}({\sigma_{1}^{k}})^{2}) .$$

The asymptotic theory of U-statistics can be found in Ferguson (2003).

Theorem 6.9.

If τ2 < 1, and kmax = o(m), then under the null model,

$$\sqrt{m} \bigg[ \frac{1}{\binom{m}{k}}\sum\limits_{|\boldsymbol{\gamma}| =k} B_{\boldsymbol{\gamma}0}(x)- 1\bigg] \rightarrow N\bigg(0,k^{2}\bigg[\frac{1}{\sqrt{1-\tau^{4}}}-1\bigg]\bigg) .$$


By Lemma 6.6, \(1/\binom {m}{k}{\sum }_{|\boldsymbol {\gamma }| =k}B_{\boldsymbol {\gamma }0}\) is a U-statistics, then evaluate the expectation and variance in Definition 6.7, one can get under the null model:

$$ \begin{array}{llll} \mathbbm{E}(B_{\boldsymbol{\gamma}0}(X))&= \mathbbm{E}({h^{k}_{1}}(X)) = 1 ,\\ \mathbbm{E}(B_{\boldsymbol{\gamma}0}(X)^{2}) &= \left\{\begin{array}{llll} \frac{1}{[(1+\tau^{2})(1-\tau^{2})]^{\frac{|\boldsymbol{\gamma}| }{2}}} &\text{ if } \tau^{2}<1 , \\ \infty &\text{ else, } \end{array}\right.\\ ({\sigma^{k}_{1}})^{2} &= \left\{\begin{array}{llll} \frac{1}{\sqrt{1-\tau^{4}}}-1 &\text{ if } \tau^{2}<1 ,\\ \infty &\text{ else, } \end{array}\right.\\ ({\sigma^{k}_{k}})^{2} &= \left\{\begin{array}{llll} \frac{1}{[(1+\tau^{2})(1-\tau^{2})]^{\frac{k}{2}}}-1 &\text{ if } \tau^{2}<1 , \\ \infty &\text{ else. } \end{array}\right. \end{array} $$

then apply Fact 6.8. □

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chang, S., Berger, J.O. Frequentist Properties of Bayesian Multiplicity Control for Multiple Testing of Normal Means. Sankhya A 82, 310–329 (2020).

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI:


  • Multiple hypothesis testing
  • spike-and-slab prior
  • multiplicity control
  • false positive probability.

AMS (2000) subject classification.

  • Primary 62C10; Secondary 62J15