Abstract
Over the last two decades, a large variety of type I error rates and control procedures have been proposed in the field of multiple hypotheses testing. This paper proposes a framework that includes many existing proposals by investigating procedures in which the ordered p-values are compared to an arbitrary positive and non-decreasing threshold sequence. For this case, we derive the error rate being controlled under different assumptions on the p-values. Our focus will be on step-up procedures. The new formulation gives insight into the relations between existing error rates and opens new perspectives for the whole field of multiple testing.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Benjamini Y (2010) Discovering the false discovery rate. J R Stat Soc Ser B 72(4):405–416
Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B Methodol 57(1):289–300
Benjamini Y, Hochberg Y (1997) Multiple hypotheses testing with weights. Scand J Stat 24(3):407–418
Benjamini Y, Yekutieli D (2001) The control of the false discovery rate in multiple testing under dependency. Ann Stat 29(4):1165–1188
Bernhard G, Klein M, Hommel G (2004) Global and multiple test procedures using ordered p-values—a review. Stat Pap 45(1):1–14
Blanchard G, Roquain E (2008) Two simple sufficient conditions for FDR control. Electron J Stat 2:963–992
Bonferroni CE (1936) Teoria statistica delle classi e calcolo delle probabilità. Pub del R Ist Sup di Sci Eco e Com di Fir 8:3–62 Bonferroni adjustment for multiple statistical tests using the same data
Dudoit S, van der Laan MJ (2008) Multiple testing procedures with applications to genomics. Springer Series in Statistics. Springer, New York
Efron B, Tibshirani R, Storey J, Tusher V (2001) Empirical Bayes analysis of a microarray experiment. J Am Stat Assoc 96:1151–1160
Finner H, Dickhaus T, Roters M (2009) On the false discovery rate and an asymptotically optimal rejection curve. Ann Stat 37:596–618
Genovese C, Wasserman L (2002) Operating characteristics and extensions of the false discovery rate procedure. J R Stat Soc Ser B 64:499–517
Genovese CR, Roeder K, Wasserman L (2006) False discovery control with p-value weighting. Biometrika 93(3):509–524
Heesen P, Janssen A (2015) Inequalities for the false discovery rate (fdr) under dependence. Electron J Stat 9(1):679–716
Hochberg Y (1988) A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75(4):800–802
Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat Theory Appl 6(2):65–70
Hommel G, Hoffmann T (1987) Controlled uncertainty. In: Bauer P, Hommel G, Sonnemann E (eds) Multiple hypotheses testing. Springer, Heidelberg, pp 154–161
Lehmann EL, Romano JP (2005) Generalizations of the familywise error rate. Ann Stat 33(3):1138–1154
Meskaldji D-E (2013) Multiple comparison procedures for large correlated data with application to brain connectivity analysis. Ph.D. thesis, Ecole Polytechnique Fédérale de Lausanne
Meskaldji D-E, Fischi-Gomez E, Griffa A, Hagmann P, Morgenthaler S, Thiran J-P (2013) Comparing connectomes across subjects and populations at different scales. Neuroimage 80:416–425 Mapping the Connectome
Meskaldji D-E, Thiran J-P, Morgenthaler S (2011) A comprehensive error rate for multiple testing. ArXiv e-prints arXiv:1112.4519
Meskaldji D-E, Vasung L, Romascano D, Thiran J-P, Hagmann P, Morgenthaler S, Ville DVD (2015) Improved statistical evaluation of group differences in connectomes by screening-filtering strategy with application to study maturation of brain connections between childhood and adolescence. Neuroimage 108:251–264
Pigeot I (2000) Basic concepts of multiple tests—a survey. Stat Pap 41(1):3–36
Roquain E (2015) Contributions to multiple testing theory for high-dimensional data. Université Pierre et Marie Curie, Paris
Roquain E, Villers F (2011) Exact calculations for false discovery proportion with application to least favorable configurations. Ann Stat 39:584–612
Sarkar SK (1998) Some probability inequalities for ordered MTP2 random variables: a proof of the simes conjecture. Ann Stat 26(2):494–504
Schweder T, Spjøtvoll E (1982) Plots of p-values to evaluate many tests simultaneously. Biometrika 69(3):493–502
Simes RJ (1986) An improved Bonferroni procedure for multiple tests of significance. Biometrika 73(3):751–754
van der Laan MJ, Dudoit S, Pollard KS (2004) Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. Stat Appl Genet Mol Biol 3:1–25 Art. 15 (electronic)
Acknowledgements
The main part of this work was done while the first author was a Ph.D student in the Signal Processing Laboratory (LTS5), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland (Meskaldji 2013). The authors would like to thank Etienne Roquain, Arnold Janssen and Bradley Efron for interesting comments and suggestions. We also thank the referees for their helpful and constructive comments.
Funding
This work was supported by the Swiss National Science Foundation [144467, PP00P2-146318].
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Meskaldji, DE., Van De Ville, D., Thiran, JP. et al. A comprehensive error rate for multiple testing. Stat Papers 61, 1859–1874 (2020). https://doi.org/10.1007/s00362-018-1008-y
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00362-018-1008-y