A comprehensive error rate for multiple testing

  • Djalel-Eddine MeskaldjiEmail author
  • Dimitri Van De Ville
  • Jean-Philippe Thiran
  • Stephan Morgenthaler
Regular Article


Over the last two decades, a large variety of type I error rates and control procedures have been proposed in the field of multiple hypotheses testing. This paper proposes a framework that includes many existing proposals by investigating procedures in which the ordered p-values are compared to an arbitrary positive and non-decreasing threshold sequence. For this case, we derive the error rate being controlled under different assumptions on the p-values. Our focus will be on step-up procedures. The new formulation gives insight into the relations between existing error rates and opens new perspectives for the whole field of multiple testing.


Family-wise error rate False discovery rate Multiple comparisons Ordered p-values Scaled false discovery rate Type I error control 



The main part of this work was done while the first author was a Ph.D student in the Signal Processing Laboratory (LTS5), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland (Meskaldji 2013). The authors would like to thank Etienne Roquain, Arnold Janssen and Bradley Efron for interesting comments and suggestions. We also thank the referees for their helpful and constructive comments.


This work was supported by the Swiss National Science Foundation [144467, PP00P2-146318].


  1. Benjamini Y (2010) Discovering the false discovery rate. J R Stat Soc Ser B 72(4):405–416MathSciNetCrossRefGoogle Scholar
  2. Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B Methodol 57(1):289–300MathSciNetzbMATHGoogle Scholar
  3. Benjamini Y, Hochberg Y (1997) Multiple hypotheses testing with weights. Scand J Stat 24(3):407–418MathSciNetCrossRefzbMATHGoogle Scholar
  4. Benjamini Y, Yekutieli D (2001) The control of the false discovery rate in multiple testing under dependency. Ann Stat 29(4):1165–1188MathSciNetCrossRefzbMATHGoogle Scholar
  5. Bernhard G, Klein M, Hommel G (2004) Global and multiple test procedures using ordered p-values—a review. Stat Pap 45(1):1–14MathSciNetCrossRefzbMATHGoogle Scholar
  6. Blanchard G, Roquain E (2008) Two simple sufficient conditions for FDR control. Electron J Stat 2:963–992MathSciNetCrossRefzbMATHGoogle Scholar
  7. Bonferroni CE (1936) Teoria statistica delle classi e calcolo delle probabilità. Pub del R Ist Sup di Sci Eco e Com di Fir 8:3–62 Bonferroni adjustment for multiple statistical tests using the same datazbMATHGoogle Scholar
  8. Dudoit S, van der Laan MJ (2008) Multiple testing procedures with applications to genomics. Springer Series in Statistics. Springer, New YorkCrossRefzbMATHGoogle Scholar
  9. Efron B, Tibshirani R, Storey J, Tusher V (2001) Empirical Bayes analysis of a microarray experiment. J Am Stat Assoc 96:1151–1160MathSciNetCrossRefzbMATHGoogle Scholar
  10. Finner H, Dickhaus T, Roters M (2009) On the false discovery rate and an asymptotically optimal rejection curve. Ann Stat 37:596–618MathSciNetCrossRefzbMATHGoogle Scholar
  11. Genovese C, Wasserman L (2002) Operating characteristics and extensions of the false discovery rate procedure. J R Stat Soc Ser B 64:499–517MathSciNetCrossRefzbMATHGoogle Scholar
  12. Genovese CR, Roeder K, Wasserman L (2006) False discovery control with p-value weighting. Biometrika 93(3):509–524MathSciNetCrossRefzbMATHGoogle Scholar
  13. Heesen P, Janssen A (2015) Inequalities for the false discovery rate (fdr) under dependence. Electron J Stat 9(1):679–716MathSciNetCrossRefzbMATHGoogle Scholar
  14. Hochberg Y (1988) A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75(4):800–802MathSciNetCrossRefzbMATHGoogle Scholar
  15. Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat Theory Appl 6(2):65–70MathSciNetzbMATHGoogle Scholar
  16. Hommel G, Hoffmann T (1987) Controlled uncertainty. In: Bauer P, Hommel G, Sonnemann E (eds) Multiple hypotheses testing. Springer, Heidelberg, pp 154–161Google Scholar
  17. Lehmann EL, Romano JP (2005) Generalizations of the familywise error rate. Ann Stat 33(3):1138–1154MathSciNetCrossRefzbMATHGoogle Scholar
  18. Meskaldji D-E (2013) Multiple comparison procedures for large correlated data with application to brain connectivity analysis. Ph.D. thesis, Ecole Polytechnique Fédérale de LausanneGoogle Scholar
  19. Meskaldji D-E, Fischi-Gomez E, Griffa A, Hagmann P, Morgenthaler S, Thiran J-P (2013) Comparing connectomes across subjects and populations at different scales. Neuroimage 80:416–425 Mapping the ConnectomeCrossRefGoogle Scholar
  20. Meskaldji D-E, Thiran J-P, Morgenthaler S (2011) A comprehensive error rate for multiple testing. ArXiv e-prints arXiv:1112.4519
  21. Meskaldji D-E, Vasung L, Romascano D, Thiran J-P, Hagmann P, Morgenthaler S, Ville DVD (2015) Improved statistical evaluation of group differences in connectomes by screening-filtering strategy with application to study maturation of brain connections between childhood and adolescence. Neuroimage 108:251–264CrossRefGoogle Scholar
  22. Pigeot I (2000) Basic concepts of multiple tests—a survey. Stat Pap 41(1):3–36MathSciNetCrossRefzbMATHGoogle Scholar
  23. Roquain E (2015) Contributions to multiple testing theory for high-dimensional data. Université Pierre et Marie Curie, ParisGoogle Scholar
  24. Roquain E, Villers F (2011) Exact calculations for false discovery proportion with application to least favorable configurations. Ann Stat 39:584–612MathSciNetCrossRefzbMATHGoogle Scholar
  25. Sarkar SK (1998) Some probability inequalities for ordered MTP2 random variables: a proof of the simes conjecture. Ann Stat 26(2):494–504CrossRefzbMATHGoogle Scholar
  26. Schweder T, Spjøtvoll E (1982) Plots of p-values to evaluate many tests simultaneously. Biometrika 69(3):493–502CrossRefGoogle Scholar
  27. Simes RJ (1986) An improved Bonferroni procedure for multiple tests of significance. Biometrika 73(3):751–754MathSciNetCrossRefzbMATHGoogle Scholar
  28. van der Laan MJ, Dudoit S, Pollard KS (2004) Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. Stat Appl Genet Mol Biol 3:1–25 Art. 15 (electronic)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Djalel-Eddine Meskaldji
    • 1
    • 3
    • 4
    Email author
  • Dimitri Van De Ville
    • 2
    • 3
  • Jean-Philippe Thiran
    • 4
  • Stephan Morgenthaler
    • 1
  1. 1.Institute of MathematicsÉcole Polytechnique Fédérale de Lausanne (EPFL)LausanneSwitzerland
  2. 2.Department of Radiology and Medical InformaticsUniversity of GenevaGenevaSwitzerland
  3. 3.Institute of BioengineeringÉcole Polytechnique Fédérale de Lausanne (EPFL)LausanneSwitzerland
  4. 4.Institute of EngineeringÉcole Polytechnique Fédérale de Lausanne (EPFL)LausanneSwitzerland

Personalised recommendations