Multiple-Hypothesis Testing Strategy

  • Mark Chang
Part of the Statistics for Biology and Health book series (SBH)


In this chapter, we will discuss multiple hypothesis-testing issues from a frequentist perspective. The Bayesian approaches for multiple-testing problems will be discussed briefly in Chap. 10. As we all know, a typical hypothesis test in the frequentist paradigm can be written as
$${H}_{o} : \delta \in {\Omega }_{0}\ \mathrm{or}\ {H}_{a} : \delta \in {\Omega }_{1},$$
where δ is a parameter such as treatment effect, the domain Ω 0 can be, for example, a set of nonpositive values, and the domain Ω 1 can be the negation of Ω 0. In this case, (1.1) becomes
$${H}_{o} : \delta \leq 0\ \mathrm{or}\ {H}_{a} : \delta > 0.$$


False Discovery Rate Closure Principle True Null Hypothesis Stepup Procedure Simultaneous Confidence Interval 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Further Readings and References

  1. Bauer, P., Rohmel, J., Maurer, W., Hothorn, L.: Testing strategies in multi-dose experiments including active control. Stat. Med. 17, 2133–2146 (1998)CrossRefGoogle Scholar
  2. Benjamini, Y., Hochberg, Y.: Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57(2), 289–300 (1995)zbMATHMathSciNetGoogle Scholar
  3. Benjamini, Y., Liu, W.: A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. J. Stat. Plan. Inference 82, 163–170 (1999)zbMATHCrossRefMathSciNetGoogle Scholar
  4. Benjamini, Y., Yekutieli, D.: The control of the false discovery rate in multiple testing under dependency. Ann. Stat. 29(4), 1165–1188 (2001)zbMATHCrossRefMathSciNetGoogle Scholar
  5. Berger, R.L.: Multiparameter hypothesis testing and acceptance sampling. Technometrics 24, 295–300 (1982)zbMATHCrossRefMathSciNetGoogle Scholar
  6. Birnbaum, A.: On the foundations of statistical inference (with discussion). J. Am. Stat. Assoc. 57, 269–326 (1962)zbMATHCrossRefMathSciNetGoogle Scholar
  7. Blanchard, G., Roquain, E.: Adaptive false discovery rate control under independence and dependence. J. Mach. Learn. Res. 10, 2837–2871 (2009)MathSciNetGoogle Scholar
  8. Bretz, F., Schmidli1, H., König, F., Racine1, A., Maurer, W.: Confirmatory seamless phase II/III clinical trials with hypotheses selection at interim: General concepts. Biom. J. 48:4.48, 623–634 (2006)Google Scholar
  9. Bretz, F., Hothorn, T., Westfall, P.: Multiple Comparison Using R. CRC Press, Boca Raton (2010)CrossRefGoogle Scholar
  10. Chang, M.: Adaptive Design Theory and Implementation Using SAS and R. Chapman and Hall/CRC, Boca Raton (2007)Google Scholar
  11. Chang, M.: Limitations of hypothesis testing in clinical trials: Discussion of “Some Controversial Multiple Testing Problems in Regulatory Applications.” J. Biopharm. Stat. 19, 35–41 (2009)CrossRefGoogle Scholar
  12. Chang, M., Chow, S.C.: Analysis strategy of multiple-endpoint adaptive design. J. Biopharm. Stat. 17, 1189–1200 (2007)CrossRefMathSciNetGoogle Scholar
  13. Chen, X., Luo, X., Capizzi, T.: The application of enhanced parallel gatekeeping strategies. Stat. Med. 24, 1385–1397 (2005)CrossRefMathSciNetGoogle Scholar
  14. Dmitrienko, A., Offen, W.W., Westfall, P.H.: Gatekeeping strategies for clinical trials that do not require all primary effects to be significant. Stat. Med. 22, 2387–2400 (2003)CrossRefGoogle Scholar
  15. Dmitrienko, A., Molenberghs, G., Chuang-Stein, C., Offen, W.: Analysis of Clinical Trials Using SAS: A Practical Guide. SAS, Cary (2005)Google Scholar
  16. Dmitrienko, A., Offen, W., Wang, O., Xiao, D.: Gatekeeping procedures in dose–response clinical trials based on the Dunnett test. Pharm. Stat. 5, 19–28 (2006a)CrossRefGoogle Scholar
  17. Dmitrienko, A., Tamhane, A.C., Wang, X., Chen, X.: Stepwise gatekeeping procedures in clinical trial applications. Biom. J. 48, 984–991 (2006b)CrossRefMathSciNetGoogle Scholar
  18. Dmitrienko, A., Wiens, B.L., Westfall, P.H.: Fallback tests in dose–response clinical trials. J. Biopharm. Stat. 16, 745–755 (2006c)CrossRefMathSciNetGoogle Scholar
  19. Dmitrienko, A., Tamhane, A.C.: Gatekeeping procedures with clinical trial applications. Pharma. Stat. 6, 171–180 (2007)CrossRefGoogle Scholar
  20. Dmitrienko, A., Wiens, B.L., Tamhane, A.C., Wang, X.: Tree-structured gatekeeping tests in clinical trials with hierarchically ordered multiple objectives. Stat. Med. 26, 2465–2478 (2007)CrossRefMathSciNetGoogle Scholar
  21. Dmitrienko, A., Tamhane, A.C., Bretz, F.: Multiple Testing Problems in Pharmaceutical Statistics. Chapman and Hall/CRC, Boca Raton (2010)Google Scholar
  22. Dunnett, C.W.: A multiple comparison procedure for comparing several treatments with a control. J. Am. Stat. Assoc. 50, 1096–1121 (1955)zbMATHCrossRefGoogle Scholar
  23. European Medicines Agency: ICH Topic E 9 Statistical Principles for Clinical Trials. Step 5. (1998). Accessed 8 Jan 2011
  24. Finner, H., Roter, M.: On the false discovery rate and expected type I errors. Biom. J. 43, 985–1005 (2001)zbMATHCrossRefMathSciNetGoogle Scholar
  25. Ghosh, J.K., Delampady, M., Samanta, T.: An Introduction to Bayesian Analysis. Springer, New York (2006)zbMATHGoogle Scholar
  26. Grechanovsky, E., Hochberg, Y.: Closed procedures are better and often admit a shortcut. J. Stat. Plan. Inference 76, 79–91 (1999)zbMATHCrossRefMathSciNetGoogle Scholar
  27. Hochberg, Y.: A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75, 800–802 (1988)zbMATHCrossRefMathSciNetGoogle Scholar
  28. Hochberg, Y., Benjamini, Y.: More powerful procedures for multiple significance testing. Stat. Med. 9, 811–818 (1990)CrossRefGoogle Scholar
  29. Hochberg, Y., Tamhane, A.C.: Multiple Comparison Procedures. Wiley, New York (1987)zbMATHCrossRefGoogle Scholar
  30. Holm, S.: A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6, 65–70 (1979)zbMATHMathSciNetGoogle Scholar
  31. Hommel, G.: A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75, 383–386 (1988)zbMATHCrossRefGoogle Scholar
  32. Hommel, G., Bretz, F.: Aesthetics and power considerations in multiple testing – a contradiction? Biom. J. 50, 657–666 (2008)CrossRefMathSciNetGoogle Scholar
  33. Hommel, G., Bretz, F., Maurer, W.: Powerful shortcuts for multiple testing procedures with special reference to gatekeeping strategies. Stat. Med. 99, 25–41 (2007)MathSciNetGoogle Scholar
  34. Hsu, J.C.: Multiple Comparisons: Theory and Methods. Chapman and Hall, London (1996)zbMATHGoogle Scholar
  35. Hung, H.M.J.: Some controversival multiple testing problems in regulatory applications. J. Biopharm. Stat. 4, 1–25 (2008)MathSciNetGoogle Scholar
  36. Huque, M., Röhmel, J.: Multiplicity problems in clinical trials: A regulatory perspective. In: Dmitrienko, A., Tamhane, A.C., Bretz, F. (eds.) Multiple Testing Problems in Pharmaceutical Statistics. Chapman and Hall/CRC, Boca Raton (2010)Google Scholar
  37. Kanji, G.K.: 100 Statistical Tests, 3rd edn. Sage Publications, London (2006)Google Scholar
  38. Lehmacher, W., Wassmer, G., Reimeir, P.: Procedure for two-sample comparisons with multiple endpoints controlling the experimentwise error rate. Biometrics 47, 511–521 (1991)CrossRefGoogle Scholar
  39. Lehmann, E.L., Romano, J.P.: Generalizations of the familywise error rate. Ann. Stat. 33, 1138–1154 (2005)zbMATHCrossRefMathSciNetGoogle Scholar
  40. Marcus, R., Peritz, E., Gabriel, K.R.: On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63, 655–660 (1976)zbMATHCrossRefMathSciNetGoogle Scholar
  41. Lemuel, A.M.: Multiple analysis in clinical trials. Springer-Verlag, New York (2003)Google Scholar
  42. Pocock, S., Geller, N., Tsiatis, A.: Analysis of multiple endpoints in clinical trials. Biometrics 43, 487–498 (1987)CrossRefMathSciNetGoogle Scholar
  43. Posch, M., Zehetmayer, S., Bauer, P.: Hunting for significance with the false discovery rate. J. Am. Stat. Assoc. 104(486), 832–840 (2009)CrossRefMathSciNetGoogle Scholar
  44. Quan, H., Luo, X., Capizzi, T.: Multiplicity adjustment for multiple endpoints in clinical trials with multiple doses of an active control. Stat. Med. 24, 2151–2170 (2005)CrossRefMathSciNetGoogle Scholar
  45. Rom, D.M.: A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika 77, 663–665 (1990)CrossRefMathSciNetGoogle Scholar
  46. Roy, S.N.: On a heuristic method for test construction and its use in multivariate analysis. Ann. Stat. 24, 220–238 (1953)zbMATHCrossRefGoogle Scholar
  47. Sarkar, S.K., Chang, C.K.: Simes’ method for multiple hypothesis testing with positively dependent test statistics. J. Am. Stat. Assoc. 92, 1601–1608 (1997)zbMATHCrossRefMathSciNetGoogle Scholar
  48. Shaffer, J.P.: Modified sequentially rejective multiple test procedures. J. Am. Stat. Assoc. 81, 826–831 (1986)zbMATHCrossRefGoogle Scholar
  49. Sidak, Z.: Rectangular confidence regions for the means of multivariate normal distributions. J. Am. Stat. Assoc. 62, 626–633 (1967)zbMATHCrossRefMathSciNetGoogle Scholar
  50. Simes, R.J.: An improved Bonferroni procedure for multiple tests of significance. Biometrika 63, 655–660 (1986)MathSciNetGoogle Scholar
  51. Storey, J., Taylor, J., Siegmund, D.: Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach. J. R. Stat. Soc. Ser. B, 66(1), 187–205 (2004)zbMATHCrossRefMathSciNetGoogle Scholar
  52. Tsong, Y., Wang, S.J., Hung, H.M.J., Cui, L.: Statistical issues on objective, design, and analysis of noninferiority active-controlled clinical trial. J. Biopharm. Stat. 13, 29–41 (2003)zbMATHCrossRefGoogle Scholar
  53. Wang, X.: Gatekeeping procedures for multiple endpoints. Ph.D. Dissertation, Department of Statistics, Northwestern University, Evanston (2006)Google Scholar
  54. Westfall, P.H., Krishen, A.: Optimally weighted, fixed-sequence and gatekeeper multiple testing procedures. J. Stat. Plan. Inference 99, 25–41 (2001)zbMATHCrossRefMathSciNetGoogle Scholar
  55. Westfall, P.H., Tobias, R.D., Rom, D., Wolfinger, R.D., Hochberg, Y.: Multiple Comparisons and Multiple Tests Using the SAS System. SAS Institute, Cary (1999)Google Scholar
  56. Westfall, P.H., Young, S.S.: Resampling-Based Multiple Testing: Examples and Methods for p-value Adjustment. Wiley, New York (1993)Google Scholar
  57. Wiens, B.L.: A fixed-sequence Bonferroni procedure for testing multiple endpoints. Pharm. Stat. 2, 211–215 (2003)CrossRefGoogle Scholar
  58. Wiens, B., Dmitrienko, A.: The fallback procedure for evaluating a single family of hypotheses. J. Biopharm. Stat. 15, 929–942 (2005)CrossRefMathSciNetGoogle Scholar
  59. Xu, H.: Using the partitioning principle to control generalized familywise error rate. Ph.D. Dissertation, The Ohio State University, Columbus (2005)Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Mark Chang
    • 1
  1. 1.BiometricsAMAG Pharmaceuticals, Inc.LexingtonUSA

Personalised recommendations