Skip to main content
SpringerLink
Log in

Basic concepts of multiple tests — A survey

  • Survey Article
  • Published:
Statistical Papers Aims and scope Submit manuscript

Abstract

It is the purpose of this paper to review the main aspects related to multiple test problems. This concerns among others the particularities of multiple tests as for instance the formulation of restrictions to avoid inconsistent decisions and of criteria to control for a multiple type I error rate. In addition, the basic principles for constructing multiple tests are introduced and their properties are summarized. The paper closes with giving a rough idea of further special multiple test problems and their corresponding test procedures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alt, R. (1988) Hierarchical test problems and the closure principle. In: P. Bauer, G. Hommel, E. Sonnermann (eds.):Multiple Hypotheses Testing. Springer-Verlag, Berlin, 163–176

    Google Scholar 

  • Banik, N., Köhne, K., andBauer, P. (1996) On the power of Fisher's combination test for two stage sampling in the presence of nuisance parameters.Biom. J. 38, 25–37

    Article  MATH  Google Scholar 

  • Bauer, P. (1987) On the assessment of the performance of multiple procedures.Biom. J. 29, 895–906

    Article  MATH  Google Scholar 

  • Bauer, P. (1989) Multistage testing with adaptive designs (with discussion).Biom. und Inf. in Med. und Biol. 20, 130–148.

    Google Scholar 

  • Bauer, P. (1997) A note on multiple testing procedures in dose finding.Biometrics 53, 1125–1128.

    Article  MATH  Google Scholar 

  • Bauer, P., Hackl, P., Hommel, G., andSonnemann, E. (1986) Multiple testing of pairs of one-sided hypotheses.Metrika 33, 121–127.

    Article  MathSciNet  Google Scholar 

  • Bauer, P., andHackl, P. (1987) Multiple testing in a set of nested hypotheses.Statistics 18, 345–349.

    Article  MathSciNet  MATH  Google Scholar 

  • Bauer, P., Pötscher, B.M., andHackl, P. (1988) Model selection by multiple test procedures.Statistics 19, 39–44.

    Article  MathSciNet  MATH  Google Scholar 

  • Bauer, P., andKöhne, K. (1994) Evaluation of experiments with adaptive interim analyses.Biometrics 50, 1029–1041.

    Article  MATH  Google Scholar 

  • Bauer, P., andRöhmel, J. (1995) An adaptive method for establishing a dose response relationship.Stat. in Med. 14, 1595–1607.

    Article  Google Scholar 

  • Begun, J.M., andGabriel, K.R. (1981) Closure of the Newman-Keuls multiple comparison procedure.J. Amer. Statist. Assoc. 76, 241–245.

    Article  MathSciNet  Google Scholar 

  • Benjamini, Y., andHochberg, Y. (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing.J. Roy. Statist. Soc., Ser. B 57, 289–300.

    MathSciNet  MATH  Google Scholar 

  • Bofinger, E. (1987) Step-down procedures for comparison with a control.Austral. J. Statist. 29, 348–364

    Article  MathSciNet  MATH  Google Scholar 

  • Budde, M., andBauer, P. (1989) Multiple test procedures in clinical dose finding studies.J. Amer. Statist. Assoc. 84, 792–796.

    Article  Google Scholar 

  • Cheng, P.H., andMeng, C.Y.K. (1995) A new formula for tail probabilities of Dunnett'sT with unequal sample sizes.Commun. Statist., Theory Methods 24, 523–532.

    Article  MATH  Google Scholar 

  • Cox, D.R. (1965) A remark on multiple comparison methods.Technometrics 2 149–156.

    Google Scholar 

  • Dixon, D.O., andDuncan, D.B. (1975) Minimum Bayes riskt-intervals for multiple comparisons.J. Amer. Statist. Assoc. 70, 822–831.

    Article  MathSciNet  MATH  Google Scholar 

  • Duncan, D.B. (1961) Bayes rules for a common multiple comparisons problem and related Student-t problems.Ann. Math. Statist. 32, 1013–1033.

    Article  MathSciNet  MATH  Google Scholar 

  • Duncan, D.B. (1965) A Bayesian approach to multiple comparisons.Technometrics 7, 171–222.

    Article  MATH  Google Scholar 

  • Duncan, D.B. (1975)t tests and intervals for comparisons suggested by the data.Biometrics 31, 339–359.

    Article  MathSciNet  MATH  Google Scholar 

  • Duncan, D.B., andGodbold, J.H. (1979) Approximatek-ratiot tests for differences between unequally replicated treatments.Biometrics 35, 749–756.

    Article  MATH  Google Scholar 

  • Dunnett, C.W. (1955) A multiple comparison procedure for comparing several treatments with a control.J. Amer. Statist. Assoc. 50, 1096–1121

    Article  MATH  Google Scholar 

  • Dunnett, C.W. (1964) New tables for multiple comparisons with a control.Biometrics 20, 482–491.

    Article  MATH  Google Scholar 

  • Dunnett, C.W., andTamhane, A.C. (1992) A step-up multiple test procedure.J. Amer. Statist. Assoc. 87, 162–170.

    Article  MathSciNet  MATH  Google Scholar 

  • Efron, B. (1979) Bootstrap methods: Another look at the jackknife.Ann. Statist. 7, 1–26.

    Article  MathSciNet  MATH  Google Scholar 

  • Finner, H. (1988a)Multiple Spannweitentests. Dissertation, University of Trier, Department of Mathematics

  • Finner, H. (1988b) Abgeschlossene multiple Spannweitentests. In: P. Bauer, G. Hommel, E. Sonnemann (eds.):Multiple Hypotheses Testing. Springer-Verlag, Berlin, 10–32.

    Google Scholar 

  • Finner, H. (1988c) Multiple Tests und Fehler III. Art. In: P. Bauer, G. Hommel, E. Sonnemann (eds.):Multiple Hypotheses Testing, Springer-Verlag, Berlin, 144–153.

    Google Scholar 

  • Finner, H. (1990) On the modifiedS-method and directional errors.Commun. Statist., Theory Methods 19, 41–53.

    Article  MathSciNet  MATH  Google Scholar 

  • Finner, H., Hayter, A.J., and Roters, M. (1993) On the joint distribution function of order statistics with reference to step-up multiple test procedures.Forschungsbericht Nr. 93-19, Mathematik/Informatik, University of Trier

  • Fisher, R.A. (1935)The Design of Experiments. Oliver & Boyd, Edingburgh, London

    Google Scholar 

  • Gabriel, K.R. (1969) Simultaneous test procedures—some theory of multiple comparisons.Ann. Math. Statist. 40, 224–250.

    Article  MathSciNet  MATH  Google Scholar 

  • Gabriel, K.R. (1978) A simple method of multiple comparisons of means.J. Amer. Statist. Assoc. 73, 724–729.

    Article  MATH  Google Scholar 

  • Gather, U., Pawlitschko, J., andPigeot, I. (1997) Unbiasedness of multiple tests.Scand. J. Statist. 23, 117–127.

    MathSciNet  Google Scholar 

  • Gather, U., Pawlitschko, J., andPigeot, I. (1997) A note on invariance of multiple tests.Statist. Neerl. 51, 366–372.

    MathSciNet  MATH  Google Scholar 

  • Hawkins, D.M. (1980)Identification of Outliers. Chapman & Hall, London.

    MATH  Google Scholar 

  • Hochberg, Y. (1974) Some conservative generalization of theT-method in simultaneous inference.J. Mult. Anal. 4, 224–234.

    Article  MathSciNet  MATH  Google Scholar 

  • Hochberg, Y. (1988) A sharper Bonferroni procedure for multiple tests of significance.Biometrika 75, 800–802.

    Article  MathSciNet  MATH  Google Scholar 

  • Hochberg, Y., Tamhane, A.C. (1987)Multiple Comparison Procedures. John Wiley & Sons, New York

    Google Scholar 

  • Holm, S. (1979) A simple sequentially rejective multiple test procedure.Scand. J. Statist. 6, 65–70.

    MathSciNet  MATH  Google Scholar 

  • Holm, S. (1985) Multiple test unbiasedness. In: M. Iosifescu, S. Grigorescu, T. Postelnicu (eds.):Proceedings of the Seventh Conference on Probability Theory. Editura Academiei, Bucuresti; VNU Science Press, Utrecht, 183–193.

    Google Scholar 

  • Hommel, G. (1986) Multiple test procedures for arbitrary dependence structures.Metrika 33, 321–336.

    Article  MathSciNet  MATH  Google Scholar 

  • Hommel, G. (1988) A stagewise rejective multiple test procedure based on a modified Bonferroni test.Biometrika 75, 383–386.

    Article  MATH  Google Scholar 

  • Hommel, G. (1989) A comparison of two modified Bonferroni procedures.Biometrika 76, 624–625

    Article  MathSciNet  MATH  Google Scholar 

  • Hommel, G., andBernhard, G. (1992) Multiple hypotheses testing. In: J. Antoch (ed.):Computational Aspects of Model Choice. Physica-Verlag, Heidelberg, 211–235

    Google Scholar 

  • Hommel, G., andBernhard, G. (1994) A multiple test procedure for nested systems of hypotheses. In: P. Dirschedl, R. Ostermann (eds):Computational Statistics. Physica-Verlag, Heidelberg, 419–433.

    Google Scholar 

  • Hommel, G., andHoffmann, T. (1988) Controlled uncertainty. In: P. Bauer, G. Hommel, E. Sonnemann (eds.):Multiple Hypotheses Testing. Springer-Verlag, Berlin, 154–161

    Google Scholar 

  • Hoover, D.R. (1991) Simultaneous comparisons of multiple treatments to two (or more) controls.Biom. J. 33, 913–921

    Article  Google Scholar 

  • Horn, M., Vollandt, R. (1995)Multiple Tests und Auswahlverfahren. Gustav Fischer Verlag, Stuttgart

    MATH  Google Scholar 

  • Hothorn, L.A., Neuhäuser, M., andKoch, H.-F. (1997) Analysis of randomized dose-finding-studies: closure test modifications based on multiple contrast tests.Biom. J. 39, 467–479.

    Article  MATH  Google Scholar 

  • Hsu, J.C. (1984) Ranking and selection and multiple comparisons with the best. In: T.J. Santner, A.C. Tamhane (eds.):Design of Experiments: Ranking and Selection. Marcel Dekker, New York.

    Google Scholar 

  • Hsu, J.C. (1985) A note on multiple comparisons with the best. In:45th Session of the International Statistical Institute, Book 2, 445–446.

  • Hsu, J.C. (1996)Multiple Comparisons: Theory and Methods. Chapman & Hall, London

    MATH  Google Scholar 

  • Keuls, M. (1952) The use of the “studentized range” in connection with an analysis of variance.Euphytica 1, 112–122

    Article  Google Scholar 

  • Lehmacher, W. (1988a) Multiples Testen bei Verlaufskurvenanalysen—T 2—Tests und Folgeanalysen mitt-Tests. In: H.-K. Selbmann, K. Dietz (eds.):Medizinische Informationsverarbeitung und Epidemiologie im Dienste der Gesundheit. Proceedings der 32. Jahrestagung der GMDS, Tübingen, 1987. Springer-Verlag, Heidelberg, 93–96.

    Google Scholar 

  • Lehmacher, W. (1988b) Analyse vonK Stichproben von Verlaufskurven. In: P. Bauer, G. Hommel, E. Sonnemann (eds.):Multiple Hypotheses Testing. Springer-Verlag, Berlin, 33–47

    Google Scholar 

  • Hehmacher, W. (1989) Schrittweises Testen a priori geordneter Hypothesen mit Kontrolle des experimentweisen Niveaus.Unpublished manuscript

  • Lehmacher, W., Wassmer, G., andReitmeir, P. (1991) Procedures for two-sample comparisons with multiple endopoints controlling the experimentwise error rate.Biometrics 47, 511–521

    Article  Google Scholar 

  • Lehmann, E.L. (1957a) A theory of some multiple decision problems. I..Ann. Math. Statist. 28, 1–25

    Article  MathSciNet  MATH  Google Scholar 

  • Lehmann, E.L. (1957b) A theory of some multiple decision problems. II.Ann. Math. Statist. 28, 547–572

    Article  MathSciNet  MATH  Google Scholar 

  • Liu, W. (1996) Multiple tests of a non-hierarchical finite family of hypotheses.J. Roy. Statist. Soc., Ser. B 58, 455–461

    MathSciNet  MATH  Google Scholar 

  • Marcus, R., Peritz, E., andGabriel, K.R. (1976) On closed testing procedures with special reference to ordered analysis of variance.Biometrika 63, 655–660

    Article  MathSciNet  MATH  Google Scholar 

  • Maurer, W., Hothorn, L.A., andLehmacher, W. (1995) Multiple comparisons in drug clinical trials and preclinical assays: a-priori ordered hypotheses. In: J. Vollmar (ed.):Biometrie in der chemisch-pharmazeutischen Industrie. Gustav Fischer Verlag, Stuttgart

    Google Scholar 

  • Maurer, W., andMellein, B. (1988) On new multiple tests based on independentp-values and the assessment of their power. In: P. Bauer, G. Hommel, E. Sonnemann (eds.):Multiple Hypotheses Testing. Springer-Verlag, Berlin, 48–66

    Google Scholar 

  • Miller, R.G. (1981)Simultaneous Statistical Inference, 2nd ed. Springer-Verlag, New York

    MATH  Google Scholar 

  • Naik, U.D. (1975) Some selection rules for comparingp processes with a standard.Commun. Statist. Theory and Methods 4, 519–535

    Article  MathSciNet  Google Scholar 

  • Newman, D. (1939) The distribution of the range in samples from a normal population, expressed in terms of an independent estimate of standard deviation.Biometrika 31, 20–30

    MathSciNet  MATH  Google Scholar 

  • O'Brien, P.C. (1984) Procedures for comparing samples with multiple endpoints.Biometrics 40 1079–1087

    Article  MathSciNet  Google Scholar 

  • Pawlitschko, J. (1992)Multiples Testen unter besonderer Berücksichtigung des Abschlußprinzips. Diploma Thesis, University of Dortmund, Department of Statistics

  • Pearce, S.C. (1983) The monstrous regiment of mathematicians.The Statistician 32, 375–378

    Article  Google Scholar 

  • Pigeot, I. (1993)Multiple Testtheorie in der Ausreißererkennung. Habilitationsschrift, University of Dortmund, Department of Statistics

  • Pigeot, I., andGather, U. (1994) Identifikation von Ausreißern als multiples Test-problem. In: S.J. Pöppl, H.-G. Lipinski, T. Mansky (eds.):Medizinische Informatik. Ein integrierender Teil arztunterstützender Technologien. Proceedings der 38. Jahrestagung der GMDS, Lübeck, 1993. MMV Medizin Verlag, München, 474–477

    Google Scholar 

  • Pocock, S.J., Geller, N.L., andTsiatis, A.A. (1987) The analysis of multiple endpoints in clinical trials.Biometrics 43, 487–498

    Article  MathSciNet  Google Scholar 

  • Rom, D.M. (1990) A sequentially rejective test procedure based on a modified Bonferroni inequality.Biometrika 77, 663–665

    Article  MathSciNet  Google Scholar 

  • Rom, D.M., andHolland, B. (1995) A new closed multiple testing procedure for hierarchical families of hypotheses.J. Statist. Planning Inf. 46, 265–275

    Article  MathSciNet  MATH  Google Scholar 

  • Rosner, B. (1975) On the detection of many outliers.Technometrics 17, 221–227

    Article  MathSciNet  MATH  Google Scholar 

  • Roy, S.N. (1953) On a heuristic method of test construction and its use in multivariate analysis.Ann. Math. Statist. 24, 220–238

    Article  MATH  Google Scholar 

  • Rüger, B. (1978) Das maximale Signifikanzniveau des Tests “LehneH 0 ab, wennk untern Tests zur Ablehnung führen”.Metrika 25, 171–178

    Article  MathSciNet  MATH  Google Scholar 

  • Savin, N.E. (1984) Multiple hypothesis testing. In: Z. Griliches, M.D. Intriligator (eds.):Handbook of Econometrics, Vol. 2. Elsevier Science Publishers B.V., Amsterdam, 827–879

    Google Scholar 

  • Scheffé, H. (1953) A method for judging all contrasts in the analysis of variance.Biometrika 40, 87–104.

    MathSciNet  MATH  Google Scholar 

  • Shaffer, J.P. (1980) Control of directional errors with stagewise multiple test procedures.Ann. Stat. 8, 1342–1348

    Article  MathSciNet  MATH  Google Scholar 

  • Shaffer, J.P. (1986) Modified sequentially rejective multiple test procedures.J. Amer. Statist. Assoc. 81, 826–831

    Article  MATH  Google Scholar 

  • Simes, R.J. (1986) An improved Bonferroni procedure for multiple tests of significance.Biometrika 73, 751–754

    Article  MathSciNet  MATH  Google Scholar 

  • Sonnemann, E. (1981) Tests zum multiplen Niveau α.ROeS-Seminar, Bad Ischl

  • Sonnemann, E. (1982) Allgemeine Lösungen multipler Testprobleme.EDV in Med. Biol. 13, 120–128

    MathSciNet  Google Scholar 

  • Spjøtvoll, E. (1972) On the optimality of some multiple comparison procedures.Ann. Math. Statist. 43, 398–411

    Article  MathSciNet  MATH  Google Scholar 

  • Spjøtvoll, E. (1974) Multiple testing in the analysis of variance.Scand. J. Statist. 1, 97–114

    MathSciNet  Google Scholar 

  • Spjøtvoll, E., andStoline, M.R. (1973) An extension of theT-method of multiple comparisons to include the cases with unequal sample sizes.J. Amer. Statist. Assoc. 68, 975–978

    Article  MathSciNet  MATH  Google Scholar 

  • Stefánsson, G., Kim, W.-C., andHsu, J.C. (1988) On confidence sets in multiple comparisons. In: S.S. Gupta, J.O. Berger (eds.):Statistical Decision Theory and Related Topics IV, Vol. 2. Springer, New York, 89–104

    Google Scholar 

  • Tamhane, A.C., Hochberg, Y., andDunnett, C.W. (1996) Multiple test procedures for dose findings.Biometrics 52, 21–37

    Article  MATH  Google Scholar 

  • Tippett, L.H.G. (1952)The Methods of Statistics.4th ed. Wiley, New York

    MATH  Google Scholar 

  • Troendle, J.F. (1995) A stepwise resampling method of multiple hypothesis testing.J. Amer. Statist. Assoc. 90, 370–378

    Article  MathSciNet  MATH  Google Scholar 

  • Tukey, J.W. (1953) The problem of multiple comparisons.Unpublished manuscript

  • Victor, N. (1982) Exploratory data analysis and clinical research.Meth. Inf. Med. 21, 53–54

    Google Scholar 

  • Waller, R.A., andDuncan, D.B. (1969) A Bayes rule for the symmetric multiple comparisons problem.J. Amer. Statist. Assoc. 64, 1484–1503

    Article  MathSciNet  Google Scholar 

  • Waller, R.A., andDuncan, D.B. (1972) A corrigendum to “A Bayes rule for the symmetric multiple comparisons problem”.J. Amer. Statist. Assoc.,67, 253–255

    Article  Google Scholar 

  • Waller, R.A., andDuncan, D.B. (1974) A Bayes rule for the symmetric multiple comparisons problem II.J. Ann. Inst. Statist. Math. 26, 247–264

    Article  MathSciNet  MATH  Google Scholar 

  • Wassmer, G. (1997) A technical note on the power determination for Fisher's combination test.Biom. J. 39, 831–838

    Article  MathSciNet  MATH  Google Scholar 

  • Wassmer, G. (1998) A comparison of two methods for adaptive interim analyses in clinical trials.Biometrics 54, 696–705

    Article  MATH  Google Scholar 

  • Wassmer, G., Reitmeir, P., Kieser, M., and Lehmacher, W. (1999) Procedures for testing multiple endpoints in clinical trials: an overview. To appear inJ. Statist. Planning Inf.

  • Welsch, R.E. (1977) Stepwise multiple comparison procedures.J. Amer. Statist. Assoc. 72, 566–575

    Article  MathSciNet  MATH  Google Scholar 

  • Westfall, P.H. (1985) Simultaneous small-sample multivariate Bernoulli confidence intervals.Biometrics 41, 1001–1013

    Article  MathSciNet  MATH  Google Scholar 

  • Westfall, P.H., andYoung, S.S. (1989)P value adjustments for multiple tests in multivariate binomial models.J. Amer. Statist. Assoc. 84, 780–786

    Article  MathSciNet  Google Scholar 

  • Westfall, P.H., andYoung, S.S. (1993)Resampling-Based Multiple Testing. John Wiley & Sons, New York

    Google Scholar 

  • Zieliński, W. (1992) Monte Carlo comparison of multiple comparison procedures.Biom. J. 34, 291–296

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institut für Statistik, Ludwig-Maximilians-Universität München, Ludwigstraße 33, 80539, München, Germany

    Iris Pigeot

Authors
  1. Iris Pigeot
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pigeot, I. Basic concepts of multiple tests — A survey. Statistical Papers 41, 3–36 (2000). https://doi.org/10.1007/BF02925674

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02925674

Key Words

Search

Navigation