Statistical Papers

, Volume 43, Issue 2, pp 197–235 | Cite as

Small sample properties of tests on homogeneity in one—way Anova and Meta—analysis

  • Joachim Hartung
  • Dogan Argaç
  • Kepher H. Makambi


In the present Monte Carlo study, the empirical Type I error properties and power of several statistics for testing the homogeneity hypothesis in a one—way classification are examined in the case of small sample sizes. We compared these tests under several scenarios: normal populations under heterogeneous variances, nonnormal populations under homogeneous variances, nonnormal populations under heterogeneous variances, balanced and unbalanced sample sizes, and increasing number of populations. Overall, none of the tests considered is uniformly dominating the others. Under normality and variance heterogeneity, the Brown—Forsythe and the Welch test perform well over a wide range of parameter configurations, the modified Brown-Forsythe test by Mehrotra keeps generally the level, but other tests may also perform well, depending on the constellation of the parameters under study. The Welch test becomes liberal when the sample sizes are small and the number of populations is large. We propose a modified version of Welch’s test that keeps the nominal level in these cases. With the understanding that methods are unacceptable if they have Type I error rates that are too high, only the testing procedure associated with the modified Brown-Forsythe test can be recommended both for normal and nonnormal data. Under normality, the modified Welch test can also be recommended.

Key words

meta—analysis balanced and unbalanced sample sizes homogeneous and heterogeneous variances nonnormality 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Asiribo, O., Gurland, J. (1990). Coping with variance heterogeneity. Commun. Statist. Theory Meth., 19, 4029–4048.CrossRefGoogle Scholar
  2. Bockenhoff, A., Hartung, J. (1998). Some corrections of the significance level in meta-analysis. Biometrical Journal, 40, 937–947.CrossRefMathSciNetGoogle Scholar
  3. Box, G. E. P. (1954). Some theorems on quadratic forms applied in the study of analysis of variance problems, I. Effect of inequality of variance in the one-way classification. Annals of Mathematical Statistics, 25, 290–403.zbMATHCrossRefMathSciNetGoogle Scholar
  4. Brown, M. B., Forsythe, A. B. (1974). The small sample behavior of some statistics which test the equality of several means. Technometrics, 16, 129–132.zbMATHCrossRefMathSciNetGoogle Scholar
  5. Chalmers, T. C. (1991). Problems induced by meta-analyses. Statistics in Medicine, 10, 971–980.CrossRefGoogle Scholar
  6. Cochran, W. G. (1937). Problems arising in the analysis of a series of similar experiments. J. Roy. Stat. Soc. Supp., 4, 102–118.CrossRefGoogle Scholar
  7. Conover, W. J., Iman, R. L. (1981). Rank transformations as a bridge between parametric and nonparametric statistics. American Statistician, 35, 124–129.zbMATHCrossRefGoogle Scholar
  8. De Beuckelaer, A. (1996). A closer examination on some parametric alternatives to the ANOVA F-test. Statistical Papers, 37, 291–305.zbMATHCrossRefGoogle Scholar
  9. Fligner, M. A. (1981). Comment on rank transformations as a bridge between parametric and nonparametric statistics. American Statistician, 35, 131–133.CrossRefGoogle Scholar
  10. Hardy, R. J., Thompson, S. G. (1998). Detecting and describing heterogeneity in meta-analysis. Statistics in Medicine, 17, 841–856.CrossRefGoogle Scholar
  11. Hartung, J., Knapp, G. (2000). On tests of the overall treatment effect in the meta-analysis with normally distributed responses. Statistics in Medicine, to appear.Google Scholar
  12. James, G. S. (1951). The comparison of several groups of observations when the ratios of population variances are unknown. Biometrika, 38, 324–329.zbMATHMathSciNetGoogle Scholar
  13. Keselman, H. J., Wilcox, R. R. (1999). The ’improved’ Brown and Forsythe test for mean equality: some things can’t be fixed. Commun. Statist. Simula., 28, 687–698.zbMATHCrossRefGoogle Scholar
  14. Lehmann, E. L. (1975) Nonparametrics. Holden Day, San Francisco.zbMATHGoogle Scholar
  15. Lehmann, E. L. (1986) Testing Statistical Hypotheses. 2nd edn., Wiley, New York.zbMATHGoogle Scholar
  16. Li, Y., Shi, L., Roth, H. D. (1994). The bias of the commonly-used estimate of variance in meta-analysis. Commun. Statist.-Theory Meth., 23, 1063–1085.zbMATHCrossRefMathSciNetGoogle Scholar
  17. Mehrotra, D. V. (1997). Improving the Brown-Forsythe solution to the generalized Behrens-Fisher problem. Commun. Statist.— Simula., 26, 1139–1145.zbMATHCrossRefGoogle Scholar
  18. Noether, G. E. (1981). Comment on rank transformations as a bridge between parametric and nonparametric statistics. American Statistician, 35, 129–130.CrossRefGoogle Scholar
  19. Normand, S. T. (1999). Meta-analysis: Formulating, evaluating, combining, and reporting. Statistics in Medicine, 18, 321–359.CrossRefGoogle Scholar
  20. Patel, J. K., Kapadia, C. P., Owen, D. B. (1976) Handbook of Statistical Distributions. Marcel Dekker, New York.zbMATHGoogle Scholar
  21. Scheffe, H. (1959) The Analysis of Variance. Wiley, New York.zbMATHGoogle Scholar
  22. Welch, B. L. (1951). On the comparison of several mean values: An alternative approach. Biometrika, 38, 330–336.zbMATHMathSciNetGoogle Scholar
  23. Whitehead, A., Whitehead, J. (1991). A general parametric approach to the meta-analysis of randomized clinical trials. Statistics in Medicine, 10, 1665–1677.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2002

Authors and Affiliations

  • Joachim Hartung
    • 1
  • Dogan Argaç
    • 1
  • Kepher H. Makambi
    • 1
  1. 1.Department of StatisticsUniversity of DortmundDortmundGermany

Personalised recommendations