How accurate and powerful are randomization tests in multivariate analysis of variance?
Multivariate analysis of variance, based on randomization (permutation) tests, has become an important tool for ecological data analyses. However, a comprehensive evaluation of the accuracy and power of available methods is still lacking. This is a thorough examination of randomization tests for multivariate group mean differences. With simulated data, the accuracy and power of randomization tests were evaluated using different test statistics in one-factor multivariate analysis of variance (MANOVA). The evaluations span a wide spectrum of data types, including specified and unspecified (field data) distributional properties, correlation structures, homogeneous to very heterogeneous variances, and balanced and unbalanced group sizes. The choice of test statistic strongly affected the results. Sums of squares between groups (Qb) computed on Euclidean distances (Qb-EUD) gave better accuracy. Qb on Bray-Curtis, Manhattan or Chord distances, the multiresponse permutation procedure (MRPP) and the sum of univariate ANOVA F produced severely inflated type I errors under increasing variance heterogeneity among groups, a common scenario in ecological data. Despite pervasive claims in the ecological literature, the evidence thus suggests caution when using test statistics other than Qb-EUD.
KeywordsCount data Distance-based MANOVA Distribution free MRPP Neyman-Pearson lemma Permutation tests Type I error Type II error
Analysis of Similarity
Analysis of variancev
Likelihood-ratio test assuming independence of variables
Multivariate analysis of variance
Multiresponse permutation procedure
Permutational multivariate analysis of variance
Sums of squares between groups
Within-groups sum of squares
Univariate ANOVA F statistic summed over all variables
- Anderson, M.J. 2001. A new method for non-parametric multivariate analysis of variance. Austral Ecol. 26: 32–46.Google Scholar
- Bradley, J.V. 1968. Distribution-Free Statistical Tests. Prentice-Hall, Englewood Cliffs.Google Scholar
- Edgington, E.S. 1969b. Statistical Inference: The Distribution-Free Approach. McGraw-Hill, New York.Google Scholar
- Edgington, E.S. 1987. Randomization Tests. Marcel Dekker, New York.Google Scholar
- Fisher, R.A. 1951. The Design of Experiments. 6th ed. Oliver and Boyd, Edinburgh.Google Scholar
- Galassi, M., Davies, J., Theiler, J., Gough, B., Jungman, G., Booth, M. and Rossi, F. 2003. GNU Scientific Library Reference Manual (2nd Ed). Available also at https://doi.org/www.gnu.org/soft-ware/gsl/.
- Hope, A.C.A. 1968. A simplified Monte Carlo significance test procedure. J. R. Stat. Soc. 30: 582–598.Google Scholar
- Kempthorne, O. 1955. The randomization theory of experimental inference. J. Amer. Statistical Assoc. 50: 946–967.Google Scholar
- Legendre, L. and Legendre, P. 1998. Numerical Ecology 2nd ed. Elsevier, New York.Google Scholar
- Manly, B.F.J. 2007. Randomization, Bootstrap, and Monte Carlo Methods in Biology. Chapman & Hall/ CRC, Boca Raton.Google Scholar
- Orlóci, L. 1978. Multivariate Analysis in Vegetation Research. Junk, The Hague.Google Scholar
- Orlóci, L. 1993. The complexities and scenarios of ecosystem analysis. In: Patil, G.P. and Rao, C.R. (eds.) Multivariate Environmental Statistics, Elsevier, Amsterdam. pp. 423–432.Google Scholar
- Pillar, V.D., Jacques, A.V.A. and Boldrini, I.I. 1992. Fatores de ambiente relacionados à variação da vegetação de um campo natural. Pesqui. Agropecu. Bras. 27: 1089–1101.Google Scholar
- Podani, J. 2000. Introduction to the Exploration of Multivariate Biological Data. Backuys Publishers, Leiden.Google Scholar
This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.