Abstract
This chapter introduces permutation methods for multiple independent variables; that is, completely-randomized designs. Included in this chapter are six example analyses illustrating computation of exact permutation probability values for multi-sample tests, calculation of measures of effect size for multi-sample tests, the effect of extreme values on conventional and permutation multi-sample tests, exact and Monte Carlo permutation procedures for multi-sample tests, application of permutation methods to multi-sample rank-score data, and analysis of multi-sample multivariate data. Included in this chapter are permutation versions of Fisher’s F test for one-way, completely-randomized analysis of variance, the Kruskal–Wallis one-way analysis of variance for ranks, the Bartlett–Nanda–Pillai trace test for multivariate analysis of variance, and a permutation-based alternative for the four conventional measures of effect size for multi-sample tests: Cohen’s \(\hat {d}\), Pearson’s η 2, Kelley’s \(\hat {\eta }^{2}\), and Hays’ \(\hat {\omega }^{2}\).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
In some disciplines tests on multiple independent samples are known as between-subjects tests and tests for multiple dependent or related samples are known as within-subjects tests.
- 2.
The terms MS Between and MS Within are only one set of descriptive labels for the numerator and denominator of the F-ratio test statistic. MS Between is often replaced by either MS Treatment or MS Factor and MS Within is often replaced by MS Error.
- 3.
It is well known that Kelley’s correlation ratio is not unbiased, but since the title of Truman Kelley’s 1935 article was “An unbiased correlation ratio measure,” the label has persisted.
- 4.
Since the sizes of the treatment groups are not equal, the average value of \(\bar {n} = 3.3333\) is used for both Cohen’s \(\hat {d}\) measure of effect size and Hays’ \(\hat {\omega }_{\text{R}}^{2}\) measure of effect size for a random-effects model. In cases where the treatment-group sizes differ greatly, a weighted average recommended by Haggard is often adopted [6].
- 5.
For a one-way completely-randomized analysis of variance, a fixed-effects model and a random-effects model yield the same F-ratio, but measures of effect size can differ under the two models.
References
Bartlett, M.S.: A note on tests of significance in multivariate analysis. Proc. Camb. Philos. Soc. 34, 33–40 (1939)
Berry, K.J., Mielke, P.W., Johnston, J.E.: Permutation Statistical Methods: An Integrated Approach. Springer, Cham (2016)
Boik, R.J.: The Fisher–Pitman permutation test: a non-robust alternative to the normal theory F test when variances are heterogeneous. Brit. J. Math. Stat. Psychol. 40, 26–42 (1987)
Box, J.F.: R. A. Fisher: The Life of a Scientist. Wiley, New York (1978)
Festinger, L.: The significance of differences between means without reference to the frequency distribution function. Psychometrika 11, 97–105 (1946)
Haggard, E.A.: Intraclass Correlation and the Analysis of Variance. Dryden, New York (1958)
Haldane, J.B.S., Smith, C.A.B.: A simple exact test for birth-order effect. Ann. Eugenic. 14, 117–124 (1948)
Hall, N.S.: R. A. Fisher and his advocacy of randomization. J. Hist. Biol. 40, 295–325 (2007)
Hotelling, H.: A generalized T test and measure of multivariate dispersion. In: Neyman, J. (ed.) Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, vol. II, pp. 23–41. University of California Press, Berkeley (1951)
Hotelling, H., Pabst, M.R.: Rank correlation and tests of significance involving no assumption of normality. Ann. Math. Stat. 7, 29–43 (1936)
Johnston, J.E., Berry, K.J., Mielke, P.W.: Permutation tests: precision in estimating probability values. Percept. Motor Skill. 105, 915–920 (2007)
Kruskal, W.H., Wallis, W.A.: Use of ranks in one-criterion variance analysis. J. Am. Stat. Assoc. 47, 583–621 (1952). [Erratum: J. Am. Stat. Assoc. 48, 907–911 (1953)]
Lawley, D.N.: A generalization of Fisher’s z test. Biometrika 30, 180–187 (1938)
Lawley, D.N.: Corrections to “A generalization of Fisher’s z test”. Biometrika 30, 467–469 (1939)
Mann, H.B., Whitney, D.R.: On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18, 50–60 (1947)
Nanda, D.N.: Distribution of the sum of roots of a determinantal equation. Ann. Math. Stat. 21, 432–439 (1950)
Olson, C.L.: On choosing a test statistic in multivariate analysis of variance. Psychol. Bull. 83, 579–586 (1976)
Olson, C.L.: Practical considerations in choosing a MANOVA test statistic: a rejoinder to Stevens. Psychol. Bull. 86, 1350–1352 (1979)
Pillai, K.C.S.: Some new test criteria in multivariate analysis. Ann. Math. Stat. 26, 117–121 (1955)
Roy, S.N.: On a heuristic method of test construction and its use in multivariate analysis. Ann. Math. Stat. 24, 220–238 (1953)
Roy, S.N.: Some Aspects of Multivariate Analysis. Wiley, New York (1957)
Snedecor, G.W.: Calculation and Interpretation of Analysis of Variance and Covariance. Collegiate Press, Ames (1934)
Tabachnick, B.G., Fidell, L.S.: Using Multivariate Statistics, 5th edn. Pearson, Boston (2007)
van der Reyden, D.: A simple statistical significance test. Rhod. Agric. J. 49, 96–104 (1952)
Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bull. 1, 80–83 (1945)
Wilks, S.S.: Certain generalizations in the analysis of variance. Biometrika 24, 471–494 (1932)
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Berry, K.J., Johnston, J.E., Mielke, P.W. (2019). Completely-Randomized Designs. In: A Primer of Permutation Statistical Methods. Springer, Cham. https://doi.org/10.1007/978-3-030-20933-9_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-20933-9_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20932-2
Online ISBN: 978-3-030-20933-9
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)