Abstract
Repeated measures ANOVA and mixed-model designs are the main classes of experimental designs used in psychology. The usual analysis relies on some parametric assumptions (typically Gaussianity). In this article, we propose methods to analyze the data when the parametric conditions do not hold. The permutation test, which is a non-parametric test, is suitable for hypothesis testing and can be applied to experimental designs. The application of permutation tests in simpler experimental designs such as factorial ANOVA or ANOVA with only between-subject factors has already been considered. The main purpose of this paper is to focus on more complex designs that include only within-subject factors (repeated measures) or designs that include both within-subject and between-subject factors (mixed-model designs). First, a general approximate permutation test (permutation of the residuals under the reduced model or reduced residuals) is proposed for any repeated measures and mixed-model designs, for any number of repetitions per cell, any number of subjects and factors and for both balanced and unbalanced designs (all-cell-filled). Next, a permutation test that uses residuals that are exchangeable up to the second moment is introduced for balanced cases in the same class of experimental designs. This permutation test is therefore exact for spherical data. Finally, we provide simulations results for the comparison of the level and the power of the proposed methods.
Similar content being viewed by others
References
Anderson M, Ter Braak C (2003) Permutation tests for multi-factorial analysis of variance. J Stat Comput Simul 73(2):85–113
Basso D (2009) Permutation tests for stochastic ordering and ANOVA: theory and applications with R. Springer, New York
Basso D, Salmaso L (2006) A discussion of permutation tests conditional to observed responses in unreplicated \(2^m\) full factorial designs. Commun Stat Theory Methods 35(1):83–97. doi: 10.1080/03610920500437277
Basso D, Chiarandini M, Salmaso L (2007) Synchronized permutation tests in replicated IxJ designs. J Stat Plan Inference 137(8):2564–2578. doi:10.1016/j.jspi.2006.04.016
Cardinal R, Aitken M (2006) ANOVA for the behavioural sciences researcher. Lawrence Erlbaum Associates, New Jersey
Cornfield J, Tukey JW (1956) Average values of mean squares in factorials. Ann Math Stat 27(4):907–949. doi:10.1214/aoms/1177728067
Field CA, Welsh A (2007) Bootstrapping clustered data. J Roy Stat Soc B 69:369–390
Fisher R (1935) The design of experiments. Oliver & Boyd, Edinburgh
Freedman D, Lane D (1983) A nonstochastic interpretation of reported significance levels. J Bus Econ Stat 1(4):292–298
Good P (1994) Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer, New York
Hirotsu C (1979) An F approximation and its application. Biometrika 66(3):577–584
IBM Corp (2013) IBM SPSS statistics for Windows, Version 22.0. Armonk, NY
Jung BC, Jhun M, Song SH (2006) A new random permutation test in ANOVA models. Stat Pap 48:47–62
Keppel G (1991) Design and analysis: a researcher’s handbook. Prentice-Hall, Englewood Cliffs
Kherad-Pajouh S (2011) Permutation tests for experimental designs, with extension to simultaneous EEG signal analysis. PhD thesis, University of Geneva. url: http://archive-ouverte.unige.ch/unige:15743
Kherad-Pajouh S, Renaud O (2010) An exact permutation method for testing any effect in balanced and unbalanced fixed effect anova. Comput Stat Data Anal 54(7):1881–1893. http://dx.doi.org/10.1016/j.csda.2010.02.015
Kirk R (1994) Experimental design: procedures for the behavioral sciences. Wadsworth Publishing Co Inc., Belmont
Mazzaro D, Pesarin F, Salmaso L (2001) Permutation tests for effects in unbalanced repeated measures factorial designs. In: Advances in model-oriented design and analysis: proceedings of the 6th international workshop on model-oriented data analysis. Springer, Berlin, p 183
Myers J, Well A (2003) Research design and statistical analysis. Lawrence Erlbaum, New Jersey
Pesarin F (2001) Multivariate permutation tests: with applications in biostatistics. Wiley, Chichester
Pesarin F, Salmaso L (2010) Permutation tests for complex data: theory, applications and software. Wiley, New York
Rouanet H, Lépine D (1970) Comparison between treatments in a repeated-measurement design: ANOVA and multivariate methods. Br J Math Stat Psychol 23:147–163
Sahai H, Ageel M (2000) The analysis of variance: fixed, random, and mixed models. Birkhauser, Boston
Still A, White A (1981) The approximate randomization test as an alternative to the \(F\) test in analysis of variance. Br J Math Stat Psychol 34(2):243–252
White A, Harris P (1978) RANOVA: a fortran IV program for four-way Monte Carlo analysis of variance. Behav Res Methods 10(5):732
Acknowledgments
The authors thank the editor and referees for their helpful comments. The work is supported by the Swiss National Foundation under the Grant number 105211-112465/1 and 105214-116269/1 and by the Ernst et Lucie Schmidheiny Fundation.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Kherad-Pajouh, S., Renaud, O. A general permutation approach for analyzing repeated measures ANOVA and mixed-model designs. Stat Papers 56, 947–967 (2015). https://doi.org/10.1007/s00362-014-0617-3
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00362-014-0617-3