Statistical Papers

, Volume 56, Issue 4, pp 947–967 | Cite as

A general permutation approach for analyzing repeated measures ANOVA and mixed-model designs

  • Sara Kherad-PajouhEmail author
  • Olivier Renaud
Regular Article


Repeated measures ANOVA and mixed-model designs are the main classes of experimental designs used in psychology. The usual analysis relies on some parametric assumptions (typically Gaussianity). In this article, we propose methods to analyze the data when the parametric conditions do not hold. The permutation test, which is a non-parametric test, is suitable for hypothesis testing and can be applied to experimental designs. The application of permutation tests in simpler experimental designs such as factorial ANOVA or ANOVA with only between-subject factors has already been considered. The main purpose of this paper is to focus on more complex designs that include only within-subject factors (repeated measures) or designs that include both within-subject and between-subject factors (mixed-model designs). First, a general approximate permutation test (permutation of the residuals under the reduced model or reduced residuals) is proposed for any repeated measures and mixed-model designs, for any number of repetitions per cell, any number of subjects and factors and for both balanced and unbalanced designs (all-cell-filled). Next, a permutation test that uses residuals that are exchangeable up to the second moment is introduced for balanced cases in the same class of experimental designs. This permutation test is therefore exact for spherical data. Finally, we provide simulations results for the comparison of the level and the power of the proposed methods.


Covariance Matrix Permutation Test Repeat Measure Design Design Matrice Balance Design 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors thank the editor and referees for their helpful comments. The work is supported by the Swiss National Foundation under the Grant number 105211-112465/1 and 105214-116269/1 and by the Ernst et Lucie Schmidheiny Fundation.

Supplementary material

362_2014_617_MOESM1_ESM.pdf (215 kb)
ESM 1 (PDF 216 kb)


  1. Anderson M, Ter Braak C (2003) Permutation tests for multi-factorial analysis of variance. J Stat Comput Simul 73(2):85–113zbMATHMathSciNetCrossRefGoogle Scholar
  2. Basso D (2009) Permutation tests for stochastic ordering and ANOVA: theory and applications with R. Springer, New YorkGoogle Scholar
  3. Basso D, Salmaso L (2006) A discussion of permutation tests conditional to observed responses in unreplicated \(2^m\) full factorial designs. Commun Stat Theory Methods 35(1):83–97. doi:  10.1080/03610920500437277 zbMATHMathSciNetCrossRefGoogle Scholar
  4. Basso D, Chiarandini M, Salmaso L (2007) Synchronized permutation tests in replicated IxJ designs. J Stat Plan Inference 137(8):2564–2578. doi: 10.1016/j.jspi.2006.04.016 zbMATHMathSciNetCrossRefGoogle Scholar
  5. Cardinal R, Aitken M (2006) ANOVA for the behavioural sciences researcher. Lawrence Erlbaum Associates, New JerseyGoogle Scholar
  6. Cornfield J, Tukey JW (1956) Average values of mean squares in factorials. Ann Math Stat 27(4):907–949. doi: 10.1214/aoms/1177728067 zbMATHMathSciNetCrossRefGoogle Scholar
  7. Field CA, Welsh A (2007) Bootstrapping clustered data. J Roy Stat Soc B 69:369–390MathSciNetCrossRefGoogle Scholar
  8. Fisher R (1935) The design of experiments. Oliver & Boyd, EdinburghGoogle Scholar
  9. Freedman D, Lane D (1983) A nonstochastic interpretation of reported significance levels. J Bus Econ Stat 1(4):292–298Google Scholar
  10. Good P (1994) Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer, New YorkzbMATHCrossRefGoogle Scholar
  11. Hirotsu C (1979) An F approximation and its application. Biometrika 66(3):577–584zbMATHMathSciNetCrossRefGoogle Scholar
  12. IBM Corp (2013) IBM SPSS statistics for Windows, Version 22.0. Armonk, NYGoogle Scholar
  13. Jung BC, Jhun M, Song SH (2006) A new random permutation test in ANOVA models. Stat Pap 48:47–62zbMATHMathSciNetCrossRefGoogle Scholar
  14. Keppel G (1991) Design and analysis: a researcher’s handbook. Prentice-Hall, Englewood CliffsGoogle Scholar
  15. Kherad-Pajouh S (2011) Permutation tests for experimental designs, with extension to simultaneous EEG signal analysis. PhD thesis, University of Geneva. url:
  16. Kherad-Pajouh S, Renaud O (2010) An exact permutation method for testing any effect in balanced and unbalanced fixed effect anova. Comput Stat Data Anal 54(7):1881–1893.
  17. Kirk R (1994) Experimental design: procedures for the behavioral sciences. Wadsworth Publishing Co Inc., BelmontGoogle Scholar
  18. Mazzaro D, Pesarin F, Salmaso L (2001) Permutation tests for effects in unbalanced repeated measures factorial designs. In: Advances in model-oriented design and analysis: proceedings of the 6th international workshop on model-oriented data analysis. Springer, Berlin, p 183Google Scholar
  19. Myers J, Well A (2003) Research design and statistical analysis. Lawrence Erlbaum, New JerseyGoogle Scholar
  20. Pesarin F (2001) Multivariate permutation tests: with applications in biostatistics. Wiley, ChichesterGoogle Scholar
  21. Pesarin F, Salmaso L (2010) Permutation tests for complex data: theory, applications and software. Wiley, New YorkCrossRefGoogle Scholar
  22. Rouanet H, Lépine D (1970) Comparison between treatments in a repeated-measurement design: ANOVA and multivariate methods. Br J Math Stat Psychol 23:147–163zbMATHCrossRefGoogle Scholar
  23. Sahai H, Ageel M (2000) The analysis of variance: fixed, random, and mixed models. Birkhauser, BostonCrossRefGoogle Scholar
  24. Still A, White A (1981) The approximate randomization test as an alternative to the \(F\) test in analysis of variance. Br J Math Stat Psychol 34(2):243–252MathSciNetCrossRefGoogle Scholar
  25. White A, Harris P (1978) RANOVA: a fortran IV program for four-way Monte Carlo analysis of variance. Behav Res Methods 10(5):732CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.Division of Biostatistics, School of Public HealthUniversity of CaliforniaBerkeleyUSA
  2. 2.Methodology & Data Analysis Group, Dept. of PsychologyGenevaSwitzerland

Personalised recommendations