Behavior Research Methods

, Volume 49, Issue 4, pp 1494–1502

Evaluating significance in linear mixed-effects models in R



Mixed-effects models are being used ever more frequently in the analysis of experimental data. However, in the lme4 package in R the standards for evaluating significance of fixed effects in these models (i.e., obtaining p-values) are somewhat vague. There are good reasons for this, but as researchers who are using these models are required in many cases to report p-values, some method for evaluating the significance of the model output is needed. This paper reports the results of simulations showing that the two most common methods for evaluating significance, using likelihood ratio tests and applying the z distribution to the Wald t values from the model output (t-as-z), are somewhat anti-conservative, especially for smaller sample sizes. Other methods for evaluating significance, including parametric bootstrapping and the Kenward-Roger and Satterthwaite approximations for degrees of freedom, were also evaluated. The results of these simulations suggest that Type 1 error rates are closest to .05 when models are fitted using REML and p-values are derived using the Kenward-Roger or Satterthwaite approximations, as these approximations both produced acceptable Type 1 error rates even for smaller samples.


Linear mixed-effects models Statistics p-values Type 1 error lme4 


  1. Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412. doi:10.1016/j.jml.2007.12.005 CrossRefGoogle Scholar
  2. Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278.CrossRefGoogle Scholar
  3. Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015a). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. doi:10.18637/jss.v067.i01 CrossRefGoogle Scholar
  4. Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015b). lme4: Linear mixed-effects models using Eigen and S4. R package version 1.1-8. Retrieved from
  5. Brown, L. D., Cai, T. T., & DasGupta, A. (2001). Interval estimation for a binomial proportion. 101–133. doi:10.1214/ss/1009213286
  6. Halekoh, U., & Højsgaard, S. (2014). pbkrtest: Parametric bootstrap and Kenward Roger based methods for mixed model comparison. R package version 0.4-2.Google Scholar
  7. Kenward, M. G., & Roger, J. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 983–997.Google Scholar
  8. Kuznetsova, A., Brockhoff, P., & Christensen, R. (2014). LmerTest: Tests for random and fixed effects for linear mixed effect models. R package, version 2.0-3.Google Scholar
  9. Pinheiro, J. C., & Bates, D. M. (2000). Mixed-effects models in S and S-PLUS. Springer.Google Scholar
  10. R Core Team. (2015). R: A Language and Environment for Statistical Computing (Version 3.2.2). Vienna, Austria: R Foundation for Statistical Computing. Retrieved from Google Scholar
  11. SAS Institute. (2008). SAS/STAT 9.2 user’s guide.Google Scholar
  12. Satterthwaite, F. E. (1941). Synthesis of variance. Psychometrika, 6(5), 309–316.CrossRefGoogle Scholar
  13. Schaalje, G. B., McBride, J. B., & Fellingham, G. W. (2002). Adequacy of approximations to distributions of test statistics in complex mixed linear models. Journal of Agricultural, Biological, and Environmental Statistics, 7(4), 512–524.CrossRefGoogle Scholar
  14. Singmann, H., Bolker, B., & Westfall, J. (2015). afex: Analysis of factorial experiments. R package, version 0.14-2.Google Scholar
  15. Stroup, W. W. (2015). Rethinking the analysis of non-normal data in plant and soil science. Agronomy Journal, 107(2), 811–827.CrossRefGoogle Scholar
  16. Westfall, J., Kenny, D. A., & Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143(5), 2020.CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2016

Authors and Affiliations

  1. 1.Department of PsychologyBrigham Young UniversityProvoUSA

Personalised recommendations