Redefine or justify? Comments on the alpha debate

  • Jan de RuiterEmail author
Theoretical Review


Benjamin et al. (Nature Human Behaviour 2, 6-10, 2017) proposed improving the reproducibility of findings in psychological research by lowering the alpha level of our conventional null hypothesis significance tests from .05 to .005, because findings with p-values close to .05 represent insufficient empirical evidence. They argued that findings with a p-value between 0.005 and 0.05 should still be published, but not called “significant” anymore. This proposal was criticized and rejected in a response by Lakens et al. (Nature Human Behavior 2, 168-171, 2018), who argued that instead of lowering the traditional alpha threshold to .005, we should stop using the term “statistically significant,” and require researchers to determine and justify their alpha levels before they collect data. In this contribution, I argue that the arguments presented by Lakens et al. against the proposal by Benjamin et al. are not convincing. Thus, given that it is highly unlikely that our field will abandon the NHST paradigm any time soon, lowering our alpha level to .005 is at this moment the best way to combat the replication crisis in psychology.


Significance Reproducibility Alpha Evidence 



The author wishes to thank Alexander Etz, Jason Noble, and Eric-Jan Wagenmakers for their helpful comments on earlier versions of this paper.


  1. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543-554.CrossRefGoogle Scholar
  2. Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., ... Johnson, V. E. (2017). Redefine statistical significance. Nature Human Behaviour, 1.Google Scholar
  3. Etz, A., & Vandekerckhove, J. (2016). A Bayesian perspective on the reproducibility project: Psychology. PloS one, 11(2), e0149794.CrossRefGoogle Scholar
  4. Held, L., & Ott, M. (In Press). On p-values and Bayes factors. Annual Review of Statistics and Its Application.Google Scholar
  5. Johnson, V. E., Payne, R.D., Wang, T., Asher, A., & Mandal, S. (2017). On the reproducibility of psychological science. Journal of the American Statistical Association 112(517), 1-10.CrossRefGoogle Scholar
  6. Lakens, D., Adolfi, F., Albers, C., Anvari, F., Apps, M., Argamon, S., ... Bradford, D. (2018). Justify your alpha. Nature Human Behavior, 2, 168-171.Google Scholar
  7. Morey, R. (2017). Redefining statistical significance: the statistical arguments [blog post]. Retrieved from
  8. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251).
  9. Royall, R. (1997). Statistical evidence: A likelihood paradigm. New York: Routledge.Google Scholar
  10. Wilson, B. M., & Wixted, J. T. (2018). The prior odds of testing a true effect in cognitive and social psychology. Advances in Methods and Practices in Psychological Science, 2515245918767122.Google Scholar
  11. Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2017). Making replication mainstream. Behavioral and Brain Sciences, 1-50.Google Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  1. 1.Departments of Computer Science and PsychologyTufts UniversityMedfordUSA

Personalised recommendations