Randomization and Permutation Tests

  • Vance W. BergerEmail author
  • Patrick Onghena
  • J. Rosser Matthews
Living reference work entry


This chapter will address the decision to use permutation tests as opposed to parametric analyses in the context of between-group analysis in randomized clinical trials designed to evaluate a medical intervention. It is important to understand at the outset that permutation tests represent a means to an end, rather than an end unto themselves. It is not so much that one seeks to use permutation tests just for the sake of doing so but, rather, that one recognizes the severe deficiencies of parametric analyses and wishes to use some other type of analysis that does not similarly suffer from these drawbacks. When viewed in this context, properly conducted permutation tests are the solution to the problem of how to compare treatments without having to rely on assumptions that cannot possibly be true. We argue that the default position would clearly be the use of exact analyses and that the burden of proof would fall to those who would argue that the approximate analyses are just as good or, as is sometimes argued, even better.


Approximations Normality Parametric analyses Precautionary principle 


  1. Baird D (1983) The fisher/Pearson chi-squared controversy: a turning point for inductive inference. Br J Philos Sci 34:105–118MathSciNetCrossRefGoogle Scholar
  2. Berger VW (1998) Admissibility of exact conditional tests of stochastic order. J Stat Plann Inference 66(1):39–50MathSciNetCrossRefGoogle Scholar
  3. Berger VW (2000) Pros and cons of permutation tests in clinical trials. Stat Med 19:1319–1328CrossRefGoogle Scholar
  4. Berger VW (2005) Selection bias and covariate imbalances in randomized clinical trials. Wiley, ChichesterCrossRefGoogle Scholar
  5. Berger VW (2015) Conflicts of interest, selective inertia, and research malpractice in randomized clinical trials: an unholy trinity. Sci Eng Ethics 21(4):857–874CrossRefGoogle Scholar
  6. Berger VW (2017) An empirical demonstration of the need for exact tests. J Mod Appl Stat Methods 16(1):34–50CrossRefGoogle Scholar
  7. Berger VW, Ivanova A (2002) Adaptive tests for ordinal data. J Mod Appl Stat Methods 1(2):269–280CrossRefGoogle Scholar
  8. Berger V, Sackrowitz H (1997) Improving tests for superior treatment in contingency tables. J Am Stat Assoc 92(438):700–705MathSciNetCrossRefGoogle Scholar
  9. Berger VW, Permutt T, Ivanova A (1998) The convex hull test for ordered categorical data. Biometrics 54(4):1541–1550CrossRefGoogle Scholar
  10. Berry KJ, Mielke PW, Johnston JE (2016) Permutation statistical methods: an integrated approach. Cham, SW: SpringerGoogle Scholar
  11. Cobb GW (2007) The introductory statistics course: a Ptolemaic curriculum? Technol Innov Stat Educ 1(1).
  12. Edgington ES, Onghena P (2007) Randomization tests, 4th edn. Chapman & Hall/CRC, Boca RatonzbMATHGoogle Scholar
  13. Efron B, Hastie T (2016) Computer age statistical inference: algorithms, evidence, and data science. Institute of mathematical statistics monographs. Cambridge University Press, Cambridge. Scholar
  14. Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman & Hall/CRC, Boca RatonCrossRefGoogle Scholar
  15. Galison P (2011) Computer simulations and the trading zone. In: Gramelsberger G (ed) From science to computational science. Diaphanes, Zürich, pp 118–157Google Scholar
  16. Geary RC (1947) Testing for normality. Biometrika 34:209–242MathSciNetCrossRefGoogle Scholar
  17. Gigerenzer G, Swijtink Z, Porter T, Daston L, Beatty J, Kruger L (1989) The empire of chance: how probability changed science and everyday life. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  18. Hanin L (2017) Why statistical inference from clinical trials is likely to generate false and irreproducible results. BMC Med Res Methodol 17:127. Scholar
  19. John LK, Loewenstein G, Prelec D (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 23:524–532CrossRefGoogle Scholar
  20. Kuhn TS (1962) The structure of scientific revolutions. University of Chicago Press, ChicagoGoogle Scholar
  21. Magnello E (2002) The introduction of mathematical statistics into medical research: the roles of Karl Pearson, Major Greenwood, and Austin Bradford Hill. In: Magnello E, Hardy A (eds) The road to medical statistics. Rodopi, AmsterdamCrossRefGoogle Scholar
  22. Matthews JR (1995) Quantification and the quest for medical certainty. Princeton University Press, PrincetonGoogle Scholar
  23. May RB, Masson MEJ, Hunter MA (1990) Applications of statistics in behavioral research. Harper & Row, New YorkGoogle Scholar
  24. Onghena P, May RB (1995) Pitfalls in computing and interpreting randomization test p values: a commentary on Chen and Dunlap. Behav Res Methods Instrum Comput 27:408–411CrossRefGoogle Scholar
  25. Perlman P, Possen BH, Legat VD, Rubenacker AS, Bockiger U, Stieben-Emmerling L (2013) When will we see people of negative height. Significance 10(1):46–48CrossRefGoogle Scholar
  26. Porter T (1994) The death of the object: Fin de siècle philosophy of physics. In: Ross D (ed) Modernist impulses in the human sciences, 1870–1930. The Johns Hopkins University Press, BaltimoreGoogle Scholar
  27. Rossman A (2015) Interview with George Cobb. J Stat Educ 23(1).
  28. Simon JL, Bruce P (1991) Resampling: a tool for everyday statistical work. Chance 4:22–32CrossRefGoogle Scholar
  29. Tintle N, Chance B, Cobb G, Rossman A, Roy S, Swanson T, Vanderstoep J (2015) Introduction to statistical investigations. Wiley, New YorkzbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Vance W. Berger
    • 1
    Email author
  • Patrick Onghena
    • 2
  • J. Rosser Matthews
    • 3
  1. 1.Biometry Research GroupNational Cancer InstituteRockvilleUSA
  2. 2.Faculty of Psychology and Educational SciencesKU LeuvenLeuvenBelgium
  3. 3.General Dynamics Health SolutionsDefense and Veterans Brain Injury CenterSilver SpringUSA

Section editors and affiliations

  • Stephen George
    • 1
  1. 1.Dept. of Biostatistics and Bioinformatics,Basic Science DivisonDuke University, School of MedicineDurhamUSA

Personalised recommendations