Computational Statistics

, Volume 33, Issue 4, pp 1847–1862 | Cite as

Applications of conditional power function of two-sample permutation test

  • Monjed H. Samuh
  • Fortunato Pesarin
Original Paper


Permutation or randomization test is a nonparametric test in which the null distribution (distribution under the null hypothesis of no relationship or no effect) of the test statistic is attained by calculating the values of the test statistic overall permutations (or by considering a large number of random permutation) of the observed dataset. The power of permutation test evaluated based on the observed dataset is called conditional power. In this paper, the conditional power of permutation tests is reviewed. The use of the conditional power function for sample size estimation is investigated. Moreover, reproducibility and generalizability probabilities are defined. The use of these probabilities for sample size adjustment is shown. Finally, an illustration example is used.


Generalizability probability Permutation test Reproducibility probability Sample size adjustment Sample size estimation 



The authors would like to thank the associated editor and referees for their comments that contribute in improving the paper. We also greatly appreciate Dr. Ibrahim Almasri, Department of Applied Mathematics and Physics - Palestine Polytechnic University, for being kind enough to read and improve the language of the paper.


  1. Aguirre-Urreta M, Rönkkö M (2015) Sample size determination and statistical power analysis in PLS using R: an annotated tutorial. Commun Assoc Inf Syst 36(3):33–51Google Scholar
  2. Akobeng AK (2016) Understanding type I and type II errors, statistical power and sample size. Acta Paediatr 105(6):605–609CrossRefGoogle Scholar
  3. Amro L, Pauly M (2017) Permuting incomplete paired data: a novel exact and asymptotic correct randomization test. J Stat Comput Simul 87(6):1148–1159MathSciNetCrossRefGoogle Scholar
  4. Barton DE (1957) A comparison of two sorts of test for a change of location applicable to truncated data. J R Stat Soc 19:119–124MathSciNetzbMATHGoogle Scholar
  5. Basso D, Pesarin F, Salmaso L, Solari A (2009) Permutation tests for stochastic ordering and ANOVA: theory and applications in R. Springer, New YorkzbMATHGoogle Scholar
  6. Bell CB, Moser JM, Thompson R (1966) Goodness criteria for two-sample distribution-free tests. Ann Math Stat 37:133–142MathSciNetCrossRefzbMATHGoogle Scholar
  7. Brewer JK, Sindelar PT (1988) Adequate sample size: a priori and post hoc considerations. J Spec Educ 21:74–84CrossRefGoogle Scholar
  8. Chow SC, Liu JP (2004) Design and analysis of clinical trials: concepts and methodologies, 2nd edn. Wiley-Blackwell, New YorkzbMATHGoogle Scholar
  9. Chow SC, Shao J, Wang H (2002) A note on sample size calculation for mean comparisons based on non-central \(t\)-statistics. J Biopharm Stat 12:441–456CrossRefGoogle Scholar
  10. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum Associates, HillsdalezbMATHGoogle Scholar
  11. Collings BJ, Hamilton MA (1988) Estimating the power of the two-sample Wilcoxon test for location shift. Biometrics 44:847–860CrossRefzbMATHGoogle Scholar
  12. Cooper H, Hedges LV (1997) The handbook of research synthesis. Russell Sage Foundation, New YorkGoogle Scholar
  13. De Capitani L, De Martini D (2016) Reproducibility probability estimation and RP-testing for some nonparametric tests. Entropy 18(4):142CrossRefGoogle Scholar
  14. De Martini D (2002) Pointwise estimate of the power and sample size determination for permutation tests. Statistica 62:779–790MathSciNetzbMATHGoogle Scholar
  15. Dixon WJ (1954) Power under normality of several nonparametric tests. Ann Math Stat 25:610–614MathSciNetCrossRefzbMATHGoogle Scholar
  16. Edgington ES (1995) Randomization tests, 3rd edn. Marcel Dekker, New YorkzbMATHGoogle Scholar
  17. Epstein B (1955) Comparison of some non-parametric tests against normal alternatives with an application to life testing. J Am Stat Assoc 50:894–900MathSciNetzbMATHGoogle Scholar
  18. Fisher RA (1934) Statistical methods for research workers. Oliver and Boyd, EdinburghzbMATHGoogle Scholar
  19. Fisher RA (1935) The design of experiments. Oliver and Boyd, EdinburghGoogle Scholar
  20. Giraudeau B, Higgins J, Tavernier E, Trinquart L (2016) Sample size calculation for meta-epidemiological studies. Stat Med 35(2):239–250MathSciNetCrossRefGoogle Scholar
  21. Good P (2005) Permutation, parametric and bootstrap tests of hypotheses, 3rd edn. Springer, New YorkzbMATHGoogle Scholar
  22. Goodman S (1992) A comment on replication, \(p\)-values and evidence. Stat Med 11:875–879CrossRefGoogle Scholar
  23. Hallahan M, Rosenthal R (1996) Statistical power: concepts, procedures, and applications. Behav Res Ther 34:489–499CrossRefGoogle Scholar
  24. Hamilton MA, Collings BJ (1991) Determining the appropriate sample size for nonparametric tests for location shift. Technometrics 33:327–337MathSciNetCrossRefzbMATHGoogle Scholar
  25. Haynam GE, Govindarajulu Z (1966) Exact power of the Mann–Whitney test for exponential and rectangular alternatives. Ann Math Stat 37:945–953MathSciNetCrossRefzbMATHGoogle Scholar
  26. Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic Press, New YorkzbMATHGoogle Scholar
  27. Hemelrijk J (1961) Experimental comparison of Student’s and Wilcoxon’s two sample test. Quantitative Methods in Pharmacology. Inter-science, New York, pp 118–133Google Scholar
  28. Hoeffding W (1952) The large-sample power of tests based on permutations of observations. Ann Math Stat 23:169–192MathSciNetCrossRefzbMATHGoogle Scholar
  29. Kraemer HC, Thiemann S (1987) How many subjects? Statistical power analysis in research. Sage Publications, Newbury ParkGoogle Scholar
  30. Lehmann EL, Romano JP (2005) Testing statistical hypotheses, 3rd edn. Springer, New YorkzbMATHGoogle Scholar
  31. Lehmann EL, Stein C (1949) On the theory of some non-parametric hypotheses. Ann Math Stat 20:28–45CrossRefzbMATHGoogle Scholar
  32. Lenth RV (2007) Post hoc power: tables and commentary. Technical Report 378, The University of Iowa - Department of Statistics and Actuarial ScienceGoogle Scholar
  33. Levine M, Ensom MHH (2001) Post hoc power analysis: an idea whose time has passed? Pharmacotherapy 21:405–409CrossRefGoogle Scholar
  34. Lipsey MW (1990) Design sensitivity: statistical power for experimental research. Sage Publications, Newbury ParkGoogle Scholar
  35. Markowski EP, Markowski CA (1999) Practical uses of statistical power in business research studies. J Educ Bus 75:122–125CrossRefzbMATHGoogle Scholar
  36. McDonald J, Gerard PD, McMahan CS, Schucany WR (2016) Exact-permutation-based sign tests for clustered binary data via weighted and unweighted test statistics. J Agric Biol Environ Stat 21(4):698–712MathSciNetCrossRefzbMATHGoogle Scholar
  37. McHugh RB (1961) Confidence interval inference and sample size determination. Am Stat 15:14–17Google Scholar
  38. Milton RC (1970) Rank order probabilities: two-sample normal shift alternatives. Wiley, New YorkzbMATHGoogle Scholar
  39. Moher D, Dulberg CS, Wells GA (1994) Statistical power, sample size, and their reporting in randomized controlled trials. J Am Med Assoc 272:122–124CrossRefGoogle Scholar
  40. Noether GE (1987) Sample size determination for some common nonparametric tests. J Am Stat Assoc 82:645–647MathSciNetCrossRefzbMATHGoogle Scholar
  41. Onwuegbuzie AJ, Leech NL (2004) Post hoc power: a concept whose time has come. Underst Stat 3:201–230CrossRefGoogle Scholar
  42. Owen DB (1965) The power of Student’s \(t\)-test. J Am Stat Assoc 60:320–333MathSciNetzbMATHGoogle Scholar
  43. Pesarin F (2001) Multivariate permutation tests: with application in biostatistics. Wiley, ChichesterzbMATHGoogle Scholar
  44. Pesarin F, Salmaso L (2010) Permutation tests for complex data: theory application and software. Wiley, ChichesterCrossRefzbMATHGoogle Scholar
  45. Pitman EJG (1937a) Significance tests which may be applied to samples from any population. J R Stat Soc B 4:119–130zbMATHGoogle Scholar
  46. Pitman EJG (1937b) Significance tests which may be applied to samples from any population. II. The correlation coefficient test. J R Stat Soc B 4:225–232zbMATHGoogle Scholar
  47. Pitman EJG (1938) Significance tests which may be applied to samples from any population. III. The analysis of variance test. Biometrika 29:322–335zbMATHGoogle Scholar
  48. Randles RH, Wolfe DA (1979) Introduction to the theory of nonparametric statistics. Wiley, New YorkzbMATHGoogle Scholar
  49. Salmaso L (2003) Synchronized permutation tests in \(2^k\) factorial designs. Commun Stat Theory Methods 32:1419–1437MathSciNetCrossRefzbMATHGoogle Scholar
  50. Samonenko I, Robinson J (2015) A new permutation test statistic for complete block designs. Ann Stat 43(1):90–101MathSciNetCrossRefzbMATHGoogle Scholar
  51. Schmitt MC (1987) The effects on an elaborated directed reading activity on the metacomprehension skills of third graders. Ph.D. Thesis, Purdue UniversityGoogle Scholar
  52. Shao J, Chow SC (2002) Reproducibility probability in clinical trials. Stat Med 21:1727–1742CrossRefGoogle Scholar
  53. Shorack GR, Wellner JA (1986) Empirical processes with applications to statistics. Wiley Series in Probability & Mathematical Statistics, New YorkzbMATHGoogle Scholar
  54. Simonoff JS, Hochberg Y, Reiser B (1986) Alternative estimation procedures for \({P}_r({X} < {Y})\) in categorized data. Biometrics 42:895–907MathSciNetCrossRefzbMATHGoogle Scholar
  55. Teichroew D (1955) Empirical power functions for nonparametric two-sample tests for small samples. Ann Math Stat 26:340–344MathSciNetCrossRefzbMATHGoogle Scholar
  56. Thomas L (1997) Retrospective power analysis. Conserv Biol 11:276–280CrossRefGoogle Scholar
  57. Wang H, Chow SC, Chen M (2005) A Bayesian approach on sample size calculation for comparing means. J Biopharm Stat 15:799–807MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Applied Mathematics and Physics DepartmentPalestine Polytechnic UniversityHebronPalestine
  2. 2.Department of Statistical SciencesUniversity of PadovaPaduaItaly

Personalised recommendations