Statistics and Computing

, Volume 26, Issue 3, pp 693–701 | Cite as

Union–intersection permutation solution for two-sample equivalence testing

  • Fortunato Pesarin
  • Luigi Salmaso
  • Eleonora Carrozzo
  • Rosa Arboretti


One of the well-known problems with testing for sharp null hypotheses against two-sided alternatives is that, when sample sizes diverge, every consistent test rejects the null with a probability converging to one, even when it is true. This kind of problem emerges in practically all applications of traditional two-sided tests. The main purpose of the present paper is to overcome this very intriguing impasse by considering a general solution to the problem of testing for an equivalence null interval against a two one-sided alternative. Our goal is to go beyond the limitations of likelihood-based methods by working in a nonparametric permutation framework. This solution requires the nonparameteric Combination of dependent permutation tests, which is the methodological tool that achieves Roy’s Union–intersection principle. To obtain practical solutions, the related algorithm is presented. To appreciate its effectiveness for practical purposes, a simple example and some simulation results are also presented. In addition, for every pair of consistent partial test statistics it is proved that, if sample sizes diverge, when the effect lies in the open equivalence interval, the Rejection probability (RP) converges to zero. Analogously, if the effect lies outside that interval, the RP converges to one.


Nonparametric combination Permutation tests Testing equivalence Two one-sided tests  Union–intersection principle 



Authors acknowledge anonymous referees and associate editor for comments which helped to improved the manuscript. This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. All authors have contributed equally to the manuscript.


  1. Basso, D., Pesarin, F., Salmaso, L., Solari, A.: Permutation Tests for Stochastic Ordering and ANOVA: Theory and Applications in R. Lecture Notes N. Springer, New York (2009)zbMATHGoogle Scholar
  2. Berger, R.L.: Multiparameter hypothesis testing and acceptance sampling. Technometrics 24, 295–300 (1982)MathSciNetCrossRefzbMATHGoogle Scholar
  3. Bertoluzzo, F., Pesarin, F., Salmaso, L.: On multi-sided permutation tests. Commun. Stat. Simul. Comput. 42(6), 1380–1390 (2013)CrossRefzbMATHGoogle Scholar
  4. Birnbaum, A.: Combining independent tests of significance. J. Am. Stat. Assoc. 49, 559–574 (1954a)MathSciNetzbMATHGoogle Scholar
  5. Birnbaum, A.: Characterization of complete classes of tests of some multiparametric hypotheses, with application to likelihood ratio tests. Ann. Math. Stat. 26, 21–36 (1954b)MathSciNetCrossRefzbMATHGoogle Scholar
  6. Brombin, C., Salmaso, L., Ferronato, G., Galzignato, P.F.: Multi-aspect procedures for paired data with application to biometric morphing. Commun. Stat. Simul. Comput. 40, 1–12 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  7. Cox, D.R., Hinkley, D.V.: Theoretical Statistics. Chapman and Hall, London (1974)CrossRefzbMATHGoogle Scholar
  8. Edgington, E.S., Onghena, P.: Randomization Tests, 4th edn. Chapman & Hall, Boca Raton (2007)zbMATHGoogle Scholar
  9. Frosini, B.V.: On Neyman–Pearson theory: information content of an experiment and a fancy paradox. Statistica 64, 271–286 (2004)Google Scholar
  10. Good, P.: Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses. Springer, New York (2000)CrossRefzbMATHGoogle Scholar
  11. Hoeffding, W.: The large-sample power of tests based on permutations of observations. Ann. Math. Stat. 23, 169–192 (1952)MathSciNetCrossRefzbMATHGoogle Scholar
  12. Hung, H.M.J., Wang, S.U.: Some controversial multiple testing problems in regulatory applications. J. Biopharm. Stat. 19, 1–11 (2009)MathSciNetCrossRefGoogle Scholar
  13. Lehmann, E.L.: Testing Statistical Hypotheses, 2nd edn. Wiley, New York (1986)CrossRefzbMATHGoogle Scholar
  14. Marozzi, M.: A bi-aspect nonparametric test for the two-sample location problem. Comput. Stat. Data Anal. 44, 639–648 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  15. Marozzi, M.: Multivariate tests based on interpoint distances with application to magnetic resonance imaging. Stat. Methods Med. Res. (2014). doi: 10.1177/0962280214529104
  16. Marozzi, M., Salmaso, L.: Multivariate bi-aspect testing for the two-sample location problem. Commun. Stat. Theory Methods 35(3), 477–488 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  17. Nunnally, J.: The place of statistics in psychology. Educ. Psychol. Measur. 20, 641–650 (1960)CrossRefGoogle Scholar
  18. Pantsulaia, G., Kintsurashvili, M.: Why is the null hypothesis rejected for almost every infinite sample by some hypothesis testing of maximal reliability. J. Stat. 11, 45–70 (2014)Google Scholar
  19. Pesarin, F.: On a nonparametric combination method for dependent permutation tests with applications. Psychom. Psychosom. 54, 172–179 (1990)CrossRefGoogle Scholar
  20. Pesarin, F.: Multivariate Permutation Tests, with Applications in Biostatistics. Wiley, Chichester (2001)zbMATHGoogle Scholar
  21. Pesarin, F., Salmaso, L.: Permutation Tests for Complex Data, Theory, Applications and Software. Wiley, Chichester (2010a)CrossRefzbMATHGoogle Scholar
  22. Pesarin, F., Salmaso, L.: Finite-sample consistency of combination-based permutation tests with application to repeated measures designs. J. Nonparametric Stat. 22(5), 669–684 (2010b)MathSciNetCrossRefzbMATHGoogle Scholar
  23. Pesarin, F., Salmaso, L.: On the weak consistency of permutation tests. Commun. Stat. Simul. Comput. 42, 1368–1397 (2013)CrossRefzbMATHGoogle Scholar
  24. Romano, J.P.: Optimal testing of equivalence hypotheses. Ann. Stat. 33, 1036–1047 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  25. Roy, S.N.: On a heuristic method of test construction and its use in multivariate analysis. Ann. Math. Stat. 24, 220–238 (1953)MathSciNetCrossRefzbMATHGoogle Scholar
  26. Salmaso, L., Solari, A.: Multiple aspect testing for case-control designs. Metrika 62, 331–340 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  27. Sen, P.K.: Union–intersection principle and constrained statistical inference. Stat. Plann. Inference 137, 3741–3752 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  28. Sen, P.K., Tsai, M.T.: Two-stage likelihood ratio and union intersection tests for one-sided alternatives multivariate mean with nuisance dispersion matrix. J. Multivar. Anal. 68(264), 282 (1999)MathSciNetzbMATHGoogle Scholar
  29. Wellek, S.: Testing Statistical Hypotheses of Equivalence and Noninferiority. Chapman & Hall, Boca Raton (2010)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Fortunato Pesarin
    • 1
  • Luigi Salmaso
    • 2
  • Eleonora Carrozzo
    • 2
  • Rosa Arboretti
    • 3
  1. 1.Department of Statistical SciencesUniversity of PadovaPadovaItaly
  2. 2.Department of Management and EngineeringUniversity of PadovaVicenzaItaly
  3. 3.Department of Land, Environment, Agriculture and ForestryUniversity of PadovaPadovaItaly

Personalised recommendations