Numerical comparison of classical and permutation statistical hypothesis testing methods

Abstract

The article is devoted to the classical problem of statistical hypothesis testing for the equality of two distributions. For normal distributions, Student’s test is optimal in many senses. However, in practice, distributions to be compared are often not normal and, generally speaking, unknown. When nothing is known about the distributions to be compared, one usually applies the nonparametric Kolmogorov–Smirnov test to solve this problem. In the present paper, methods are considered that are based on permutations and, in recent years, have attracted interest for their simplicity, universality, and relatively high efficiency. Methods of stochastic simulation are applied to the comparative analysis of the power of a few permutation tests and classical methods (such as the Kolmogorov–Smirnov test, Student’s test, and the Mann–Whitney test) for a wide class of distribution functions. Normal distributions, Cauchy distributions, and their mixtures, as well as exponential, Weibull, Fisher’s, and Student’s distributions are considered. It is established that, for many typical distributions, the permutation method based on the sum of the absolute values of differences is the most powerful one. The advantage of this method over other ones is especially large when one compares symmetric distributions with the same centers. Thus, this permutation method can be recommended for application in cases when the distributions to be compared are different from normal ones.