Using p-values for the comparison of classifiers: pitfalls and alternatives

Abstract

The statistical comparison of machine learning classifiers is frequently underpinned by null hypothesis significance testing. Here, we provide a survey and analysis of underrated problems that significance testing entails for classification benchmark studies. The p-value has become deeply entrenched in machine learning, but it is substantially less objective and less informative than commonly assumed. Even very small p-values can drastically overstate the evidence against the null hypothesis. Moreover, the p-value depends on the experimenter’s intentions, irrespective of whether these were actually realized or not. We show how such intentions can lead to experimental designs with more than one stage, and how to calculate a valid p-value for such designs. We discuss two widely used statistical tests for the comparison of classifiers, the Friedman test and the Wilcoxon signed rank test. Some improvements to the use of p-values, such as the calibration with the Bayes factor bound, and alternative methods for the evaluation of benchmark studies are discussed as well.

Appendix: Bayes factor bound

Under the null hypothesis, the p-value is known to be a random variable that is uniformly distributed over [0, 1]. Under the alternative hypothesis $H_1$, the p-value has density $f(p\,|\,\xi )$, with $\xi$ being an unknown parameter (Sellke et al. 2001; Benjamin and Berger 2019),

$$\begin{aligned} H_0: p \sim {\mathcal {U}}(0,1) \end{aligned}$$

(17)

$$\begin{aligned} H_1: p \sim f(p\,|\,\xi ) \end{aligned}$$

(18)

In significance testing, larger absolute values of the test statistic (cf. Eq. 1) are taken as casting more doubt on the null hypothesis and thereby providing more evidence in favor of the alternative hypothesis. Under $H_1$, the density should therefore be decreasing for increasing values of p. Sellke et al. (2001) propose the class of Beta($\xi ,1$) densities, with $0 < \xi \le 1$, for the distribution of the p-value,

$$\begin{aligned} f(p\,|\,\xi ) = \xi p^{\xi -1} \end{aligned}$$

(19)

For $\xi = 1$, we obtain the uniform distribution under the null hypothesis, $f(p\,|\,\xi = 1) = 1$. For a given prior density $\pi (\xi )$ under the alternative hypothesis, the odds of $H_1$ to $H_0$ (i.e., the Bayes factor) are

$$\begin{aligned} \mathrm {BF}_{\theta }(p)&= \frac{{\mathbb {P}}(H_1\,|\,p)}{{\mathbb {P}}(H_0\,|\,p)} \nonumber \\&= \frac{\int _{0}^{1} f(p\,|\,\xi )~\pi (\xi )~d\xi }{f(p\,|\, 1)} \end{aligned}$$

(20)

The upper bound of $\mathrm {BF}_{\pi }(p)$ is

$$\begin{aligned} \overline{\mathrm {BF}}_{\pi }(p) = \sup _{\mathrm {all}\,\pi } \mathrm {BF}_{\pi }(p) = \frac{\sup _{\xi } \xi \,p^{\xi -1}}{f(p\,|\, 1)} \end{aligned}$$

(21)

Thus,

$$\begin{aligned} \frac{\partial \,\xi \,p^{\xi -1}}{\partial \xi } = p^{\xi -1} + \xi \,p^{\xi -1}\,\ln (p)&\overset{!}{=} 0 \\ 1 + \xi \,\ln (p)&= 0 \\ \xi&= - \frac{1}{\ln (p)}\quad \mathrm {for}\,p \le e^{-1}\,\mathrm {since}\, \xi \le 1 \end{aligned}$$

Hence,

$$\begin{aligned} f\left( p\,|-\frac{1}{\ln (p)}\right)&= -\frac{1}{\ln (p)}\,p^{-\frac{1}{\ln (p)}-1} \\&= -\frac{1}{\ln (p)}\,p^{-\left( \frac{1}{\ln (p)}+1\right) } \\&= -\frac{1}{\ln (p)\,p^{\frac{1}{\ln (p)}} \, p} \\&= -\frac{1}{\ln (p)\,e\,p} \quad \mathrm {because}\,p^{\frac{1}{\ln (p)}} = p^{\log _pe} = e \end{aligned}$$

The second derivative w.r.t. $\xi$ is

$$\begin{aligned} \frac{\partial ^2 f(p\,|\,\xi )}{\partial \xi ^2} = 2\,p^{\xi -1}\,\ln (p) + [\ln (p)]^2 \,\xi \,p^{\xi -1} \end{aligned}$$

and for $\xi = - \frac{1}{\ln (p)}$,

$$\begin{aligned} \frac{\partial ^2 f\left( p\,|- \frac{1}{\ln (p)}\right) }{\partial \xi ^2} = \frac{\ln (p)}{e\,p} < 0 \end{aligned}$$

Thus, $[-\ln (p)\,e\,p]^{-1}$ is an upper bound on the odds of the alternative to the null hypothesis for $p \le e^{-1}$, and this bound holds for any reasonable prior distribution on $\xi$ (Sellke et al. 2001).