Model Predictivity Assessment: Incremental Test-Set Selection and Accuracy Evaluation

Abstract

Unbiased assessment of the predictivity of models learnt by supervised machine learning (ML) methods requires knowledge of the learned function over a reserved test set (not used by the learning algorithm). The quality of the assessment depends, naturally, on the properties of the test set and on the error statistic used to estimate the prediction error. In this work we tackle both issues, proposing a new predictivity criterion that carefully weights the individual observed errors to obtain a global error estimate, and using incremental experimental design methods to “optimally” select the test points on which the criterion is computed. Several incremental constructions are studied, including greedy-packing (coffee-house design), support points and kernel herding techniques. Our results show that the incremental and weighted versions of the latter two, based on Maximum Mean Discrepancy concepts, yield superior performance. An industrial test case provided by the historical French electricity supplier (EDF) illustrates the practical relevance of the methodology, indicating that it is an efficient alternative to expensive cross-validation techniques.

Appendix

1.1 Appendix A: Maximum Mean Discrepancy

Let K be a positive definite kernel on $\mathcal {X}\times \mathcal {X}$, defining a reproducing kernel Hilbert space (RKHS) $\mathcal {H}_K$ of functions on $\mathcal {X}$, with scalar product $\langle f,g\rangle _{\mathcal {H}_K}$ and norm $\Vert f\Vert _{\mathcal {H}_K}$; see, e.g., [2]. For any $f\in \mathcal {H}_K$ and any probability measures $\mu$ and $\xi$ on $\mathcal {X}$, we have

$$\begin{aligned} \left| \int _\mathcal {X}f(\textbf{x})\, \text{ d }\xi (\textbf{x}) - \int _\mathcal {X}f(\textbf{x})\, \text{ d }\mu (\textbf{x}) \right|= & {} \left| \int _\mathcal {X}\langle f,K_\textbf{x}\rangle _{\mathcal {H}_K}\, \text{ d }(\xi -\mu )(\textbf{x}) \right| \nonumber \\= & {} \left| \langle f,(P_{K,\xi }-P_{K,\mu }\rangle _{\mathcal {H}_K} \right| \,, \end{aligned}$$

(16)

where we have denoted $K_\textbf{x}(\cdot )=K(\textbf{x},\cdot )$ and used the reproducing property $f(\textbf{x})=\langle f,K_\textbf{x}\rangle _{\mathcal {H}_K}$ for all $\textbf{x}\in \mathcal {X}$, and where, for any probability measure $\nu$ on $\mathcal {X}$ and $\textbf{x}\in \mathcal {X}$,

$$\begin{aligned} P_{K,\nu }(\textbf{x}) = \int _\mathcal {X}K(\textbf{x}, \textbf{x}') \, \text{ d }\nu (\textbf{x}') \,, \end{aligned}$$

(17)

is the potential of $\nu$ at $\textbf{x}$. $P_{K,\nu }\in \mathcal {H}_K$ and is called kernel embedding of $\nu$ in ML. In some cases, the potential can be expressed analytically (see. Appendix 6), otherwise it can be estimated by numerical quadrature (Quasi Monte Carlo). Cauchy-Schwartz inequality applied to (16) gives

$$ \left| \int _\mathcal {X}f(\textbf{x})\, \text{ d }\xi (\textbf{x}) - \int _\mathcal {X}f(\textbf{x})\, \text{ d }\mu (\textbf{x}) \right| \le \Vert f\Vert _{\mathcal {H}_K}\,\Vert P_{K,\xi }-P_{K,\mu }\Vert _{\mathcal {H}_K} $$

and therefore

$$ \Vert P_{K,\xi }-P_{K,\mu }\Vert _{\mathcal {H}_K}=\sup _{f\in \mathcal {H}_K:\ \Vert f\Vert _{\mathcal {H}_K}=1} \left| \int _\mathcal {X}f(\textbf{x})\, \text{ d }\xi (\textbf{x}) - \int _\mathcal {X}f(\textbf{x})\, \text{ d }\mu (\textbf{x}) \right| \,. $$

The Maximum Mean Discrepancy (MMD) between $\xi$ and $\mu$ (for the kernel K and set $\mathcal {X}$) is $d_K(\xi ,\mu )=\Vert P_{K,\xi }-P_{K,\mu }\Vert _{\mathcal {H}_K}$. Direct calculation gives

$$\begin{aligned} d_K^2(\xi ,\mu )= & {} \Vert P_{K,\xi }-P_{K,\mu }\Vert _{\mathcal {H}_K}^2 = \int _{\mathcal {X}^2} K(\textbf{x},\textbf{x}')\, \text{ d }(\xi -\mu )(\textbf{x})\text{ d }(\xi -\mu )(\textbf{x}') \end{aligned}$$

(18)

$$\begin{aligned}= & {} \mathbb {E}_{\zeta ,\zeta '\sim \xi } K(\zeta ,\zeta ') + \mathbb {E}_{\zeta ,\zeta '\sim \mu } K(\zeta ,\zeta ') - 2\mathbb {E}_{\zeta \sim \xi , \zeta '\sim \mu } K(\zeta ,\zeta ') \,, \end{aligned}$$

(19)

where the random variables $\zeta$ and $\zeta '$ in (19) are independent, see [49]. When K is the energy distance kernel (10), one recovers the expression (11) for the corresponding MMD. One may refer to [51] for an illuminating exposition on MMD, kernel embedding, and conditions on K (the notion of characteristic kernel) that make $d_K$ a metric on the space of probability measures on $\mathcal {X}$. The distance and Matérn kernels considered in this paper are characteristic.

1.2 Appendix B: Analytical Computation of Potentials for Matérn Kernels

As for tensor-product kernels, the potential is the product of the one-dimensional potentials, we only consider one-dimensional input spaces.

For $\mu$ the uniform distribution on [0, 1] and K the Matérn kernel $K_{5/2,\theta }$ with smoothness $\nu =5/2$ and correlation length $\theta$, see (15), we get

$$\begin{aligned} P_{K_{5/2,\theta },\mu }(x) = \frac{16 \theta }{3 \sqrt{5}} - \frac{1}{15 \theta } (S_\theta (x) + S_\theta (1-x)), \end{aligned}$$

where

$$\begin{aligned} \nonumber S_\theta (x) = \exp \left( - \frac{\sqrt{5}}{\theta } x \right) \left( 5 \sqrt{5} x^2 + 25 \theta x + 8 \sqrt{5} \theta ^2 \right) . \end{aligned}$$

The expressions $P_{K_{\nu ,\theta },\mu }(x)$ for $\nu =1/2$ and $\nu =3/2$ can be found in [40].

When $\mu$ is the standard normal distribution $\mathcal {N}(0,1)$, the potential $P_{K_{5/2,\theta },\mathcal {N}(0,1)}$ is $P_{K_{5/2,\theta },\mathcal {N}(0,1)}(x) = T_\theta (x) + T_\theta (-x),$ where

$$\begin{aligned} T_\theta (x)= & {} \frac{1}{6} \left( \frac{5}{\theta ^2} x^2 + \left( 3 - \frac{10}{\theta ^2} \right) \frac{\sqrt{5}}{\theta } x + \frac{5}{\theta ^2} \left( \frac{5}{\theta ^2} -2 \right) + 3 \right) \\{} & {} \times \, \textrm{erfc} \left( \frac{\frac{\sqrt{5}}{\theta } - x}{\sqrt{2}} \right) \exp \left( \frac{5}{2 \theta ^2} - \frac{\sqrt{5}}{\theta }x\right) \\ {}{} & {} + \frac{1}{3 \sqrt{2 \pi }} \frac{\sqrt{5}}{\theta } \left( 3 - \frac{5}{\theta ^2} \right) \exp \left( -\frac{x^2}{2}\right) . \end{aligned}$$