Accelerating kernel classifiers through borders mapping

Abstract

Support vector machine (SVM) and other kernel techniques represent a family of powerful statistical classification methods with high accuracy and broad applicability. Because they use all or a significant portion of the training data, however, they can be slow, especially for large problems. Piecewise linear classifiers are similarly versatile, yet have the additional advantages of simplicity, ease of interpretation and, if the number of component linear classifiers is not too large, speed. Here we show how a simple, piecewise linear classifier can be trained from a kernel-based classifier in order to improve the classification speed. The method works by finding the root of the difference in conditional probabilities between pairs of opposite classes to build up a representation of the decision boundary. When tested on 17 different datasets, it succeeded in improving the classification speed of a SVM for 12 of them by up to two orders of magnitude. Of these, two were less accurate than a simple, linear classifier. The method is best suited to problems with continuum features data and smooth probability functions. Because the component linear classifiers are built up individually from an existing classifier, rather than through a simultaneous optimization procedure, the classifier is also fast to train.

Appendix A: Subsampling

Let $n_i$ be the number of samples of the ith class such that:

$$\begin{aligned} n_i \ge n_{i-1} \end{aligned}$$

Let $0 \le \alpha (n) \le 1$ be a function used to subsample each of the class distributions in turn:

$$\begin{aligned} n_i^\prime = \alpha (n_i) n_i \end{aligned}$$

We wish to retain the rank ordering of the class sizes:

$$\begin{aligned} \alpha (n_i) n_i \ge \alpha (n_{i-1}) n_{i-1} \end{aligned}$$

while ensuring that the smallest classes have some minimum representation:

$$\begin{aligned} \alpha (n_i) \le \alpha (n_{i-1}) \end{aligned}$$

(11)

Thus:

$$\begin{aligned} \frac{{\rm d}}{{\rm d} n} \left[ n \alpha (n) \right]&= \alpha (n) + n \frac{{\rm d} \alpha }{{\rm n}} \ge 0\nonumber \\ \frac{{\rm d}\alpha }{{\rm d}n}&\ge - \frac{\alpha (n)}{n} \end{aligned}$$

(12)

The simplest means of ensuring that both (11) and (12) are fulfilled is to multiply the right side of (12) with a constant, $0 \le \zeta \le 1$, and equate it with the left side:

$$\begin{aligned} \frac{{\mathrm{d}} \alpha }{{\mathrm{d}} n} = - \frac{\zeta \alpha (n)}{n} \end{aligned}$$

Integrating:

$$\begin{aligned} \alpha (n)=Cn^{-\zeta } \end{aligned}$$

The parameter, $C$, is set such that $n_1^\prime =n_1$:

$$\begin{aligned} C = n_1^\zeta \end{aligned}$$

while $\zeta$ is set such that:

$$\begin{aligned} f\sum _i n_i= & {} \sum _i \alpha (n_i) n_i \\= & {} n_1^\zeta \sum _i n_i^{1-\zeta } \end{aligned}$$

where $0< f< 1$ is the desired fraction of training data.