Manifold Based Local Classifiers: Linear and Nonlinear Approaches

Abstract

In case of insufficient data samples in high-dimensional classification problems, sparse scatters of samples tend to have many ‘holes’—regions that have few or no nearby training samples from the class. When such regions lie close to inter-class boundaries, the nearest neighbors of a query may lie in the wrong class, thus leading to errors in the Nearest Neighbor classification rule. The K-local hyperplane distance nearest neighbor (HKNN) algorithm tackles this problem by approximating each class with a smooth nonlinear manifold, which is considered to be locally linear. The method takes advantage of the local linearity assumption by using the distances from a query sample to the affine hulls of query’s nearest neighbors for decision making. However, HKNN is limited to using the Euclidean distance metric, which is a significant limitation in practice. In this paper we reformulate HKNN in terms of subspaces, and propose a variant, the Local Discriminative Common Vector (LDCV) method, that is more suitable for classification tasks where the classes have similar intra-class variations. We then extend both methods to the nonlinear case by mapping the nearest neighbors into a higher-dimensional space where the linear manifolds are constructed. This procedure allows us to use a wide variety of distance functions in the process, while computing distances between the query sample and the nonlinear manifolds remains straightforward owing to the linear nature of the manifolds in the mapped space. We tested the proposed methods on several classification tasks, obtaining better results than both the Support Vector Machines (SVMs) and their local counterpart SVM-KNN on the USPS and Image segmentation databases, and outperforming the local SVM-KNN on the Caltech visual recognition database.

Appendix

• Theorem 1: Let P and $P_{{\text{NS}}}^{\left( i \right)}$ be the projection matrices of the subspaces $R\left( {S_T^K } \right)$ and $N\left( {S_i^K } \right)$, $i = 1, \ldots ,C$, respectively. Then P and $P_{{\text{NS}}}^{\left( i \right)}$ commute, i.e.:

  $$P_{{\text{NS}}}^{\left( i \right)} P = PP_{{\text{NS}}}^{\left( i \right)} ,\quad i = 1,...,C.$$

  Proof of the theorem is omitted since it can be derived as in the proof of Theorem 1 in [21].

• Theorem 2: Assume that there are C classes in the training set. For a query x _q $\left\| {P_{{\text{NS}}}^{\left( i \right)} \left( {x_q - \mu _i } \right)} \right\| \leqslant \left\| {P_{{\text{NS}}}^{\left( j \right)} \left( {x_q - \mu _j } \right)} \right\|$ implies that $\left\| {P_{{\text{int}}}^{\left( i \right)} \left( {x_q - \mu _i } \right)} \right\| \leqslant \left\| {P_{{\text{int}}}^{\left( j \right)} \left( {x_q - \mu _j } \right)} \right\|$ for $i,j = 1,...,C$, and $i \ne j$.

Proof: We first recall several facts from [13] (see Lemma 1 of [13]). For each $i = 1, \ldots ,C,$ it holds $N\left( {S_T^K } \right) \subset N\left( {S_i^K } \right)$, where N(A) denotes the null space of a matrix A. Consequently, $N\left( {S_T^K } \right)$ and $R\left( {S_i^K } \right)$ are orthogonal, where $R\left( {S_i^K } \right)$ is the range of $S_i^K$. This implies the identity $\left( {I - P} \right)\left( {I - P_{{\text{NS}}}^{\left( i \right)} } \right) = 0$ or $\left( {I - P} \right) = \left( {I - P} \right)P_{{\text{NS}}}^{\left( i \right)}$.

Thus, we can write:

$$\begin{array}{*{20}c} {\left\| {P_{{\text{NS}}}^{\left( i \right)} \left( {x_q - \mu _i } \right)} \right\| = \left\| {PP_{{\text{NS}}}^{\left( i \right)} \left( {x_q - \mu _i } \right) + \left( {I - P} \right)P_{{\text{NS}}}^{\left( i \right)} \left( {x_q - \mu _i } \right)} \right\| = \left\| {PP_{{\text{NS}}}^{\left( i \right)} \left( {x_q - \mu _i } \right)} \right\| + } \\ {\left\| {\left( {I - P} \right)P_{{\text{NS}}}^{\left( i \right)} \left( {x_q - \mu _i } \right)} \right\| = \left\| {PP_{{\text{NS}}}^{\left( i \right)} \left( {x_q - \mu _i } \right)} \right\| + \left\| {\left( {I - P} \right)\left( {x_q - \mu _i } \right)} \right\|.} \\ \end{array} $$

(25)

We now note that the vector $\left( {I - P} \right)\left( {x_q - \mu _i } \right)$ is the same for each class (i.e., it does not depend on the class index i) since we have shown in [21] that (I − P)μ _i is a so-called common vector for the class consisting of all samples in $V = \left\{ {x_m^i } \right\}_{m = 1,i = 1}^{K,C}$ and that in fact (I − P)x is the same vector for all x in the affine hull of V.

Thus, we have shown that:

$$\left\| {P_{{\text{NS}}}^{\left( i \right)} \left( {x_q - \mu _i } \right)} \right\| = \left\| {PP_{{\text{NS}}}^{\left( i \right)} \left( {x_q - \mu _i } \right)} \right\| + \left\| v \right\|,$$

(26)

for some vector v independent of the class index i. The assertion of Theorem 2 now immediately follows from this fact. □