Distance-based discriminant analysis method and its applications

Abstract

This paper proposes a method of finding a discriminative linear transformation that enhances the data’s degree of conformance to the compactness hypothesis and its inverse. The problem formulation relies on inter-observation distances only, which is shown to improve non-parametric and non-linear classifier performance on benchmark and real-world data sets. The proposed approach is suitable for both binary and multiple-category classification problems, and can be applied as a dimensionality reduction technique. In the latter case, the number of necessary discriminative dimensions can be determined exactly. Also considered is a kernel-based extension of the proposed discriminant analysis method which overcomes the linearity assumption of the sought discriminative transformation imposed by the initial formulation. This enhancement allows the proposed method to be applied to non-linear classification problems and has an additional benefit of being able to accommodate indefinite kernels.

Appendix

This section focuses on the intuition behind the definitions of design matrices R and G specified in (14) and (22). The derivations listed here are mostly based on those developed for the SMACOF multi-dimensional scaling algorithm [6].

Let us consider matrix R that is used in calculation of the majorizing expression of S _W (T) represented by a weighted sum of within-distances. In the derivations that follow, we will assume all weights to be equal to unity, and show afterwards how this assumption can be easily corrected for. We, thus, begin by rewriting a squared within-distance in the vector form:

$$ \left(d_{ij}^W(T)\right)^2 = \sum_{a=1}^m (x_{ia}^{\prime}-x_{ja}^{\prime})^2 = ({\mathbf {x}}_i^{\prime} - {{\mathbf{x}}}_j^{\prime})({{\mathbf{x}}}_i^{\prime} - {{\mathbf{x}}}_j^{\prime})^{\rm T}, $$

(50)

where x ^′ _i and x ^′ _j denote rows i and j from matrix X′ = XT, representing the corresponding observations transformed by T. Noticing that x ^′ _i − x ^′ _j =  (e _i −e _j )^T X′, (50) becomes:

$$ \begin{aligned} \left(d_{ij}^W(T)\right)^2 &= (e_i-e_j)^{\rm T} X^{\prime} X^{\prime {\rm T}} (e_i-e_j)\\ &= {{\mathbf{tr}}} \left(X^{\prime {\rm T}} (e_i-e_j) (e_i-e_j)^{\rm T} X^{\prime}\right)\\ & = {{\mathbf{tr}}} \left(X^{\prime {\rm T}} A_{ij} X^{\prime} \right), \end{aligned} $$

(51)

where A _ij is a square symmetric matrix whose elements are all zeros, except for those four indexed by the combinations of i and j that are either 1 (diagonal) or −1 (off-diagonal). For instance, A ₁₃ for i = 1, j = 3 and N _X  = 3 will have the following form:

$$ A_{13} = \left[{\begin{array}{*{20}c} 1 & 0 & -1\\ 0 & 0 & 0\\ -1 & 0 & 1\\ \end{array}}\right]. $$

(52)

Taking into account (51), the sum of the squared within-distances can be expressed as:

$$ \begin{aligned} \sum_{i < j}^{N_X} \left(d_{ij}^W(T)\right)^2 &= \sum_{i < j}^{N_X} {{\mathbf{tr}}} \left(X^{\prime {\rm T}} A_{ij} X^{\prime}\right)\\ & = {{\mathbf{tr}}}\left(X^{\prime {\rm T}} V X^{\prime}\right)\\ &= {{\mathbf{tr}}} \left(T^{\rm T} X^{\rm T} V X T\right), \end{aligned} $$

(53)

where $V=\sum_{i < j}^{N_X} A_{ij},$ for which there exists an easy computational shortcut. Namely, V is obtained by placing −1 in all off-diagonal entries of the matrix, while the diagonal elements are calculated as negated sums of their corresponding off-diagonal values in rows or columns. That is:

$$ v_{ij} = \left\{{\begin{array}{ll} -1,& \hbox{if }\, i \ne j;\\ -\sum\limits_{k=1,k \ne i}^{N_X} v_{ik} = N_X-1, & \hbox{if}\,i = j;\\ \end{array}} \right. $$

(54)

For instance, coming back to our previous N _X  = 3 example, this technique produces:

$$ \begin{aligned} V &= \sum_{i < j}^{N_X=3} A_{ij}\\ &= \left(\left[{\begin{array}{*{20}c} 1 &-1 & 0\\ -1 & 1 & 0\\ 0 & 0 & 0\\ \end{array}}\right] + \left[ {\begin{array}{*{20}c} 1 & 0 & -1\\ 0 & 0 & 0\\ -1 & 0 & 1\\ \end{array}} \right] + \left[{\begin{array}{*{20}c} 0 & 0 & 0\\ 0 & 1 &-1\\ 0 &-1 & 1\\ \end{array}} \right]\right)\\ & = \left[{\begin{array}{*{20}c} 2 & -1 & -1\\ -1 & 2 & -1\\ -1 & -1 &2\\ \end{array}} \right]. \end{aligned} $$

(55)

It is not difficult to see that the same result applies to the case of non-unitary weights associated with each distance, the only difference being that instead of −1 placed into the off-diagonal elements of V, one should use the negated values of the corresponding weights. And this is exactly how the matrix formulation of $\mu_{S_W}(T,{\bar T}),$ (15), and design matrix R, (14), are obtained:

$$ \begin{aligned} \mu_{S_W}(T,{\bar T}) &= \sum_{i < j}^{N_X} \frac{{\bar w}_{ij}\cdot\left(d^W_{ij}(T)\right)^2}{2\Psi\left(d^W_{ij}({\bar T})\right)} + K_1\\ &= \sum_{i < j}^{N_X} \frac{{\bar w}_{ij}} {\Psi\left(d^W_{ij}({\bar T})\right)} \left[\frac{1}{2} {{\mathbf{tr}}} \left(T^{\rm T} X^{\rm T} A_{ij} X T \right) + K_1^{\prime} \right]\\ & = \frac{1}{2}{{\mathbf{tr}}} \left(T^{\rm T} X^{\rm T} \sum_{i < j}^{N_X} \frac{{\bar w}_{ij}} {\Psi\left(d^W_{ij}({\bar T}) \right)} A_{ij} X T \right) + K_1\\ & =\frac{1}{2}{{\mathbf{tr}}} \left(T^{\rm T} X^{\rm T} R X T \right) + K_1\\ \end{aligned} $$

(56)

In order to derive the formulation of matrix G, as specified for the majorizer of −S _B (T) based on Taylor series expansion (23), we rewrite (22) using the same techniques as we did in (51) arriving at:

$$ -d_{ij}^B(T) \le - \frac{{{\mathbf{tr}}}\left(T^{\rm T} Z^{\rm T} C_{ij} Z {\bar T} \right) }{d^B_{ij}({\bar T})}, $$

(57)

where C _ij  = (e _i −e _{N_X+j} ) (e _i −e _{N_X+j} )^T is a between-class analog of matrix A _ij . From (55), it is apparent that the same type of a computational shortcut used above to obtain V may be exploited here too. Indeed, matrix $F=\sum_{i=1}^{N_X} \sum_{j=1}^{N_Y} C_{ij}$ can be quickly constructed by placing −1 in the off-diagonal elements that correspond to index locations of the between-distances, and subsequently summing with negation to obtain the diagonal entries. An illustration of the technique for N _X  = 2, N _Y  = 3 is shown below:

$$ \begin{aligned} F &= \sum_{i=1}^{N_X=2} \sum_{j=1}^{N_Y=3} C_{ij}\\ &=\left[{\begin{array}{*{20}c} 3 & 0 & -1 & -1 & -1\\ 0 & 3 & -1 & -1 & -1\\ -1 &-1 & 2 & 0 & 0 \\ -1 & -1 & 0 & 2 & 0\\ -1 & -1 & 0 & 0 & 2\\ \end{array}} \right]. \end{aligned} $$

(58)

This is the case of unitary weights. Again, the extension to the non-unitary weight formulation is trivial, and will involve pre-multiplying the off-diagonal entries by the appropriate quantities, which in the case of G are the reciprocals of the squares of the corresponding distances, as shown in (23).