Semi-supervised Kernel Fisher discriminant analysis based on exponential-adjusted geometric distance

Abstract

Fisher discriminant analysis (FDA) is a widely used dimensionality reduction tool in pattern recognition. However, FDA cannot obtain an optimal subspace for classification without sufficient labeled samples. Thus, semi-supervised discriminant analysis has attracted great attention in recent years. In this paper, the proposed method employs the exponential-adjusted geometric distance as the measure of similarity, which modifies the exponential function and the scaling factor. The distance not only satisfies the global and local consistency requirements, but also the similarity matrix obtained is more consistent with the real data distribution, thus improves the dimensionality reduction performance. First, in order to deal with the nonlinear separated data, the kernel function is used to map the original data into the high-dimensional feature space. Then, both labeled and unlabeled data in feature space are used to capture the consistence assumption of geometrical structure based on exponential-adjusted geometric distance, which are incorporated into the objection function of local Fisher discriminant analysis as a regularization term. Eventually, the optimal projection matrix is obtained by maximizing the objective function. Experiments on artificial datasets, UCI benchmark datasets, and high-dimensional recognition problems indicate that the presented technique has a significantly improvement in discriminant performance compared with the-state-of-art dimensionality reduction techniques.

Appendix

The proposed distance measure can simultaneously satisfy the following four properties, and the specific proof process is as follows:

• (a) Reflexivity.

If and only if ${{\varvec{x}}}_{i}={{\varvec{x}}}_{j}$;

Proof

When ${\varvec{x}}_{i} = {\varvec{x}}_{j}$, since $d\left( {{\varvec{p}}_{k} ,{\varvec{p}}_{k + 1} } \right)$ is the Euclidean distance of any two adjacent data points on the minimum path between the vertices ${\varvec{x}}_{i}$ and ${\varvec{x}}_{j}$ on graph ${\varvec{G}}$. ${\varvec{x}}_{i} = {\varvec{x}}_{j}$, then the minimum path between ${\varvec{x}}_{i}$ and ${\varvec{x}}_{j}$ is 0, and $d\left( {{\varvec{p}}_{k} ,{\varvec{p}}_{k + 1} } \right) = 0,k = 1, \cdots ,\left| {\varvec{p}} \right|$

$$\begin{aligned} D_{i,j}^{\rho } & = \frac{1}{\rho }\ln \left( {1 + \mathop \sum \limits_{k = 1}^{{\left| {\varvec{p}} \right|}} \left( {e^{{\frac{{\rho d\left( {{\varvec{p}}_{k} ,{\varvec{p}}_{k + 1} } \right)}}{{\delta_{k} \delta_{k + 1} }}}} - 1} \right)} \right) \\ & = \frac{1}{\rho }\ln \left( {1 + 0} \right) = 0 \\ \end{aligned}$$

• (b) Nonnegative

Proof

$$\begin{aligned} & e^{{\frac{{\rho d\left( {{\varvec{p}}_{k} ,{\varvec{p}}_{k + 1} } \right)}}{{\delta_{k} \delta_{k + 1} }}}} \ge 1,\quad \left( {e^{{\frac{{\rho d\left( {{\varvec{p}}_{k} ,{\varvec{p}}_{k + 1} } \right)}}{{\delta_{k} \delta_{k + 1} }}}} - 1} \right) \ge 0, \\ & 1 + \mathop \sum \limits_{k = 1}^{{\left| {\varvec{p}} \right|}} \left( {e^{{\rho d\left( {{\varvec{p}}_{k} ,{\varvec{p}}_{k + 1} } \right)/\left( {\delta_{k} \delta_{k + 1} } \right)}} - 1} \right) \ge 1 \\ & \frac{1}{\rho }ln\left( {1 + \mathop \sum \limits_{k = 1}^{{\left| {\varvec{p}} \right|}} \left( {e^{{\rho d\left( {{\varvec{p}}_{k} ,{\varvec{p}}_{k + 1} } \right)/\left( {\delta_{k} \delta_{k + 1} } \right)}} - 1} \right)} \right) \ge 0 \\ \end{aligned}$$

• (c) Symmetry

Proof

Since,

$$\frac{{d\left( {{\varvec{p}}_{k} ,{\varvec{p}}_{k + 1} } \right)}}{{\delta_{k} \delta_{k + 1} }} = \frac{{d({\varvec{p}}_{k + 1} ,{\varvec{p}}_{k} )}}{{\delta_{k + 1} \delta_{k} }}$$

So,

$$D_{i,j}^{\rho } = D_{j,i}^{\rho }$$

• (d) Triangle inequality

Proof

Assume,

$$D_{i,j}^{\rho } > D_{i,m}^{\rho } + D_{m,j}^{\rho }$$

According to the definition,

$$\mathop {{\text{min}}}\limits_{{{\varvec{p}} \in {\varvec{P}}_{ij} }} \mathop \sum \limits_{k = 1}^{{\left| {\varvec{p}} \right|}} \left( {e^{{\rho d\left( {{\varvec{p}}_{k} ,{\varvec{p}}_{k + 1} } \right)/\left( {\delta_{k} \delta_{k + 1} } \right)}} - 1} \right)$$

should at least take

$$\mathop {{\text{min}}}\limits_{{{\varvec{p}} \in {\varvec{P}}_{im} }} \mathop \sum \limits_{k = 1}^{{\left| {\varvec{p}} \right|}} \left( {e^{{\rho d\left( {{\varvec{p}}_{k} ,{\varvec{p}}_{k + 1} } \right)/\left( {\delta_{k} \delta_{k + 1} } \right)}} - 1} \right) + \mathop {{\text{min}}}\limits_{{{\varvec{p}} \in {\varvec{P}}_{mj} }} \mathop \sum \limits_{k = 1}^{{\left| {\varvec{p}} \right|}} \left( {e^{{\rho d\left( {{\varvec{p}}_{k} ,{\varvec{p}}_{k + 1} } \right)/\left( {\delta_{k} \delta_{k + 1} } \right)}} - 1} \right)$$

So the assumption is not true,

Proving $D_{i,j}^{\rho } < D_{i,m}^{\rho } + D_{m,j}^{\rho }$.