Two-dimensional k-subspace clustering and its applications on image recognition

Abstract

Image clustering plays an important role in computer vision and machine learning. However, most of the existing clustering algorithms flatten the image into one-dimensional vector as an image representation for subsequent learning without fully considering the spatial relationship between pixels, which may lose some useful intrinsic structural information of the matrix data samples and result in high computational complexity. In this paper, we propose a novel two-dimensional k-subspace clustering (2DkSC). By projecting data samples into a discriminant low-dimensional space, 2DkSC maximizes the between-cluster difference and meanwhile minimizes within-cluster distance of matrix data samples in the projected space, thus dimensionality reduction and clustering can be realized simultaneously. The weight between the between-cluster and within-cluster terms is derived from a Bhattacharyya upper bound, which is determined by the involved input data samples. This weighting constant makes the proposed 2DkSC adaptive without setting any parameters, which improves the computational efficiency. Moreover, 2DkSC can be effectively solved by a standard eigenvalue decomposition problem. Experimental results on three different types of image datasets show that 2DkSC achieves the best clustering results in terms of average clustering accuracy and average normalized mutual information, which demonstrates the superiority of the proposed method.

Appendix

In the appendix, we present the proof procedure of the relevant Bhattacharyya error bound. It is further explained that the weighting constant $\Delta _{i}$ balances the importance between clusters and the importance within clusters, which is derived by minimizing an upper bound of theoretical framework of the Bhattacharyya error bound optimality.

The Bhattacharyya error [43] is a close upper bound to the Bayes error, which is given by

$$\begin{aligned} \epsilon _B=\sum \limits _{i<j}^k\sqrt{P_iP_{j}}\int \sqrt{p_i({\textbf {X}})p_{j}({\textbf {X}})}d{\textbf {X}}, \end{aligned}$$

(1)

where ${\textbf {X}}$ is a data sample, $P_i$ is the prior probability, and $p_i({\textbf {X}})$ is the probability density function of the i-th class of the data.

Proposition 1

Assume $P_i$ and $p_i({\textbf {X}})$ are the prior probability and the probability density function of the i-th class for the training data set T, respectively, and the data samples in each class are independent and identically normally distributed. Let $p_1({\textbf {X}}), p_2({\textbf {X}}),\ldots , p_k({\textbf {X}})$ be the Gaussian functions given by $p_i({\textbf {X}})=\mathcal {N}({\textbf {X}}\mid {\overline{{\textbf {X}}}}_i, \varvec{\Sigma }_i)$, where ${\overline{{\textbf {X}}}}_i$ and $\varvec{\Sigma }_i$ are the class mean and the class covariance matrix, respectively. We further suppose $\varvec{\Sigma }_i=\varvec{\Sigma }$, $i=1,2,\ldots ,k$, where $\varvec{\Sigma }$ is the covariance matrix of the data set T, and ${\overline{{\textbf {X}}}}_i$ and $\varvec{\Sigma }$ can be estimated accurately from T. Then for arbitrary projection vector ${\textbf {w}}\in \mathbb {R}^{m}$, the Bhattacharyya error bound $\epsilon _B$ defined by (1) on the data set $\widetilde{T}=\{\widetilde{{\textbf {X}}}_i\mid \widetilde{{\textbf {X}}}_i={\textbf {w}}^T{\textbf {X}}_i\in \mathbb {R}^{1\times n}\}$ satisfies the following [34]:

$$\begin{aligned} \begin{aligned} \epsilon _B \le&-\frac{a}{8}\sum \limits _{i<j}^k\sqrt{P_iP_j}{\Vert {\textbf {w}}^T({\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}})\Vert _2^2}+\frac{a}{8}\Delta \sum _{i=1}^{k}\sum _{s=1}^{N_i}\Vert {\textbf {w}}^T({\textbf {X}}_{is}-\overline{{\textbf {X}}}_i)\Vert _2^2\\&+\sum \limits _{i<j}^k\sqrt{P_iP_j},\\ \end{aligned} \end{aligned}$$

(2)

where $\Delta =\frac{1}{4}\sum \nolimits _{i<j}^k\frac{\sqrt{N_iN_j}}{N}\Vert {\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}}\Vert _F^2$, $P_i=\frac{N_i}{N}$, $P_j=\frac{N_j}{N}$, and $a>0$ is some constant.

Proof

We first note that $p_i(\widetilde{{{\textbf {X}}}})=\mathcal {N}(\widetilde{{{\textbf {X}}}}\mid \widetilde{{\overline{{\textbf {X}}}}}_i, \widetilde{\varvec{\Sigma }})$, where $\widetilde{ {{\textbf {X}}}}_i={\textbf {w}}^T{\textbf {X}}_i$, $\widetilde{{\overline{{\textbf {X}}}}}_i={\textbf {w}}^T\overline{{\textbf {X}}}_i\in \mathbb {R}^{1\times n}$ is the i-class mean, and $\widetilde{\varvec{\Sigma }}$ is the covariance matrix in the $1\times n$ projected space. Denote

$$\begin{aligned} {\textbf {D}}=\begin{pmatrix}{} {\textbf {w}}^T{\textbf {X}}_1\\ \vdots \\ {\textbf {w}}^T{\textbf {X}}_N\end{pmatrix}^T\in \mathbb {R}^{n\times N} ~\text {and}~\widetilde{\overline{{\textbf {X}}}}_{{\textbf {I}}}=\begin{pmatrix} {\textbf {w}}^T\overline{{\textbf {X}}}_{t_1}\\ \vdots \\ {\textbf {w}}^T\overline{{\textbf {X}}}_{t_N}\end{pmatrix}^T\in \mathbb {R}^{n\times N}. \end{aligned}$$

(3)

Then $\widetilde{\varvec{\Sigma }}=({\textbf {D}}-\widetilde{\overline{{\textbf {X}}}}_{{\textbf {I}}}) ({\textbf {D}}-\widetilde{\overline{{\textbf {X}}}}_{{\textbf {I}}})^T$.

According to [44], we have

$$\begin{aligned} \begin{aligned} \int \sqrt{p_i(\widetilde{{\textbf {X}}})p_{j}(\widetilde{{\textbf {X}}})}= e^{-\frac{1}{8}(\widetilde{{\overline{{\textbf {X}}}}_{i}}-\widetilde{{\overline{{\textbf {X}}}}_{j}})\widetilde{\varvec{\Sigma }}^{-1}(\widetilde{{\overline{{\textbf {X}}}}_{i}}-\widetilde{{\overline{{\textbf {X}}}}_{j}})^T}. \end{aligned} \end{aligned}$$

(4)

The upper bound of the error $\epsilon _B$ can be estimated as

$$\begin{aligned} \begin{aligned} \epsilon _B&=\sum \limits _{i<j}^k \sqrt{P_iP_j} e^{-\frac{1}{8}(\widetilde{{\overline{{\textbf {X}}}}_{i}}-\widetilde{{\overline{{\textbf {X}}}}_{j}})\widetilde{\varvec{\Sigma }}^{-1}(\widetilde{{\overline{{\textbf {X}}}}_{i}}-\widetilde{{\overline{{\textbf {X}}}}_{j}})^T}\\&=\sum \limits _{i<j}^k \sqrt{P_iP_j} e^{-\frac{1}{8}\Vert (\widetilde{{\overline{{\textbf {X}}}}_{i}}-\widetilde{{\overline{{\textbf {X}}}}_{j}})\widetilde{\varvec{\Sigma }}^{-\frac{1}{2}}\Vert _2^2}\\&\le \sum \limits _{i<j}^k \sqrt{P_iP_j} \left( 1-\frac{a}{8}\Vert (\widetilde{{\overline{{\textbf {X}}}}_{i}}-\widetilde{{\overline{{\textbf {X}}}}_{j}})\widetilde{\varvec{\Sigma }}^{-\frac{1}{2}}\Vert _2^2\right) \\&=\sum \limits _{i<j}^k \sqrt{P_iP_j} -\frac{a}{8}\sum \limits _{i<j}^k\sqrt{P_iP_j}\cdot \Vert ({\textbf {w}}^T{{\overline{{\textbf {X}}}}_{i}}-{\textbf {w}}^T{{\overline{{\textbf {X}}}}_{j}})\widetilde{\varvec{\Sigma }}^{-\frac{1}{2}}\Vert _2^2\\&\le \sum \limits _{i<j}^k \sqrt{P_iP_j} -\frac{a}{8}\sum \limits _{i<j}^k\sqrt{P_iP_j}\cdot \frac{\Vert ({\textbf {w}}^T{{\overline{{\textbf {X}}}}_{i}}-{\textbf {w}}^T{{\overline{{\textbf {X}}}}_{j}})\Vert _2^2}{\Vert \widetilde{\varvec{\Sigma }}^{\frac{1}{2}}\Vert _F^2}\\&\le \sum \limits _{i<j}^k \sqrt{P_iP_j} -\frac{a}{8}\sum \limits _{i<j}^k\sqrt{P_iP_j}\cdot \Vert {\textbf {w}}^T({\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}})\Vert _2^2\\&\quad +\frac{a}{8}\sum \limits _{i<j}^k \sqrt{P_iP_j}\cdot \Delta _{ij}'\Vert \widetilde{\varvec{\Sigma }}^{\frac{1}{2}}\Vert _F^2, \end{aligned} \end{aligned}$$

(5)

where $\Delta _{ij}'= \frac{1}{4}\Vert {\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}}\Vert _F^2$, $a>0$ is some constant.

For the first inequality of (5), note that the real value function $f(z)=e^{-z}$ is concave when $z\in [0,b]$, $b>0$; therefore, $e^{-z}\le 1-\frac{1-e^{-b}}{b}z$. By taking $a=\frac{1-e^{-b}}{b}$ and noting $\widetilde{{\overline{{\textbf {X}}}}_{i}}={\textbf {w}}^T{\overline{{\textbf {X}}}}_{i}$, the first inequality is obtained. For the second inequality, we first note that for any ${\textbf {z}}\in \mathbb {R}^{1\times {n}}$ and an invertible ${\textbf {A}}\in \mathbb {R}^{n\times n}$, $\Vert {\textbf {z}}\Vert _2=\Vert ({\textbf {z}}{} {\textbf {A}}){\textbf {A}}^{-1}\Vert _2\le \Vert {\textbf {z}}{} {\textbf {A}}\Vert _2\cdot \Vert {\textbf {A}}^{-1}\Vert _F$, which implies $\Vert {\textbf {z}}{} {\textbf {A}}\Vert _2\ge \frac{\Vert {\textbf {z}}\Vert _2}{\Vert {\textbf {A}}^{-1}\Vert _F}$. By taking ${\textbf {z}}={\textbf {w}}^T{{\overline{{\textbf {X}}}}_{i}}-{\textbf {w}}^T{{\overline{{\textbf {X}}}}_{j}}$ and ${\textbf {A}}=\widetilde{\varvec{\Sigma }}^{-\frac{1}{2}}$, we get the second inequality. For the last inequality, since $\Vert {\textbf {w}}\Vert _2=1$, $\Vert {\textbf {w}}^T({\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}})\Vert _2^2\le \Vert {\textbf {w}}\Vert _2^2\cdot \Vert {\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}}\Vert _F^2 = \Vert {\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}}\Vert _F^2$ and $\frac{1}{\Vert \widetilde{\varvec{\Sigma }}^{\frac{1}{2}}\Vert _F^2} \left( 1-\frac{1}{\Vert \widetilde{\varvec{\Sigma }}^{\frac{1}{2}}\Vert _F^2}\right) \le \frac{1}{4}$, we have

$$\begin{aligned} \begin{aligned}&\left( \Vert {\textbf {w}}^T({\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}})\Vert _2^2 -\frac{\Vert {\textbf {w}}^T({\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}})\Vert _2^2}{\Vert \widetilde{\varvec{\Sigma }}^{\frac{1}{2}}\Vert _F^2}\right) \cdot \frac{1}{\Vert \widetilde{\varvec{\Sigma }}^{\frac{1}{2}}\Vert _F^2}\\&\quad =\Vert {\textbf {w}}^T({\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}})\Vert _2^2\cdot \frac{1}{\Vert \widetilde{\varvec{\Sigma }}^{\frac{1}{2}}\Vert _F^2} \left( 1-\frac{1}{\Vert \widetilde{\varvec{\Sigma }}^{\frac{1}{2}}\Vert _F^2}\right) \\&\quad \le \frac{1}{4}\Vert {\textbf {w}}^T({\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}})\Vert _2^2\\&\quad \le \frac{1}{4}\Vert {\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}}\Vert _F^2\\&\quad = \Delta _{ij}'. \end{aligned} \end{aligned}$$

(6)

which implies

$$\begin{aligned} \begin{aligned}&-\frac{\Vert {\textbf {w}}^T({\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}})\Vert _2^2}{\Vert \widetilde{\varvec{\Sigma }}^{\frac{1}{2}}\Vert _F^2} \le -\Vert {\textbf {w}}^T({\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}})\Vert _2^2 +\Delta _{ij}'\cdot \Vert \widetilde{\varvec{\Sigma }}^{\frac{1}{2}}\Vert _F^2. \end{aligned} \end{aligned}$$

(7)

By multiplying $\frac{a}{8}\sqrt{P_iP_j}$ to both sides of (7) and summing it over all $1\le i<j\le k$, we obtain the last inequality of (5).

Take $\Delta =\sum \nolimits _{i<j}^k\sqrt{P_iP_j}\Delta _{ij}'= \frac{1}{4}\sum \nolimits _{i<j}^k\frac{\sqrt{N_iN_j}}{N}\Vert {\overline{{\textbf {X}}}_{i}-\overline{{\textbf {X}}}_{j}}\Vert _F^2$, and note that $\Vert \widetilde{\varvec{\Sigma }}^{\frac{1}{2}}\Vert _F^2=\sum \nolimits _{i=1}^{k}\sum \nolimits _{s=1}^{N_i}\Vert {\textbf {w}}^T({\textbf {X}}_{s}^{i}-\overline{{\textbf {X}}}_i)\Vert _2^2$, we then obtain (2). $\square$