Joint learning affinity matrix and representation matrix for robust low-rank multi-kernel clustering

Abstract

Multi-kernel subspace clustering has attracted widespread attention, because it can process nonlinear data effectively. It usually solves the representation coefficient between data by the subspace clustering optimization model, and then the constructed affinity matrix is input into the spectral clustering method to get the final clustering result. Obviously, the quality of the affinity matrix (graph) has a significant impact on the final clustering result. Unfortunately, there are two deficiencies in the previous multi-kernel subspace clustering methods as follows: 1) this typical two-phase method restricts the learning of the affinity matrix; 2) it does not fully extract the data global structure mapped to the kernel space. In order to solve these two problems simultaneously, a novel low-rank multi-kernel subspace clustering method incorporating a joint learning scheme, namely JALSC, is proposed. The innovation of this method is reflected in the following two aspects: 1) the adaptive local structure is used to learn the representation of the data and the affinity graph in the integrated objective function at the same time. The optimal affinity graph obtained by the one-step learning scheme helps to improve the clustering performance; 2) our method uses a non-convex low-rank approximation function to constrain the consensus kernel to preserve the global structure of the data after mapping to the feature space. A mass of experiments on several commonly used datasets show that JALSC obtains the best clustering performance and has better robustness compared with several advanced multi-kernel clustering methods.

Appendices

Appendix A

Lemma 1

Let the singular value decomposition (SVD) of $\boldsymbol {P}\in \mathbb {R}^{m\times n}$ be P = UΣV^T with Σ = diag(σ₁,σ₂,…,σ_i),where $r={\min \limits } (m,n)$.If we have σ₁ > σ₂ > ⋯ > σ_r,and the corresponding weights ω₁ ≤ ω₂ ≤⋯ ≤ ω_r,for the following problem:

$$ \min_{\boldsymbol{Q} \in \boldsymbol{R}^{m \times n}}\left( \frac{1}{2}\|\boldsymbol{Q}-\boldsymbol{P}\|_{F}^{2}+\lambda\|\boldsymbol{Q}\|_{w, S_{p}}^{p}\right) $$

(A.1)

its closed solution is Q = UΛV^T,and Λ is the singular value diagonal matrix of the matrix Q.Then, problem (A.1) can be translated into the solution of Λ = diag(δ₁,δ₂,...,δ_r) in the following problem:

$$ \begin{array}{c} \underset{\delta_{1}, \delta_{2}, \ldots, \delta_{r}} \min {\sum}_{i=1}^{r}\left[\frac{1}{2}\left( \delta_{i}-\sigma_{i}\right)^{2}+\lambda w_{i} {\delta_{i}^{p}}\right], \\ \text { s.t. } \delta_{i} \geq 0 ,i=1,2, {\ldots} r \end{array} $$

(A.2)

The Generalized Soft Threshold (GST) algorithm can be implemented to solve (A.2) by decoupling it into r independent sub-problems, which is described in Lemma 2.

Lemma 2

Given y ∈R and λ > 0, we have the following optimization problem of x:

$$ \min_{x} \left[ \frac{1}{2}(x-y)^{2}+\lambda\left|x\right|^{p} \right] $$

(A.3)

Let $\tau _p^{GST}(\lambda )$ and x^∗ be the thresholds and the corresponding minimum values in the GST method, and their solutions are obtained by substituting them into (A.3), i.e

$$ \frac{1}{2}\left( x^{*}-\tau_{p}^{GST}(\lambda)\right)^{2}+\lambda (x^{*})^{p}=\frac{1}{2}(\tau_{p}^{GST}(\lambda))^{2} $$

(A.4)

After a series of simple mathematical substitution of the above equation, we can get:

$$ \begin{array}{c} x^{*}=[2\lambda(1-p)]^{\frac{1}{2-p}}\\ \tau_{p}^{GST}(\lambda)=[2\lambda(1-p)]^{\frac{1}{2-p}}+\lambda p [2\lambda(1-p)]^{\frac{p-1}{2-p}} \end{array} $$

(A.5)

Then, we get the solution of formula (A.3):

$$ \begin{array}{c} x^{*}=\left\{\begin{array}{ll}0 & |y| \leq \tau_{p}^{\text{GST}}(\lambda) \\\operatorname{sgn}(y) S_{p}^{GST}(|y|, \lambda) & |y|>\tau_{p}^{GST}(\lambda) \end{array}\right. \end{array} $$

(A.6)

where $S_{p}^{GST}(|y|, \lambda )$ can be obtained by solving follow problem:

$$ \lambda p (S_{p}^{GST}(|y|, \lambda))^{p-1}+S_{p}^{GST}(|y|, \lambda)-|y|=0 $$

(A.7)

Appendix B

Given the following vector form problem:

$$ \underset{\boldsymbol{c}}{\arg\min}\|\boldsymbol{c}+\boldsymbol{d}\|_{F}^{2} \quad \text { s.t. } \boldsymbol{c}^{T} \boldsymbol{1}=1,\boldsymbol{c} \succeq 0 $$

(A.8)

where $\boldsymbol {c}\in \mathbb {R}^{n\times {1}}$ and $\boldsymbol {d}\in \mathbb {R}^{n\times {1}}$ are target vector and known distance vector,respectively. Specifying that the learned affinity graph c has the first k important edges,so in theory there are k non-zero terms in c.The solution of the (A.8) is:

$$ \boldsymbol{c}=\left( \frac{1+{\sum}_{i=1}^{k}\boldsymbol{d}_{i}^{\to}}{k}\boldsymbol{1}-\boldsymbol{d}\right)_{+} $$

(A.9)

where $\boldsymbol {d}_{i}^{\to }$ is the ascending order of the elements in d.

Proof

Due to the existence of constraint c^T1 = 1,c ≽ 0 in problem (A.8), its Lagrangian function is written as:

$$ \mathcal{L}(\boldsymbol{c},\kappa,\rho)=\frac{1}{2}\|\boldsymbol{c}+\boldsymbol{d}\|_{F}^{2}-\kappa\left( \boldsymbol{c}^{T} \boldsymbol{1}-1\right)-\rho^{T} \boldsymbol{c} $$

(A.10)

where κ ≽ 0 and ρ ≽ 0 are the Lagrangian multipliers.By means of the KKT condition, we can obtain the following relation:

$$ \left\{\begin{array}{l}\boldsymbol{c}+\boldsymbol{d}-\kappa \boldsymbol{1}-\rho=0 \\ \boldsymbol{c}^{T} \mathbf{1}-1=0 \\ \rho^{T} \boldsymbol{c}=0 \end{array}\right. $$

(A.11)

Only when the conditions c_i > 0 and ρ_i = 0 are met, the third equation holds.We have

$$ \boldsymbol{c}=(\kappa \boldsymbol{1}-\boldsymbol{d})_{+} $$

(A.12)

There are k positive elements in c ≽ 0. Namely:

$$ \kappa-{\boldsymbol{d}_{k}^{\to}}>0 ,\kappa-{\boldsymbol{d}_{k+1}^{\to}} \leq 0 $$

(A.13)

According to (A.12) together with c^T1 = 1, we have

$$ {\sum}_{i=1}^{k}\left( \kappa-{\boldsymbol{d}_{i}^{\to}}\right)=1 $$

(A.14)

$$ \kappa=\frac{1+{\sum}_{i=1}^{k} \boldsymbol{d}_{i}^{\to}}{k} $$

(A.15)

Finally, by simply substituting the κ in above equation into (A.12) , a closed-form solution in the same form as (A.8) is obtained. □