An algorithm of non-negative matrix factorization with the nearest neighbor after per-treatments

Abstract

Clustering is a hot topic in machine learning. For high dimension data, nonnegative matrix factorization (NMF) is a crucial technology in clustering. However, NMF has some disadvantages. First, NMF clusters data in original space while outliers and noise will weaken NMF clustering results. Second, NMF does not take local structure which is beneficial for clustering of data into consideration. To address these two disadvantages, a new algorithm is proposed called nonnegative matrix factorization with the nearest neighbor after per-treatments (PNNMF). Per-treatments are used to alleviate effects of outliers and noise. After per-treatments, some credible connected components generated by the nesrest neighbor of data are chosen to capture local structure. Moreover a new initialization for basis matrix is proposed basing these credible connected components. Experiments on real data sets confirm the effectiveness of PNNMF.

Appendix

To proof Theorem 1, we first give a definition and two lemmas.

Definition 1: $G(v, v^{\prime })$ is an auxiliary function of F(v) if $G(v, v^{\prime })$ sastisfies: 1.$G(v, v^{\prime })$ $\geqslant$ F(v); 2.G(v,v) = F(v).

Lemma 1: If G is the auxiliary function of F, F is unincreased under the condition

$$ \begin{array}{@{}rcl@{}} v^{t+1} = \mathop{\arg\min}_{v} G(v, v^{t}). \end{array} $$

Proof: F(v^t+ 1) = G(v^t+ 1,v^t+ 1) $\leqslant$ G(v^t+ 1,v^t) $\leqslant$ G(v^t,v^t) = F(v^t). $_{\square }$

We can rewrite the cost function (16) as

$$ \begin{array}{@{}rcl@{}} O = tr({X^{P}}^{T}X^{P}) - 2\sum\limits^{n}_{i=1}\sum\limits^{k}_{j=1} V_{ij}(U^{T}X)_{ji} + \sum\limits^{n}_{i=1}\sum\limits^{k}_{j=1} V_{ij}(U^{T}UV^{T})_{ji} \end{array} $$

$$ \begin{array}{@{}rcl@{}} + \lambda \sum\limits^{k}_{i=1}\sum\limits^{n}_{j=1} V^{T}_{ij}((H^{T}-B^{T})(H-B)V)_{ji}. \end{array} $$

Now we consider the element v_ab in V. To update the v_ab, we denote the related part of v_ab in O as:

$$ \begin{array}{@{}rcl@{}} F_{ab} = 2v_{ab}(U^{T}X^{P})_{ab}+v_{ab}(U^{T}UV^{T})_{ab}+(VU^{T}U)_{ab}v^{T}_{ab}+\lambda v^{T}_{ba}((H^{T}-B^{T})(H-B)V)_{ab} \end{array} $$

$$ \begin{array}{@{}rcl@{}} +\lambda (V^{T}(H^{T}-B^{T})(H-B))_{ba}v_{ab}. \end{array} $$

Basing F_ab, we can calculate its first-order derivative and second-order derivative as follow:

$$ \begin{array}{@{}rcl@{}} F_{ab}^{\prime} = -2({X^{P}}^{T}U)_{ab}+2(VU^{T}U)_{ab}+2\lambda ((H^{T}-B^{T})(H-T)V)_{ab}. \end{array} $$

$$ \begin{array}{@{}rcl@{}} F_{ab}^{\prime\prime} = 2\sum\limits_{i=1}^{m}U_{ib}^{2}+2\lambda \sum\limits_{i=1}^{n}(H_{ia}-B_{ia})^{2}. \end{array} $$

Lemma 2: Function

$$ \begin{array}{@{}rcl@{}} G(v, v^{t}_{ab}) = F_{ab}(v^{t}_{ab})+F^{\prime}_{ab}(v^{t}_{ab})(v-v^{t}_{ab})+\frac{(VU^{T}U+\lambda H^{T}HV+\lambda B^{T}BV)_{ab}}{v^{t}_{ab}}(v-v^{t}_{ab})^{2} \end{array} $$

is the auxiliary function of F_ab.

Proof: It is obvious that $G(v^{t}_{ab}, v^{t}_{ab}) = F_{ab}(v)$. Next we will prove that $G(v, v^{t}_{ab}) \geqslant F_{ab}(v)$.

Because F_ab(v) is a quadratic function, the Taylor expansion of F_ab(v) on $v_{ab}^{t}$ is

$$ \begin{array}{@{}rcl@{}} F_{ab}(v) = F_{ab}(v^{t}_{ab})+F^{\prime}_{ab}(v^{t}_{ab})(v-v^{t}_{ab})+(\sum\limits_{i=1}^{m}U_{ib}^{2}+\lambda \sum\limits_{i=1}^{n}(H_{ia}-B_{ia})^{2})(v-v^{t}_{ab})^{2}. \end{array} $$

If $G(v, v^{t}_{ab}) \geqslant F_{ab}(v)$, we only need

$$ \begin{array}{@{}rcl@{}} \frac{(VU^{T}U+\lambda H^{T}HV+\lambda B^{T}BV)_{ab}}{v^{t}_{ab}} \geqslant \sum\limits_{i=1}^{m}U_{ib}^{2}+\lambda \sum\limits_{i=1}^{n}(H_{ia}-B_{ia})^{2}. \end{array} $$

(20)

We can rewrite (20) as

$$ \begin{array}{@{}rcl@{}} (VU^{T}U+\lambda H^{T}HV+\lambda E^{T}EV)_{ab} \geqslant v^{t}_{ab}(U^{T}U)_{bb}+\lambda (H^{T}H-H^{T}B-B^{T}H+B^{T}B)_{aa}v^{t}_{ab}. \end{array} $$

(21)

Because V and U are nonnegative, we have

$$ \begin{array}{@{}rcl@{}} (VU^{T}U+\lambda H^{T}HV+\lambda E^{T}EV)_{ab} \geqslant v_{ab}^{t}(U^{T}U)_{bb}+\lambda (H^{T}H+B^{T}B)_{aa}v^{t}_{ab} \end{array} $$

$$ \begin{array}{@{}rcl@{}} \geqslant v^{t}_{ab}(U^{T}U)_{bb}+\lambda (H^{T}H-H^{T}B-B^{T}H+B^{T}B)_{aa}v^{t}_{ab}. \end{array} $$

It means the (20) is hold and the Lemma 2 is proved. $_{\square }$

Now we give the proof of Theorem 1.

Proof: For each element in V, we can find its auxiliary function $G(v, v^{t}_{ab})$. Because $G(v, v^{t}_{ab})$ is a quadratic function, calculate the first-order derivative as follow:

$$ \begin{array}{@{}rcl@{}} G^{\prime}(v, v^{t}_{ab})=F^{\prime}_{ab}(v_{ab}^{t})+\frac{2(VU^{T}U+\lambda H^{T}HV+\lambda B^{T}BV)_{ab}}{v_{ab}^{t}}(v-v^{t}_{ab}). \end{array} $$

Let the derivative equal 0 and we can get the update rule:

$$ \begin{array}{@{}rcl@{}} v^{t+1}_{ab} = v^{t}_{ab}-\frac{v^{t}_{ab}}{2} \frac{F^{\prime}_{ab}(v_{ab}^{t})}{(VU^{T}U+\lambda H^{T}HV+\lambda B^{T}BV)_{ab}} \end{array} $$

$$ \begin{array}{@{}rcl@{}} =v^{t}_{ab}-\frac{v^{t}_{ab}}{2} \frac{2((-{X^{P}}^{T}U+VU^{T}U+\lambda (H^{T}-B^{T})(H-B)V))_{ab}}{(VU^{T}U+\lambda H^{T}HV+\lambda B^{T}BV)_{ab}} \end{array} $$

$$ \begin{array}{@{}rcl@{}} =\frac{({X^{P}}^{T}U+\lambda H^{T}BV+\lambda B^{T}HV)_{ab}}{(VU^{T}U+\lambda H^{T}HV+\lambda B^{T}BV)_{ab}}v^{t}_{ab}. \end{array} $$

It is same as the update rule (19). For U, we can use the similar method to prove.

So, the proof of Theorem 1 is done. $_{\square }$