Cauchy balanced nonnegative matrix factorization

Abstract

Nonnegative Matrix Factorization (NMF) plays an important role in many data mining and machine learning tasks. Standard NMF uses the Frobenius norm as the loss function which is well-known to be sensitive to noise. To address this issue, we propose a robust formulation of NMF, i.e., Cauchy-NMF, which is derived based on the assumption that the noise generally follows identical independent distributed (i.i.d.) Cauchy distribution. In particular, we derive the Cauchy Balanced NMF model (Cauchy-B-NMF) using Cauchy distribution, where (a) the numerical value of each element in the coefficient matrix is viewed as the posterior probability, which allows the clustering result to be obtained directly from the coefficient matrix without any additional post-processing; (b) a novel manifold regularization term is incorporated into the loss function, explicitly making the distant data points have dissimilar embeddings, while implicitly making the neighbouring data points have similar embeddings; (c) a balanced clustering term is enforced to achieve the desired equal number of data points across different clusters. We derive an efficient computational algorithm to solve the resultant optimization problem, and also provide a rigorous analysis of the algorithm convergence. Experimental results on several benchmarks demonstrate the effectiveness of our algorithms, which consistently provides better clustering results compared to many other NMF variants.

Appendices

Appendix 1. Proof of Lemma 3

Proof

We utilize an auxiliary function to prove Lemma 1.

If $R(\textbf{G}, \textbf{G}^\prime )$ satisfies the following conditions, it is an auxiliary function for $P(\textbf{G})$.

$$\begin{aligned} P(\textbf{G})&\le R(\textbf{G}, \textbf{G}^\prime )\quad \forall \textbf{G}^\prime , \end{aligned}$$

(29)

$$\begin{aligned} P(\textbf{G})&=R(\textbf{G}, \textbf{G}). \end{aligned}$$

(30)

Define

$$\begin{aligned} \textbf{G}^{t+1}=\arg \min _{\textbf{G}}R(\textbf{G}, \textbf{G}^t), \end{aligned}$$

(31)

then we have

$$\begin{aligned} P(\textbf{G}^{t+1})\le R(\textbf{G}^{t+1}, \textbf{G}^t)\le R(\textbf{G}^t, \textbf{G}^t)=P(\textbf{G}^t). \end{aligned}$$

(32)

Equation (32) indicates that $P(\textbf{G})$ doesn’t increase under the updating rule of Eq. (31).

Let

$$\begin{aligned} \begin{aligned} P(\textbf{G})=\,&\textrm{Tr}(\textbf{X}-\textbf{F}\textbf{G})\textbf{D}(\textbf{X}-\textbf{F}\textbf{G})^{{\textrm{T}}}\\&+\lambda \textrm{Tr}(\textbf{G}^{{\textrm{T}}}\textbf{G}\textbf{Q})+\mu \textrm{Tr}((\textbf{G}^{{\textrm{T}}}\textbf{M})\otimes \textbf{N}^{{\textrm{T}}}\textbf{G})\\ =\,&\textrm{Tr}(\textbf{X}\textbf{D}\textbf{X}^{{\textrm{T}}}-2\textbf{F}\textbf{G}\textbf{D}\textbf{X}^{{\textrm{T}}})+\textrm{Tr}(\textbf{G}^{{\textrm{T}}}\textbf{F}^{{\textrm{T}}}\textbf{F}\textbf{G}\textbf{D})\\&+\lambda \textrm{Tr}(\textbf{G}^{{\textrm{T}}}\textbf{G}\textbf{Q})+\mu \textrm{Tr}((\textbf{G}^{{\textrm{T}}}\textbf{M})\otimes \textbf{N}^{{\textrm{T}}}\textbf{G}),\\ \end{aligned} \end{aligned}$$

(33)

then Eq. (21) can be rewritten as:

$$\begin{aligned} P(\textbf{G}^{t+1})\le P(\textbf{G}^{t}). \end{aligned}$$

(34)

In the remainder of the proof, we have to find an auxiliary function for $P(\textbf{G})$ and the global minima of the auxiliary function.

An auxiliary function for $P(\textbf{G})$ is showed as follows:

$$\begin{aligned} \begin{aligned} R(\textbf{G}, \textbf{G}^\prime )=&\textrm{Tr}(\textbf{X}\textbf{D}\textbf{X}^{{\textrm{T}}}-2\textbf{F}\textbf{G}\textbf{D}\textbf{X}^{{\textrm{T}}})+\sum _{ki}\frac{(\textbf{F}^{{\textrm{T}}}\textbf{F}\textbf{G}^\prime \textbf{D})_{ki}G_{ki}^2}{G_{ki}^\prime }\\&\quad +\sum _{ki}\frac{(\textbf{G}^\prime \textbf{Q})_{ki}G_{ki}^2}{G_{ki}^\prime }+\mu \textrm{Tr}((\textbf{G}^{{\textrm{T}}}\textbf{M})\otimes \textbf{N}^{{\textrm{T}}}\textbf{G}). \end{aligned} \end{aligned}$$

(35)

The following matrix inequality in Ding et al. (2010) is utilized in our proof.

$$\begin{aligned} \textrm{Tr}(\textbf{H}{^{\textrm{T}}}\textbf{A}\textbf{H}\textbf{B})\le \sum _{ki}(\textbf{A}\textbf{H}^\prime \textbf{B})_{ki}\frac{H_{ki}^2}{H_{ki}^\prime }, \end{aligned}$$

(36)

where $\textbf{A}, \textbf{B}, \textbf{H}$ are nonnegative matrices with appropriate sizes and $\textbf{A}=\textbf{A}^{{\textrm{T}}}, \textbf{B}=\textbf{B}^{{\textrm{T}}}$. The equality holds when $\textbf{H}=\textbf{H}^\prime$.

In the inequality Eq. (36), set $\textbf{H}=\textbf{G}, \textbf{A}=\textbf{F}^{{\textrm{T}}}\textbf{F}, \textbf{B}=\textbf{D}$, then

$$\begin{aligned} \textrm{Tr}(\textbf{G}^{{\textrm{T}}}\textbf{F}^{{\textrm{T}}}\textbf{F}\textbf{G}\textbf{D})\le \sum _{ki}\frac{(\textbf{F}^{{\textrm{T}}}\textbf{F}\textbf{G}^\prime \textbf{D})_{ki}G_{ki}^2}{G_{ki}^\prime }. \end{aligned}$$

(37)

In the inequality Eq. (36), set $\textbf{H}=\textbf{G}, \textbf{A}=\textbf{I}, \textbf{B}=\textbf{Q}$, then

$$\begin{aligned} \textrm{Tr}(\textbf{G}^{{\textrm{T}}}\textbf{G}\textbf{Q})\le \sum _{ki}\frac{(\textbf{G}^\prime \textbf{Q})_{ki}G_{ki}^2}{G_{ki}^\prime }. \end{aligned}$$

(38)

Based on Eqs. (37) and (38), $R(\textbf{G}, \textbf{G}^\prime )$ of Eq. (35) is an auxiliary function for $P(\textbf{G})$ of Eq. (33).

Now we should find the global minima for $R(\textbf{G},\textbf{G}^\prime )$ under the constraint condition $G_{ki}\ge 0$ and $\sum _{k}G_{ki}=1$, the problem can be formulated as:

$$\begin{aligned} \min _{G_{ki}\ge 0,\sum _{k}G_{ki}=1}R(\textbf{G},\textbf{G}^\prime ). \end{aligned}$$

(39)

The Lagrangian function of the problem (39) is

$$\begin{aligned} \begin{aligned}&{\mathcal {L}}(G_{ki},\alpha _{i},\beta _{ki})\\ =\,&\textrm{Tr}(\textbf{X}\textbf{D}\textbf{X}^{{\textrm{T}}}-2\textbf{F}\textbf{G}\textbf{D}\textbf{X}^{{\textrm{T}}})+\sum _{ki}\frac{(\textbf{F}^{{\textrm{T}}}\textbf{F}\textbf{G}^\prime \textbf{D})_{ki}G_{ki}^2}{G_{ki}^\prime }\\&+\sum _{ki}\frac{(\textbf{G}^\prime \textbf{Q})_{ki}G_{ki}^2}{G_{ki}^\prime }+\mu \textrm{Tr}((\textbf{G}^{{\textrm{T}}}\textbf{M})\otimes \textbf{N}^{{\textrm{T}}}\textbf{G})\\&+\alpha _i\sum _{k}(G_{ki}-1)+\beta _{ki}G_{ki}, \end{aligned} \end{aligned}$$

(40)

where $\alpha _{i}$ and $\beta _{ki}\ge 0$ are the Lagrangian multipliers.

The derivative of Eq. (40) w.r.t. $G_{ki}$ is:

$$\begin{aligned} \begin{aligned} \frac{\partial {\mathcal {L}}(G_{ki},\alpha _{i},\beta _{ki})}{\partial G_{ki}}&= -2(\textbf{F}^{{\textrm{T}}}\textbf{X}\textbf{D})_{ki}+2\frac{(\textbf{F}^{{\textrm{T}}}\textbf{F}\textbf{G}^\prime \textbf{D})_{ki}G_{ki}}{G_{ki}^\prime }\\&\quad +2\frac{(\textbf{G}^\prime \textbf{Q})_{ki}G_{ki}}{G_{ki}^\prime }+2\mu G_{ki}M_{kk}N_{ki}\\&\quad +\alpha _{i}+\beta _{ki}. \end{aligned} \end{aligned}$$

(41)

Set $\frac{\partial {\mathcal {L}}(G_{ki},\alpha _{i},\beta _{ki})}{\partial G_{ki}}=0$ and note that $\beta _{ki}G_{ki}=0$ according to the KKT condition, then we have

$$\begin{aligned} G_{ki}=\left( \frac{((\textbf{F}^{{\textrm{T}}}\textbf{X}\textbf{D})_{ki}-0.5\alpha _{i})G_{ki}^\prime }{(\textbf{F}^{{\textrm{T}}}\textbf{F}\textbf{G}^\prime \textbf{D})_{ki}+G_{ki}^\prime M_{kk}N_{ki}+(\textbf{G}^\prime \textbf{Q})_{ki}}\right) _{+}, \end{aligned}$$

(42)

where $(e)_{+}=\max (0,e)$.

According to the constraint condition $\sum _{k}G_{ki}=1$, we have the following equation:

$$\begin{aligned} h_i(\alpha _i)=0, \end{aligned}$$

(43)

where $h_i(\alpha _i)$ is defined as Eq. (19).

Therefore, the value of $\alpha _i$ is the root of the above equation which can be obtained by Newton’s method. And $G_{ki}$ is computed by substituting the value of $\alpha _i$ into Eq. (42).

Noting $G_{ki}^{t+1}\leftarrow G_{ki}$ and $G_{ki}^t\leftarrow G_{ki}^\prime$, Eq. (42) recovers the updating rule of Eq. (17). Therefore, under the updating rule of Eq. (17), $P(\textbf{G})$ doesn’t increase. $\square$

Appendix 2. Proof of Lemma 4

Proof

First of all, the following inequality is required:

$$\begin{aligned} \ln z\le z-1. \end{aligned}$$

(44)

We show the proof of Eq. (44) as follows.

Let $f(z)=\ln z-z+1$, then the first-order and second-order gradient of f(z) are:

$$\begin{aligned} \frac{\partial f(z)}{\partial z}=\frac{1}{z}-1, \;\; \frac{\partial ^{2} f(z)}{\partial z^2}=-\frac{1}{z^2}. \end{aligned}$$

(45)

Therefore, the global maxima of f(z) is obtained by setting the gradient of f(z) to zero, i.e., $z=1$.

$$\begin{aligned} f(z)\le f(1)=0. \end{aligned}$$

(46)

This implies that Eq. (44) holds.

Let

$$\begin{aligned} z=\frac{\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t+1}\Vert ^2+\gamma }{\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t}\Vert ^2+\gamma }, \end{aligned}$$

(47)

then Eq. (44) turns into the following inequality:

$$\begin{aligned} \begin{aligned}&\ln (\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t+1}||^2+\gamma )-\ln (\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t}\Vert ^2+\gamma )\le \\&\frac{\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t+1}\Vert ^2}{\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t}\Vert ^2+\gamma }-\frac{\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t}\Vert ^2}{\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t}\Vert ^2+\gamma }. \end{aligned} \end{aligned}$$

(48)

On the other hand, Eq. (22) can be written as:

$$\begin{aligned} \begin{aligned}&\sum _{i=1}^n\ln (\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t+1}\Vert ^2+\gamma )-\sum _{i=1}^n\ln (\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t}\Vert ^2+\gamma )\le \\&\sum _{i=1}^n\frac{\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t+1}\Vert ^2}{\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t}||^2+\gamma }-\sum _{i=1}^n\frac{\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t}\Vert ^2}{\Vert \textbf{x}_i-\textbf{F}\textbf{g}_i^{t}\Vert ^2+\gamma }. \end{aligned} \end{aligned}$$

(49)

Equation (48) implies that in Eq. (49) each i-th term of the LHS is not more than the i-th term of the RHS. Thus, add all the terms together, Eq. (48) becomes into Eq. (49). $\square$

Appendix 3. Proof of Lemma 5

Proof

The following inequality holds based on cauchy inequality:

$$\begin{aligned} \left( \sum _{i} G_{ki}^{t+1}\right) ^2\le \sum _{i} \frac{(G_{ki}^{t+1})^2}{G_{ki}^t}\sum _{i} G_{ki}^t. \end{aligned}$$

(50)

Then we have

$$\begin{aligned} \begin{aligned} \sum _k\left( \sum _{i} G_{ki}^{t+1}\right) ^2&\le \sum _k\left( \sum _{i}\frac{(G_{ki}^{t+1})^2}{G_{ki}^t}\sum _{i} G_{ki}^t\right) \\&=\textrm{Tr}(((\textbf{G}^{t+1})^{\textrm{T}}\textbf{M})\otimes \textbf{N}^{\textrm{T}}\textbf{G}^{t+1}). \end{aligned} \end{aligned}$$

(51)

Note that

$$\begin{aligned} \sum _k\left( \sum _{i} G_{ki}^{t}\right) ^2=\textrm{Tr}(((\textbf{G}^{t})^{\textrm{T}}\textbf{M})\otimes \textbf{N}^{\textrm{T}}\textbf{G}^{t}). \end{aligned}$$

(52)

Based on Eqs. (51) and (52), Eq. (23) holds. $\square$