A Matrix Factorization Framework for Jointly Analyzing Multiple Nonnegative Data Sources

Abstract

Nonnegative matrix factorization based methods provide one of the simplest and most effective approaches to text mining. However, their applicability is mainly limited to analyzing a single data source. In this chapter, we propose a novel joint matrix factorization framework which can jointly analyze multiple data sources by exploiting their shared and individual structures. The proposed framework is flexible to handle any arbitrary sharing configurations encountered in real world data. We derive an efficient algorithm for learning the factorization and show that its convergence is theoretically guaranteed. We demonstrate the utility and effectiveness of the proposed framework in two real-world applications—improving social media retrieval using auxiliary sources and cross-social media retrieval. Representing each social media source using their textual tags, for both applications, we show that retrieval performance exceeds the existing state-of-the-art techniques. The proposed solution provides a generic framework and can be applicable to a wider context in data mining wherever one needs to exploit mutual and individual knowledge present across multiple data sources.

Appendix

1.1 Proof of Convergence

We prove the convergence of multiplicative updates given by Eqs.  (10) and (11). We avoid lengthy derivations and only provide a sketch of the proof. Following Ref. [11], the auxiliary function $G(w,w^{t})$ is defined as an upper bound function for $J(w^{t})$. For our MS-NMF case, we prove the following lemma extended from Ref. [11]:

Lemma.

If $\left( W_{\nu }\right) _{p}$ is $p$th row of matrix $W_{\nu }$, $\nu \in S\left( n,i\right)$ and $C\left( \left( W_{\nu }\right) _{p}\right)$ is the diagonal matrix with its $\left( l,k\right)$th element

$$ C_{lk}\left( \left( W_{\nu }\right) _{p}\right) =\mathbf {1}_{l,k}\frac{\left( \sum \limits _{i\in \nu }\lambda _{i}H_{i,\nu }\left( \sum \limits _{u\in S\left( n,i\right) }H_{i,u}^{\mathsf {T}}\left( W_{u}\right) _{p}\right) \right) _{l}}{\left( W_{\nu }\right) _{pl}} $$

then

$$ G\left( \left( W_{\nu }\right) _{p},\left( W_{\nu }\right) _{p}^{t}\right) =J\left( \left( W_{\nu }\right) _{p}^{t}\right) +\left( \left( W_{\nu }\right) _{p}-\left( W_{\nu }\right) _{p}^{t}\right) ^{\mathsf {T}}\nabla _{\left( W_{\nu }\right) _{p}^{t}}J\left( \left( W_{\nu }\right) _{p}^{t}\right) \\ +\frac{1}{2}\left( \left( W_{\nu }\right) _{p}-\left( W_{\nu }\right) _{p}^{t}\right) ^{\mathsf {T}}C\left( \left( W_{\nu }\right) _{p}^{t}\right) \left( \left( W_{\nu }\right) _{p}^{t}-\left( W_{\nu }\right) _{p}^{t}\right) $$

is an auxiliary function for $J\left( \left( W_{\nu }\right) _{p}^{t}\right)$, cost function defined for $p$th row of the data.

Proof.

The second derivative of $J\left( \left( W_{\nu }\right) _{p}^{t}\right)$ i.e. $\nabla _{\left( W_{\nu }\right) _{p}^{t}}^{2}J\left( \left( W_{\nu }\right) _{p}^{t}\right) =\sum _{i\in \nu }\lambda _{i}H_{i,\nu } H_{i,\nu }^{\mathsf {T}}$. Comparing the expression of $G\left( \left( W_{\nu }\right) _{p},\left( W_{\nu }\right) _{p}^{t}\right)$ in the lemma with the Taylor series expansion of $G\left( \left( W_{\nu }\right) _{p},\left( W_{\nu }\right) _{p}^{t}\right)$ at $\left( W_{\nu }\right) _{p}^{t}$, it can be seen that all we need to prove is the following

$$ \left( \left( W_{\nu }\right) _{p}-\left( W_{\nu }\right) _{p}^{t}\right) ^{\mathsf {T}}T_{W_{\nu }}\left( \left( W_{\nu }\right) _{p}^{t}-\left( W_{\nu }\right) _{p}^{t}\right) \ge 0 $$

where $T_{W_{\nu }}\triangleq C\left( \left( W_{\nu }\right) _{p}^{t}\right) -\sum _{i\in \nu }\lambda _{i}H_{i,\nu }H_{i,\nu }^{\mathsf {T}}$. Similar to Ref. [11], instead of showing it directly, we show the positive definiteness of matrix $E$ with elements

$$\begin{aligned} E_{lk}\left( \left( W_{\nu }\right) _{p}^{t}\right)&= \left( \left( W_{\nu }\right) _{p}-\left( W_{\nu }\right) _{p}^{t}\right) _{l}^{\mathsf {T}}\left( T_{W_{\nu }}\right) _{lk}\left( \left( W_{\nu }\right) _{p}^{t}-\left( W_{\nu }\right) _{p}^{t}\right) _{k} \end{aligned}$$

For positive definiteness of matrix $E$, for every nonzero $z$, we have to show that $z^{\mathsf {T}}Mz$ is positive. To avoid lengthy derivation, we only show main step here :

$$\begin{aligned} z^{\mathsf {T}}Mz&=\sum _{l,k}z_{l}\left( W_{\nu }\right) _{pl}^{t}\left( T_{W_{\nu }}\right) _{lk}\left( W_{\nu }\right) _{pk}^{t}z_{k}\\&=\sum _{l,k}z_{l}^{2}\left( W_{\nu }\right) _{pl}^{t}\left( \sum _{u\in S\left( n,i\right) ,u\ne \nu }H_{i,u}^{\mathsf {T}}\left( W_{u}\right) _{p}\right) _{l}\\&\quad +\lambda \sum _{l,k}\left( W_{\nu }\right) _{pl}^{t}\left( \sum _{i\in \nu }\lambda _{i}\left( H_{i,\nu }H_{i,\nu }^{\mathsf {T}}\right) _{lk}\right) \left( W_{\nu }\right) _{pk}^{t}\frac{\left( z_{l}-z_{k}\right) ^{2}}{2}\ge 0 \end{aligned}$$

At the local minimum of $G\left( \left( W_{\nu }\right) _{p},\left( W_{\nu }\right) _{p}^{t}\right)$ for iteration $\left( t\right)$, by comparing $\nabla _{\left( W_{\nu }\right) _{p}^{t}}G\left( \left( W_{\nu }\right) _{p},\left( W_{\nu }\right) _{p}^{t}\right)$ with gradient-descent update of Eq. (8), we get the step size $\eta _{\left( W_{\nu }\right) _{lk}^{t}}$ as in Eq. (9).