Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions

Nonnegative matrix factorization (NMF), which is a dimension reduction technique for decomposing a data matrix into two factor matrices, in both of which all entries are nonnegative, has been applied to many fields and extended to various forms (Lee and Seung 1999, 2001; Berry et al. 2007; Wang and Zhang 2013). One of best-known extensions is orthogonal NMF (ONMF), which imposes column orthogonality on one side’s nonnegative factor matrix (Ding et al. 2006; Yoo and Choi 2008; Choi 2008; Yoo and Choi 2010a; Li et al. 2010; Pompili et al. 2014; Mirzal 2014; Kimura et al. 2014). Because a nonnegative column orthogonal matrix plays a role analogous to an indicator matrix in k-means clustering, and in fact one can obtain the sparse factor matrix from ONMF, it has mainly been adopted for nearest-neighbor clustering tasks such as document and term clustering (Mauthner et al. 2010; Kim et al. 2011; Wang et al. 2016). Another extended version of NMF is nonnegative matrix tri-factorization (NMTF), which decomposes a nonnegative matrix into three nonnegative factor matrices. Because constraint-free NMTF is known to be generally equivalent to NMF, some constraints are often imposed. One popular constraint is column orthogonality of the left- and right-side nonnegative factor matrices, similar to ONMF. This NMTF is referred to as orthogonal NMTF (ONMTF) (Ding et al. 2006; Yoo and Choi 2010b, 2009). Owing to the relationship between column orthogonal nonnegative factor matrices and clustering mentioned above, ONMTF is considered to be a biclustering method. The objective of biclustering is to simultaneously detect any row or column clusters of a data matrix (Govaert and Nadif 2013). The sample objects and variables are classified using data matrix at a time. It has been adopted for use in document-term clustering, collaborative filtering, etc. (Costa and Ortale 2014; Chen et al. 2009).

In ONMF and ONMTF, it is often assumed that the error is normal or Poisson distributed. However, in NMF, various algorithms have been proposed based on various error distributions, including generalized ones. Tweedie family distributions are well-known generalized distributions for NMF (Févotte and Idier 2011; Nakano et al. 2010). It uses the index parameter \(\beta \) to identify the distribution, and includes normal (\(\beta =2\)), Poisson (\(\beta =1\)), compound Poisson (CP) (\(\beta \in (0,1)\)), gamma (\(\beta =0\)), and inverse normal (\(\beta =-1\)) as special cases (Jørgensen 1997; Dunn and Smyth 2001). The assumption of a Tweedie family distribution as the error distribution implies that the error criterion has the form of a \(\beta \)-divergence (Simsekli et al. 2013; Tan and Févotte 2013). \(\beta \)-divergence is a generalized divergence that includes Euclidean distance (\(\beta =2\)), KL-divergence (\(\beta =1\)), and Itakura–Saito divergence (\(\beta =0\))(Cichocki and Amari 2010). One of the merits of Tweedie family distributions is their flexibility with real nonnegative data. The assumption of normality for nonnegative data means that the variance is the same whenever the expected value is small or large. However, one can adjust the relationship between expected value and variance in a Tweedie distribution assumption by changing \(\beta \), because variance is proportional to a power of the expected value, such as in \(V(y)=\phi E(y)^{2-\beta }\), where V(y) and E(y) are the expected value and variance, respectively, of the random variable y, and \(\phi \) is the dispersion parameter in the Tweedie family distribution. For example, if we assume a CP distribution (i.e., choose \(\beta \) in (0,1)), it is implicitly suggested that the variance will be large when the expected value is also large. In fact, a CP distribution is defined as a Poisson mixture of gamma distributions; it is absolutely continuous on the positive axis and has a positive mass at zero, from the aspect of the generative model. In other words, a CP distributed random variable is the sum of n independent identically gamma-distributed random variables, and the number of random variables n is Poisson distributed. This assumption is familiar with a gross summation of the nonnegative values and associated with various types of real-world nonnegative data (e.g., precipitation, insurance, and purchase volume data) (Ohnishi and Dunn 2007; Smyth and Jørgensen 2002). From an NMF parameter estimation aspect, this assumption is related to robust estimation in the presence of outliers that have extremely large positive values (Li et al. 2017; Virtanen et al. 2015; Carabias-Orti et al. 2013; Weninger and Schuller 2012; Févotte et al. 2009; Virtanen 2007).

In this paper, we propose a new ONMTF method that generalizes the error distribution to a Tweedie family distribution. Our new method has two advantages. First, it facilitates interpretations from nonnegative data matrices by simultaneously detecting the clusters of row and column objects and the relationships among them. Second, the assumption of the error distribution is plausible with some nonnegative data, which leads to robust estimation against extremely large positive values.

One of the ways to develop the new method is to derive an iterative algorithm for estimating factor matrices using the same techniques as the previous ONMTF methods proposed by Ding et al. (2006) and Yoo and Choi (2010b). In both methods, the factor matrices are estimated using a multiplicative updating algorithm, in which they are iteratively updated by element-wise multiplication. However, this algorithm suffers from two drawbacks. First, column orthogonality is approximately (not precisely) obtained despite the column orthogonality constraints. Second, although the objective function value tends to be non-increasing in the early stages, it is not exactly monotonically non-increasing. These problems are caused by the difficulty of obtaining the optimal nonnegative column orthogonal factor matrices by constrained optimization. Mirzal (2014) pointed out the second drawback in the multiplicative algorithm for ONMF, and proposed a new convergent ONMF method using an additive updating algorithm; however, there is no guarantee that an orthogonal factor matrix will be obtained. Kimura et al. (2014) proposed a new ONMF method using a hierarchical alternating least-squares algorithm; this algorithm is faster than the previous multiplicative algorithms, but still has these two drawbacks. On the other hand, Pompili et al. (2014) proposed a k-means like method for ONMF, which exactly maintains orthogonality and ensures monotonicity for the objective function. Pompili et al. (2014) found that the optimization problem of ONMF is similar to that of spherical k-means introduced by Banerjee et al. (2003), and referred to this problem as a weighted variant of the spherical k-means (WVSK). This is an ONMF method, not ONMTF, and its error distribution is normal. Thus, we extend it to the ONMTF methods based on normal and Tweedie family distributions. Of course, our ONMTF method guarantees orthogonality and the monotonically non-increasing property of the objective function value.

Several biclustering methods based on k-means have been proposed in the literature as double k-means (Vichi 2001; Wang et al. 2011; Van Mechelen et al. 2004). These methods consider the following hard clustering problem: each object belongs to exactly one cluster, and only belongs or does not belong, like 0 or 1. Although we also consider a hard clustering problem, our proposed methods identify a membership degree. In that sense, our methods are somewhat more flexible than the double k-means. Li and Peng (2005) relaxed the hard double k-means method using orthogonal constrained factor matrices instead of indicator matrices. However, they did not impose nonnegativity constraints on these matrices. Moreover, these former double k-means approaches did not consider non-normal error distributions. The contribution of our work is the proposal of flexible hard bi-clustering approaches that consider the nonnegativity constraints and robust estimation with respect to extremely large positive values.

First, we introduce ONMTF-N, our new ONMTF method based on normal error distribution; this method is an ONMTF version of ONMF by Pompili et al. (2014). We then introduce two other new ONMTF methods (ONMTF-P and ONMTF-CP), which are respectively based on Poisson and CP error distributions. Poisson distribution is often used for integer data, e.g., count data, especially in the linear regression framework on the assumption that the expected value and the variance are equal; thereby, ONMTF-P can also be applied for count data, such as a contingency table. On the other hand, ONMTF-CP can be applied to nonnegative data that are collected as a summation of nonnegative values, such as a matrix containing the purchase volumes of individual customers in a store. To the best of our knowledge, ONMTF based on a CP distribution has not been proposed in any previous studies. Our two simulation studies demonstrate the increased consistency of ONMTF-N (compared to ONMTF based on the multiplicative algorithm), and the robustness of ONMTF-CP. In addition, we apply our ONMTF to document-term matrix to examine its goodness of clustering.

Because the previous methods are not relatively accurete, as shown in a simulation described later on, the estimates obtained by these methods can result in misinterpretations. However, our methods are more reliable because the estimates obtained by our methods are more accurate as compared to the previous methods. Moreover, our methods, especially ONMTF-P and ONMTF-CP, can be applied to real-world data because nonnegative data cannot be assumed to have a normal distribution; for example, purchase volume data contains a few extremely large positive values, which indicates that a large volume was purchased by only a few people. In such situations, a right-tailed distribution is more appropriate than a normal distribution. Therefore, our methods can significantly contribute in solving problems encountered in real-world data.

The notation employed in this paper is as follows. A matrix is represented in uppercase (bold type), e.g., \({\varvec{M}}\); its i, j element is represented in lowercase, e.g., \(m_{ij}\). An element of a complicated matrix is expressed as \([\cdot ]_{ij}\). Further, \({\varvec{m}}_{i}\) and \({\varvec{m}}_{(j)}\) are column vectors with elements \(m_{ij}\) of the i-th row and the j-th column of \({\varvec{M}}\), respectively. We use the prime symbol and ‘\(-1\)’ to express a transpose matrix and inverse matrix, e.g., \({\varvec{M}}^{\prime }\) and \({\varvec{M}}^{-1}\), respectively. The trace and diagonal parts of a square matrix \({\varvec{M}}\) are denoted by \(\text {tr}({\varvec{M}})\) and \(\text {diag}({\varvec{M}})\), respectively. The Euclidean norm of a matrix or vector is represented as \(\Vert {\varvec{M}}\Vert =\sqrt{\text {tr}({\varvec{M}}^{\prime }{\varvec{M}})}\). \({\varvec{D}}_{{\varvec{M}}}\) is a diagonal matrix in which each diagonal element is \(\Vert {\varvec{m}}_{(j)}\Vert \), while \(\varDelta ({\varvec{M}})\) is the vector with the absolute values of elements in eigenvector \({\varvec{v}}\) with the largest eigenvalue of square matrix \({\varvec{M}}\): \({\varvec{M}}{\varvec{v}} = \lambda {\varvec{v}}\). Finally, \(\mathbb {R}^{n \times p}_{+}\) is a set of \(n \times p\) nonnegative matrices.

We introduce herein our bi-clustering problem as a motivation for our ONMTF models, which are presented in later sections, and discuss the connection between the bi-clustering problem and ONMTF. Let \(y_{ij}\) be the i, j element of a nonnegative data matrix \({\varvec{Y}} \in \mathbb {R}^{n \times p}_{+}\) and \(R=\)\(\{ r_{1},\dots ,r_{n} \}\) and \(C=\{ c_{1},\dots ,c_{p} \}\) be the sets of the n row and p column objects, respectively. One of our aims is to detect a k-partition \(\mathcal {R}=\{ R_{1},\dots ,R_{k} \}\) of R and an \(\ell \)-partition \(\mathcal {C}=\{ C_{1},\dots ,C_{\ell } \}\) of C, where \(R_{m}\) and \(C_{q}\) are the m- and q-th classes of \(\mathcal {R}\) and \(\mathcal {C}\), respectively. Classes \(R_{m}\) and \(C_{q}\) are defined as sets of row and column objects, respectively, and \(\mathcal {R}\) and \(\mathcal {C}\) are sets of k and \(\ell \) disjoint non-empty classes that cover R and C, respectively. This definition implies that an object that belongs to one class does not belong to any other classes (i.e., when \(r_{i} \in R_{m}\) then \(r_{i} \notin R_{m^{*}}\) for all \(m^{*} \ne m\) and when \(c_{j} \in C_{q}\) then \(c_{j} \notin C_{q^{*}}\) for all \(q^{*} \ne q\)). We refer to \(R_{m}\) as an “m-th row cluster” and \(C_{q}\) as a “q-th column cluster.” We consider the so-called membership of objects for a cluster to which each of the objects belong to. Let \({\varvec{F}}=(f_{im})\) be the \(n \times k\) membership matrix of row objects such that \(f_{im}>0\) when \(r_{i} \in R_{m}\), while \(f_{im}=0\) when \(r_{i} \notin R_{m}\). Let \({\varvec{A}}=(a_{jq})\) be the \(p \times \ell \) membership matrix of column objects such that \(a_{jq}>0\) when \(c_{j} \in C_{q}\), while \(a_{jq}=0\) when \(c_{j} \notin C_{q}\). Note that this definition leads to the orthogonality of \({\varvec{F}}\) and \({\varvec{A}}\): \({\varvec{f}}_{(m)}^{\prime }{\varvec{f}}_{(m^{*})}=0\) for all \(m^{*} \ne m\) and \({\varvec{a}}_{(q)}^{\prime }{\varvec{a}}_{(q^{*})}=0\) for all \(q^{*} \ne q\). If \({\varvec{f}}_{(m)}\) and \({\varvec{a}}_{(q)}\) for all m, q have the unit length, the orthogonality is changed to orthonormality: \({\varvec{F}}^{\prime }{\varvec{F}}={\varvec{I}}_{k}\) and \({\varvec{A}}^{\prime }{\varvec{A}}={\varvec{I}}_{\ell }\). We also consider a relationship between row and column clusters. Let \({\varvec{S}} = (s_{mq})\) be a \(k \times \ell \) matrix such that \(s_{mq} > 0\) for all entries. This study aims to estimate the best unknown parameters \(\theta =\)\(\{ \mathcal {R}, \mathcal {C}, {\varvec{F}}, {\varvec{A}}, {\varvec{S}}\}\) by given \({\varvec{Y}}\). We consider the following approximation problem for this aim: the best \(\theta \) is obtained such that for all i, j there exist \(m: R_{m} \ni r_{i}\) and \(q: C_{q} \ni c_{j}\) such that \(y_{ij} \approx x_{ij} = f_{im}s_{mq}a_{jq}\). From the definition of the membership matrices \({\varvec{F}}\) and \({\varvec{A}}\), we can rewrite the approximation problem as \(y_{ij} \approx x_{ij} = \sum _{i=1}^{n}\sum _{j=1}^{p}f_{im}s_{mq}a_{jq}\). In matrix forms, we can describe it as \({\varvec{Y}} \approx {\varvec{X}} = {\varvec{F}}{\varvec{S}}{\varvec{A}}^{\prime }\). Note that \({\varvec{F}}\) and \({\varvec{A}}\) have orthogonality, and all entries in all matrices are nonnegative. Hence, the approximation problem is equivalent to the ONMTF problem. We consider herein that the numbers of classes k and \(\ell \) are chosen in advance.

In this section, a new method for NMTF, namely, ONMTF-N, is introduced. This method is based on the WVSK algorithm proposed by Pompili et al. (2014); it is a fundamental algorithm for ONMTF methods described in the following sections, where it is assumed that data follows Poisson and CP distributions.

This optimization problem is formally derived from a maximum likelihood (ML) problem under the assumption of normality: \(y_{ij} \sim N(x_{ij},\sigma ^{2})\) for all i, j independently, where \(N(\mu ,\sigma ^{2})\) is a normal distribution with mean \(\mu \) and variance \(\sigma ^{2}\), which is why we named the method “ONMTF-N.” However, this assumption is incompatible because we consider that the given data matrix only contains nonnegative entries. For this issue, we start with (2) to describe the ONMTF-N problem. The explanation on the other two methods that will be described later starts with the ML problem.

From (9), (11), (12), (13), (14) and (15), we can formulate an alternating optimization algorithm for model ONMTF-N as Algorithm2.1 given below: Note that lines 7 and 8 mean the normalization of the length of each column vector of \({\varvec{F}}\) and the modification of the length of each row vector of \({\varvec{S}}\) maintaining the objective function value, respectively. These two steps are needed because the update of \(C_{q}\) in step 9 is under the assumption of \({\varvec{F}}^{\prime }{\varvec{F}}={\varvec{I}}_{k}\). This holds for steps 12 and 13.

Open image in new window

In some former double k-means approaches (e.g., Vichi (2001); Wang et al. (2011); Li and Peng (2005)), the middle factor matrix \({\varvec{S}}\) is updated such that \({\varvec{S}} = {\varvec{F}}^{\prime }{\varvec{Y}}{\varvec{A}}\) or \({\varvec{S}} = ({\varvec{F}}^{\prime }{\varvec{F}})^{-1}{\varvec{F}}^{\prime }{\varvec{Y}}{\varvec{A}}({\varvec{A}}^{\prime }{\varvec{A}})^{-1}\). We can utilize this update in our methods, and it may lead to a better performance of the estimation accuracy or the computation time. However, we adopt the two-time update described in Algorithm2.1 to emphasize ONMTF-N as an expansion of the WVSK (Pompili et al. 2014). This expansion provides the idea of ONMTF-P and ONMTF-CP introduced in the sections that follow. Approaches for updating \({\varvec{S}}\) within the algorithm will be investigated in the future.

In this section, we introduce a new ONMTF method, namely, ONMTF-P. It is a modified version of ONMTF-N described in Sect. 2, and it assumes that data follow a Poisson distribution. Although a multiplicative updating algorithm for NMTF under this assumption was proposed by Yoo and Choi (2009), the orthogonal constraints were not imposed on it. In contrast, our algorithm is not based on a multiplicative updating algorithm but on the WVSK algorithm, and the orthogonality constraints are imposed on it. Unfortunately, we only provide the model and the algorithm of ONMTF-P because of space limitations. For the derivation of the update equations, please see Abe H, Yadohisa H (2017) Supplementary material to “Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions.”.

In formulating the bi-clustering problem described in Sect. 1 , we consider \(y_{ij}\) as the independent Poisson distributed random variables with density:

$$\begin{aligned} f(y_{ij}|x_{ij}) = x_{ij}^{y_{ij}}\exp \left\{ -x_{ij} \right\} /y_{ij}! \quad (i=1,\dots ,n;\;j=1,\dots ,p), \end{aligned}$$

(16)

where \(x_{ij} = \sum _{i=1}^{n}\sum _{j=1}^{p}f_{im}s_{mq}a_{jq}\). The objective function to be minimized is derived from (16) by an ML procedure as follows:

$$\begin{aligned}&Q_{\text {P}}(\theta ) =\, d_{\text {KL}}({\varvec{Y}},{\varvec{F}}{\varvec{S}}{\varvec{A}}^{\prime }) = \textstyle \sum \limits _{i=1}^{n}\sum \limits _{j=1}^{p}\left( y_{ij}\log \left( y_{ij}/x_{ij} \right) - y_{ij} + x_{ij} \right) \nonumber \\&\quad =\textstyle \sum \limits _{i=1}^{n}\sum \limits _{j=1}^{p}\left[ -y_{ij}\log \left\{ \sum \limits _{m=1}^{k}\sum \limits _{q=1}^{\ell }f_{im}s_{mq}a_{jq}\right\} + \sum \limits _{m=1}^{k}\sum \limits _{q=1}^{\ell }f_{im}s_{mq}a_{jq} \right] + \text {cst}. \end{aligned}$$

(17)

where \(d_{\text {KL}}(\cdot ,\cdot )\) is KL-divergence, and “cst” is a constant with respect to \(\theta \) to be estimated. Hence, the optimization problem of ONMTF-P is

$$\begin{aligned}&\min Q_{\text {P}}(\theta ) \text { subject to } {\varvec{F}} \in \mathbb {R}^{n \times k}_{+},\; {\varvec{S}} \in \mathbb {R}^{k \times \ell }_{+},\;{\varvec{A}} \in \mathbb {R}^{p \times \ell }_{+}, \nonumber \\&\quad {\varvec{f}}_{(m)}^{\prime }{\varvec{f}}_{(m^{*})} = 0 \; (m\ne m^{*}),\; {\varvec{a}}_{(q)}^{\prime }{\varvec{a}}_{(q^{*})} = 0 \; (q\ne q^{*}). \end{aligned}$$

(18)

The two constraints at the bottom are from the bi-clustering problem described in Sect. 1. The procedure to solve the optimization problem (18) of ONMTF-P is completed as Algorithm3.1. Here, the \(\lambda _{m} \; (m=1,\dots ,k)\) and \(\lambda _{q}^{*} \; (q=1,\dots ,\ell )\) are defined as:

$$\begin{aligned} \lambda _{m} = \textstyle \sum \limits _{\gamma = 1}^{p}\sum \limits _{q=1}^{\ell }{s_{mq}}a_{\gamma q} \qquad \text {and} \qquad \lambda _{q}^{*} = \sum \limits _{i=1}^{n}\sum \limits _{m=1}^{k}{s_{mq}}f_{im}. \end{aligned}$$

(19)

Open image in new window

In this section, we introduce the other new method for ONMTF, namely, ONMTF-CP, where it is assumed that data follows a CP distribution. This method is also based on the WVSK algorithm, as in the case of ONMTF-N and ONMTF-P described in Sects. 2 and 3, respectively. ONMTF-CP has a hyperparameter \(\beta \) that determines the robustness of estimation against the extremely large positive values. It is noteworthy that we derive a new auxiliary function for updating the middle factor matrix \({\varvec{S}}\).

From (26), (40), (41), (42), (43), and (44), we can formulate an alternating optimization algorithm for model ONMTF-CP as Algorithm4.1. Here, \(\lambda _{m}\), \(\eta _{im}\), \(\lambda _{q}^{*}\), and \(\eta _{jq}^{*}\) are defined as the same formulas in (37) and (44), in which \(s_{mq}^{*}\) is converted to \(s_{mq}\).

Open image in new window

In this section, we describe two simulation studies. The first study compares ONMTF-N and previous ONMTF methods in terms of estimation accuracy. The second study analyzes the characteristics of the estimates given by ONMTF-N, ONMTF-P, and ONMTF-CP.

5.1 Simulation study 1: estimation accuracy of ONMTF-N

We conduct a simulation study to examine the estimation accuracy of ONMT- F-N. We compare ONMTF-N with the previous ONMTF techniques proposed by Ding et al. (2006) and Yoo and Choi (2010b). We refer to their methods as “Ding’s method” and “Yoo’s method,” respectively. The details of the procedure are presented as follows: First, we generate synthetic data matrix as \({\varvec{Y}}=\tilde{{\varvec{F}}}^{*}\tilde{{\varvec{S}}}^{*}\tilde{{\varvec{A}}}^{*\prime } + {\varvec{E}}\), where \(\tilde{{\varvec{F}}}^{*}\), \(\tilde{{\varvec{S}}}^{*}\), and \(\tilde{{\varvec{A}}}^{*}\) denote three nonnormalized true factor matrices generated as described below and \({\varvec{E}}\) denotes the error matrix whose entries are normal random number from \(N(0,\sigma ^{2})\). The entries are converted to zero if \(y_{ij}<0\). We generate each entries of these three nonnormalized true factor matrices \(\tilde{{\varvec{F}}}^{*}\), \(\tilde{{\varvec{S}}}^{*}\), and \(\tilde{{\varvec{A}}}^{*}\) as follows:

$$\begin{aligned}&{\left\{ \begin{array}{ll} \tilde{f}_{im}^{*} \sim Ex(\mu ^{1/3}) &{} (r_{i} \in \tilde{R}_{m}) \\ \tilde{f}_{im}^{*} = 0 &{} (r_{i} \notin \tilde{R}_{m})\\ \end{array}\right. } \quad (i=1,\dots ,n;\;m=1,\dots ,k), \end{aligned}$$

(45)

$$\begin{aligned}&{\left\{ \begin{array}{ll} \tilde{a}_{jm}^{*} \sim Ex(\mu ^{1/3}) &{} (c_{j} \in \tilde{C}_{q}) \\ \tilde{a}_{jm}^{*} = 0 &{} (c_{j} \notin \tilde{C}_{q})\\ \end{array}\right. } \quad (j=1,\dots ,p;\;q=1,\dots ,\ell ), \end{aligned}$$

(46)

$$\begin{aligned}&\tilde{s}_{mq}^{*} \sim Ex(\mu ^{1/3}) \quad (m=1,\dots ,k;\;q=1,\dots ,\ell ), \end{aligned}$$

(47)

where \(Ex(\cdot )\) denotes an exponential distribution, \(\mu \) denotes the expected value of each entry of \({\varvec{Y}}\), and \(\tilde{\mathcal {R}}=\{\tilde{R}_{1},\dots ,\tilde{R}_{k}\}\) and \(\tilde{\mathcal {C}}=\{\tilde{C}_{1},\dots ,\tilde{C}_{\ell }\}\) denote true partitions of row and column objects. These partitions are determined using discrete uniform random numbers. It is noted that the \(\mu \) undoubtedly represents the expected value of \(y_{ij}\) from the following equation:

$$\begin{aligned}&\displaystyle E[y_{ij}] = E[\tilde{f}_{im}^{*}\tilde{s}_{mq}^{*}\tilde{a}_{jq}^{*}] = E[\tilde{f}_{im}^{*}]E[\tilde{s}_{mq}^{*}]E[\tilde{a}_{jq}^{*}] = \left( \mu ^{1/3}\right) ^{3} = \mu \nonumber \\&\quad \displaystyle (i=1,\dots ,n;\;j=1,\dots ,p;\;\tilde{R}_{m} \ni r_{i};\;\tilde{C}_{q}\ni c_{j}). \end{aligned}$$

(48)

Here, we get the normalized true factor matrices, \(\tilde{{\varvec{F}}}\), \(\tilde{{\varvec{S}}}\), and \(\tilde{{\varvec{A}}}\) for the later step as follows:

$$\begin{aligned} \tilde{{\varvec{F}}} = \tilde{{\varvec{F}}}^{*}{\varvec{D}}_{\tilde{{\varvec{F}}}^{*}}^{-1}, \quad \tilde{{\varvec{A}}} =\tilde{{\varvec{A}}}^{*}{\varvec{D}}_{\tilde{{\varvec{A}}}^{*}}^{-1}, \quad \tilde{{\varvec{S}}} = {\varvec{D}}_{\tilde{{\varvec{F}}}^{*}}\tilde{{\varvec{S}}}^{*}{\varvec{D}}_{\tilde{{\varvec{A}}}^{*}}. \end{aligned}$$

(49)

where \({\varvec{D}}_{\tilde{{\varvec{F}}}^{*}}\) and \({\varvec{D}}_{\tilde{{\varvec{A}}}^{*}}\) denote the \(k \times k\) and \(\ell \times \ell \) diagonal matrix, in which each diagonal element is \(\Vert \tilde{{\varvec{f}}}_{m}^{*}\Vert \; (m=1,\dots ,k)\) and \(\Vert \tilde{{\varvec{a}}}_{q}^{*}\Vert \; (q=1,\dots ,\ell )\), respectively. Second, we execute our proposed method ONMTF-N and the previous methods (Ding’s and Yoo’s methods) using given \({\varvec{Y}}\), initialized factor matrices, \(\tau \), and \(\nu \), where \(\tau \) denotes the threshold for the stopping algorithm and \(\nu \) denotes the maximum number of iterative cycles. Here, we use the initial inputs \(\bar{\mathcal {R}}\), \(\bar{\mathcal {C}}\), \(\bar{{\varvec{F}}}\), \(\bar{{\varvec{S}}}\), and \(\bar{{\varvec{A}}}\) for ONMTF-N which is generated in the same manner of generating true ones as described above. In this initialization, instead of the \(\mu \), \(\bar{y} = \sum _{i=1}^{n}\sum _{j=1}^{p}y_{ij}/(np)\) is used in generating exponential random numbers. For Ding’s and Yoo’s methods, we generate initial inputs as follows:

$$\begin{aligned}&\bar{f}_{im}, \; \bar{s}_{mq}, \; \bar{a}_{jq} \sim Ex(\bar{y}^{1/3}) \nonumber \\&\quad (i=1,\dots ,n;\;j=1,\dots ,p;\;m=1,\dots ,k;\;q=1,\dots ,\ell ) \end{aligned}$$

(50)

From this execution, we get the estimated parameters, that is, estimated partitions of row and column objects \(\hat{\mathcal {R}}=\{\hat{R}_{1},\dots ,\hat{R}_{k}\}\) and \(\hat{\mathcal {C}}=\{\hat{C}_{1},\dots ,\hat{C}_{\ell }\}\), respectively, and three factor matrices \(\hat{{\varvec{F}}}\), \(\hat{{\varvec{S}}}\), and \(\hat{{\varvec{A}}}\). Note that the factor matrices are normalized such as (49). Finally, we measure the similarity (or dissimilarity) between each of the five pairs of true and the estimated parameters. We calculate the following five indices: (a) \(\text {ARI}(\tilde{\mathcal {R}},\hat{\mathcal {R}})\), (b) \(\text {ARI}(\tilde{\mathcal {C}},\hat{\mathcal {C}})\), (c) \(\displaystyle \Vert \tilde{{\varvec{F}}} - \hat{{\varvec{F}}} \Vert /{(nk)}\), (d) \(\displaystyle \Vert \tilde{{\varvec{A}}} - \hat{{\varvec{A}}} \Vert /{(p\ell )}\), and (e) \(\displaystyle \Vert \tilde{{\varvec{S}}} - \hat{{\varvec{S}}} \Vert /{(k\ell )}\), where \(\text {ARI}(\cdot ,\cdot )\) denotes the adjusted Rand index (ARI) (Hubert and Arabie 1985), which is a similarity measure between two partitions of objects. ARI is 1 if the two partitions are completely the same and is close to 0 if they are different. For the three factor matrices, we calculate the mean square error (MSE). The parameters for generating the synthetic data are set as follows: (n, p, k, \(\ell \)) \(=\) (1000, 600, 5, 3), \(\sigma = 1\), \(\mu =10\), \(\tau = np \times 10^{-7}\), \(\nu = 1000\). For more results of the other conditions, please see Abe H, Yadohisa H (2017) Supplementary material to “Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions.” It is noted that the true numbers of row and column clusters, and the estimated ones, are the same as k and \(\ell \), respectively. We generate 100 synthetic data matrices; from among the candidates of the estimates given by 20 executions of each ONMTF, we select the best estimates for which the objective function value is minimized. The results are shown in Fig. 1. ONMTF-N has the highest ARI for row clusters (see (a)), followed by Yoo’s and Ding’s methods in that order. In contrast, ARIs for the column clusters (see (b)) of both Ding’s and Yoo’s methods are as large as those of ONMTF-N. This result suggests that all three methods can accurately detect the smaller side clusters (in our simulation, this is the column side), but two previous methods cannot accurately detect the larger side clusters. The results of the MSE for the row and column factor matrices (see the (c) and (d)) are similar to the ARI results. The MSE for the middle factor matrix (see (e)) obtained by our method is smaller than those obtained by the other two methods. Overall, from this simulation study, we can conclude that ONMTF-N provides better estimation consistency than the other two ONMTF methods.

Open image in new window

Open image in new window

In this section, we describe an application involving a document-term data matrix. The data is used to compare the goodness of clustering of ONMTF-N with those of the previous ONMTF methods and other methods used for the document clustering task. In addition, we attempt to apply our methods to point-of-sale data to check the differences among ONMTF-N, ONMTF-P, and ONMTF-CP. However, the application is not described herein because of space limitations. For the application, please see Abe H, Yadohisa H (2017) Supplementary material to “Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions.”

Open image in new window

The data matrices we used were obtained from the open data CLUTO¹ website. Table 2 lists the selected data matrices and statistics. The list of datasets in Table 2 are ordered by the number of elements. The tr11, tr12, tr23, tr31, tr41, and tr45 datasets are derived from the TREC² collections. The true categories of the documents in the tr31 and tr41 datasets are obtained by particular queries. The re0 and re1 datasets are from the Reuters-21578 text categorization test collection, distribution 1.0.³ The fbis data set is from the Foreign Broadcast Information Service data of TREC-5. The hitech is a dataset of San Jose Mercury Newspaper articles and contains documents about computers, electronics, health, medicine, research, and technology. The k1a, k1b, and wap datasets are used for the WebACE project (Boley et al. 1999) and contain web pages in various subject directories of Yahoo!.⁴ Datasets k1a and k1b contain the same documents, but the true labels are different.

Table 3 shows ARI of the 12 methods for each dataset. Yoo’s method has four datasets (tr12, re0k1a, and k1b) with the best ARI and this number is the largest amongst those of the 12 methods. The second best performance is shown in SP, which has three datasets (tr23, tr41, and hitech) with the best ARI and the method has high accuracy in many datasets. The third best performance is shown in N, Mod, and WV, each of which show the best ARI in two datasets. Although Yoo’s method seems to show the best performance from the perspective of which method shows the best ARI in any of the datasets, the method has poor ARI on some datasets, e.g., re1, tr41, tr31, and hitech. In fact, Yoo’s method shows higher performance than N in no more than 5 datasets of all 13 datasets. On the other hand, N is not poor for almost all datasets in the biclustering methods. However, it is poorer than one-side clustering methods, SP and WV. A possible reason for the result is as follows. Applying one-side clustering approach to document-term clustering task leads to the clustering of documents and terms into the same number of clusters; on the other hand, biclustering approach leads to clustering of the two into a different number of clusters. In the biclustering approach, a number of term clusters may be short or too much for clustering documents, and this mismatch could cause the poorer performance of N in comparison to one-side clustering methods. We also show the worse performance of P and CPs, which are robust methods, in comparison to that of other methods. The cause of this result can be robustness against the large positive values. We use a document-term matrix converted using tf-idf conversion, which strongly weights terms in a few documents. The entries of such terms have a large positive value. The effect of the weighted entries disappear when these robust ONMTFs are used, implying that an ONMTF based on the Euclidean distance is not used, and interpretable clusters cannot be obtained. Note that good performance of the robust ONMTF can be obtained using new standardizations other than tf-idf conversion. However, we will undertake this challenge in our future works. DK and LP have the poor result in most of the datasets. The accuracy of LP can be improved by selection more appropriate hyperparameters for each dataset.

We obtain the results of the computational time, degree of approximation (measured as Euclidean distance at convergence), and convergence behavior. However, we do not report these herein because of space limitation. Please see Abe H, Yadohisa H (2017) Supplementary material to “Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions for the results.”

We now focus on the term clustering. Each of the term cluster obtained using the two methods could be interpreted as Table 7. We find that both methods extract similar term clusters from each other (e.g., TC 1, 2, 4, 5, and 6 of both.) However, TC 3, which is strongly related to DC 3 (see Table 6) in both methods, has different terms. TC 3 of ONMTF-N seems to mean the chart of movie by the box office and that of Yoo’s method seems to mean sports. Unfortunately, the detail of document clusters cannot be shown due to space limitations, but in fact, DC 3 of ONMTF-N includes some “entertainment” documents, while that of Yoo’s method includes almost all of “sports” documents. From the fact that Yoo’s TC 1 includes some words related to box office , documents related to box office chart can be integrated into DC 1 in Yoo’s estimates. On the other hand, there are no word cluster on sports in ONMTF-N. The “sports” documents are indeed misclassified to DC 1, which includes many entertainment documents, by ONMTF-N. Although this misclassifying of “sports” documents leads to poorer ARI of ONMTF-N than Yoo’s method, ONMTF-N detects the meaningful cluster related to box office chart instead of “sports.” However, ONMTF-N has irrelevantly estimated the document clusters unfamiliar with the real document labels. Moreover, ONMTF-N has some assorted term clusters that are not affected by document clustering. The difference of the two can be explained from the aspect of their estimating algorithms. In fact, ONMTF-N exhibits less freedom than Yoo’s method for estimating factor matrices owing to their orthogonality. Indeed its objective function value at convergence is larger than Yoo’s: ONMTF-N has 241.8; and Yoo’s method has 241.1. This estimating problem can be solved by more random starts thanks to the higher computational speed of ONMTF-N in comparison to Yoo’s method.

In this paper, we proposed a new method for ONMTF, namely, ONMTF-N, in which the objective function value is monotonically non-increasing and the orthogonality of the factor matrices is maintained. In addition, we proposed two other ONMTF methods, namely, ONMTF-P and ONMTF-CP. The main contributions of this paper are as follows. First, our simulation study and an application involving some document-term matrices indicated that ONMTF-N shows higher estimation accuracy than previous methods. Second, we derived a new auxiliary function to optimize the middle factor matrix in ONMTF-CP using an inequality of a bivariate concave function. Third, another simulation study indicated that ONMTF-CP may be robust against the effect of extremely large positive values.

NMFs with orthogonality, including our methods, should be used considering the trade-off between its easy-to-understand estimates and its under fitting. An orthogonal constraint simplifies a factor matrix, thereby facilitating result interpretation. However, a factor matrix with a simplified structure leads to a poor approximation to data matrix (e.g., \({\varvec{Y}}\).)

Two issues remain to be addressed in the future. First, cluster degeneration tends to occur in the three proposed methods. Therefore, further investigation is required to develop an approach for rapidly seeking alternative initial parameters, in order to avoid cluster degeneration. Second, it is necessary to develop an approach for estimating the best number of clusters for both row objects and column objects.