Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions

Abstract

Orthogonal nonnegative matrix tri-factorization (ONMTF) is a biclustering method using a given nonnegative data matrix and has been applied to document-term clustering, collaborative filtering, and so on. In previously proposed ONMTF methods, it is assumed that the error distribution is normal. However, the assumption of normal distribution is not always appropriate for nonnegative data. In this paper, we propose three new ONMTF methods, which respectively employ the following error distributions: normal, Poisson, and compound Poisson. To develop the new methods, we adopt a k-means based algorithm but not a multiplicative updating algorithm, which was the main method used for obtaining estimators in previous methods. A simulation study and an application involving document-term matrices demonstrate that our method can outperform previous methods, in terms of the goodness of clustering and in the estimation of the factor matrix.

1 Introduction

Nonnegative matrix factorization (NMF), which is a dimension reduction technique for decomposing a data matrix into two factor matrices, in both of which all entries are nonnegative, has been applied to many fields and extended to various forms (Lee and Seung 1999, 2001; Berry et al. 2007; Wang and Zhang 2013). One of best-known extensions is orthogonal NMF (ONMF), which imposes column orthogonality on one side’s nonnegative factor matrix (Ding et al. 2006; Yoo and Choi 2008; Choi 2008; Yoo and Choi 2010a; Li et al. 2010; Pompili et al. 2014; Mirzal 2014; Kimura et al. 2014). Because a nonnegative column orthogonal matrix plays a role analogous to an indicator matrix in k-means clustering, and in fact one can obtain the sparse factor matrix from ONMF, it has mainly been adopted for nearest-neighbor clustering tasks such as document and term clustering (Mauthner et al. 2010; Kim et al. 2011; Wang et al. 2016). Another extended version of NMF is nonnegative matrix tri-factorization (NMTF), which decomposes a nonnegative matrix into three nonnegative factor matrices. Because constraint-free NMTF is known to be generally equivalent to NMF, some constraints are often imposed. One popular constraint is column orthogonality of the left- and right-side nonnegative factor matrices, similar to ONMF. This NMTF is referred to as orthogonal NMTF (ONMTF) (Ding et al. 2006; Yoo and Choi 2010b, 2009). Owing to the relationship between column orthogonal nonnegative factor matrices and clustering mentioned above, ONMTF is considered to be a biclustering method. The objective of biclustering is to simultaneously detect any row or column clusters of a data matrix (Govaert and Nadif 2013). The sample objects and variables are classified using data matrix at a time. It has been adopted for use in document-term clustering, collaborative filtering, etc. (Costa and Ortale 2014; Chen et al. 2009).

In ONMF and ONMTF, it is often assumed that the error is normal or Poisson distributed. However, in NMF, various algorithms have been proposed based on various error distributions, including generalized ones. Tweedie family distributions are well-known generalized distributions for NMF (Févotte and Idier 2011; Nakano et al. 2010). It uses the index parameter $\beta$ to identify the distribution, and includes normal ($\beta =2$), Poisson ($\beta =1$), compound Poisson (CP) ($\beta \in (0,1)$), gamma ($\beta =0$), and inverse normal ($\beta =-1$) as special cases (Jørgensen 1997; Dunn and Smyth 2001). The assumption of a Tweedie family distribution as the error distribution implies that the error criterion has the form of a $\beta$-divergence (Simsekli et al. 2013; Tan and Févotte 2013). $\beta$-divergence is a generalized divergence that includes Euclidean distance ($\beta =2$), KL-divergence ($\beta =1$), and Itakura–Saito divergence ($\beta =0$)(Cichocki and Amari 2010). One of the merits of Tweedie family distributions is their flexibility with real nonnegative data. The assumption of normality for nonnegative data means that the variance is the same whenever the expected value is small or large. However, one can adjust the relationship between expected value and variance in a Tweedie distribution assumption by changing $\beta$, because variance is proportional to a power of the expected value, such as in $V(y)=\phi E(y)^{2-\beta }$, where V(y) and E(y) are the expected value and variance, respectively, of the random variable y, and $\phi$ is the dispersion parameter in the Tweedie family distribution. For example, if we assume a CP distribution (i.e., choose $\beta$ in (0,1)), it is implicitly suggested that the variance will be large when the expected value is also large. In fact, a CP distribution is defined as a Poisson mixture of gamma distributions; it is absolutely continuous on the positive axis and has a positive mass at zero, from the aspect of the generative model. In other words, a CP distributed random variable is the sum of n independent identically gamma-distributed random variables, and the number of random variables n is Poisson distributed. This assumption is familiar with a gross summation of the nonnegative values and associated with various types of real-world nonnegative data (e.g., precipitation, insurance, and purchase volume data) (Ohnishi and Dunn 2007; Smyth and Jørgensen 2002). From an NMF parameter estimation aspect, this assumption is related to robust estimation in the presence of outliers that have extremely large positive values (Li et al. 2017; Virtanen et al. 2015; Carabias-Orti et al. 2013; Weninger and Schuller 2012; Févotte et al. 2009; Virtanen 2007).

In this paper, we propose a new ONMTF method that generalizes the error distribution to a Tweedie family distribution. Our new method has two advantages. First, it facilitates interpretations from nonnegative data matrices by simultaneously detecting the clusters of row and column objects and the relationships among them. Second, the assumption of the error distribution is plausible with some nonnegative data, which leads to robust estimation against extremely large positive values.

One of the ways to develop the new method is to derive an iterative algorithm for estimating factor matrices using the same techniques as the previous ONMTF methods proposed by Ding et al. (2006) and Yoo and Choi (2010b). In both methods, the factor matrices are estimated using a multiplicative updating algorithm, in which they are iteratively updated by element-wise multiplication. However, this algorithm suffers from two drawbacks. First, column orthogonality is approximately (not precisely) obtained despite the column orthogonality constraints. Second, although the objective function value tends to be non-increasing in the early stages, it is not exactly monotonically non-increasing. These problems are caused by the difficulty of obtaining the optimal nonnegative column orthogonal factor matrices by constrained optimization. Mirzal (2014) pointed out the second drawback in the multiplicative algorithm for ONMF, and proposed a new convergent ONMF method using an additive updating algorithm; however, there is no guarantee that an orthogonal factor matrix will be obtained. Kimura et al. (2014) proposed a new ONMF method using a hierarchical alternating least-squares algorithm; this algorithm is faster than the previous multiplicative algorithms, but still has these two drawbacks. On the other hand, Pompili et al. (2014) proposed a k-means like method for ONMF, which exactly maintains orthogonality and ensures monotonicity for the objective function. Pompili et al. (2014) found that the optimization problem of ONMF is similar to that of spherical k-means introduced by Banerjee et al. (2003), and referred to this problem as a weighted variant of the spherical k-means (WVSK). This is an ONMF method, not ONMTF, and its error distribution is normal. Thus, we extend it to the ONMTF methods based on normal and Tweedie family distributions. Of course, our ONMTF method guarantees orthogonality and the monotonically non-increasing property of the objective function value.

Several biclustering methods based on k-means have been proposed in the literature as double k-means (Vichi 2001; Wang et al. 2011; Van Mechelen et al. 2004). These methods consider the following hard clustering problem: each object belongs to exactly one cluster, and only belongs or does not belong, like 0 or 1. Although we also consider a hard clustering problem, our proposed methods identify a membership degree. In that sense, our methods are somewhat more flexible than the double k-means. Li and Peng (2005) relaxed the hard double k-means method using orthogonal constrained factor matrices instead of indicator matrices. However, they did not impose nonnegativity constraints on these matrices. Moreover, these former double k-means approaches did not consider non-normal error distributions. The contribution of our work is the proposal of flexible hard bi-clustering approaches that consider the nonnegativity constraints and robust estimation with respect to extremely large positive values.

First, we introduce ONMTF-N, our new ONMTF method based on normal error distribution; this method is an ONMTF version of ONMF by Pompili et al. (2014). We then introduce two other new ONMTF methods (ONMTF-P and ONMTF-CP), which are respectively based on Poisson and CP error distributions. Poisson distribution is often used for integer data, e.g., count data, especially in the linear regression framework on the assumption that the expected value and the variance are equal; thereby, ONMTF-P can also be applied for count data, such as a contingency table. On the other hand, ONMTF-CP can be applied to nonnegative data that are collected as a summation of nonnegative values, such as a matrix containing the purchase volumes of individual customers in a store. To the best of our knowledge, ONMTF based on a CP distribution has not been proposed in any previous studies. Our two simulation studies demonstrate the increased consistency of ONMTF-N (compared to ONMTF based on the multiplicative algorithm), and the robustness of ONMTF-CP. In addition, we apply our ONMTF to document-term matrix to examine its goodness of clustering.

Because the previous methods are not relatively accurete, as shown in a simulation described later on, the estimates obtained by these methods can result in misinterpretations. However, our methods are more reliable because the estimates obtained by our methods are more accurate as compared to the previous methods. Moreover, our methods, especially ONMTF-P and ONMTF-CP, can be applied to real-world data because nonnegative data cannot be assumed to have a normal distribution; for example, purchase volume data contains a few extremely large positive values, which indicates that a large volume was purchased by only a few people. In such situations, a right-tailed distribution is more appropriate than a normal distribution. Therefore, our methods can significantly contribute in solving problems encountered in real-world data.

The notation employed in this paper is as follows. A matrix is represented in uppercase (bold type), e.g., ${\varvec{M}}$; its i, j element is represented in lowercase, e.g., $m_{ij}$. An element of a complicated matrix is expressed as $[\cdot ]_{ij}$. Further, ${\varvec{m}}_{i}$ and ${\varvec{m}}_{(j)}$ are column vectors with elements $m_{ij}$ of the i-th row and the j-th column of ${\varvec{M}}$, respectively. We use the prime symbol and ‘$-1$’ to express a transpose matrix and inverse matrix, e.g., ${\varvec{M}}^{\prime }$ and ${\varvec{M}}^{-1}$, respectively. The trace and diagonal parts of a square matrix ${\varvec{M}}$ are denoted by $\text {tr}({\varvec{M}})$ and $\text {diag}({\varvec{M}})$, respectively. The Euclidean norm of a matrix or vector is represented as $\Vert {\varvec{M}}\Vert =\sqrt{\text {tr}({\varvec{M}}^{\prime }{\varvec{M}})}$. ${\varvec{D}}_{{\varvec{M}}}$ is a diagonal matrix in which each diagonal element is $\Vert {\varvec{m}}_{(j)}\Vert$, while $\varDelta ({\varvec{M}})$ is the vector with the absolute values of elements in eigenvector ${\varvec{v}}$ with the largest eigenvalue of square matrix ${\varvec{M}}$: ${\varvec{M}}{\varvec{v}} = \lambda {\varvec{v}}$. Finally, $\mathbb {R}^{n \times p}_{+}$ is a set of $n \times p$ nonnegative matrices.

We introduce herein our bi-clustering problem as a motivation for our ONMTF models, which are presented in later sections, and discuss the connection between the bi-clustering problem and ONMTF. Let $y_{ij}$ be the i, j element of a nonnegative data matrix ${\varvec{Y}} \in \mathbb {R}^{n \times p}_{+}$ and $R=$$\{ r_{1},\dots ,r_{n} \}$ and $C=\{ c_{1},\dots ,c_{p} \}$ be the sets of the n row and p column objects, respectively. One of our aims is to detect a k-partition $\mathcal {R}=\{ R_{1},\dots ,R_{k} \}$ of R and an $\ell$-partition $\mathcal {C}=\{ C_{1},\dots ,C_{\ell } \}$ of C, where $R_{m}$ and $C_{q}$ are the m- and q-th classes of $\mathcal {R}$ and $\mathcal {C}$, respectively. Classes $R_{m}$ and $C_{q}$ are defined as sets of row and column objects, respectively, and $\mathcal {R}$ and $\mathcal {C}$ are sets of k and $\ell$ disjoint non-empty classes that cover R and C, respectively. This definition implies that an object that belongs to one class does not belong to any other classes (i.e., when $r_{i} \in R_{m}$ then $r_{i} \notin R_{m^{*}}$ for all $m^{*} \ne m$ and when $c_{j} \in C_{q}$ then $c_{j} \notin C_{q^{*}}$ for all $q^{*} \ne q$). We refer to $R_{m}$ as an “m-th row cluster” and $C_{q}$ as a “q-th column cluster.” We consider the so-called membership of objects for a cluster to which each of the objects belong to. Let ${\varvec{F}}=(f_{im})$ be the $n \times k$ membership matrix of row objects such that $f_{im}>0$ when $r_{i} \in R_{m}$, while $f_{im}=0$ when $r_{i} \notin R_{m}$. Let ${\varvec{A}}=(a_{jq})$ be the $p \times \ell$ membership matrix of column objects such that $a_{jq}>0$ when $c_{j} \in C_{q}$, while $a_{jq}=0$ when $c_{j} \notin C_{q}$. Note that this definition leads to the orthogonality of ${\varvec{F}}$ and ${\varvec{A}}$: ${\varvec{f}}_{(m)}^{\prime }{\varvec{f}}_{(m^{*})}=0$ for all $m^{*} \ne m$ and ${\varvec{a}}_{(q)}^{\prime }{\varvec{a}}_{(q^{*})}=0$ for all $q^{*} \ne q$. If ${\varvec{f}}_{(m)}$ and ${\varvec{a}}_{(q)}$ for all m, q have the unit length, the orthogonality is changed to orthonormality: ${\varvec{F}}^{\prime }{\varvec{F}}={\varvec{I}}_{k}$ and ${\varvec{A}}^{\prime }{\varvec{A}}={\varvec{I}}_{\ell }$. We also consider a relationship between row and column clusters. Let ${\varvec{S}} = (s_{mq})$ be a $k \times \ell$ matrix such that $s_{mq} > 0$ for all entries. This study aims to estimate the best unknown parameters $\theta =$$\{ \mathcal {R}, \mathcal {C}, {\varvec{F}}, {\varvec{A}}, {\varvec{S}}\}$ by given ${\varvec{Y}}$. We consider the following approximation problem for this aim: the best $\theta$ is obtained such that for all i, j there exist $m: R_{m} \ni r_{i}$ and $q: C_{q} \ni c_{j}$ such that $y_{ij} \approx x_{ij} = f_{im}s_{mq}a_{jq}$. From the definition of the membership matrices ${\varvec{F}}$ and ${\varvec{A}}$, we can rewrite the approximation problem as $y_{ij} \approx x_{ij} = \sum _{i=1}^{n}\sum _{j=1}^{p}f_{im}s_{mq}a_{jq}$. In matrix forms, we can describe it as ${\varvec{Y}} \approx {\varvec{X}} = {\varvec{F}}{\varvec{S}}{\varvec{A}}^{\prime }$. Note that ${\varvec{F}}$ and ${\varvec{A}}$ have orthogonality, and all entries in all matrices are nonnegative. Hence, the approximation problem is equivalent to the ONMTF problem. We consider herein that the numbers of classes k and $\ell$ are chosen in advance.

2 Orthogonal NMTF based on normal error distribution

In this section, a new method for NMTF, namely, ONMTF-N, is introduced. This method is based on the WVSK algorithm proposed by Pompili et al. (2014); it is a fundamental algorithm for ONMTF methods described in the following sections, where it is assumed that data follows Poisson and CP distributions.

To formulate the bi-clustering problem as the approximation problem described in Sect. 1, we define the objective function by the following Euclidean distance:

$$\begin{aligned} Q_{\text {N}}(\theta ) = d_{\text {EUC}}({\varvec{Y}},{\varvec{F}}{\varvec{S}}{\varvec{A}}^{\prime }) = \Vert {\varvec{Y}} - {\varvec{F}}{\varvec{S}}{\varvec{A}}^{\prime } \Vert ^{2}/2, \end{aligned}$$

(1)

and define ONMTF-N as the following optimization problem with respect to $\theta$:

$$\begin{aligned}&\min Q_{\text {N}}(\theta ) \nonumber \\&\text {subject to } {\varvec{F}} \in \mathbb {R}^{n \times k}_{+},\; {\varvec{S}} \in \mathbb {R}^{k \times \ell }_{+},\;{\varvec{A}} \in \mathbb {R}^{p \times \ell }_{+},\; {\varvec{F}}^{\prime }{\varvec{F}}={\varvec{I}}_{k},\; {\varvec{A}}^{\prime }{\varvec{A}}={\varvec{I}}_{\ell }. \end{aligned}$$

(2)

We set the two orthogonal constraints for ${\varvec{F}}$ and ${\varvec{A}}$ according to the discussion in Sect. 1.

This optimization problem is formally derived from a maximum likelihood (ML) problem under the assumption of normality: $y_{ij} \sim N(x_{ij},\sigma ^{2})$ for all i, j independently, where $N(\mu ,\sigma ^{2})$ is a normal distribution with mean $\mu$ and variance $\sigma ^{2}$, which is why we named the method “ONMTF-N.” However, this assumption is incompatible because we consider that the given data matrix only contains nonnegative entries. For this issue, we start with (2) to describe the ONMTF-N problem. The explanation on the other two methods that will be described later starts with the ML problem.

The objective function (1) is invariant under the changing of length of each row vector of ${\varvec{S}}$ because the following is satisfied:

$$\begin{aligned} 2Q_{\text {N}}&= \Vert {\varvec{Y}} - {\varvec{F}}{\varvec{S}}{\varvec{A}}^{\prime } \Vert ^{2} = \Vert {\varvec{Y}} - {\varvec{F}}{\varvec{D}}_{{\varvec{S}}^{\prime }}{\varvec{D}}_{{\varvec{S^{\prime }}}}^{-1}{\varvec{S}}{\varvec{A}}^{\prime } \Vert ^{2} = \Vert {\varvec{Y}} - {{\varvec{F}}^{*}{\varvec{S}}^{*}{\varvec{A}}}^{\prime } \Vert ^{2}, \end{aligned}$$

(3)

where ${\varvec{D}}_{{\varvec{S^{\prime }}}}$ is the $k \times k$ diagonal matrix in which each diagonal element is $\Vert {\varvec{s}}_{m}\Vert \; (m=1,\dots ,k)$, ${\varvec{F}}^{*} = {\varvec{F}}{\varvec{D}}_{{\varvec{S}}^{\prime }}$, and ${\varvec{S}}^{*}={\varvec{D}}_{{\varvec{S^{\prime }}}}^{-1}{\varvec{S}}$. Now,

$$\begin{aligned} \text {diag}\left( {{\varvec{S}}^{*}}{\varvec{S}}^{*\prime } \right) = {\varvec{I}}_{k} \end{aligned}$$

(4)

is satisfied and ${\varvec{f}}_{(m)}^{*}\;(m=1,\dots ,k)$ no longer has to be a unit length. According to (3) and (4), the optimization problem (2) is the same as

$$\begin{aligned}&\min Q_{\text {N}}(\theta ) \text { subject to } {\varvec{F}} \in \mathbb {R}^{n \times k}_{+},\; {\varvec{S}} \in \mathbb {R}^{k \times \ell }_{+},\;{\varvec{A}} \in \mathbb {R}^{p \times \ell }_{+}, \text {diag}\left( {{\varvec{S}}}{\varvec{S}}^{\prime } \right) ={\varvec{I}}_{k}, \nonumber \\&{\varvec{f}}_{(m)}^{\prime }{\varvec{f}}_{(m^{*})}=0\;(m\ne m^{*}),\; {\varvec{A}}^{\prime }{\varvec{A}}={\varvec{I}}_{\ell }. \end{aligned}$$

(5)

The parameters are updated using alternating optimization. First, we derive the update equation of ${\varvec{F}}$ given $\mathcal {R}$, ${\varvec{S}}$, and ${\varvec{A}}$. Note that (1) can be expanded such that

$$\begin{aligned} 2Q_{\text {N}}&= \textstyle \sum \limits _{i=1}^{n}\left\| {\varvec{y}}_{i}-\textstyle \sum \limits _{m=1}^{k}f_{im}{\varvec{A}}{\varvec{s}}_{m} \right\| ^{2} =\textstyle \sum \limits _{m=1}^{k}\textstyle \sum \limits _{r_{i} \in R_{m}}\left\| {\varvec{y}}_{i}-f_{im}{\varvec{A}}{\varvec{s}}_{m} \right\| ^{2} \nonumber \\&= \textstyle \sum \limits _{m=1}^{k}\sum \limits _{r_{i} \in R_{m}}\left\{ {\varvec{y}}_{i}^{\prime }{\varvec{y}}_{i} - 2f_{im}{\varvec{y}}_{i}^{\prime }{\varvec{A}}{\varvec{s}}_{m} + f_{im}^{2}{\varvec{s}}_{m}^{\prime }{\varvec{A}}^{\prime }{\varvec{A}}{\varvec{s}}_{m} \right\} \nonumber \\&= \textstyle \sum \limits _{m=1}^{k}\textstyle \sum \limits _{r_{i} \in R_{m}}\left\{ {\varvec{y}}_{i}^{\prime }{\varvec{y}}_{i} - 2f_{im}{\varvec{y}}_{i}^{\prime }{\varvec{A}}{\varvec{s}}_{m} + f_{im}^{2} \right\} \quad (\because (4)~ \text {and}~ {\varvec{A}}^{\prime }{\varvec{A}}={\varvec{I}}_{\ell }). \end{aligned}$$

(6)

Hence, the minimizer of $f_{im}$ for (6) is

$$\begin{aligned} f_{im} = {\varvec{y}}_{i}^{\prime }{\varvec{A}}{\varvec{s}}_{m} \quad (r_{i} \in R_{m};\;m=1,\dots ,k). \end{aligned}$$

(7)

Substituting (7) into (6) and rearranging the terms proportional to the parameters, we obtain

$$\begin{aligned} Q^{*}_{\text {N}}(\mathcal {R},{\varvec{S}})&= \textstyle \sum \limits _{m=1}^{k}\sum \limits _{r_{i} \in R_{m}}\left\{ -\left( {\varvec{y}}_{i}^{\prime }{\varvec{A}}{\varvec{s}}_{m} \right) ^{2} \right\} . \end{aligned}$$

(8)

Therefore, the problem of minimizing $Q_{\text {N}}$ is the same as the problem of minimizing $Q^{*}_{\text {N}}$. Note that (8) can be observed to be similar to the objective function of WVSK when ${\varvec{A}}^{\prime }{\varvec{y}}_{i} \; (i=1,\dots ,n)$ and ${\varvec{s}}_{m} \; (m=1,\dots ,k)$ are considered as n data vectors and k normalized centroids in $\mathbb {R}_{+}^{\ell }$, respectively. In addition, when all n data vectors are normalized (i.e., $\Vert {\varvec{A}}^{\prime }{\varvec{y}}_{i} \Vert ^{2} = 1$$(i=1,\dots ,n)$), (8) is observed to be the same as the objective function of spherical k-means in $\mathbb {R}_{+}^{\ell }$. Given ${\varvec{S}}$ and ${\varvec{A}}$, the minimizers of $\mathcal {R}$ for $Q^{*}_{\text {N}}$ are derived by, e.g., a k-means algorithm such that

$$\begin{aligned} {R_{m} = \left\{ r_{i} \; \left| \; \mathop {\mathrm{argmax}}\limits _{m^{*}=1,\dots ,k}\left( {\varvec{y}}_{i}^{\prime }{\varvec{A}}{\varvec{s}}_{m^{*}} \right) \right. = m \right\} } \; (m=1,\dots ,k). \end{aligned}$$

(9)

Because (8) can be rewritten such that

$$\begin{aligned} Q^{*}_{\text {N}}&= -\textstyle \sum \limits _{m=1}^{k}\Vert {\varvec{Y}}_{m}{\varvec{A}}{\varvec{s}}_{m} \Vert ^{2} = -\sum \limits _{m=1}^{k}\text {tr}\left\{ {\varvec{s}}_{m}^{\prime }{\varvec{A}}^{\prime } {\varvec{Y}}_{m}^{\prime }{\varvec{Y}}_{m}{\varvec{A}}{\varvec{s}}_{m} \right\} , \end{aligned}$$

(10)

where ${\varvec{Y}}_{m}\;(m=1,\dots ,k)$ is a $|R_{m}|\times p$ matrix consisting of the row vectors ${\varvec{y}}_{i}^{\prime }$, such that $r_{i} \in R_{m}$. The minimizer of ${\varvec{S}}$ given $\mathcal {R}$ and ${\varvec{A}}$ can be obtained by an eigenvector of ${\varvec{A}}^{\prime }{\varvec{Y}}_{m}^{\prime }{\varvec{Y}}_{m}{\varvec{A}}$ as follows:

$$\begin{aligned} {{\varvec{s}}_{m} = \varDelta ({\varvec{A}}^{\prime }{\varvec{Y}}_{m}^{\prime }{\varvec{Y}}_{m}{\varvec{A}})} \quad (m=1,\dots ,k). \end{aligned}$$

(11)

Finally, the minimizer of ${\varvec{F}}$ given $\mathcal {R}$, ${\varvec{S}}$, and ${\varvec{A}}$ can be expressed from (7) as

$$\begin{aligned} f_{im} = {\left\{ \begin{array}{ll} {\varvec{y}}_{i}^{\prime }{\varvec{A}}{\varvec{s}}_{m} &{} (r_{i} \in R_{m}) \\ 0 &{} (r_{i} \notin R_{m}) \end{array}\right. } \quad (i=1,\dots ,n;\;m=1,\dots ,k). \end{aligned}$$

(12)

If we regard the model as ${\varvec{Y}}^{\prime } \approx {\varvec{A}}{\varvec{S}}^{\prime }{\varvec{F}}^{\prime }$, the update equations of $\mathcal {C}$, ${\varvec{S}}$, and ${\varvec{A}}$ can be derived similarly to (9), (11), and (12) as follows:

$$\begin{aligned} C_{q}&= {\left\{ c_{j} \left| \mathop {\mathrm{argmax}}\limits _{q^{*}=1,\dots ,\ell }\left\{ {\varvec{y}}_{(j)}^{\prime }{\varvec{F}}{\varvec{s}}_{(q^{*})} \right\} = q \right. \right\} } \; (q=1,\dots ,\ell ) \end{aligned}$$

(13)

$$\begin{aligned} {\varvec{s}}_{(q)}&= {\varDelta ({\varvec{F}}^{\prime }{\varvec{Y}}_{(q)}{\varvec{Y}}_{(q)}^{\prime }{\varvec{F}})} \quad (q=1,\dots ,\ell ) \end{aligned}$$

(14)

$$\begin{aligned} a_{jq}&= {\left\{ \begin{array}{ll} {\varvec{y}}_{(j)}^{\prime }{\varvec{F}}{\varvec{s}}_{(q)} &{} (c_{j} \in C_{q}) \\ 0 &{} (c_{j} \notin C_{q}) \end{array}\right. } \quad (j=1,\dots p;\;q=1,\dots ,\ell ), \end{aligned}$$

(15)

where ${\varvec{Y}}_{(q)}\;(q=1,\dots ,\ell )$ is an $n \times |C_{q}|$ matrix consisting of the column vectors ${\varvec{y}}_{(j)}$, such that $c_{j} \in C_{q}$.

From (9), (11), (12), (13), (14) and (15), we can formulate an alternating optimization algorithm for model ONMTF-N as Algorithm2.1 given below: Note that lines 7 and 8 mean the normalization of the length of each column vector of ${\varvec{F}}$ and the modification of the length of each row vector of ${\varvec{S}}$ maintaining the objective function value, respectively. These two steps are needed because the update of $C_{q}$ in step 9 is under the assumption of ${\varvec{F}}^{\prime }{\varvec{F}}={\varvec{I}}_{k}$. This holds for steps 12 and 13.

In some former double k-means approaches (e.g., Vichi (2001); Wang et al. (2011); Li and Peng (2005)), the middle factor matrix ${\varvec{S}}$ is updated such that ${\varvec{S}} = {\varvec{F}}^{\prime }{\varvec{Y}}{\varvec{A}}$ or ${\varvec{S}} = ({\varvec{F}}^{\prime }{\varvec{F}})^{-1}{\varvec{F}}^{\prime }{\varvec{Y}}{\varvec{A}}({\varvec{A}}^{\prime }{\varvec{A}})^{-1}$. We can utilize this update in our methods, and it may lead to a better performance of the estimation accuracy or the computation time. However, we adopt the two-time update described in Algorithm2.1 to emphasize ONMTF-N as an expansion of the WVSK (Pompili et al. 2014). This expansion provides the idea of ONMTF-P and ONMTF-CP introduced in the sections that follow. Approaches for updating ${\varvec{S}}$ within the algorithm will be investigated in the future.

3 Orthogonal NMTF based on Poisson error distribution

In this section, we introduce a new ONMTF method, namely, ONMTF-P. It is a modified version of ONMTF-N described in Sect. 2, and it assumes that data follow a Poisson distribution. Although a multiplicative updating algorithm for NMTF under this assumption was proposed by Yoo and Choi (2009), the orthogonal constraints were not imposed on it. In contrast, our algorithm is not based on a multiplicative updating algorithm but on the WVSK algorithm, and the orthogonality constraints are imposed on it. Unfortunately, we only provide the model and the algorithm of ONMTF-P because of space limitations. For the derivation of the update equations, please see Abe H, Yadohisa H (2017) Supplementary material to “Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions.”.

In formulating the bi-clustering problem described in Sect. 1 , we consider $y_{ij}$ as the independent Poisson distributed random variables with density:

$$\begin{aligned} f(y_{ij}|x_{ij}) = x_{ij}^{y_{ij}}\exp \left\{ -x_{ij} \right\} /y_{ij}! \quad (i=1,\dots ,n;\;j=1,\dots ,p), \end{aligned}$$

(16)

where $x_{ij} = \sum _{i=1}^{n}\sum _{j=1}^{p}f_{im}s_{mq}a_{jq}$. The objective function to be minimized is derived from (16) by an ML procedure as follows:

$$\begin{aligned}&Q_{\text {P}}(\theta ) =\, d_{\text {KL}}({\varvec{Y}},{\varvec{F}}{\varvec{S}}{\varvec{A}}^{\prime }) = \textstyle \sum \limits _{i=1}^{n}\sum \limits _{j=1}^{p}\left( y_{ij}\log \left( y_{ij}/x_{ij} \right) - y_{ij} + x_{ij} \right) \nonumber \\&\quad =\textstyle \sum \limits _{i=1}^{n}\sum \limits _{j=1}^{p}\left[ -y_{ij}\log \left\{ \sum \limits _{m=1}^{k}\sum \limits _{q=1}^{\ell }f_{im}s_{mq}a_{jq}\right\} + \sum \limits _{m=1}^{k}\sum \limits _{q=1}^{\ell }f_{im}s_{mq}a_{jq} \right] + \text {cst}. \end{aligned}$$

(17)

where $d_{\text {KL}}(\cdot ,\cdot )$ is KL-divergence, and “cst” is a constant with respect to $\theta$ to be estimated. Hence, the optimization problem of ONMTF-P is

$$\begin{aligned}&\min Q_{\text {P}}(\theta ) \text { subject to } {\varvec{F}} \in \mathbb {R}^{n \times k}_{+},\; {\varvec{S}} \in \mathbb {R}^{k \times \ell }_{+},\;{\varvec{A}} \in \mathbb {R}^{p \times \ell }_{+}, \nonumber \\&\quad {\varvec{f}}_{(m)}^{\prime }{\varvec{f}}_{(m^{*})} = 0 \; (m\ne m^{*}),\; {\varvec{a}}_{(q)}^{\prime }{\varvec{a}}_{(q^{*})} = 0 \; (q\ne q^{*}). \end{aligned}$$

(18)

The two constraints at the bottom are from the bi-clustering problem described in Sect. 1. The procedure to solve the optimization problem (18) of ONMTF-P is completed as Algorithm3.1. Here, the $\lambda _{m} \; (m=1,\dots ,k)$ and $\lambda _{q}^{*} \; (q=1,\dots ,\ell )$ are defined as:

$$\begin{aligned} \lambda _{m} = \textstyle \sum \limits _{\gamma = 1}^{p}\sum \limits _{q=1}^{\ell }{s_{mq}}a_{\gamma q} \qquad \text {and} \qquad \lambda _{q}^{*} = \sum \limits _{i=1}^{n}\sum \limits _{m=1}^{k}{s_{mq}}f_{im}. \end{aligned}$$

(19)

4 Orthogonal NMTF based on CP error distribution

In this section, we introduce the other new method for ONMTF, namely, ONMTF-CP, where it is assumed that data follows a CP distribution. This method is also based on the WVSK algorithm, as in the case of ONMTF-N and ONMTF-P described in Sects. 2 and 3, respectively. ONMTF-CP has a hyperparameter $\beta$ that determines the robustness of estimation against the extremely large positive values. It is noteworthy that we derive a new auxiliary function for updating the middle factor matrix ${\varvec{S}}$.

In formulating the bi-clustering problem described in Sect. 1, we consider $y_{ij}$ as the independent CP distributed random variables with density:

$$\begin{aligned}&f(y_{ij}|x_{ij},\phi ,\beta ) = h_{\beta }(y_{ij},\phi )\exp \left\{ \frac{1}{\phi }\left( \frac{1}{\beta -1}y_{ij}x_{ij}^{\beta -1} - \frac{1}{\beta }x_{ij}^{\beta } \right) \right\} \nonumber \\&\quad (i=1,\dots ,n;\;j=1,\dots ,p), \end{aligned}$$

(20)

where $x_{ij} = \sum _{i=1}^{n}\sum _{j=1}^{p}f_{im}s_{mq}a_{jq}$, $\phi$ is a dispersion parameter, and $\beta \in (0,1)$ is an index parameter that determines the shape of the distribution. The objective function to be minimized is derived from (20) by a ML procedure as follows:

$$\begin{aligned} Q_{\text {CP}}(\theta ) =\,&d_{\beta }({\varvec{Y}},{\varvec{X}}) = \textstyle \sum \limits _{i=1}^{n}\sum \limits _{j=1}^{p}\left( y_{ij}\displaystyle \frac{y_{ij}^{\beta -1}-x_{ij}^{\beta -1}}{\beta -1} - \frac{y_{ij}^{\beta } - x_{ij}^{\beta }}{\beta } \right) \nonumber \\ =&\frac{1}{\beta }\textstyle \sum \limits _{i=1}^{n}\sum \limits _{j=1}^{p}\left( \sum \limits _{m=1}^{k}\sum \limits _{q=1}^{\ell }f_{im}s_{mq}a_{jq}\right) ^{\beta } \nonumber \\&- \displaystyle \frac{1}{\beta -1}\textstyle \sum \limits _{i=1}^{n}\sum \limits _{j=1}^{p}y_{ij}\left( \sum \limits _{m=1}^{k}\sum \limits _{q=1}^{\ell }f_{im}s_{mq}a_{jq}\right) ^{\beta -1} + \text {cst}. \end{aligned}$$

(21)

where $d_{\beta }(\cdot ,\cdot )$ is a $\beta$-divergence with parameter $\beta$ that corresponding to the index parameter of CP (20). Hence, the optimization problem of ONMTF-CP is

$$\begin{aligned}&\min Q_{\text {CP}}(\theta ) \text { subject to } {\varvec{F}} \in \mathbb {R}^{n \times k}_{+},\; {\varvec{S}} \in \mathbb {R}^{k \times \ell }_{+},\;{\varvec{A}} \in \mathbb {R}^{p \times \ell }_{+}, \nonumber \\&\quad {\varvec{f}}_{(m)}^{\prime }{\varvec{f}}_{(m^{*})} = 0 \; (m\ne m^{*}),\; {\varvec{a}}_{(q)}^{\prime }{\varvec{a}}_{(q^{*})} = 0 \; (q\ne q^{*}). \end{aligned}$$

(22)

The two constraints at the bottom are from the bi-clustering problem described in Sect. 1.

The parameters are updated using alternating optimization. First, we derive the update equation of ${\varvec{F}}$ given $\mathcal {R}$, ${\varvec{S}}$, and ${\varvec{A}}$. Note that (21) can be rewritten as

$$\begin{aligned} Q_{\text {CP}} =&\textstyle \sum \limits _{i=1}^{n}\sum \limits _{j=1}^{p}\left\{ \displaystyle \frac{1}{\beta }\left( \textstyle \sum \limits _{m=1}^{k}f_{im}[{\varvec{S}}{\varvec{A}}^{\prime }]_{mj}\right) ^{\beta }\right. \nonumber \\&\left. - \frac{1}{\beta -1}y_{ij}\left( \textstyle \sum \limits _{m=1}^{k}f_{im}[{\varvec{S}}{\varvec{A}}^{\prime }]_{mj}\right) ^{\beta -1} \right\} + \text {cst} \nonumber \\ =&{\textstyle \sum \limits _{m=1}^{k}\sum \limits _{r_{i} \in R_{m}}\left\{ \displaystyle \frac{1}{\beta }f_{im}^{\beta }\textstyle \sum \limits _{j=1}^{p} [{\varvec{S}}{\varvec{A}}^{\prime }]_{mj}^{\beta } - \displaystyle \frac{1}{\beta -1}f_{im}^{\beta -1}\textstyle \sum \limits _{j=1}^{p}y_{ij}[{\varvec{S}}{\varvec{A}}^{\prime }]_{mj}^{\beta -1} \right\} + \text {cst}.} \end{aligned}$$

(23)

Hence, the minimizer of $f_{im}$ for (23) is

$$\begin{aligned} f_{im} = \frac{\sum _{j=1}^{p}y_{ij}[{\varvec{S}}{\varvec{A}}^{\prime }]_{mj}^{\beta -1}}{\sum _{j=1}^{p} [{\varvec{S}}{\varvec{A}}^{\prime }]_{mj}^{\beta }}\quad (r_{i} \in R_{m}\;m=1,\dots ,k). \end{aligned}$$

(24)

Substituting (24) into (23) and rearranging the terms proportional to the parameters, we obtain

$$\begin{aligned}&Q^{*}_{\text {CP}}(\mathcal {R},{\varvec{S}}) = -\frac{1}{\beta (\beta -1)}\textstyle \sum \limits _{m=1}^{k}\textstyle \sum \limits _{r_{i} \in R_{m}}\displaystyle \frac{\left\{ \sum _{j=1}^{p}y_{ij}[{\varvec{S}}{\varvec{A}}^{\prime }]_{mj}^{\beta -1}\right\} ^{\beta }}{\left\{ \sum _{j=1}^{p} [{\varvec{S}}{\varvec{A}}^{\prime }]_{mj}^{\beta }\right\} ^{\beta -1}}. \end{aligned}$$

(25)

Therefore, the problem of minimizing $Q_{\text {CP}}$ is the same as the problem of minimizing $Q^{*}_{\text {CP}}$. Given ${\varvec{S}}$ and ${\varvec{A}}$, the minimizers of $\mathcal {R}$ for $Q^{*}_{\text {CP}}$ are derived by, e.g., a k-means algorithm such that

$$\begin{aligned} R_{m} = {\left\{ r_{i} \left| \mathop {\mathrm{argmin}}\limits _{m^{*}=1,\dots ,k}\left\{ \frac{\left\{ \sum _{j=1}^{p}y_{ij}[{\varvec{S}}{\varvec{A}}^{\prime }]_{m^{*}j}^{\beta -1}\right\} ^{\beta }}{\left\{ \sum _{j=1}^{p} [{\varvec{S}}{\varvec{A}}^{\prime }]_{m^{*}j}^{\beta }\right\} ^{\beta -1}} \right\} = m \right. \right\} } \; (m=1,\dots ,k). \end{aligned}$$

(26)

Then, we derive the optimal ${\varvec{S}}$ for $Q^{*}_{\text {CP}}(\mathcal {R},{\varvec{S}})$ given $\mathcal {R}$ and ${\varvec{A}}$. Note that (25) is rewritten as

$$\begin{aligned} Q^{*}_{\text {CP}}&= -\frac{1}{\beta (\beta -1)}\textstyle \sum \limits _{m=1}^{k}\sum \limits _{r_{i} \in R_{m}}\displaystyle \frac{\left\{ \sum _{q=1}^{\ell }\sum _{c_{j}\in C_{q}}y_{ij}s_{mq}^{\beta -1}a_{jq}^{\beta -1}\right\} ^{\beta }}{\left\{ \sum _{q=1}^{\ell }\sum _{c_{j}\in C_{q}} s_{mq}^{\beta }a_{jq}^{\beta }\right\} ^{\beta -1}}. \end{aligned}$$

(27)

It is difficult to directly obtain the minimizer of $s_{mq}$ for (27) because the summation of $s_{mq}$ is in the power function. However, we can obtain the optimal $s_{mq}$ using the auxiliary function method, as in the case of ONMTF-P. First, we can find that the following bivariate function

$$\begin{aligned} f(x,y)=-\frac{1}{\beta (\beta -1)}(y^{\beta }/x^{\beta -1}) \quad ((x,y)\in \mathbb {R}^{2}_{+}; \; \beta \in (0,1)) \end{aligned}$$

(28)

is involved in equation (27). We need the following corollary to obtain the appropriate inequality for (28):

Corollary 1

The bivariate function f(x, y) in (28) is concave if $(x,y)\in \mathbb {R}^{2}_{+}$ and $\beta >0$.

Proof

The first- and second-order differentials of f(x, y) are as follows:

$$\begin{aligned} f_{x}(x,y)&=(1/\beta )y^{\beta }x^{-\beta }, \end{aligned}$$

(29)

$$\begin{aligned} f_{y}(x,y)&=(1/(1-\beta ))y^{\beta -1}x^{1-\beta }, \end{aligned}$$

(30)

$$\begin{aligned} f_{xx}(x,y)&=-y^{\beta }x^{-\beta -1}, \end{aligned}$$

(31)

$$\begin{aligned} f_{yy}(x,y)&=-y^{\beta -2}x^{1-\beta }, \text { and} \end{aligned}$$

(32)

$$\begin{aligned} f_{xy}(x,y)&=y^{\beta -1}x^{-\beta } . \end{aligned}$$

(33)

From (31), (32), and (33), we have

$$\begin{aligned} f_{xx}(x,y)< 0, \; f_{yy}(x,y) < 0, \text { and } f_{xx}(x,y)f_{yy}(x,y)-\{ f_{xy}(x,y) \} = 0. \end{aligned}$$

(34)

Hence, the function f(x, y) is concave. $\square$

Therefore, we have

$$\begin{aligned} f(x,y) \le f(\lambda ,\eta ) + f_{x}(\lambda ,\eta )(x-\lambda ) + f_{y}(\lambda ,\eta )(y-\eta ) \end{aligned}$$

(35)

for any $\lambda$ and $\eta$ with equality if and only if $x=\lambda$ and $y=\eta$. From this inequality, we obtain the following auxiliary function of (27) for $s_{mq}$:

$$\begin{aligned}&Q_{\text {CP}}^{\text {aux}}({\varvec{S}},{\varvec{S}}^{*}) \nonumber \\&\quad = \textstyle \sum \limits _{m=1}^{k}\sum \limits _{r_{i} \in R_{m}}\left\{ -\displaystyle \frac{1}{\beta (\beta -1)}\frac{\eta _{im}^{\beta }}{\lambda _{m}^{\beta -1}} + \frac{1}{\beta }\left( \frac{\eta _{im}}{\lambda _{m}} \right) ^{\beta }\left( \textstyle \sum \limits _{q=1}^{\ell }\sum \limits _{c_{j}\in C_{q}}s_{mq}^{\beta }a_{jq}^{\beta } - \lambda _{m} \right) \right. \nonumber \\&\qquad \left. + \frac{1}{1-\beta }\left( \frac{\eta _{im}}{\lambda _{m}} \right) ^{\beta -1}\left( \textstyle \sum \limits _{q=1}^{\ell }\sum \limits _{c_{j}\in C_{q}}y_{ij}s_{mq}^{\beta -1}a_{jq}^{\beta -1} - \eta _{im} \right) \right\} \end{aligned}$$

(36)

with

$$\begin{aligned}&\lambda _{m} = \textstyle \sum \limits _{q=1}^{\ell }\sum \limits _{c_{j}\in C_{q}}s_{mq}^{*\beta }a_{jq}^{\beta }, \quad \text { and }\quad \eta _{im} = \sum \limits _{q=1}^{\ell }\sum \limits _{c_{j}\in C_{q}}y_{ij}s_{mq}^{*\beta -1}a_{jq}^{\beta -1} \nonumber \\&\quad (i=1,\dots n;\;m=1,\dots ,k), \end{aligned}$$

(37)

where $s_{mq}^{*}$ is the current value of $s_{mq}$. Let $Q^{*}_{\text {CP}}({\varvec{S}})$ be the same function as $Q^{*}_{\text {CP}}(\mathcal {R},{\varvec{S}})$. We then have:

$$\begin{aligned} Q^{\text {aux}}_{\text {CP}}({\varvec{S}},{\varvec{S}}^{*}) \ge Q^{*}_{\text {CP}}({\varvec{S}}) \quad \text {and} \quad Q^{\text {aux}}_{\text {CP}}({\varvec{S}},{\varvec{S}}) = Q^{*}_{\text {CP}}({\varvec{S}}) \end{aligned}$$

(38)

for any ${\varvec{S}}$ and ${\varvec{S}}^{*}=(s_{mq}^{*})$, and if we can find $\hat{{\varvec{S}}} = \mathop {\mathrm{argmin}}\limits _{{\varvec{S}}}Q^{\text {aux}}_{\text {CP}}({\varvec{S}},{\varvec{S}}^{*})$, we have

$$\begin{aligned} Q^{*}_{\text {CP}}({\varvec{S}}^{*}) = Q^{\text {aux}}_{\text {CP}}({\varvec{S}}^{*},{\varvec{S}}^{*}) \ge Q^{\text {aux}}_{\text {CP}}(\hat{{\varvec{S}}},{\varvec{S}}^{*}) \ge Q^{*}_{\text {CP}}(\hat{{\varvec{S}}}). \end{aligned}$$

(39)

Hence, the minimizer of ${\varvec{S}}$ for (36) is at least the optimal ${\varvec{S}}$, which is monotonically non-increasing for (27). We obtain the following update equation of $s_{mq}$ as a minimizer for $Q^{\text {aux}}_{\text {CP}}({\varvec{S}},{\varvec{S}}^{*})$:

$$\begin{aligned} s_{mq} = \frac{\sum _{r_{i}\in R_{m}}\sum _{c_{j}\in C_{q}}\left( \eta _{im}/\lambda _{m} \right) ^{\beta -1}y_{ij}a_{jq}^{\beta -1}}{\sum _{r_{i}\in R_{m}}\sum _{c_{j}\in C_{q}}\left( \eta _{im}/\lambda _{m} \right) ^{\beta }a_{jq}^{\beta }}, \end{aligned}$$

(40)

Finally, the minimizer of ${\varvec{F}}$ given $\mathcal {R}$, ${\varvec{S}}$, and ${\varvec{A}}$ can be expressed from (24) as follows:

$$\begin{aligned} f_{im} = {\left\{ \begin{array}{ll} \displaystyle \frac{\sum _{j=1}^{p}y_{ij}[{\varvec{S}}{\varvec{A}}^{\prime }]_{mj}^{\beta -1}}{\sum _{j=1}^{p} [{\varvec{S}}{\varvec{A}}^{\prime }]_{mj}^{\beta }} &{} (r_{i} \in R_{m}) \\ 0 &{} (r_{i} \notin R_{m}) \end{array}\right. } \quad (i=1,\dots ,n;\;m=1,\dots ,k). \end{aligned}$$

(41)

If we regard the model as ${\varvec{Y}}^{\prime } \approx {\varvec{A}}{\varvec{S}}^{\prime }{\varvec{F}}^{\prime }$, the update equations of $\mathcal {C}$, ${\varvec{S}}$, and ${\varvec{A}}$ can be derived similarly to (26), (40), and (41) as follows:

$$\begin{aligned} C_{q} =&{\left\{ c_{j} \left| \mathop {\mathrm{argmin}}\limits _{q^{*}=1,\dots ,\ell }\left\{ \frac{\left\{ \sum _{i=1}^{n}y_{ij}[{\varvec{F}}{\varvec{S}}]_{iq^{*}}^{\beta -1}\right\} ^{\beta }}{\left\{ \sum _{i=1}^{n} [{\varvec{F}}{\varvec{S}}]_{iq^{*}}^{\beta }\right\} ^{\beta -1}} \right\} = q \right. \right\} } \quad (q=1,\dots ,\ell ), \end{aligned}$$

(42)

$$\begin{aligned} s_{mq} =&\frac{\sum _{r_{i}\in R_{m}}\sum _{c_{j}\in C_{q}}\left( \eta _{jq}^{*}/\lambda _{q}^{*} \right) ^{\beta -1}y_{ij}f_{im}^{\beta -1}}{\sum _{r_{i}\in R_{m}}\sum _{c_{j}\in C_{q}}\left( \eta _{jq}^{*}/\lambda _{q}^{*} \right) ^{\beta }f_{im}^{\beta }} \; (m=1,\dots ,k;\;q=1,\dots ,\ell ) \nonumber \\&{\text {with }} \lambda _{q}^{*} = \textstyle \sum \limits _{m=1}^{k}\sum \limits _{r_{i}\in R_{m}}s_{mq}^{*\beta }f_{im}^{\beta }, \; \eta _{jq}^{*} = \sum \limits _{m=1}^{k}\sum \limits _{r_{i}\in R_{m}}y_{ij}s_{mq}^{*\beta -1}f_{im}^{\beta -1} \nonumber \\&(j=1,\dots p;\;q=1,\dots ,\ell ), \end{aligned}$$

(43)

$$\begin{aligned} a_{jq} =&{\left\{ \begin{array}{ll} \displaystyle \frac{\sum _{i=1}^{n}y_{ij}[{\varvec{F}}{\varvec{S}}]_{iq}^{\beta -1}}{\sum _{i=1}^{n} [{\varvec{F}}{\varvec{S}}]_{iq}^{\beta }} &{} (c_{j} \in C_{q}) \\ 0 &{} (c_{j} \notin C_{q}) \end{array}\right. } \quad (j=1,\dots p;\;q=1,\dots ,\ell ). \end{aligned}$$

(44)

A novelty of this paper is the derivation of the updating equation for ${\varvec{S}}$ using the auxiliary function obtained by an inequality of the bivariate concave function in ONMTF-CP. To the best of our knowledge, the corollary on the inequality and its proof have not yet been presented in previous studies.

From (26), (40), (41), (42), (43), and (44), we can formulate an alternating optimization algorithm for model ONMTF-CP as Algorithm4.1. Here, $\lambda _{m}$, $\eta _{im}$, $\lambda _{q}^{*}$, and $\eta _{jq}^{*}$ are defined as the same formulas in (37) and (44), in which $s_{mq}^{*}$ is converted to $s_{mq}$.

4.1 Some issues

The proposed ONMTF methods introduced earlier are based on the k-means algorithm from the row and column sides. In other words, these methods have the disadvantages of double k-means clustering: initialization, local minima, empty clusters. As for the initialization, we randomly assign each column object to a cluster in $\mathcal {C}$ and use an exponential random number as an initial value of $a_{jq}\;(c_{j} \in C_{q})$ for all j and all entries of ${\varvec{S}}$. Note that the initial $\mathcal {R}$ and ${\varvec{F}}$ are not randomly given, but updated by the randomly initialized $\mathcal {C}$, ${\varvec{A}}$, and ${\varvec{S}}$. Concerning the local minima and empty clusters, it is recommended to run these algorithms with efficiently large random starts. We restart the update iteration from another random start if the empty clusters occur. We select the estimates with the least objective function value from the non-empty cluster estimates. Any initialization with the k-means (e.g., Xue et al. (2008)) is not used in our proposed methods because of the two following reasons: 1) its initialization also needs initialization to compute and 2) its initialization does not always lead to the best estimates; therefore, we use large random starts.

5 Simulation studies

In this section, we describe two simulation studies. The first study compares ONMTF-N and previous ONMTF methods in terms of estimation accuracy. The second study analyzes the characteristics of the estimates given by ONMTF-N, ONMTF-P, and ONMTF-CP.

5.1 Simulation study 1: estimation accuracy of ONMTF-N

We conduct a simulation study to examine the estimation accuracy of ONMT- F-N. We compare ONMTF-N with the previous ONMTF techniques proposed by Ding et al. (2006) and Yoo and Choi (2010b). We refer to their methods as “Ding’s method” and “Yoo’s method,” respectively. The details of the procedure are presented as follows: First, we generate synthetic data matrix as ${\varvec{Y}}=\tilde{{\varvec{F}}}^{*}\tilde{{\varvec{S}}}^{*}\tilde{{\varvec{A}}}^{*\prime } + {\varvec{E}}$, where $\tilde{{\varvec{F}}}^{*}$, $\tilde{{\varvec{S}}}^{*}$, and $\tilde{{\varvec{A}}}^{*}$ denote three nonnormalized true factor matrices generated as described below and ${\varvec{E}}$ denotes the error matrix whose entries are normal random number from $N(0,\sigma ^{2})$. The entries are converted to zero if $y_{ij}<0$. We generate each entries of these three nonnormalized true factor matrices $\tilde{{\varvec{F}}}^{*}$, $\tilde{{\varvec{S}}}^{*}$, and $\tilde{{\varvec{A}}}^{*}$ as follows:

$$\begin{aligned}&{\left\{ \begin{array}{ll} \tilde{f}_{im}^{*} \sim Ex(\mu ^{1/3}) &{} (r_{i} \in \tilde{R}_{m}) \\ \tilde{f}_{im}^{*} = 0 &{} (r_{i} \notin \tilde{R}_{m})\\ \end{array}\right. } \quad (i=1,\dots ,n;\;m=1,\dots ,k), \end{aligned}$$

(45)

$$\begin{aligned}&{\left\{ \begin{array}{ll} \tilde{a}_{jm}^{*} \sim Ex(\mu ^{1/3}) &{} (c_{j} \in \tilde{C}_{q}) \\ \tilde{a}_{jm}^{*} = 0 &{} (c_{j} \notin \tilde{C}_{q})\\ \end{array}\right. } \quad (j=1,\dots ,p;\;q=1,\dots ,\ell ), \end{aligned}$$

(46)

$$\begin{aligned}&\tilde{s}_{mq}^{*} \sim Ex(\mu ^{1/3}) \quad (m=1,\dots ,k;\;q=1,\dots ,\ell ), \end{aligned}$$

(47)

where $Ex(\cdot )$ denotes an exponential distribution, $\mu$ denotes the expected value of each entry of ${\varvec{Y}}$, and $\tilde{\mathcal {R}}=\{\tilde{R}_{1},\dots ,\tilde{R}_{k}\}$ and $\tilde{\mathcal {C}}=\{\tilde{C}_{1},\dots ,\tilde{C}_{\ell }\}$ denote true partitions of row and column objects. These partitions are determined using discrete uniform random numbers. It is noted that the $\mu$ undoubtedly represents the expected value of $y_{ij}$ from the following equation:

$$\begin{aligned}&\displaystyle E[y_{ij}] = E[\tilde{f}_{im}^{*}\tilde{s}_{mq}^{*}\tilde{a}_{jq}^{*}] = E[\tilde{f}_{im}^{*}]E[\tilde{s}_{mq}^{*}]E[\tilde{a}_{jq}^{*}] = \left( \mu ^{1/3}\right) ^{3} = \mu \nonumber \\&\quad \displaystyle (i=1,\dots ,n;\;j=1,\dots ,p;\;\tilde{R}_{m} \ni r_{i};\;\tilde{C}_{q}\ni c_{j}). \end{aligned}$$

(48)

Here, we get the normalized true factor matrices, $\tilde{{\varvec{F}}}$, $\tilde{{\varvec{S}}}$, and $\tilde{{\varvec{A}}}$ for the later step as follows:

$$\begin{aligned} \tilde{{\varvec{F}}} = \tilde{{\varvec{F}}}^{*}{\varvec{D}}_{\tilde{{\varvec{F}}}^{*}}^{-1}, \quad \tilde{{\varvec{A}}} =\tilde{{\varvec{A}}}^{*}{\varvec{D}}_{\tilde{{\varvec{A}}}^{*}}^{-1}, \quad \tilde{{\varvec{S}}} = {\varvec{D}}_{\tilde{{\varvec{F}}}^{*}}\tilde{{\varvec{S}}}^{*}{\varvec{D}}_{\tilde{{\varvec{A}}}^{*}}. \end{aligned}$$

(49)

where ${\varvec{D}}_{\tilde{{\varvec{F}}}^{*}}$ and ${\varvec{D}}_{\tilde{{\varvec{A}}}^{*}}$ denote the $k \times k$ and $\ell \times \ell$ diagonal matrix, in which each diagonal element is $\Vert \tilde{{\varvec{f}}}_{m}^{*}\Vert \; (m=1,\dots ,k)$ and $\Vert \tilde{{\varvec{a}}}_{q}^{*}\Vert \; (q=1,\dots ,\ell )$, respectively. Second, we execute our proposed method ONMTF-N and the previous methods (Ding’s and Yoo’s methods) using given ${\varvec{Y}}$, initialized factor matrices, $\tau$, and $\nu$, where $\tau$ denotes the threshold for the stopping algorithm and $\nu$ denotes the maximum number of iterative cycles. Here, we use the initial inputs $\bar{\mathcal {R}}$, $\bar{\mathcal {C}}$, $\bar{{\varvec{F}}}$, $\bar{{\varvec{S}}}$, and $\bar{{\varvec{A}}}$ for ONMTF-N which is generated in the same manner of generating true ones as described above. In this initialization, instead of the $\mu$, $\bar{y} = \sum _{i=1}^{n}\sum _{j=1}^{p}y_{ij}/(np)$ is used in generating exponential random numbers. For Ding’s and Yoo’s methods, we generate initial inputs as follows:

$$\begin{aligned}&\bar{f}_{im}, \; \bar{s}_{mq}, \; \bar{a}_{jq} \sim Ex(\bar{y}^{1/3}) \nonumber \\&\quad (i=1,\dots ,n;\;j=1,\dots ,p;\;m=1,\dots ,k;\;q=1,\dots ,\ell ) \end{aligned}$$

(50)

From this execution, we get the estimated parameters, that is, estimated partitions of row and column objects $\hat{\mathcal {R}}=\{\hat{R}_{1},\dots ,\hat{R}_{k}\}$ and $\hat{\mathcal {C}}=\{\hat{C}_{1},\dots ,\hat{C}_{\ell }\}$, respectively, and three factor matrices $\hat{{\varvec{F}}}$, $\hat{{\varvec{S}}}$, and $\hat{{\varvec{A}}}$. Note that the factor matrices are normalized such as (49). Finally, we measure the similarity (or dissimilarity) between each of the five pairs of true and the estimated parameters. We calculate the following five indices: (a) $\text {ARI}(\tilde{\mathcal {R}},\hat{\mathcal {R}})$, (b) $\text {ARI}(\tilde{\mathcal {C}},\hat{\mathcal {C}})$, (c) $\displaystyle \Vert \tilde{{\varvec{F}}} - \hat{{\varvec{F}}} \Vert /{(nk)}$, (d) $\displaystyle \Vert \tilde{{\varvec{A}}} - \hat{{\varvec{A}}} \Vert /{(p\ell )}$, and (e) $\displaystyle \Vert \tilde{{\varvec{S}}} - \hat{{\varvec{S}}} \Vert /{(k\ell )}$, where $\text {ARI}(\cdot ,\cdot )$ denotes the adjusted Rand index (ARI) (Hubert and Arabie 1985), which is a similarity measure between two partitions of objects. ARI is 1 if the two partitions are completely the same and is close to 0 if they are different. For the three factor matrices, we calculate the mean square error (MSE). The parameters for generating the synthetic data are set as follows: (n, p, k, $\ell$) $=$ (1000, 600, 5, 3), $\sigma = 1$, $\mu =10$, $\tau = np \times 10^{-7}$, $\nu = 1000$. For more results of the other conditions, please see Abe H, Yadohisa H (2017) Supplementary material to “Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions.” It is noted that the true numbers of row and column clusters, and the estimated ones, are the same as k and $\ell$, respectively. We generate 100 synthetic data matrices; from among the candidates of the estimates given by 20 executions of each ONMTF, we select the best estimates for which the objective function value is minimized. The results are shown in Fig. 1. ONMTF-N has the highest ARI for row clusters (see (a)), followed by Yoo’s and Ding’s methods in that order. In contrast, ARIs for the column clusters (see (b)) of both Ding’s and Yoo’s methods are as large as those of ONMTF-N. This result suggests that all three methods can accurately detect the smaller side clusters (in our simulation, this is the column side), but two previous methods cannot accurately detect the larger side clusters. The results of the MSE for the row and column factor matrices (see the (c) and (d)) are similar to the ARI results. The MSE for the middle factor matrix (see (e)) obtained by our method is smaller than those obtained by the other two methods. Overall, from this simulation study, we can conclude that ONMTF-N provides better estimation consistency than the other two ONMTF methods.

Fig. 1

Boxplots of five indices for the 3 methods. Each of the panels represents a$\text {ARI}(\tilde{\mathcal {R}},\hat{\mathcal {R}})$, b$\text {ARI}(\tilde{\mathcal {C}},\hat{\mathcal {C}})$, c$\displaystyle \Vert \tilde{{\varvec{F}}} - \hat{{\varvec{F}}} \Vert /{(nk)}$, d$\displaystyle \Vert \tilde{{\varvec{A}}} - \hat{{\varvec{A}}} \Vert /{(p\ell )}$, and e$\displaystyle \Vert \tilde{{\varvec{S}}} - \hat{{\varvec{S}}} \Vert /{(k\ell )}$. The left, middle, and right boxplots represent Ding’s, Yoo’s, and our proposed method, respectively. The vertical axis ranges are not same because the magnitude of $f_{im}$ and $a_{jq}$ differs depending on n and p

Fig. 2

Graph of $d_{\beta }(x,y)$ for various $\beta$ values. The left side represents $y = 10$ and the right side represents $y = 100$. The horizontal axis denotes x and the vertical axis denotes $d_{\beta }(x,y)$

5.2 Simulation study 2: robustness of ONMTF-CP against extremely large positive values

We conducted another simulation study to demonstrate the characteristics of the estimates given by ONMTF-N, ONMTF-P, and ONMTF-CP. As mentioned in previous sections, it is assumed that $y_{ij}$ follows normal, Poisson, and CP distributions, respectively, in these three ONMTF methods. These distributions belong to the Tweedie family, which is described by (20), and the value of $\beta$ determines the distribution: it is normal if $\beta =2$, Poisson if $\beta =1$, and CP if $\beta \in (0,1)$. The index parameter $\beta$ is related to the robust estimation against extremely large positive values. Figure 2 shows $\beta$-divergence, which is derived from the log-likelihood of Tweedie distributions for various $\beta$ values when $y=10$ or $y=100$. The $\beta$-divergence around a small y is larger than that around a large y, except $\beta =2$. This means that extremely large positive values in the data are not considered for parameter estimation when $\beta$ is smaller than 2. To examine these characteristics in ONMTF, we measure the estimation accuracy of ONMTF-N, ONMTF-P, and ONMTF-CP for synthetic data matrices generated using normal, Poisson, and CP distributions of data. The accuracy is calculated using the ARI between true clusters and estimated clusters of row and column objects.

We now explain how to generate a synthetic data matrix. First, we generate $\tilde{\mathcal {R}}$, $\tilde{{\varvec{F}}}^{*}$, $\tilde{\mathcal {C}}$, $\tilde{{\varvec{A}}}^{*}$, and $\tilde{{\varvec{S}}}^{*}$ as in Sect. 5.1. Next, we generate each element of the synthetic data matrix ${\varvec{Y}}$ as a random number from $y_{ij} \sim TW(x_{ij},\phi ,\tilde{\beta })$, where $TW(x,\phi ,\beta )$ denotes a Tweedie distribution, and the mean $x_{ij}$ is the corresponding i, j element of ${\varvec{X}} = \tilde{{\varvec{F}}}^{*}\tilde{{\varvec{S}}}^{*}\tilde{{\varvec{A}}}^{*\prime }$. Note that $TW(x_{ij},\phi ,\tilde{\beta })$ is normal if $\tilde{\beta }=2$ and Poisson if $\tilde{\beta }=1$. Negative values of $y_{ij}$ can be generated when $\tilde{\beta }=2$. In this case, $y_{ij}$ is converted to zero. The parameters for generating synthetic data are set as follows: $(n,p,k,\ell ) = (100,100,5,5)$, $\phi = 2$, $\tilde{\beta }=\{ 2,1,0.8,0.5,0.2 \}$, $\mu =10$, $\tau = np \times 10^{-7}$ (threshold for the stopping algorithm), and $\nu = 1000$ (the maximum number of iterative cycles.) It is noted that the true numbers of row and column clusters, and the estimated ones, are the same as k and $\ell$, respectively. We generate 100 synthetic data matrices for each of five conditions. Then, from among the estimate candidates given by 20 executions, we select the best estimates, $\hat{\mathcal {R}}$, for which the objective function value is minimized. We then calculate $\text {ARI}(\tilde{\mathcal {R}},\hat{\mathcal {R}})$ of each ONMTF. We execute ONMTF-CP for three cases: $\beta =\{0.2,0.5,0.8\}$. We refer to the procedures as ONMTF-CP2, ONMTF-CP5, and ONMTF-CP8, respectively. The results are shown in Fig. 3. Note that we do not show the result of ARI for column clustering because it is very similar to that for row clustering. When $\tilde{\beta }=2$ (normal), ONMTF-N has the best accuracy, followed by ONMTF-P, ONMTF-CP8, ONMTF-CP5, and ONMTF-CP2, in that order. When $\tilde{\beta }=0.5$, ONMTF-N has the worst accuracy; when $\tilde{\beta }=0.2$, the accuracy deteriorates in the order of ONMTF-N, ONMTF-P, and ONMTF-CP8. Because more extremely large positive values are generated from a CP distribution with small $\tilde{\beta }$ values, these results imply that ONMTF-N, ONMTF-P, and ONMTF-CP procedures with relatively larger $\beta$ values do not fit a data matrix containing some extremely large positive values. This does not mean that an ONMTF-CP procedure with a small $\beta$ value is the best in any case. It may be worse for a data matrix having a normal error, as shown in the case of $\tilde{\beta }=2$ in Fig. 3. However, it may be preferable to use ONMTF-CP because the inaccuracy of ONMTF-CP with small $\beta$ values is smaller than that of ONMTF-N for a data matrix containing some extremely large positive values.

6 Applications

In this section, we describe an application involving a document-term data matrix. The data is used to compare the goodness of clustering of ONMTF-N with those of the previous ONMTF methods and other methods used for the document clustering task. In addition, we attempt to apply our methods to point-of-sale data to check the differences among ONMTF-N, ONMTF-P, and ONMTF-CP. However, the application is not described herein because of space limitations. For the application, please see Abe H, Yadohisa H (2017) Supplementary material to “Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions.”

Fig. 3

$\text {ARI}(\tilde{\mathcal {R}},\hat{\mathcal {R}})$ obtained by five ONMTF methods for five conditions. The five boxplots for ONMTF-N, ONMTF-P, ONMTF-CP8, ONMTF-CP5, and ONMTF-CP2 are lined up from left to right in that order in each of the panels

We describe an application involving a matrix containing document-term data to compare the goodness of clustering of ONMTF-N with previous ONMTF, Ding’s and Yoo’s methods, and those of methods used previously for the document clustering task. Two reasons are considered to apply the three ONMTF methods for document-term data: (1) the ONMTF is compatible with the document-term clustering; (2) these two previous ONMTF methods have some problems, as described above, and we are interested in its performance in real data application. In addition to the three ONMTF methods based on normal distribution, we also applied various methods described in Table 1. Double k-means (DK), LP-FNMTF (LP), and Graph modularity maximization (Mod) were applied as the base line biclustering methods. LP was proposed by Wang et al. (2011) as extended double k-means. Mod was proposed by Ailem et al. (2016) for direct maximization of bipartite graph modularity. We applied ONMTF-P (P) and ONMTF-CP (CP8, CP5, and CP2) to confirm the robust performance of them for the document clustering task. SPKM (SP) and WVSPKM (WV) were also applied for comparison with one-side clustering with biclustering.

Table 1 List of methods for comparison in the application for document clustering

Table 2 Stats of some text-word datasets in CLUTO

The data matrices we used were obtained from the open data CLUTO¹ website. Table 2 lists the selected data matrices and statistics. The list of datasets in Table 2 are ordered by the number of elements. The tr11, tr12, tr23, tr31, tr41, and tr45 datasets are derived from the TREC² collections. The true categories of the documents in the tr31 and tr41 datasets are obtained by particular queries. The re0 and re1 datasets are from the Reuters-21578 text categorization test collection, distribution 1.0.³ The fbis data set is from the Foreign Broadcast Information Service data of TREC-5. The hitech is a dataset of San Jose Mercury Newspaper articles and contains documents about computers, electronics, health, medicine, research, and technology. The k1a, k1b, and wap datasets are used for the WebACE project (Boley et al. 1999) and contain web pages in various subject directories of Yahoo!.⁴ Datasets k1a and k1b contain the same documents, but the true labels are different.

We conduct tf-idf conversion for all data matrices. The intuitive idea for the document clustering is to classify the documents using “unique words” that appear in each of the documents. Although we count the appearance frequency of every word in each of the documents, the counting does not represent the “uniqueness” of the word. Well used words (e.g., “run”, “come”, and “get”) are not important for document clustering. We can add weight to words, which do not appear in many documents, by tf-idf conversion. Using tf-idf values provides a well intuitive document clustering. The row and column objects are assigned to documents and words, respectively; then, ${\varvec{F}}$ and ${\varvec{A}}$ can be considered as the factor matrices of documents and words, respectively. The number of document clusters (the number of columns of ${\varvec{F}}$, e.g., k) is set to the number of document classes provided in advance, and the number of word clusters (the number of columns of ${\varvec{A}}$, e.g., $\ell$) is set to 10 for all the data matrices. Although the best number of word cluster must be estimated from the dataset, we set the number to 10 in advance to avoid the time consuming computation. Note that the k=$\ell$ in Mod algorithm, that is , the numbers of word and document clusters are same. The convergence threshold is set as $\tau =10^{-5}\Vert {\varvec{Y}} \Vert ^{2}/2$ for all the methods except for co-clustering based on graph modularity whose computation is repeated until no change is observed in its objective function value. The goodness of clustering is measured using the ARI between the given clusters and estimated clusters of the documents. It is noted that some clusters occasionally become empty in the update iteration for some methods. In this case, we restart the update iteration from another initial parameter. From among the candidates of the estimates given by 10 executions of each method, we select the best estimates for which the objective function value is minimized. The initial inputs of each method is as follows: for Ding, Yoo, N, P, and CP methods, we use initial inputs as described in Sect. 5.1. For DK, we generate initial partition of row objects as described in Sect. 5.1 and initial middle factor matrix (which means center of cluster) such as $s_{mq} \sim Ex(\bar{y})$. For SP, we generate the initial right side factor matrix (which means the normalized center of cluster) as follows: first we generate random number $a_{jk} \sim Ex(1)$ and then normalize square norm of each row of the ${\varvec{A}}$ to 1. For WV, we generate the initial right side factor matrix such as $a_{jk} \sim Ex(\bar{y}^{1/2})$. For LP, we generate initial partition of row and column objects as described in Sect. 5.1. For Mod, we use the default initialization procedure of the “CoclustMod” function which is distributed in Python package “coclust.”⁵ We apply the LP in the same way described in Wang et al. (2011). We use nearest neighbor graph as affinity matrices. We try to compute with all patterns of hyperparameters: the neighborhood size $k=\{ 1,2, \dots , 10 \}$ and each of the regularization parameters, $\alpha$ and $\beta$ (which is different from $\beta$ of Tweedie distribution or $\beta$-divergence), is $\{ 0.1, 1, 10, 100, 500, 1000 \}$. We select the hyperparameter pattern with the highest average of ARI for relatively small datasets (tr23, tr12, tr11, re0, fbis, and tr45) and then we apply for the remaining datasets with the hyperparameter pattern. The selected pattern is $k=9$, $\alpha =100$, and $\beta =500$. The result is in Table 3.

Table 3 ARI between the given and estimated clusters of the documents

Table 3 shows ARI of the 12 methods for each dataset. Yoo’s method has four datasets (tr12, re0k1a, and k1b) with the best ARI and this number is the largest amongst those of the 12 methods. The second best performance is shown in SP, which has three datasets (tr23, tr41, and hitech) with the best ARI and the method has high accuracy in many datasets. The third best performance is shown in N, Mod, and WV, each of which show the best ARI in two datasets. Although Yoo’s method seems to show the best performance from the perspective of which method shows the best ARI in any of the datasets, the method has poor ARI on some datasets, e.g., re1, tr41, tr31, and hitech. In fact, Yoo’s method shows higher performance than N in no more than 5 datasets of all 13 datasets. On the other hand, N is not poor for almost all datasets in the biclustering methods. However, it is poorer than one-side clustering methods, SP and WV. A possible reason for the result is as follows. Applying one-side clustering approach to document-term clustering task leads to the clustering of documents and terms into the same number of clusters; on the other hand, biclustering approach leads to clustering of the two into a different number of clusters. In the biclustering approach, a number of term clusters may be short or too much for clustering documents, and this mismatch could cause the poorer performance of N in comparison to one-side clustering methods. We also show the worse performance of P and CPs, which are robust methods, in comparison to that of other methods. The cause of this result can be robustness against the large positive values. We use a document-term matrix converted using tf-idf conversion, which strongly weights terms in a few documents. The entries of such terms have a large positive value. The effect of the weighted entries disappear when these robust ONMTFs are used, implying that an ONMTF based on the Euclidean distance is not used, and interpretable clusters cannot be obtained. Note that good performance of the robust ONMTF can be obtained using new standardizations other than tf-idf conversion. However, we will undertake this challenge in our future works. DK and LP have the poor result in most of the datasets. The accuracy of LP can be improved by selection more appropriate hyperparameters for each dataset.

We obtain the results of the computational time, degree of approximation (measured as Euclidean distance at convergence), and convergence behavior. However, we do not report these herein because of space limitation. Please see Abe H, Yadohisa H (2017) Supplementary material to “Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions for the results.”

We now show the estimates given by ONMTF-N using the k1a datasets as an example to demonstrate the features from the bi-clustering aspect. The k1a dataset consists of web news documents obtained from the Reuters news service in October, 1997 (Boley 1998). The documents in k1a are labeled in advance under six categories: “business,” “entertainment,” “health,” “politics,” “sports,” and “tech”. Table 4 presents the estimated 10 clusters of the terms given by ONMTF-N. For comparison, we also show them by Yoo’s method in Table 5. Table 6 shows the center factor matrices ${\varvec{S}}$ that enable us to grasp the relationship between the estimated document and the term clusters. We label the document clusters herein as DC 1 to DC 6 and the term clusters as TC 1 to TC 10.

Table 4 Terms of 10 clusters in the k1b datasets obtained using ONMTF-N

Table 5 Terms of 10 clusters in the k1b datasets by Yoo’s method. The value of the right side of each of the term is the value of $a_{jq}$ on each of the term. All $a_{jq}$ are standardized such that the length of each column vector in ${\varvec{A}}$ is 1. Only terms are shown which have top 10 values of $a_{jq}$ in each cluster

Table 6 Middle factor matrix ${\varvec{S}}$ of the k1a dataset

Table 7 Our interpretation for term clusters obtained by two methods

We now focus on the term clustering. Each of the term cluster obtained using the two methods could be interpreted as Table 7. We find that both methods extract similar term clusters from each other (e.g., TC 1, 2, 4, 5, and 6 of both.) However, TC 3, which is strongly related to DC 3 (see Table 6) in both methods, has different terms. TC 3 of ONMTF-N seems to mean the chart of movie by the box office and that of Yoo’s method seems to mean sports. Unfortunately, the detail of document clusters cannot be shown due to space limitations, but in fact, DC 3 of ONMTF-N includes some “entertainment” documents, while that of Yoo’s method includes almost all of “sports” documents. From the fact that Yoo’s TC 1 includes some words related to box office , documents related to box office chart can be integrated into DC 1 in Yoo’s estimates. On the other hand, there are no word cluster on sports in ONMTF-N. The “sports” documents are indeed misclassified to DC 1, which includes many entertainment documents, by ONMTF-N. Although this misclassifying of “sports” documents leads to poorer ARI of ONMTF-N than Yoo’s method, ONMTF-N detects the meaningful cluster related to box office chart instead of “sports.” However, ONMTF-N has irrelevantly estimated the document clusters unfamiliar with the real document labels. Moreover, ONMTF-N has some assorted term clusters that are not affected by document clustering. The difference of the two can be explained from the aspect of their estimating algorithms. In fact, ONMTF-N exhibits less freedom than Yoo’s method for estimating factor matrices owing to their orthogonality. Indeed its objective function value at convergence is larger than Yoo’s: ONMTF-N has 241.8; and Yoo’s method has 241.1. This estimating problem can be solved by more random starts thanks to the higher computational speed of ONMTF-N in comparison to Yoo’s method.

7 Conclusion

In this paper, we proposed a new method for ONMTF, namely, ONMTF-N, in which the objective function value is monotonically non-increasing and the orthogonality of the factor matrices is maintained. In addition, we proposed two other ONMTF methods, namely, ONMTF-P and ONMTF-CP. The main contributions of this paper are as follows. First, our simulation study and an application involving some document-term matrices indicated that ONMTF-N shows higher estimation accuracy than previous methods. Second, we derived a new auxiliary function to optimize the middle factor matrix in ONMTF-CP using an inequality of a bivariate concave function. Third, another simulation study indicated that ONMTF-CP may be robust against the effect of extremely large positive values.

NMFs with orthogonality, including our methods, should be used considering the trade-off between its easy-to-understand estimates and its under fitting. An orthogonal constraint simplifies a factor matrix, thereby facilitating result interpretation. However, a factor matrix with a simplified structure leads to a poor approximation to data matrix (e.g., ${\varvec{Y}}$.)

Two issues remain to be addressed in the future. First, cluster degeneration tends to occur in the three proposed methods. Therefore, further investigation is required to develop an approach for rapidly seeking alternative initial parameters, in order to avoid cluster degeneration. Second, it is necessary to develop an approach for estimating the best number of clusters for both row objects and column objects.