Mutual information, phi-squared and model-based co-clustering for contingency tables

Abstract

Many of the datasets encountered in statistics are two-dimensional in nature and can be represented by a matrix. Classical clustering procedures seek to construct separately an optimal partition of rows or, sometimes, of columns. In contrast, co-clustering methods cluster the rows and the columns simultaneously and organize the data into homogeneous blocks (after suitable permutations). Methods of this kind have practical importance in a wide variety of applications such as document clustering, where data are typically organized in two-way contingency tables. Our goal is to offer coherent frameworks for understanding some existing criteria and algorithms for co-clustering contingency tables, and to propose new ones. We look at two different frameworks for the problem of co-clustering. The first involves minimizing an objective function based on measures of association and in particular on phi-squared and mutual information. The second uses a model-based co-clustering approach, and we consider two models: the block model and the latent block model. We establish connections between different approaches, criteria and algorithms, and we highlight a number of implicit assumptions in some commonly used algorithms. Our contribution is illustrated by numerical experiments on simulated and real-case datasets that show the relevance of the presented methods in the document clustering field.

Appendices

Appendix A: Proof of Proposition 1

Denoting $e_{ij}=p_{ij}-p_{i.} p_{.j}$ $\forall i,j$ and using the relation $\log (1+x)=x-x^2/2 + O(x^2)$, it can be written $\log (1+\frac{e_{ij}}{p_{i.}p_{.j}})=\frac{e_{ij}}{p_{i.}p_{.j}} - \frac{1}{2} \left( \frac{e_{ij}}{p_{i.}p_{.j}}\right) ^2 + O(e_{ij}^2)$ for $e_{ij} \rightarrow 0$ and

$$\begin{aligned} (p_{ij} +e_{ij})\log \left( 1+\frac{e_{ij}}{p_{i.}p_{.j}}\right)= & {} e_{ij} - \frac{1}{2} \frac{e_{ij}^2}{p_{i.}p_{.j}} + \frac{e_{ij}^2}{p_{i.}p_{.j}} -\frac{1}{2} \frac{e_{ij}^3}{p_{i.}^2 p_{.j}^2}\\&+\, O(e_{ij}^2) =e_{ij}+ \frac{1}{2} \frac{e_{ij}^2}{p_{i.}p_{.j}} + O(e_{ij}^2)\\ \sum _{i,j} (p_{ij} +e_{ij})\log \left( 1+\frac{e_{ij}}{p_{i.}p_{.j}}\right)= & {} \underbrace{\sum _{i,j} e_{ij}}_{=0} + \frac{1}{2} \sum _{i,j}\frac{e_{ij}^2}{p_{i.}p_{.j}} + O\left( \sum _{i,j}e_{ij}^2\right) \end{aligned}$$

this yields

$$\begin{aligned} \mathcal {I}(P_{IJ})=\frac{1}{2} \Phi ^2(P_{IJ}) + O\left( \sum _{i,j}(p_{ij}-p_{i.}p_{.j})^2\right) \quad \text{ for } \quad (p_{ij}-p_{i.}p_{.j}) \rightarrow 0 \end{aligned}$$

Appendix B: Properties of $P_{KL}^{\mathbf {z}\mathbf {w}}$

$$\begin{aligned} \sum _{k\ell } p_{k\ell }^{\mathbf {z}\mathbf {w}}=1 \quad (P_{KL}^{\mathbf {z}\mathbf {w}} \hbox {is a distribution} ) \sum _{\ell } p_{k\ell }^{\mathbf {z}\mathbf {w}}=\sum _i z_{ik} p_{i.} \quad \text {and} \quad \sum _{k} p_{k\ell }^{\mathbf {z}\mathbf {w}}=\sum _j w_{j\ell } p_{.j}. \end{aligned}$$

Proof

We have

$$\begin{aligned} \sum _{k,\ell } p_{k\ell }^{\mathbf {z}\mathbf {w}}= \sum _{i,j,k,\ell } z_{ik}w_{j\ell } p_{ij} =\sum _{ij}p_{ij} \sum _{k,\ell } z_{ik}w_{j\ell }=1\quad \text{ since } \sum _{k,\ell } z_{ik}w_{j\ell }=1 \end{aligned}$$

$$\begin{aligned} \sum _\ell p_{k\ell }^{\mathbf {z}\mathbf {w}}= & {} \sum _{i,j,\ell } z_{ik}w_{j\ell } \; p_{ij} = \sum _i\left( z_{ik} \sum _j \left( p_{jj} \sum _\ell w_{j\ell }\right) \right) \\ {}= & {} \sum _i z_{ik} p_{i.}\quad \text{ since } \sum _\ell w_{j\ell }=1 \end{aligned}$$

$$\begin{aligned} \sum _k p_{k\ell }^{\mathbf {z}\mathbf {w}}= & {} \sum _{i,j,k} z_{ik}w_{j\ell } \; p_{ij} = \sum _j\left( w_{j\ell } \sum _i \left( p_{jj} \sum _k z_{ik}\right) \right) \\ {}= & {} \sum _j w_{j\ell } p_{i.j}\quad \text{ since } \sum _k z_{ik}=1 \end{aligned}$$

$\square$

Appendix C: Properties of $Q_{IJ}^{\mathbf {z}\mathbf {w}}$

$$\begin{aligned} \sum _{i,j} q_{ij}^{\mathbf {z}\mathbf {w}}=1, \quad q_{i.}^{\mathbf {z}}=p_{i.} \quad \text {and} \quad q_{.j}^{\mathbf {w}}=p_{.j} \qquad \forall i,j, \end{aligned}$$

Proof

$$\begin{aligned} \sum _{i,j} q_{ij}^{\mathbf {z}\mathbf {w}}= \sum _{i,j} p_{i.}p_{.j} \sum _{k,\ell } z_{ik} w_{j\ell }\frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}} =\sum _{k,\ell } \frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}} \underbrace{\sum _{i,j} p_{i.}p_{.j}z_{ik} w_{j\ell }}_{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}=\sum _{k,\ell } p_{k\ell }^{\mathbf {z}\mathbf {w}}=1 \end{aligned}$$

$$\begin{aligned} q_{i.}^{\mathbf {z}}=\sum _j q_{ij}^{\mathbf {z}\mathbf {w}}= \sum _j p_{i.}p_{.j} \sum _{k,\ell } z_{ik} w_{j\ell }\frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}= p_{i.} \sum _k \frac{z_{ik}}{p_{k.}} \underbrace{\sum _{\ell } \frac{\overbrace{\sum _j (w_{j\ell } p_{.j} )}^{p_{.\ell }^{\mathbf {w}}} p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{.\ell }^{\mathbf {w}}}}_{=p_{k.}^{\mathbf {z}}}=p_{i.}. \end{aligned}$$

and, symmetrically, $q_{.j}^{\mathbf {w}}=p_{.j}.$ $\square$

Appendix D: Proof of Proposition 2

Lemma 1

$$\begin{aligned} \sum _{k,\ell } \frac{(p_{k\ell }^{\mathbf {z}\mathbf {w}})^2}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}= \sum _{i,j} \frac{p_{ij} q_{ij}^{\mathbf {z}\mathbf {w}}}{p_{i.}p_{.j}}=\sum _{i,j} \frac{(q_{ij}^{\mathbf {z}\mathbf {w}})^2}{p_{i.}p_{.j}} \end{aligned}$$

Proof

$$\begin{aligned} \sum _{i,j} \frac{p_{ij} q_{ij}^{\mathbf {z}\mathbf {w}}}{p_{i.}p_{.j}}= & {} \sum _{i,j} p_{ij} \frac{p_{i.}p_{.j} \sum _{k,\ell }z_{ik} w_{j\ell } \frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}}{p_{i.}p_{.j}}= \sum _{k,\ell } \frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}} \sum _{i,j} z_{ik}w_{j\ell } p_{ij}\\ {}= & {} \sum _{k,\ell } \frac{(p_{k\ell }^{\mathbf {z}\mathbf {w}})^2}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}} \end{aligned}$$

$$\begin{aligned} \sum _{i,j} \frac{(q_{ij}^{\mathbf {z}\mathbf {w}})^2}{p_{i.}p_{.j}}= & {} \sum _{i,j} \frac{ \left( p_{i.}p_{.j} \sum _{k,\ell } z_{ik}w_{j\ell } \frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}\right) ^2}{p_{i.}p_{.j}} =\sum _{i,j} p_{i.}p_{.j}\sum _{k,\ell } z_{ik}w_{j\ell } \frac{(p_{k\ell }^{\mathbf {z}\mathbf {w}})^2}{(p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}})^2}\\= & {} \sum _{k,\ell } \frac{(p_{k\ell }^{\mathbf {z}\mathbf {w}})^2}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}. \end{aligned}$$

$\square$

Proof of Proposition 2

• The first equation (a) can easily be deduced from Lemma 1

• Second equation (b):

  $$\begin{aligned} \Phi ^2(P_{IJ})-\Phi ^2(Q_{IJ}^{\mathbf {z}\mathbf {w}})= & {} \left( \sum _{i,j} \frac{p_{ij}^2}{p_{i.}p_{.j}} -1\right) - \left( \sum _{i,j} \frac{(q_{ij}^{\mathbf {z}\mathbf {w}})^2}{q_{i.}^{\mathbf {z}}q_{.j}^{\mathbf {w}}} -1\right) \\= & {} \sum _{i,j} \frac{p_{ij}^2}{p_{i.}p_{.j}} - \sum _{i,j} \frac{(q_{ij}^{\mathbf {z}\mathbf {w}})^2}{p_{i.}p_{.j}}. \end{aligned}$$

  Using Lemma 1, this relation can be written

  $$\begin{aligned} \sum _{i,j} \frac{p_{ij}^2}{p_{i.}p_{.j}} - \sum _{i,j} \frac{p_{ij}q_{ij}^{\mathbf {z}\mathbf {w}}}{p_{i.}p_{.j}} = \sum _{i,j} p_{ij} \left( \frac{p_{ij}}{p_{i.}p_{.j}} - \frac{q_{ij}^{\mathbf {z}\mathbf {w}}}{p_{i.}p_{.j}}\right) \end{aligned}$$

  and

  $$\begin{aligned}&\sum _{i,j} \frac{p_{ij}^2}{p_{i.}p_{.j}} - 2 \sum _{i,j} \frac{p_{ij}q_{ij}^{\mathbf {z}\mathbf {w}}}{p_{i.}p_{.j}} + \sum _{i,j} \frac{(q_{ij}^{\mathbf {z}\mathbf {w}})^2}{p_{i.}p_{.j}}=\sum _{i,j} \frac{(p_{ij}-q_{ij}^{\mathbf {z}\mathbf {w}})^2}{p_{i.}p_{.j}}\\&\quad =\chi ^2(P_{IJ}||Q_{IJ}^{\mathbf {z}\mathbf {w}})\ge 0. \end{aligned}$$

• The two inequalities (c) can easily be deduced from the previous relation.

Appendix E: Proof of Proposition 3

• First equation (d):

  $$\begin{aligned} \mathcal {I}(Q_{IJ}^{\mathbf {z}\mathbf {w}})= & {} \sum _{i,j} q_{ij}^{\mathbf {z}\mathbf {w}} \log \frac{q_{ij}^{\mathbf {z}\mathbf {w}}}{p_{i.}p_{.j}} =\sum _{i,j}\left( p_{i.}p_{.j}\left( \sum _{k,\ell }z_{ik} w_{j\ell }\frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}\right) \right. \\&\left. \times \log \left( \sum _{k,\ell } z_{ik}w_{j\ell }\frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}\right) \right) \\= & {} \sum _{i,j}\left( p_{i.}p_{.j}\left( \sum _{k,\ell }z_{ik} w_{j\ell }\frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {w}}p_{.\ell }^{\mathbf {w}}} \log \frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}\right) \right) \\= & {} \sum _{k,\ell } p_{k\ell }^{\mathbf {z}\mathbf {w}} \log \frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}\\= & {} \mathcal {I}(P_{KL}^{\mathbf {z}\mathbf {w}}). \end{aligned}$$

• Second equation (e):

  $$\begin{aligned} \mathcal {I}(P_{IJ})-\mathcal {I}(Q_{IJ}^{\mathbf {z}\mathbf {w}})= & {} \mathcal {I}(P_{IJ})-\mathcal {I}(P_{KL}^{\mathbf {z}\mathbf {w}}) =\sum _{i,j} p_{ij} \log \frac{p_{ij}}{p_{i.}p_{.j}} -\sum _{k,\ell } p_{k\ell }^{\mathbf {z}\mathbf {w}} \log \frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}\\= & {} \sum _{i,j,k,\ell } z_{ik}w_{j\ell } p_{ij} \log \frac{p_{ij}}{p_{i.}p_{.j}} -\sum _{i,j,k,\ell }(z_{ik}w_{j\ell } p_{ij}) \log \frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}\\= & {} \sum _{i,j,k,\ell } z_{ik}w_{j\ell } p_{ij} \log \frac{p_{ij}}{p_{i.}p_{.j}} \frac{p_{k.}^{\mathbf {z}} p_{.\ell }^{\mathbf {w}}}{p_{k\ell }^{\mathbf {z}\mathbf {w}}}\\= & {} \sum _{i,j,k,\ell } z_{ik}w_{j\ell } p_{ij} \log \frac{p_{ij}}{p_{i.}p_{.j} \frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}} p_{.\ell }^{\mathbf {w}}}}\\= & {} \sum _{i,j} p_{ij} \log \frac{p_{ij}}{p_{i.}p_{.j}\sum _{k,\ell } z_{ik} w_{j\ell } \frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}} p_{.\ell }^{\mathbf {w}}}}\\= & {} \sum _{i,j} p_{ij} \log \frac{p_{ij}}{q_{ij}^{\mathbf {z}\mathbf {w}}} =\text {KL}(P_{IJ}||Q_{IJ}^{\mathbf {z}\mathbf {w}}) \ge 0. \end{aligned}$$

• The two inequalities (f) can easily be deduced from the previous relation.

Appendix: F Properties of $R_{IJ}^{\mathbf {z}\mathbf {w}\varvec{\delta }}$

$$\begin{aligned} \sum _{i,j} r_{ij}^{\mathbf {z}\mathbf {w}\varvec{\delta }}=1 \quad r_{i.}^{\mathbf {z}\mathbf {w}\varvec{\delta }}=p_{i.} \quad \text {and} \quad r_{.j}^{\mathbf {z}\mathbf {w}\varvec{\delta }}=p_{.j} \qquad \forall i,j \end{aligned}$$

Proof

$$\begin{aligned} \sum _{ij}r_{ij}^{\mathbf {z}\mathbf {w}\varvec{\delta }}= & {} \sum _{i,j}p_{i.}p_{.j}\sum _{k,\ell }z_{ik}w_{j\ell }\delta _{k\ell }= \sum _{k,\ell } \delta _{k\ell }\sum _{i,j}p_{i.}p_{.j}z_{ik}w_{j\ell }= \sum _{k,\ell }p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}} \delta _{k\ell }=1\\ r_{i.}^{\mathbf {z}\mathbf {w}\varvec{\delta }}= & {} \sum _j r_{ij}^{\mathbf {z}\mathbf {w}\varvec{\delta }}= \sum _j p_{i.}p_{.j} \sum _{k,\ell } z_{ik} w_{j\ell }\delta _{k\ell }=p_{i.} \sum _j p_{.j}=p_{i.} \end{aligned}$$

and, symmetrically, $r_{.j}^{\mathbf {z}\mathbf {w}\varvec{\delta }}=p_{.j}.$ $\square$

Appendix G: Proof of Eq. (7)

$$\begin{aligned} \widetilde{W}_{\Phi ^2}(\mathbf {z},\mathbf {w},\varvec{\delta })&=D_{\Phi ^2}(P_{IJ}||R_{KL}^{\mathbf {z}\mathbf {w}\varvec{\delta }}) =\sum _{i,j} \frac{(p_{ij}-r_{ij}^{\mathbf {z}\mathbf {w}\varvec{\delta }})^2}{p_{i.}p_{.j}} \\&=\sum _{i,j} \frac{(p_{ij}-p_{i.}p_{.j}\sum _{k,\ell }z_{ik}w_{j\ell }\delta _{k\ell })^2}{p_{i.}p_{.j}} \\&=\sum _{i,j} p_{i.}p_{.j} \left( \frac{p_{ij}}{p_{i.}p_{.j}}-\sum _{k,\ell }z_{ik}w_{j\ell }\delta _{k\ell }\right) ^2\\&=\sum _{i,j}p_{i.}p_{.j}\left( \frac{(\sum _{k,\ell }z_{ik}w_{j\ell })p_{ij}}{p_{i.}p_{.j}} -\sum _{k,\ell }z_{ik}w_{j\ell }\delta _{k\ell }\right) ^2 \end{aligned}$$

and since $z_{ik}w_{j\ell } \in \{0,1\}$, we obtain $\widetilde{W}_{\Phi ^2}(\mathbf {z},\mathbf {w},\varvec{\delta }) =\sum _{i,j,k,\ell } z_{ik} w_{j\ell } \,p_{i.}p_{.j} \left( \frac{p_{ij}}{p_{i.}p_{.j}}-\delta _{k\ell }\right) ^2$

Appendix H: Proof of Eq. (8)

$$\begin{aligned} \widetilde{W}_{\mathcal {I}}(\mathbf {z},\mathbf {w},\varvec{\delta })&=KL(P_{IJ}||R_{IJ}^{\mathbf {z}\mathbf {w}\varvec{\delta }}) =\sum _{i,j} p_{ij} \log \frac{p_{ij}}{r_{ij}^{\mathbf {z}\mathbf {w}\varvec{\delta }}}\\&=\sum _{i,j} p_{ij} \log \frac{p_{ij}}{p_{i.}p_{.j} \sum _{k,\ell } z_{ik} w_{j\ell }\delta _{k\ell }}\\&=\sum _{i,j} p_{ij} \log \frac{p_{ij}}{p_{i.}p_{.j}} - \sum _{i,j} p_{ij} \log \sum _{k,\ell } z_{ik} w_{j\ell }\delta _{k\ell }\\&=\sum _{i,j} p_{ij} \log \frac{p_{ij}}{p_{i.}p_{.j}} -\sum _{i,k} z_{ik} \sum _\ell p_{i\ell }^{\mathbf {w}} \log \delta _{k\ell }\\&=\sum _{i,j} p_{ij} \log \frac{p_{ij}}{p_{i.}p_{.j}} -\sum _{k,\ell } p_{k\ell }^{\mathbf {z}\mathbf {w}} \log \delta _{k\ell }. \end{aligned}$$

Appendix I: Proof of Proposition 4

Using Eq. (7), the problem can be formulated ${{\mathrm{argmin}}}_{\delta _{k\ell }} F(\delta _{k\ell }) \quad \forall k,\ell$ where

$$\begin{aligned} F(\delta _{k\ell })= & {} \sum _{i,j}z_{ik}w_{j\ell }p_{i.}p_{.j}\left( \frac{p_{ij}}{p_{i.}p_{.j}}-\delta _{k\ell }\right) ^2 =p_{k.}^{\mathbf {z}} p_{.\ell }^{\mathbf {w}} \delta _{k\ell }^2 -2 p_{k\ell }^{\mathbf {z}\mathbf {w}} \delta _{k\ell }\\&+\sum _{i,j}z_{ik}w_{j\ell } \frac{p_{ij}^2}{p_{i.}p_{.j}} \end{aligned}$$

is a quadratic function of $\delta _{k\ell }$ which has its maximum value for $\delta _{k\ell }=-\frac{- 2 p_{k\ell }^{\mathbf {z}\mathbf {w}}}{2 (p_{k.}^{\mathbf {z}} p_{.\ell }^{\mathbf {w}})} =\frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}.$

Appendix J: Proof of Proposition 5

The problem can be formulated as ${{\mathrm{argmax}}}_{\delta _{k\ell }} \sum _{k,\ell } p_{k\ell }^{\mathbf {z}\mathbf {w}} \log \delta _{k\ell } \ \text {with} \ \sum _{k,\ell }p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}} \delta _{k\ell }=1.$ We can solve this problem by using the method of Lagrange multiplier, we introduce a new variable $\lambda$ and study the Lagrange function defined by $F(k,\ell )=\sum _{k,\ell } p_{k\ell }^{\mathbf {z}\mathbf {w}}\log \delta _{k\ell } -\lambda (\sum _{k,\ell }p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}\delta _{k\ell }-1).$ Then we have $\forall k,\ell$ $\frac{\partial F(k,\ell )}{\partial \delta _{k\ell }}=0 \Rightarrow \frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{\delta _{k\ell }} - \lambda p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}=0 \Rightarrow \delta _{k\ell }=\frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{\lambda p_{k.}^{\mathbf {z}} p_{.\ell }^{\mathbf {w}}}.$ The constraint $\sum _{k,\ell } p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}} \delta _{k\ell }=1$ yields $\lambda =\sum _{k,\ell }p_{k\ell }^{\mathbf {z}\mathbf {w}}=1$ and therefore we obtain $\delta _{k\ell }=\frac{p_{k\ell }^{\mathbf {z}\mathbf {w}}}{p_{k.}^{\mathbf {z}}p_{.\ell }^{\mathbf {w}}}.$

Appendix K: VEM algorithm

Knowing that $y_{k\ell }^{\widetilde{\mathbf {z}}\widetilde{\mathbf {w}}}=\sum _{i,j} \widetilde{z}_{ik} \widetilde{w}_{j\ell } \, x_{ij}$, $y_{k.}^{\widetilde{\mathbf {z}}}=\sum _{\ell } y_{k\ell }^{\widetilde{\mathbf {z}}\widetilde{\mathbf {w}}}$ and $y_{.\ell }^{\widetilde{\mathbf {w}}}=\sum _k y_{k\ell }^{\widetilde{\mathbf {z}}\widetilde{\mathbf {w}}}$, VEM alternates the following steps

• Update of $\varvec{\theta }$: $\pi _k=\frac{\widetilde{z}_{.k}}{n}$, $\rho _\ell =\frac{\widetilde{w}_{.\ell }}{d}$ and $\gamma _{k\ell }=\frac{y_{k\ell }^{\widetilde{\mathbf {z}}\widetilde{\mathbf {w}}}}{y_{k.}^{\widetilde{\mathbf {z}}}y_{.\ell }^{\widetilde{\mathbf {w}}}}$

• Update of $\widetilde{\mathbf {z}}$: $\tilde{z}_{ik} \propto \pi _k \exp (\sum _\ell y_{.\ell }^{\widetilde{\mathbf {w}}} \log \gamma _{k\ell })$

• Update of $\widetilde{\mathbf {w}}$: $\tilde{w}_{j\ell } \propto \rho _\ell \exp (\sum _k y_{k.}^{\widetilde{\mathbf {z}}} \log \gamma _{k\ell })$

Proof

• Update of $\varvec{\theta }$:

  • Equations (16) and (17) lead to ${{\mathrm{argmax}}}_{\varvec{\pi }} \sum _k \widetilde{z}_k \log \pi _k$ and then to $\pi _k=\frac{\widetilde{z}_{.k}}{n}$ $\forall k.$

  • Similarly, we have $\rho _\ell =\frac{\widetilde{w}_{.\ell }}{n}\qquad \forall \ell .$

  • For $\gamma _{k\ell }$, we have $\forall k,\ell$,

  $$\begin{aligned} \hat{\gamma }_{k\ell }= & {} {{\mathrm{argmax}}}_{\gamma _{k\ell }} \sum _{i,j} \widetilde{z}_{ik} \widetilde{w}_{j\ell }(x_{ij} \log \gamma _{k\ell }-x_{i.} x_{.j} \gamma _{k\ell })\\= & {} {{\mathrm{argmax}}}_{\gamma _{k\ell }} \frac{y_{k\ell }^{\widetilde{\mathbf {z}}\widetilde{\mathbf {w}}}}{\gamma _{k\ell }}-y_{k.}^{\widetilde{\mathbf {z}}}y_{.\ell }^{\widetilde{\mathbf {w}}} =\frac{y_{k\ell }^{\widetilde{\mathbf {z}}\widetilde{\mathbf {w}}}}{y_{k.}^{\widetilde{\mathbf {z}}}y_{.\ell }^{\widetilde{\mathbf {w}}}}. \end{aligned}$$

• Update of $\widetilde{\mathbf {z}}$: Eqs. (16) and (17) lead, for all i, to

  $$\begin{aligned} {{\mathrm{argmax}}}_{\widetilde{z}_{ik}} \sum _k \left( \widetilde{z}_{ik} \log \pi _k + \widetilde{z}_{ik}\sum _{j,\ell } \widetilde{w}_{j\ell } (x_{ij} \log \gamma _{k\ell } -x_{i.}x_{.j} \gamma _{k\ell }) - \widetilde{z}_{ik} \log \widetilde{z}_{ik}\right) \end{aligned}$$

  $\forall i,k$ under the constraint $\sum _k \widetilde{z}_{ik}=1$. This takes the following form

  $$\begin{aligned} {{\mathrm{argmax}}}_{\widetilde{z}_{ik}}\sum _k\left( \widetilde{z}_{ik} \log \pi _k + \widetilde{z}_{ik} \sum _\ell (y_{.\ell }^{\mathbf {w}} \log \gamma _{k\ell } - x_{i.} y_{.\ell }^{\mathbf {w}} \gamma _{k\ell }) - \widetilde{z}_{ik} \log \widetilde{z}_{ik}\right) \end{aligned}$$

  and since $\gamma _{k\ell }=\frac{y_{k\ell }^{\widetilde{\mathbf {z}}'\widetilde{\mathbf {w}}}}{y_{k.}^{\widetilde{\mathbf {z}}'}y_{.\ell }^{\widetilde{\mathbf {w}}}}$ where $\widetilde{\mathbf {z}}'$ means the vector of membership values computed above, we obtain

  $$\begin{aligned} \sum _\ell x_{i.} y_{.\ell }^{\widetilde{\mathbf {w}}} \gamma _{k\ell }= \sum _\ell x_{i.} y_{.\ell }^{\widetilde{\mathbf {w}}} \frac{y_{k\ell }^{\widetilde{\mathbf {z}}'\widetilde{\mathbf {w}}}}{y_{k.}^{\widetilde{\mathbf {z}}'}y_{.\ell }^{\widetilde{\mathbf {w}}}}=x_{i.} \frac{\sum _\ell y_{k\ell }^{\widetilde{\mathbf {z}}'\widetilde{\mathbf {w}}}}{y_{k.}^{\widetilde{\mathbf {z}}'}}=x_{i.} \end{aligned}$$

  which does not depend on k and then

  $$\begin{aligned} {{\mathrm{argmax}}}_{\widetilde{z}_{ik}}\sum _k\left( \widetilde{z}_{ik} \log \pi _k + \widetilde{z}_{ik} \sum _\ell y_{.\ell }^{\widetilde{\mathbf {w}}} \log \gamma _{k\ell } - \widetilde{z}_{ik} \log \widetilde{z}_{ik}\right) \end{aligned}$$

  under the constraint $\sum _k \widetilde{z}_{ik}=1$. Using the lagrange multiplier, we obtain

  $$\begin{aligned} \tilde{z}_{ik} =\frac{\pi _k \exp (\sum _\ell y_{.\ell }^{\widetilde{\mathbf {w}}} \log \gamma _{k\ell })}{\sum _{k'} \pi _{k'} exp (\sum _\ell y_{.\ell }^{\widetilde{\mathbf {w}}} \log \gamma _{k'\ell })}\propto \pi _k \exp \left( \sum _\ell y_{.\ell }^{\widetilde{\mathbf {w}}} \log \gamma _{k\ell }\right) \end{aligned}$$

• The update of $\widetilde{\mathbf {w}}$ is proven in a similar way.

$\square$