The dynamic stochastic topic block model for dynamic networks with textual edges

Abstract

The present paper develops a probabilistic model to cluster the nodes of a dynamic graph, accounting for the content of textual edges as well as their frequency. Vertices are clustered in groups which are homogeneous both in terms of interaction frequency and discussed topics. The dynamic graph is considered stationary on a latent time interval if the proportions of topics discussed between each pair of node groups do not change in time during that interval. A classification variational expectation–maximization algorithm is adopted to perform inference. A model selection criterion is also derived to select the number of node groups, time clusters and topics. Experiments on simulated data are carried out to assess the proposed methodology. We finally illustrate an application to the Enron dataset.

Proofs

1.1 Proof of Proposition 1

Proof

The VEM update step for the distribution $R(Z^{iju}_n)$, for all i, j, u and n, is given by

$$\begin{aligned}&\log R(Z^{iju}_n)\nonumber \\&\quad =\mathbf {E}_{R(Z^{\smallsetminus i,j,u,n}, \theta )}[\log p(W|Z,D,\beta ) \nonumber \\&\quad \quad + \log p(Z|D,Y,X,\theta )] + C \nonumber \\&\quad =\sum _{k=1}^K Z^{iju}_{nk}\sum _{v=1}^V W^{iju}_{nv} \log \beta _{kv}\nonumber \\&\quad \quad + \sum _{q,r}^Q\sum _{l=1}^{L} Y_{iq}Y_{jr} X_{ul}\sum _{k=1}^KZ^{iju}_{nk}\mathbf {E}_{\theta _{qrl}}[\log \theta _{qrl}] + C\nonumber \\&\quad =\sum _{k=1}^K Z^{iju}_{nk}\left( \sum _{v=1}^V W^{iju}_{nv} \log \beta _{kv} \right. \nonumber \\&\quad \quad \left. + \sum _{q,r}^Q \sum _{l=1}^L Y_{iq}Y_{jr} X_{ul} \left( \psi (\gamma _{qrlk}) - \psi \left( \sum _{k=1}^K \gamma _{qrlk}\right) \right) \right) + C, \end{aligned}$$

(18)

where the expectation is taken with respect to the distribution $R(Z, \theta )$ conditional on $Z^{iju}_n$ to be fixed, C includes all the terms not depending on $Z^{iju}_n$ and $\psi (\cdot )$ denotes the digamma function. The functional form of a multinomial distribution can be recognised

$$\begin{aligned} R(Z^{iju}_n) = \mathcal {M}\left( Z^{iju}_n; 1, \phi ^{iju}_n = \{\phi ^{iju}_{n1}, \dots , \phi ^{iju}_{nK}\} \right) , \end{aligned}$$

where

$$\begin{aligned}&\phi ^{iju}_{nk} \propto \left( \prod _{v=1}^V \beta _{kv}^{W^{iju}_{nv}}\right) \prod _{q,r}^Q \prod _{l=1}^L \exp \left( \psi (\gamma _{qrlk}) \right. \\&\quad \left. - \psi \left( \sum _{k=1}^K \gamma _{qrlk}\right) \right) ^{Y_{iq} Y_{jr} X_{ul}}. \end{aligned}$$

$\square$

1.2 Proof of Proposition 2

Proof

The VEM update step for distribution the distribution $R(\theta )$ is given by

$$\begin{aligned} \log R(\theta )&= \mathbf {E}_{R(Z)}[\log p(Z|D, Y, X, \theta )] + C \nonumber \\&= \sum _{j\ne i}^{M} \sum _{u=1}^U\sum _{n=1}^{N_{iju}} \sum _{q,r}^Q \sum _{l=1}^L Y_{iq} Y_{jr} X_{ul}\nonumber \\&\quad \sum _{k=1}^K\mathbf {E}_{R(Z)}[Z^{iju}_{nk}] \log \theta _{qrlk} \nonumber \\&\quad + \sum _{q,r}^Q \sum _{l=1}^L \sum _{k=1}^K(\alpha _k - 1)\log \theta _{qrlk} + C \nonumber \\&=\sum _{q,r}^Q \sum _{l=1}^L \sum _{k=1}^K\left( \alpha _k \right. \nonumber \\&\quad \left. + \sum _{j\ne i}^M\sum _{u=1}^U \sum _{n=1}^{N_{iju}} Y_{iq} Y_{jr} X_{ul}\phi ^{iju}_{nk} - 1 \right) \log \theta _{qrlk} \nonumber \\&\quad + C, \end{aligned}$$

(19)

where C contains the terms not depending on $\theta$. The functional form of a Dirichlet distribution can be recognized

$$\begin{aligned} R(\theta ) = \prod _{q,r}^Q \prod _{l=1}^L \text {Dir}(\theta _{qrl}; \gamma _{qrl} = \{\gamma _{qrl1}, \dots , \gamma _{qrlK}\}), \end{aligned}$$

with

$$\begin{aligned} \gamma _{qrlk} = \alpha _k + \sum _{j\ne i}^M \sum _{u=1}^U\sum _{n=1}^{N_{iju}} Y_{iq} Y_{jr} X_{ul}\phi ^{iju}_{nk}. \end{aligned}$$

$\square$

1.3 Derivation of the lower bound

The functional $\tilde{\mathcal {L}}(R(\cdot ); D,Y, X,W, \beta )$ in (12) given in Propositions 2 and 3, is given by

$$\begin{aligned}&\tilde{\mathcal {L}}(R(\cdot );D,Y,X,W,\beta )\\&\quad =\sum _{j\ne i}^M \sum _{u=1}^U \sum _{n=1}^{N_{iju}}\sum _{k=1}^K\sum _{v=1}^V W^{iju}_{nv}\phi ^{iju}_{nk}\log (\beta _{kv})\\&\qquad +\sum _{j\ne i}^M \sum _{u=1}^U \sum _{n=1}^{N_{iju}}\sum _{k=1}^K\phi ^{iju}_{nk}\\&\qquad \times \left( \sum _{q,r}^Q \sum _{l}^LY_{iq}Y_{jr}X_{ul}\left( \psi (\gamma _{qrlk}) - \psi (\sum _{k=1}^K \gamma _{qrlk})\right) \right) \\&\qquad +\sum _{q,r}^Q \sum _l^L\left( \log \varGamma \left( \sum _{k=1}^K \alpha _k\right) - \sum _{k=1}^K\log \varGamma (\alpha _k) \right. \\&\qquad \left. + \sum _{k=1}^K(\alpha _k - 1)\left( \psi (\gamma _{qrlk}) - \psi \left( \sum _{k=1}^K \gamma _{qrlk}\right) \right) \right) \\&\qquad -\sum _{j\ne i}^M\sum _{u=1}^U\sum _{n=1}^{N_{iju}}\sum _{k=1}^K\phi ^{iju}_{nk}\log (\phi ^{iju}_{nk}) \\&\qquad -\sum _{q,r}^{Q} \sum _{l}^L \left( \log \varGamma \left( \sum _{k=1}^K \gamma _{qrlk}\right) - \sum _{k=1}^K\log \varGamma (\gamma _{qrlk}) \right. \\&\qquad \left. + \sum _{k=1}^K(\gamma _{qrlk} - 1)\left( \psi (\gamma _{qrlk}) - \psi \left( \sum _{k=1}^K \gamma _{qrlk}\right) \right) \right) . \end{aligned}$$

1.4 Proof of Proposition 3

Proof

The maximization of the functional in (12) with respect to $\beta$ is considered at first. By isolating the terms depending on $\beta$ and introducing K Lagrange multipliers accounting for the constraints $\sum _{v=1}^V \beta _{kv}=1$, $\forall k$, we obtain the following objective function

$$\begin{aligned} f(\beta ):= & {} \sum _{j \ne i}^M\sum _{u=1}^U\sum _{n=1}^{N_{iju}}\sum _{k=1}^K\sum _{v=1}^V \phi ^{iju}_{nk}\log \beta _{kv} \\&+ \sum _{k=1}^K \lambda _k\left( \sum _{k=1}^K\beta _{kv}-1\right) , \end{aligned}$$

whose gradient can be easily computed and set equal to zero to find the $\beta _{kv}$ in (13).

In a similar fashion, when optimizing with respect to $\rho$, the following objective function is introduced

$$\begin{aligned} f(\rho ):= \sum _{i=1}^M \sum _{q=1}^Q Y_{iq} \log \rho _{q} + \lambda \left( \sum _{q=1}^Q\rho _q-1\right) , \end{aligned}$$

(20)

and its first derivative with respect to $\rho _q$ is set equal to zero to obtain the stationary point in equation (15). The optimization with respect to $\delta$ is analogous and (14) is a consequence of the likelihood in (4). $\square$

1.5 Proof of Proposition 4

Proof

A factorizing prior distribution being attached to the model parameters, $(\varLambda , \rho , \delta , \beta )$, the integrated complete-data log-likelihood $\log p(W,D,Y,X|Q,L,K)$ can easily be written as

$$\begin{aligned} \begin{aligned}&\log p(W,D,Y,X|Q,L,K)\\&\quad = \log \int _{\beta } p(W|D,Y,X,\beta ,Q,L,K)p(\beta |K)d\beta \\&\qquad + \log \int _{\varLambda } p(D|Y, X, \varLambda ,Q,L)p(\varLambda |Q,L)d\varLambda \\&\qquad + \log \int _{\rho } p(Y|\rho ,Q)p(\rho |Q)d\rho \\&\qquad + \log \int _{\delta } p(X| \delta ,L)p(\delta |L)d\delta , \end{aligned} \end{aligned}$$

(21)

where the dependency on (Q, L, K) is made explicit and the pair $(Z, \theta )$ is integrated out as in Sect. 3.1. Following the derivation of the ICL criterion (Biernacki et al. 2000) we rely on a BIC-like approximation of the second term on the right hand side of the above equation to obtain

$$\begin{aligned} \begin{aligned}&\log \int _{\varLambda } p(D|Y, X, \varLambda ,Q,L)p(\varLambda |Q,L)d\varLambda \\&\quad \approx \max _{\varLambda } \log p(D|Y,X,\varLambda , Q,L) \\&\qquad -\, \frac{Q^2L}{2}\log (MU(M-1)). \end{aligned} \end{aligned}$$

Similarly the last two terms can be approximated as

$$\begin{aligned}&\log \int _{\rho } p(Y|\rho ,Q)p(\rho |Q)d\rho \\&\quad \approx \max _{\rho } \log p(Y|\rho , Q) - \frac{Q-1}{2}\log (M) \end{aligned}$$

and

$$\begin{aligned}&\log \int _{\delta } p(X|\delta ,L)p(\delta |L)d\delta \\&\quad \approx \max _{\delta } \log p(X|\delta , L) - \frac{L-1}{2}\log (U). \end{aligned}$$

Notice that the last three approximations lead to the ICL criterion for the dSBM model

$$\begin{aligned} \begin{aligned} ICL_{dSBM}&:=\max _{\varLambda } \log p(D|Y,X,\varLambda , Q,L)\\&\quad - \frac{Q^2L}{2}\log (MU(M-1)) \\&\quad + \max _{\rho } \log p(Y|\rho , Q) - \frac{Q-1}{2}\log (M) \\&\quad + \max _{\delta } \log p(X|\delta , L) - \frac{L-1}{2}\log (U). \end{aligned} \end{aligned}$$

The exact version of this criterion is maximized relying on a greedy search approach in Corneli et al. (2016b).

Consider now the first term on the right hand side of (21). Recalling that the documents W can be organized as $W=(\tilde{W}_{qrl})_{q,r,l}$ such that all words in $\tilde{W}_{qrl}$ follow the same mixture distribution over topics, we adopt the BIC-like approximation obtained in Bouveyron et al. (2016) corrected by the number of documents in dSTBM

$$\begin{aligned} \begin{aligned}&\log \int _{\beta } p(W|D,Y,X,\beta ,Q,L,K)p(\beta |K)d\beta \\&\quad \approx \max _{\beta }\log p(W|D,Y,X,\beta ,Q,L,K)\\&\quad - \frac{K(V-1)}{2} \log (Q^2L). \end{aligned} \end{aligned}$$

Since the first term on the right hand side of the above approximation is not tractable, it is replaced by its variational approximation $\mathcal {\tilde{L}}(R(\cdot );D,Y,X,W,\beta )$, defined in (12), and the proposition is proven. $\square$