A block-based generative model for attributed network embedding

Abstract

Attributed network embedding has attracted plenty of interest in recent years. It aims to learn task-independent, low-dimensional, and continuous vectors for nodes preserving both topology and attribute information. Most of the existing methods, such as random-walk based methods and GCNs, mainly focus on the local information, i.e., the attributes of the neighbours. Thus, they have been well studied for assortative networks (i.e., networks with communities) but ignored disassortative networks (i.e., networks with multipartite, hubs, and hybrid structures), which are common in the real world. To model both assortative and disassortative networks, we propose a block-based generative model for attributed network embedding from a probability perspective. Specifically, the nodes are assigned to several blocks wherein the nodes in the same block share the similar linkage patterns. These patterns can define assortative networks containing communities or disassortative networks with the multipartite, hub, or any hybrid structures. To preserve the attribute information, we assume that each node has a hidden embedding related to its assigned block. We use a neural network to characterize the nonlinearity between node embeddings and node attributes. We perform extensive experiments on real-world and synthetic attributed networks. The results show that our proposed method consistently outperforms state-of-the-art embedding methods for both clustering and classification tasks, especially on disassortative networks.

Appendices

Appendix

In this section, we give some details for the derivation of likelihood of complete-data and the update rules of the parameters of our model.

Derivation of likelihood of complete-data

According to the generative process of attributed networks, the joint probability or the likelihood of complete-data is

$$ p(\pmb{X},\pmb{A},\pmb{Z},\pmb{c}|\pmb{\Pi},\pmb{\omega},\pmb{\sigma},\pmb{\mu})=p(\pmb{A}|\pmb{c},\pmb{\Pi})p(\pmb{X}|\pmb{Z})p(\pmb{Z}|\pmb{c},\pmb{\sigma},\pmb{\mu})p(\pmb{C}|\pmb{\omega}) $$

(13)

and each factor is defined as follows.

First, we know that the node assignment follows a multinomial distribution. The probability of node i belongs to block k is ω_k and the assignment for each node is independent. Thus, the probability of assigning all nodes, i.e., obtaining vector c =< c₁, c₂,...c_n >, is

$$ p(\pmb{c}|\pmb{\omega}) = \prod\limits_{i}\omega_{c_{i}}. $$

(14)

Then, the embedding of node i follows a Gaussian distribution with mean $\pmb {\mu }_{c_{i}}$ and standard derivation $\pmb {\sigma }_{c_{i}}$ if we know that node i belongs to block c_i. Thus, we have

$$ p(\pmb{Z}|\pmb{c},\pmb{\sigma},\pmb{\mu}) = \prod\limits_{id}\frac{1}{\sqrt{2\pi}\sigma_{c_{i}d}}e^{-\frac{(z_{id}-\mu_{c_{i}d})^{2}}{2\sigma_{c_{i}d}^{2}}}. $$

(15)

As for the probability of generating node attributes, if X ∈{0, 1}^n×M, it follows a Bernoulli distribution, i.e., the probability of node i having m-th attribute is υ_im. Thus,

$$ p(\pmb{X}|\pmb{Z}) = \prod\limits_{im}\upsilon_{im}^{x_{im}}(1-\upsilon_{im})^{1-x_{im}}, $$

(16)

Similarly, if $\pmb {X} \in \mathbb {R}^{n\times M}$, we can obtain

$$ p(\pmb{X}|\pmb{Z}) = \prod\limits_{im}\frac{1}{\sqrt{2\pi}\lambda_{im}}e^{-\frac{(x_{im}-\upsilon_{im})^{2}}{2\lambda_{im}^{2}}}. $$

(17)

Finally, generating links between each pair of nodes follows a Bernoulli distribution and the generation process of each pair of nodes is independent. The probability of node i connecting to node j is $\pi _{c_{i}c_{j}}$ if the node assignment is known. Thus, the probability of generating links is

$$ p(\pmb{A}|\pmb{c},\pmb{\Pi}) = \prod\limits_{ij}\pi_{c_{i}c_{j}}^{a_{ij}}(1-\pi_{c_{i}c_{j}})^{1-a_{ij}}. $$

(18)

Using a network with binary attributes as an example, we substitute (14)-(16) and (18) to (13), we obtain

$$ \begin{array}{@{}rcl@{}} &&p(\pmb{X},\pmb{A},\pmb{Z},\pmb{c}|\pmb{\Pi},\pmb{\omega},\pmb{\sigma},\pmb{\mu})\\ &&=\prod\limits_{ij}\pi_{c_{i}c_{j}}^{a_{ij}}(1-\pi_{c_{i}c_{j}})^{1-a_{ij}}\times\prod\limits_{im}\upsilon_{im}^{x_{im}}(1-\upsilon_{im})^{(1-x_{im})}\\ &&\quad\times\prod\limits_{id}\frac{1}{\sqrt{2\pi}\sigma_{c_{i}d}}e^{-\frac{(z_{id}-\mu_{c_{i}d})^{2}}{2\sigma_{c_{i}d}^{2}}}\times\prod\limits_{i}\omega_{c_{i}}. \end{array} $$

(19)

Derivation of update rules of the parameters

First, the items related to τ on (7) are:

$$ \begin{array}{@{}rcl@{}} \mathcal{L}_{[\tau_{ik}]} &= &\sum\limits_{j}\sum\limits_{l}\tau_{ik}\tau_{jl}[a_{ij}\log\pi_{kl}+(1-a_{ij})\log(1-\pi_{kl})] \end{array} $$

(20)

$$ \begin{array}{@{}rcl@{}} && -\frac{1}{2}\sum\limits_{d=1}^{D}\tau_{ik}(\log\sigma_{kd}^{2}+\frac{\hat{\sigma}_{id}^{2}}{\sigma_{kd}^{2}} +\frac{(\hat{\mu}_{id}-\mu_{kd})^{2}}{\sigma_{kd}^{2}}) \end{array} $$

(21)

$$ \begin{array}{@{}rcl@{}} &&+ \tau_{ik}\log\frac{\omega_{k}}{\tau_{ik}}. \end{array} $$

(22)

Set $\frac {\partial {\mathscr{L}}_{[\tau _{ik}]}}{\partial \tau _{ik}}=0$, then we can update τ_ik by

$$ \begin{array}{@{}rcl@{}} \tau_{ik} \propto &\exp& (\sum\limits_{j}\sum\limits_{l}\tau_{jl}[a_{ij}\log\pi_{kl}+(1-a_{ij})\log(1-\pi_{kl})] \end{array} $$

(22)

$$ \begin{array}{@{}rcl@{}} && \quad-\frac{1}{2}{\sum\limits_{d}^{D}}(\log\sigma_{kd}^{2}+\frac{\hat{\sigma}_{id}^{2}}{\sigma_{kd}^{2}} +\frac{(\hat{\mu}_{id}-\mu_{kd})^{2}}{\sigma_{kd}^{2}}) + \log\omega_{k}). \end{array} $$

(22)

Then, we optimize π_kl:

$$ \mathcal{L}_{[\pi_{kl}]} = \sum\limits_{ij}\tau_{ik}\tau_{jl}[A_{ij}\log\pi_{kl}+(1-A_{ij})\log(1-\pi_{kl})]. $$

Set $\frac {\partial {\mathscr{L}}_{[\pi _{kl}]}}{\partial \pi _{kl}}=0$, we obtain

$$ \pi_{kl} = \frac{{\sum}_{ij}\tau_{ik}\tau_{jl}A_{ij}}{{\sum}_{ij}\tau_{ik}\tau_{jl}}. $$

(21)

Next, the items related to ω_k are

$$ \mathcal{L}_{[\omega_{k}]}= \sum\limits_{i}\gamma_{ik}\log\omega_{k}. $$

(21)

Since ${\sum }_{k=1}^{K}=1$, we take the derivative of ${\mathscr{L}}_{[\omega _{k}]} + \upbeta ({\sum }_{k}\omega _{k} - 1)$ of ω_k, and make the derivative to zero. Then, we can obtain the update formula for ω_k as follows:

$$ \omega_{k} = \frac{1}{n}\sum\limits_{i}\gamma_{ik}. $$

(21)

In the same way, we can obtain the items related to μ_kd and σ_kd as follows:

$$ \mathcal{L}_{[\mu_{kd}]}=-\frac{1}{2}{\sum\limits_{i}^{n}}\gamma_{ik}\frac{(\hat{\mu}_{id}-\mu_{kd})^{2}}{\sigma_{kd}^{2}}, $$

(21)

and

$$ \mathcal{L}_{[\sigma_{kd}]}=-\frac{1}{2}{\sum\limits_{i}^{n}}\gamma_{ik}(\log\sigma_{kd}^{2}+\frac{\hat{\sigma}_{id}^{2}}{\sigma_{kd}^{2}}+\frac{(\hat{\mu}_{id}-\mu_{kd})^{2}}{\sigma_{kd}^{2}}). $$

(21)

We set $\frac {\partial {\mathscr{L}}_{[\mu _{kd}]}}{\partial \mu _{kd}}=0$ and $\frac {\partial {\mathscr{L}}_{[\sigma _{kd}]}}{\partial \sigma _{kd}}=0$, then we derive the update rules for μ_kd and σ_kd are

$$ \mu_{kd}= \frac{{{\sum}_{i}^{n}}\tau_{ik}\hat{\mu}_{id}}{{{\sum}_{i}^{n}}\tau_{ik}}, $$

(21)

and

$$ \sigma_{kd}= \frac{{{\sum}_{i}^{n}}\tau_{ik}(\hat{\sigma}_{id}+(\hat{\mu}_{id}-\mu_{kd})^{2})}{{{\sum}_{i}^{n}}\tau_{ik}}, $$

(21)

respectively.