Probabilistic Graphical Model Based Highly Scalable Directed Community Detection Algorithm

Abstract

Community detection algorithms have essential applications for character statistics in complex network which could contribute to the study of the real network, such as the online social network and the logistics distribution network. But traditional community detection algorithms could not handle the significant characteristic of directionality in real network for only concentrating on undirected network. Based on Information Transfer Probability method of classic Probabilistic Graphical Model (PGM) theory from Turing Award Owner Pearl, we propose an efficient local directed community detection method named Information Transfer Gain (ITG) from basic information transfer triangles which composed the core structure of community. Then, aiming at processing the large scale directed social network with high efficiency, we propose the scalable and distributed algorithm of Distributed Information Transfer Gain (DITG) based on GraphX model in Spark. Finally, with extensive experiment on directed artificial network dataset and real social network dataset, we prove that our algorithm have good precision and efficiency in distributed environment compared with some classical directed detection algorithms such as FastGN, OSLOM and Infomap.

Appendix

A. Triple and Triangle Structure Statistic Method

Tables 1 and 2 give the repeat times of the edge function calculation. This appendix shows the statistic method of the triple and triangle structure.

Figure 10 shows the structure basis with three vertices. For the statistic of triple, two numbers are used to represent the two edges connect vertex and its two neighbours vertex and vertex. The number has three versions which are 0, 1 and 2 exactly. 0 represents the bidirectional edge while 1 and 2 represent the out direction edge and in direction edge respectively. The in and out attribute is observed by the focus vertex. When it comes to triangle structure, three numbers are used. The first two numbers remains the meaning. While the third number represents the edge attribute of the opposite edge of vertex. 0 is for bidirectional edge as well. 1 and 2 represent the directions from vertex to vertex and from vertex to vertex respectively. So the statistic of the triple and triangle structure is obvious for the counting of edge function repeat times. We set up the model of the situation in Fig. 10(a), and we can get the all nine ITG figures respectively.

Fig. 10.

Two triple forms based vertex i in directed graph

It can be found in Fig. 11 the basic nine sub graphs of Fig. 10(a) and the other eighteen sub graphs of Fig. 10(b) can be found in Fig. 12. All the twenty seven sub graphs are classified to two types of weighted triangles which is the computation fundamental of Formula (1).

Fig. 11.

Vertex based directed triples (all sub graphs of Fig. 10(a))

Fig. 12.

Vertex based directed triples (all sub graphs of Fig. 10(b))

Table 11. All ITG computation in sub graphs of Figure A.10.

B. Parameters Details of Formula ( 2 )

$$ \Theta _{1} = \frac{{((r - 1)\delta + 1 + q)(d_{in} - 1)\delta }}{{(r + q)((r - 1)(r - 2)\delta^{3} + (d_{in} - 1)\delta + q(q - 1)\delta \omega + q(q + 1)\omega + d_{out} \omega )}} $$

(2-1)

$$ \Theta _{2} = - \frac{{(r - 1)(r - 2)\delta^{3} }}{{(r - 1)(r - 2)\delta^{3} + q(q - 1)\omega + q(r - 1)\delta \omega }} \cdot \frac{(r + 1)\delta + q}{(r + q)(r - 1 + q)} $$

(2-2)

$$ \Theta _{3} = \frac{{d_{in} (d_{in} - 1)\delta }}{{d_{in} (d_{in} - 1)\delta + d_{out} (d_{out} - 1)\omega + d_{out} d_{in} \omega }} \cdot \frac{{d_{in} + d_{out} }}{{r + d_{out} }} $$

(23)

$$ q = {{(b - d_{in} )} \mathord{\left/ {\vphantom {{(b - d_{in} )} r}} \right. \kern-0pt} r} $$

(2-4)