Efficient algorithms for incremental all pairs shortest paths, closeness and betweenness in social network analysis

Abstract

One of the biggest challenges in today’s social network analysis (SNA) is handling dynamic data. Real-world social networks evolve with time, forcing their corresponding graph representations to dynamically update by addition or deletion of edges/nodes. Consequently, a researcher is often interested in fast recomputation of important SNA metrics pertaining to a network. Recomputations of SNA metrics are expensive. Use of dynamic algorithms has been found as a solution to this problem. For calculating closeness and betweenness centrality metrics, computations of all pairs shortest paths (APSP) are needed. Thus, to compute these SNA metrics dynamically, APSP are needed to be computed dynamically. This paper presents fast incremental updating algorithms along with the time complexity results for APSP, closeness centrality and betweenness centrality, considering two distinct cases: edge addition and node addition. The following time complexity results are presented: (1) The incremental APSP algorithm runs in $O(n^2)$ time ($\Omega (n^2)$ is the theoretical lower bound of the APSP problem), (2) The incremental closeness algorithm that runs in $O(n^2)$ time, and (3) The incremental betweenness algorithm runs in $O(nm + n^2\, {\mathrm{log}} \, n)$ time. Here, $m$ is the number of edges and $n$ is the number of nodes in the network. Though the time complexity of the presented incremental betweenness algorithm is no better than its static counterpart (Brandes, J Math Sociol 25(2):163–177, 2001), the experimental comparisons demonstrate that the former performs better than the latter. All the presented methods are applicable to weighted, directed or undirected graphs with non-negative real-valued edge weights. An alternate version of the incremental APSP algorithm is presented in the Appendix section. It is demonstrated that this version works better on large graphs.

Appendix

1.1 The modified incremental APSP algorithm

This section presents a modified version of the designed incremental APSP algorithm. In the incremental algorithm for node addition, the shortest paths passing through the new node $z$ are compared with all the $n^2$ shortest paths in the original network, in the last step of the algorithm. In this modified version, we try to avoid some of these comparisons with the aim to improve the empirical performance of the algorithm.

A new shortest path distance between a pair of nodes $(i,j)$ and passing through the newly added $z$ has two parts; $\min\nolimits_{k^{\text{ in }} \in T_1}\{d^{k^{\text{ in }}}_{iz}\}$ and $\min\nolimits_{k^{\text{ out }} \in T_2}\{d^{k^{\text{ out }}}_{zj}\}$. Now, if any of these two parts is greater than the original shortest path distance, $d_{ij}$, i.e., if $\min\nolimits_{k^{\text{ in }} \in T_1}\{d^{k^{\text{ in }}}_{iz}\} > d_{ij}$ or $\min\nolimits_{k^{\text{ out }} \in T_2}\{d^{k^{\text{ out }}}_{zj}\} > d_{ij}$, then the new shortest path distance of a path passing through $z$ will not be shorter than the original shortest path distance $d_{ij}$ and in this way further computations can be avoided. This idea provides a foundation to the concept of the modified version of the incremental algorithm. For this algorithm, we sort all rows and columns of the original distance matrix, which takes $O(n^2{\mathrm{log}}n)$ time. However, this is only one time investment, made at first addition of a new node, for original distance matrix. For each consecutive addition of a node, new insertion of shortest path distance into the distance matrix takes only $O({\mathrm{log}}n)$ time and hence the method turns out to be useful, as it saves computational efforts in the long run.

• 1. Sort shortest path distances starting from each node: Let $d_{i\cdot }$ be the set of the shortest paths that start from node $i$ and end at any other node. $d_{i\cdot }$ represents a single row in the distance matrix of shortest paths. Sort this row in the descending order according to path lengths. Remove all elements in this row starting from the element of the highest path length, that satisfy the condition $d_{i\cdot } >\min\nolimits_{k^{\text{ in }} \in T_1}\{d^{k^{\text{ in }}}_{iz}\}$ and store in a new matrix. Follow this procedure for all $i$ nodes.

• 2. Sort shortest path distances ending at each node: Similarly, let $d_{\cdot j}$ be the set of shortest paths ending at node $j$. $d_{\cdot j}$ represents a single column in the distance matrix of shortest paths. Sort this column in the descending order according to the path lengths from a copy of the distance matrix. Remove all elements in this column starting from the element of the highest path length, that satisfy the condition $d_{\cdot j} > \min\nolimits_{k^{\text{ out }} \in T_2}\{d^{k^{\text{ out }}}_{zj}\}$ and store in the same new matrix. Repeat this procedure for all columns.

• 3. Update shortest path distances: Now, in this new matrix, the old shortest path distances that are selected by row as well as column operations will try to overlap in the same element of the matrix. Select only those paths for the final comparison of checking if $\min\nolimits_{k^{\text{ in }} \in T_1}\{d^{k^{\text{ in }}}_{iz}\} + \min\nolimits_{k^{\text{ out }} \in T_2}\{d^{k^{\text{ out }}}_{zj}\} < d_{ij}$. All non-overlapping elements do not qualify for this comparison. In this way, empirical performance can be further improved.

In the worst case when all $d_{ij}$ are greater than both $\min\nolimits_{k^{\text{ in }} \in T_1}\{d^{k^{\text{ in }}}_{iz}\}$ and $\min\nolimits_{k^{\text{ out }} \in T_2}\{d^{k^{\text{ out }}}_{zj}\}$ each time, this algorithm also performs $n^2$ comparisons and thus does not prove to be an improvement over the original version of our algorithm. However, usually the number comparisons are almost always smaller than $n^2$ and hence this algorithm can prove to be empirically effective (see also Sect. 4). Algorithm 4 below shows the pseudocode of the algorithm.

1.2 Snijder’s stochastic actor-based model for network dynamics

This section briefly describes Snijder’s stochastic actor-based model for network dynamics, which has been used to generate a set of graphs to test the two betweenness algorithms.

A set of longitudinal observed networks is an input to such a model. They are assumed to be outcomes of a Markov process. These models are “actor-based” models because it is assumed that actors in a network control their outgoing ties. Given an opportunity, an actor can form a tie, delete one of its existing ties, or do nothing. An opportunity to make such a change is probabilistically given to an actor. This opportunity may depend on the network position of the actors (e.g., centrality) and on actor covariates (e.g., age and sex). The probabilities of an actor’s choices depend on the so-called objective function. The objective function expresses how likely it is for the actor to change her/his network in a particular way. On average, each actor “tries to” move into a direction of higher values of her/his objective function, subject to constraints of the current network structure, the changes made by other actors in the network, and subject to random influence. The objective function is assumed to be a linear combination of effects.

$$\begin{aligned} f_{i}(\beta ,x) = \sum _{k}{\beta _{k}s_{ki}(x)}. \end{aligned}$$

(13)

Here, $f_{i}(\beta , x)$ is the value of the objective function for actor $i$ depending on the state $x$ of the network. The function $s_{ki}(x)$ are effects, chosen based on subject-matter knowledge and correspond to the tendencies of the actors. The weights $\beta _{k}$ are the statistical parameters. The effects represent aspects of the network as viewed from the point of view of actor $i$. If $\beta _{k}$ equals 0, the corresponding effect plays no role in the network dynamics; if $\beta _{k}$ is positive, then there will be a higher probability of moving into directions where the corresponding effect is higher, and the converse is true of $\beta _{k}$ is negative.

In the model that is used to generated graphs for testing the two betweenness algorithms, we have considered outdegree, reciprocity and transitive triplets as the effects in the model. Hence, the objective function of actor $i$ in our model would be,

$$\begin{aligned} f_{i}(\beta ,x) = \beta _{1}s_{1i}(x) + \beta _{2}s_{2i}(x) + \beta _{3}s_{3i}(x). \end{aligned}$$

(14)

Here, $s_{1i}(x) = \sum _{j}{x_{ij}}$, the outdegree effect; $s_{2i}(x) = \sum _{j}{x_{ij}x_{ji}}$, the reciprocity effect; and $s_{3i}(x) = \sum _{j,h}{x_{ih}x_{ij}x_{jh}}$, the transitive triplets effect.

The parameters of such a model are estimated by a maximum likelihood method. Once the parameters are estimated, we can simulate different networks having same effects and same strength. For the purpose of generating graphs, an R script mentioned in Ripley et al. (2011) is used. It is assumed that the model parameters are already estimated. 60 different graphs are generated with their node sizes ranging from 100 to 1000 and outdegree weights$(\beta _1)$ ranging from −2.8 to −1.8, keeping all other parameter values fixed (rate = 2, reciprocity parameter$(\beta _2) = 2$, transitive triplets parameter$(\beta _3) = 0.3$). These values are decided as per guidance given in the script. Note that, the outdegree weights have negative values. This parameter generally has negative value in large networks to hinder link formations between nodes that are remotely connected to each other. A node with a total degree (outgoing plus incoming) of approximately 10 is removed from each generated graph with its attached edges; it is then treated as an increment.