Memory-aware framework for fast and scalable second-order random walk over billion-edge natural graphs

Abstract

Second-order random walk is an important technique for graph analysis. Many applications including graph embedding, proximity measure and community detection use it to capture higher-order patterns in the graph, thus improving the model accuracy. However, the memory explosion problem of this technique hinders it from analyzing large graphs. When processing a billion-edge graph like Twitter, existing solutions (e.g., alias method) of the second-order random walk may take up 1796TB memory. Such high memory consumption comes from the memory-unaware strategies for the node sampling during the random walk. In this paper, to clearly compare the efficiency of various node sampling methods, we first design a cost model and propose two new node sampling methods: one follows the acceptance-rejection paradigm to achieve a better balance between memory and time cost, and the other is optimized for fast sampling the skewed probability distributions existed in natural graphs. Second, to achieve the high efficiency of the second-order random walk within arbitrary memory budgets, we propose a novel memory-aware framework on the basis of the cost model. The framework applies a cost-based optimizer to assign desirable node sampling method for each node or edge in the graph within a memory budget meanwhile minimizing the time cost of the random walk. Finally, the framework provides general programming interfaces for users to define new second-order random walk models easily. The empirical studies demonstrate that our memory-aware framework is robust with respect to memory and is able to achieve considerable efficiency by reducing 90% of the memory cost.

Appendices

The proof of proposition 1

Proof

Let G(V, E) be an unweighted graph G(V, E), u and v be the previous node and current node. Considering that the nodes in the same group have the same probabilities, we simply use $p_i, i=1..3$, to denote the e2e probability of a node in the ith group, and $p_i^g, i=1..3$ to denote the probability of the ith group.

According to the definition of groups, we have $|G_I|=1, |G_II|=\theta _{uv}, |G_{III}|=d_v-1-\theta _{uv}$. Therefore, $p_{i}^g=\sum _{1}^{|G_i|}p_i$.

Based on the analysis in Section 5.1, the time cost of rejection node sampler $T_r$ is $C_vcK$, and the time cost of group-based node sampler $T_g$ is $(1+p_1^gc+p_2^gC_{v}^{g_2}c+ p_3^gC_{v}^{g_3}c)K$.

In the context of unweighted graph, we have $C_v = d_v max\{$ $p_1, p_2, p_3\}$, $C_v^{g_2}=1$, $C_v^{g_3}=\frac{d_v}{d_v-\theta _{uv}-1}$.

To derive the condition of $T_r > T_g$, we should have

$$\begin{aligned}&T_r-T_g = C_vcK-(1+p_1^gc+p_2^gC_{v}^{g_2}c+ p_3^{g}C_{v}^{g_3}c)K \\&\quad = (d_v max_{i=1..3}\{p_i\}c - (1+p_1c+p_2\theta _{uv}c+ p_3d_vc))K \\&\quad = (d_v (max_{i=1..3}\{p_i\}-p_3)c - (1+p_1c+p_2\theta _{uv}c))K \\&\quad > 0 \end{aligned}$$

Therefore, the above inequation holds when

$$\begin{aligned} d_v (max_{i=1..3}\{p_i\}-p_3) - (p_1+p_2\theta _{uv}) > \frac{1}{c} \end{aligned}$$

is satisfied. And the proposition is proved. $\square$

Table 10 The distribution of types of node samplers and the concrete node samplers for the nodes with top-10 largest degrees when running NV(0.25, 4) with different greedy algorithms over Youtube. The values in parentheses are the average degree for the nodes having the same node sampler. N: Naive, R: Rejection, A: Alias

LP-domination analysis

In this section, we show that there is no LP domination among the alias, rejection and naive sampling methods. Here, we give the proof with a common setting $d_f=4$, $d_i=4$, $c=1$.

Proof

Following the cost model in Table 2. To prove no LP-domination among the three sampling methods, we need to show that $\frac{T_{r}-T_{n}}{M_{r}-M_{n}}-\frac{T_{a}-T_{r}}{M_{a}-M_{r}}\le 0$ holds.

$$\begin{aligned}&\frac{T_{r}-T_{n}}{M_{r}-M_{n}}-\frac{T_{a}-T_{r}}{M_{a}-M_{r}}\\&\quad =\frac{C_vcK-d_v(c+1)K}{(2b_f+b_i)d_v-M_{n}}-\frac{K-C_vcK}{(b_f+b_i)d^2_v-b_fd_v}\\&\quad =\frac{C_vK-2d_vK}{12d_v-M_{n}}-\frac{K-C_vK}{8d^2_v-4d_v} \\&\quad =K\frac{(C_v-2d_v)(8d_v^2-4d_v)-(1-C_v)(12d_v-M_n)}{(12d_v-M_n)(8d_v^2-4d_v)} \end{aligned}$$

Let $0<M_n=\frac{b_fd_{max}}{|V|}<b_f=4$ and $C_v\le d_v$, it is easy to figure out $(12d_v-M_n)(8d_v^2-4d_v) > 0$ when $d_v \ge 1$. Then, we only need to compute the bound of $(C_v-2d_v)(8d_v^2-4d_v)-(1-C_v)(12d_v-M_n)$ as below:

$$\begin{aligned}&(C_v-2d_v)(8d_v^2-4d_v)-(1-C_v)(12d_v-M_n) \\&\quad \text {//let~} C_v=d_v\\&\quad \le -d_v(8d_v^2-4d_v)-(1-d_v)(12d_v-M_n) \\&\quad =-8d_v^3+16d_v^2-12d_v+M_n-d_vM_n \\&\quad \text {//let~} M_n=4~\text {and~omit}~-d_vM_v\\&\quad < -8d_v^3+16d_v^2-12d_v+4 ~~~~~~~\text {//}(d_v \ge 1)\\&\quad \le 0 . \end{aligned}$$

$\square$

Analysis about the results of Deg-inc on Youtube

In Fig. 8a, b, when memory budget is larger than 7.5 GB, Dec-inc has similar performance to the LP-std and LP-est on Youtube. To clearly analyze the reasons behind this results, we take Fig. 8a as an example and profile the distribution of types of node samplers. And we also give the concrete node samplers for the nodes with top-10 largest degrees. The statistics are reported in Table 10. From the table, we clearly see that when memory budget is 7.5 GB, the distribution of types of node samplers are all most the same between LP-std and Deg-inc. After checking the complete node sampler assignment, we find only two nodes have different node samplers. Recall that Deg-inc processes the nodes with small degree first, due to the sparsity of Youtube, even all the nodes with small degrees are assigned alias method, there are enough memory budget left which allows nodes with large degrees to use rejection method. But when memory budget is 2.5 GB, nodes with large degrees are assigned naive node sampler by Deg-inc, resulting poor efficiency. Unlike Deg-inc, Deg-dec is able to assign alias method or rejection method to nodes with large degrees no matter memory budget is 2.5 GB or 7.5 GB. However, Deg-dec always processes the largest nodes first, thus consuming a lot of memory budget. Finally, Deg-dec leads to many other nodes using naive method, and the average degree of naive method for Deg-dec in Table 10 implicitly demonstrates such node sampler assignment.