Ingress: an automated incremental graph processing system

Abstract

The graph data keep growing over time in real life. The ever-growing amount of dynamic graph data demands efficient techniques of incremental graph computation. However, incremental graph algorithms are challenging to develop. Existing approaches usually require users to manually design nontrivial incremental operators, or choose different memoization strategies for certain specific types of computation, limiting the usability and generality. In light of these challenges, we propose $\textsf{Ingress}$, an automated system for incremental graph proc essing. $\textsf{Ingress}$ is able to deduce the incremental counterpart of a batch vertex-centric algorithm, without the need of redesigned logic or data structures from users. Underlying $\textsf{Ingress}$ is an automated incrementalization framework equipped with four different memoization policies, to support all kinds of vertex-centric computations with optimized memory utilization. We identify sufficient conditions for the applicability of these policies. $\textsf{Ingress}$ chooses the best-fit policy for a given algorithm automatically by verifying these conditions. In addition to the ease-of-use and generalization, $\textsf{Ingress}$ outperforms state-of-the-art incremental graph systems by $12.14\times$ on average (up to $49.23\times$) in efficiency.

Appendices

Appendix

A1. Proof of Lemma 1

Proof

Following the computation process of an $\textsf{MF}$-applicable $\mathcal {A}$ as shown in Eq. (4), we have that

In the above equations, lines (a) and (b) are due to the property (C2) as defined in Sect. 4.1; line (c) follows from a simple induction; and line (d) is true because of Eq. (4) and the property (C3). $\square$

A2. Proof of Lemma 2

Proof

The incremental $\textsf{MF}$-applicable $\mathcal {A}$ on $G\oplus \varDelta G$ starts from the previous computation status (see Algorithm 1). By Lemma 1, we have that $\hat{\mathbb {X}}^0=\mathbb {X}^k={\mathcal {U}}\left( \mathbb {X}^0\cup \bigcup _{\ell =0}^{k-1} \mathcal {G}^\ell (\mathbb {M}^0)\right)$. It remains to verify $\hat{\mathbb {M}}^{0}$. Let $\mathbb {M}_v = \bigcup _{\ell =0}^{k-1} m_v^{\ell }$ be the messages received by v in the computation over G. Observe the following.

(1) Algorithm 1 generates two sets of messages, $M_v^{+}$ and $M_v^{-}$, in case that the edges related to v are evolved, i.e., $\mathcal {G}(\mathbb {M}_{v}) \ne \hat{\mathcal {G}}(\mathbb {M}_{v})$. By Algorithm 1, we have that ${\mathcal {U}}\left( \hat{\mathcal {G}}(\mathbb {M}_{v}) \ominus \mathcal {G}(\mathbb {M}_{v})\right) = {\mathcal {U}}(M_v^{+}\cup M_v^{-})$.

(2) If the edges related to v are not evolved, we have that $\mathcal {G}(\mathbb {M}_v) = \hat{\mathcal {G}}(\mathbb {M}_v)$. By (C1) and (C2), it holds that:

$$\begin{aligned} \begin{aligned} {\mathcal {U}}\left( \hat{\mathcal {G}}(\mathbb {M}_v) \ominus \mathcal {G}(\mathbb {M}_v)\right)&= {\mathcal {U}}\left( \hat{\mathcal {G}}(\mathbb {M}_v) \cup {\mathcal {U}}^{-}\circ {\mathcal {U}}\circ \mathcal {G}(\mathbb {M}_v)\right) \\&= {\mathcal {U}}\left( \hat{\mathcal {G}}(\mathbb {M}_v) \setminus \mathcal {G}(\mathbb {M}_v)\right) = {\mathcal {U}}({\textbf {0}}). \end{aligned} \end{aligned}$$

That is, the messages in $\hat{\mathcal {G}}(\mathbb {M}_v)$ and $\mathcal {G}(\mathbb {M}_v)$ can be canceled out w.r.t. function ${\mathcal {U}}$. Thus there is no need to generate these messages in incremental computation over $G\oplus \varDelta G$.

According to Algorithm 1, the initial messages $\hat{\mathbb {M}}^{0}$ for the incremental computation consist of ${\mathcal {U}}(M_v^{+}\cup M_v^{-})$ for vertices with evolved edges. By the above analysis, vertices without evolved edges contribute empty even if the corresponding messages are also generated. As a consequence, we can express $\hat{\mathbb {M}}^{0}$ as follows:

$$\begin{aligned} \begin{aligned} \hat{\mathbb {M}}^{0}&= {\mathcal {U}}\left( \bigcup _{v\in V}\hat{\mathcal {G}}(\mathbb {M}_v) \ominus \bigcup _{v\in V}\mathcal {G}(\mathbb {M}_v)\right) \\&= {\mathcal {U}}\left( \hat{\mathcal {G}}\left( \bigcup _{v\in V}\bigcup _{\ell =0}^{k-1} m_v^{\ell }\right) \ominus \mathcal {G}\left( \bigcup _{v\in V}\bigcup _{\ell =0}^{k-1} m_v^{\ell }\right) \right) \\&= {\mathcal {U}}\left( \hat{\mathcal {G}}\left( \bigcup _{\ell =0}^{k-1} \mathcal {G}^\ell (\mathbb {M}^0)\right) \ominus \mathcal {G}\left( \bigcup _{\ell =0}^{k-1} \mathcal {G}^\ell (\mathbb {M}^0)\right) \right) .\end{aligned} \end{aligned}$$

$\square$

A3. Proof of Lemma 3

Proof

Since the batch algorithm $\mathcal {A}$ is $\textsf{MP}$-applicable, i.e., condition (C4) holds, each state $x_v^k$ of an unreset vertex v in $\mathbb {X}_N^k$ is determined by an effective message generated when running $\mathcal {A}$ on G, denoted as $m_v^c$. Hence $x_v^k = m_v^c$. Observe that the right-hand side of Eq. (8) refers to the result of executing $\mathcal {A}$ over the reserved graph $G_\mathcal {R}$ (see Lemma 1). Then if for each unreset vertex v, the effective message $m_v^c$ is the same as the corresponding effective message generated for v when invoking $\mathcal {A}$ over $G_\mathcal {R}$, then Eq. (8) holds. To show this precondition, it suffices to prove that $m_v^c$ is transmitted from another unreset vertex $v'$ and $(v', v)$ is not a deleted edge in $\varDelta G$. This is because the computation of $\mathcal {A}$ strictly follows the propagation of messages and it can be formally verified by induction on the rounds of the computation.

We next prove that each effective message $m_v^c$ is sent from another unreset vertex via an edge that is not deleted by contradiction. Assume by contradiction that (i) a reset vertex $v'$ sends $m_v^c$ to the unreset v or (ii) $m_v^c$ is transmitted along a deleted edge $(v',v)$. (i) If $v'$ has been reset, then v must also be a reset vertex as Algorithm 2 propagates $\bot$ along the paths formed by all effective messages, including $m_v^c$. (ii) If $(v',v)$ is a deleted edge, then the state of v should also be reset since the deleted $(v',v)$ initiates a cancelation message in Algorithm 2. Hence either case leads to a contradiction. The correctness of Eq. 8 follows. $\square$

A4. Proof of Lemma 4

Proof

When messages $\bot$ are propagated in Algorithm 2, the initial $\hat{\mathbb {X}}^0$ includes the current states for both reset and unreset vertices. By Lemma 3 and the definition of changed graph $G_\mathcal {C}$, we have that $\hat{\mathbb {X}}^0 = {\mathcal {U}}\big (\mathbb {X}_\mathcal {R}^0 \cup \bigcup ^{k-1}_{\ell =0}\mathcal {G}_\mathcal {R}^\ell (\mathbb {M}_\mathcal {R}^0) \big ) \cup \mathbb {X}_\mathcal {C}^0 = {\mathcal {U}}\left( \mathbb {X}_\mathcal {R}^0 \cup \bigcup ^{k}_{\ell =0}\mathbb {M}_\mathcal {R}^\ell \right) \cup \mathbb {X}_\mathcal {C}^0$.

We next analyze the initial compensation messages $\hat{\mathbb {M}}_0$ initiated by Algorithm 2. By its operations, we can see that each initial compensation message can be sent from either an unreset vertex in $V_\mathcal {R}$ or a reset vertex in $V_\mathcal {C}$. In light of this, we denote by $\hat{\mathbb {M}}^r$ and $\hat{\mathbb {M}}^c$ the collections of initial compensation messages sent from unreset and reset vertices, respectively.

Suppose that an unreset vertex v sends an initial compensation message to $v'$. Due to the logic of Algorithm 2 (line 4), $v'$ must be a reset vertex or $(v, v')$ is an evolved edge. Hence edge $(v, v')$ cannot appear in the reserved graph $G_\mathcal {R}$ under both cases. Recall that $M_v^+$ denotes the initial compensation messages sent from v. Then for each unreset vertex $v \in V_R$, $M_v^+ = {\mathcal {U}}\circ \hat{\mathcal {G}}(x_v^k) {\setminus } {\mathcal {U}}\circ \mathcal {G}_\mathcal {R}(x_v^k)$. This expression also indicates that no actual message is created on unreset v if none of v’s adjacent edges is evolved or covers reset vertices. Based on this observation and Lemma 3, we have

$$\begin{aligned} \begin{aligned} \hat{\mathbb {M}}^r&= \bigcup _{v\in V_\mathcal {R}} \big ( {\mathcal {U}}\circ \hat{\mathcal {G}}(x_v^k) \setminus {\mathcal {U}}\circ \mathcal {G}_\mathcal {R}(x_v^k) \big ) \\&= \bigcup _{v\in V_\mathcal {R}} \bigcup _{\ell = 0}^k \big ( {\mathcal {U}}\circ \hat{\mathcal {G}}(m_v^\ell ) \setminus {\mathcal {U}}\circ \mathcal {G}_\mathcal {R}(m_v^\ell ) \big ) \\&= \bigcup _{\ell = 0}^k \big ( {\mathcal {U}}\circ \hat{\mathcal {G}}(\mathbb {M}_\mathcal {R}^\ell ) \setminus {\mathcal {U}}\circ \mathcal {G}_\mathcal {R}(\mathbb {M}_\mathcal {R}^\ell ) \big ). \end{aligned} \end{aligned}$$

Putting this and the fact that $\hat{\mathbb {M}}^c {=} \mathbb {M}_\mathcal {C}^0$ together, i.e., the initial messages constructed at reset vertices are the same as their counterparts created in the changed graph, we have $\hat{\mathbb {M}}^0 = \mathbb {M}^0_\mathcal {C}\cup \bigcup ^{k}_{\ell =0}\left( {\mathcal {U}}\circ \hat{\mathcal {G}}( \mathbb {M}^\ell _\mathcal {R}){\setminus } {\mathcal {U}}\circ \mathcal {G}_\mathcal {R}(\mathbb {M}^\ell _\mathcal {R})\right)$. $\square$