Improved Algorithms for Distributed Entropy Monitoring

Abstract

Modern data management systems often need to deal with massive, dynamic and inherently distributed data sources. We collect the data using a distributed network, and at the same time try to maintain a global view of the data at a central coordinator using a minimal amount of communication. Such applications have been captured by the distributed monitoring model which has attracted a lot of attention in recent years. In this paper we investigate the monitoring of the entropy functions, which are very useful in network monitoring applications such as detecting distributed denial-of-service attacks. Our results improve the previous best results by Arackaparambil et al. in ICLP 1: 95–106 (2009). Our technical contribution also includes implementing the celebrated AMS sampling method (by Alon et al. in J Comput Syst Sci 58(1): 137–147 1999) in the distributed monitoring model, which could be of independent interest.

Appendices

Appendix 1: Proof of Lemma 4

Proof

We can assume that $U_1,\ldots ,U_m$ are distinct, since the event that $U_i = U_j$ for some $i \ne j$ has measure 0. Let $T_i = \min \{U_1,\ldots U_m \}$. Let $\text {OD}_i$ denote the order of $U_1,\ldots ,U_i$. Let $\Sigma _i=\{ \text {Permutation of}~ [i]\}$. It is clear that for any $\sigma \in \Sigma _i$, $\mathbf {Pr}[\text {OD}_i = \sigma \ |\ T_i > t] = \mathbf {Pr}[\text {OD}_i = \sigma ] = \frac{1}{i!}$. Since the order of $U_1,\ldots ,U_i$ does not depend on the minimal value in that sequence, we have

$$\begin{aligned} \mathbf {Pr}[\text {OD}_i = \sigma , T_i> t]&= \mathbf {Pr}[\text {OD}_i = \sigma \ |\ T_i> t] \cdot \mathbf {Pr}[T_i> t] \\&= \mathbf {Pr}[\text {OD}_i = \sigma ] \cdot \mathbf {Pr}[T_i > t]. \end{aligned}$$

Therefore, the events $\{\text {OD}_i=\sigma \}$ and $\{ T_i > t\}$ are independent.

For any given $\sigma \in \Sigma _{i-1}$ and $z \in \{0,1\}$ :

$$\begin{aligned} \mathbf {Pr}[J_i = z\ |\ \text {OD}_{i-1}=\sigma ]= & {} \lim _{t \rightarrow 0} \mathbf {Pr}[J_i = z, T_{i-1}> t\ |\ \text {OD}_{i-1}=\sigma ] \nonumber \\= & {} \lim _{t \rightarrow 0}\frac{\mathbf {Pr}[J_i = z\ |\ T_{i-1}> t, \text {OD}_{i-1}=\sigma ] }{\mathbf {Pr}[\text {OD}_{i-1}=\sigma ]/ \mathbf {Pr}[ T_{i-1} > t, \text {OD}_{i-1}=\sigma ]}\nonumber \\ \end{aligned}$$

(11)

$$\begin{aligned}= & {} \lim _{t \rightarrow 0}\frac{\mathbf {Pr}[J_i = z\ |\ T_{i-1}> t] \cdot \mathbf {Pr}[T_{i-1}> t] }{\mathbf {Pr}[\text {OD}_{i-1}=\sigma ] / \mathbf {Pr}[\text {OD}_{i-1}=\sigma ]} \\= & {} \lim _{t \rightarrow 0} \mathbf {Pr}[J_i = z, T_{i-1} > t]\nonumber \\= & {} \mathbf {Pr}[J_i = z],\nonumber \end{aligned}$$

(12)

where (11) to (12) holds because the events $\{ J_i = z\}$ and $\{ \text {OD}_{i-1}=\sigma \}$ are conditionally independent given $\{ T_{i-1} > t\}$, and the events $\{\text {OD}_i=\sigma \}$ and $\{ T_i > t\}$ are independent.

Therefore, $J_i$ and $\text {OD}_{i-1}$ are independent. Consequently, $J_i$ is independent of $J_1,\ldots , J_{i-1}$, since the latter sequence is fully determined by $\text {OD}_{i-1}$.

$$\begin{aligned} \mathbf {Pr}[J_i = 1]= & {} \mathbf {Pr}[U_i < \min \{U_1, \ldots , U_{i-1} \}] \\= & {} \int _0^1(1-x)^{i-1}dx = \frac{1}{i}. \end{aligned}$$

Thus $\mathbf {E}[J_i] = \mathbf {Pr}[J_i = 1] = \frac{1}{i}$. By the linearity of expectation, $\mathbf {E}[J] = \sum _{i \in [m]} \frac{1}{i} \approx \log m$.

Since $J_1, \ldots , J_m$ are independent, $\mathbf {Pr}[J > 2 \log m] < m^{-1/3}$ follows from a Chernoff Bound. $\square$

Appendix 2: The Assumption of Tracking m Exactly

We explain here why it suffices to assume that m can be maintained at the coordinator exactly without any cost. First, note that we can always use CountEachSimple to maintain a $(1+\epsilon ^2)$-approximation of m using $O(\frac{k}{\epsilon ^2}\log m)$ bits of communication, which will be dominated by the cost of other parts of the algorithm for tracking the Shannon entropy. Second, the additional error introduced for the Shannon entropy by the $\epsilon ^2 m$ additive error of m is negligible: let $g_x(m) = f_m(x) - f_m(x-1) = x\log \frac{m}{x} - (x-1)\log \frac{m}{x-1}$, and recall (in the proof of Lemma 7) that $X > 0.5$ under any $A\in \mathcal {A'}$. It is easy to verify that

$$\begin{aligned} \left| g_x((1\pm \epsilon ^2)m) - g_x(m)\right| = O(\epsilon ^2) \le O(\epsilon ^2) X, \end{aligned}$$

which is negligible compared with $|X - \hat{X}| \le O(\epsilon ) X$ (the error introduced by using $\hat{R}$ to approximate R). Similar arguments also apply to $X'$, and to the analysis of the Tsallis Entropy.

Appendix 3: Pseudocode for CountEachSimple

Algorithm 10, 11 describe how we can maintain a $(1 + \epsilon )$-approximation to the frequency of element e.