Randomized Algorithms for Tracking Distributed Count, Frequencies, and Ranks

Abstract

We show that randomization can lead to significant improvements for a few fundamental problems in distributed tracking. Our basis is the count-tracking problem, where there are k players, each holding a counter $n_i$ that gets incremented over time, and the goal is to track an $\varepsilon$-approximation of their sum $n=\sum _i n_i$ continuously at all times, using minimum communication. While the deterministic communication complexity of the problem is ${\varTheta }(k/\varepsilon \cdot \log N)$, where N is the final value of n when the tracking finishes, we show that with randomization, the communication cost can be reduced to ${\varTheta }(\sqrt{k}/\varepsilon \cdot \log N)$. Our algorithm is simple and uses only O(1) space at each player, while the lower bound holds even assuming each player has infinite computing power. Then, we extend our techniques to two related distributed tracking problems: frequency-tracking and rank-tracking, and obtain similar improvements over previous deterministic algorithms. Both problems are of central importance in large data monitoring and analysis, and have been extensively studied in the literature.

Appendix: Lower Bound for the Sampling Problem

Claim

To solve the sampling problem we need to probe at least ${\varOmega }(k)$ sites.

Proof

Suppose that the coordinator only samples $z = o(k)$ sites. Let X be the number of sites that are sampled with bit 1. Then X is chosen from the hypergeometric distribution with probability density function (pdf) $\mathsf {Pr}[X = x] = {s' \atopwithdelims ()x}{k' - s' \atopwithdelims ()z - x}/{k' \atopwithdelims ()z}$. The expected value of X is $\frac{z}{k'} \cdot s'$, which is $\frac{z}{k'}\left( \frac{k}{2} - y + \sqrt{k}\right)$ or $\frac{z}{k'}\left( \frac{k}{2} - y - \sqrt{k}\right)$, depending on the value of $s'$. Let $p = \left( \frac{k}{2} - y\right) /k' = \frac{1}{2} \pm o(1)$ and $\alpha = \sqrt{k}/k' = 1/\sqrt{k} \pm o(1/\sqrt{k})$. To avoid tedious calculation, we assume that X is picked randomly from one of the two normal distributions $\mathcal {N}_1(\mu _1, \sigma _1^2)$ and $\mathcal {N}_2(\mu _2, \sigma _2^2)$ with equal probability, where $\mu _1 = z(p-\alpha ), \mu _2 = z(p+\alpha ), \sigma _1, \sigma _2 = {\varTheta }(\sqrt{zp(1-p)}) = {\varTheta }(\sqrt{z})$. In Feller [11] it is shown that the normal distribution approximates the hypergeometric distribution very well when z is large and $p \pm \alpha$ are constants in (0, 1).⁵ Now our task is to decide from which of the two distributions X is drawn based on the value of X with success probability at least 0.7.

Fig. 2

Differentiating two distributions

Let $f_1(x; \mu _1, \sigma _1^2)$ and $f_2(x; \mu _2, \sigma _2^2)$ be the pdf of the two normal distributions $\mathcal {N}_1, \mathcal {N}_2$, respectively. It is easy to see that the best deterministic algorithm of differentiating the two distributions based on the value of a sample X will do the following.

• If $X > x_0$, then X is chosen from $\mathcal {N}_2$, otherwise X is chosen from $\mathcal {N}_1$, where $x_0$ is the value such that $f_1(x_0; \mu _1, \sigma _1^2) = f_2(x_0; \mu _2, \sigma _2^2)$ (thus $\mu _1< x_0 < \mu _2$).

Indeed, if $X > x_0$ and the algorithm decides that “X is chosen from $\mathcal {N}_1$”, we can always flip this decision and improve the success probability of the algorithm.

The error comes from two sources: (1) $X > x_0$ but X is actually drawn from $\mathcal {N}_2$; (2) $X \le x_0$ but X is actually drawn from $\mathcal {N}_1$. The total error is

$$\begin{aligned} 1/2 \cdot ({\varPhi }(-\ell _1/\sigma _1) + {\varPhi }(-\ell _2/\sigma _2)), \end{aligned}$$

where $\ell _1 = x_0 - \mu _1$ and $\ell _2 = \mu _2 - x_0$. (Thus $\ell _1 + \ell _2 = \mu _2 - \mu _1 = 2 \alpha z$). ${\varPhi }(\cdot )$ is the cumulative distribution function (cdf) of the normal distribution. See Fig. 2.

Finally note that $\ell _1/\sigma _1 = O(\alpha z / \sqrt{z}) = O(\sqrt{z/k}) = o(1)$ and $\ell _2/\sigma _2 = O(\alpha z / \sqrt{z}) = o(1)$, so ${\varPhi }(-\ell _1/\sigma _1) + {\varPhi }(-\ell _2/\sigma _2) > 0.99$. Therefore, the failure probability is at least 0.49, contradicting our success probability guarantee. Thus we must have $z = {\varOmega }(k)$. $\square$