Constructing fading histograms from data streams

Abstract

The ability to collect data is changing drastically. Nowadays, data are gathered in the form of transient and finite data streams. Memory restrictions preclude keeping all received data in memory. When dealing with massive data streams, it is mandatory to create compact representations of data, also known as synopses structures or summaries. Reducing memory occupancy is of utmost importance when handling a huge amount of data. This paper addresses the problem of constructing histograms from data streams under error constraints. When constructing online histograms from data streams there are two main characteristics to embrace: the updating facility and the error of the histogram. Moreover, in dynamic environments, besides the need of compact summaries to capture the most important properties of data, it is also essential to forget old data. Therefore, this paper presents sliding histograms and fading histograms, an abrupt and a smooth strategies to forget outdated data.

Appendix: Fading histograms

This appendix presents some computations on the error of approximating the fading sliding histogram with the fading histogram. Considering the definition of histogram frequencies (1), the frequencies of a sliding histogram (with $k$ buckets) constructed over a sliding window of length $w$ and computed at observation $i$ with an exponential fading factor $\alpha$ ($0 \ll \alpha < 1$) can be defined as follows:

$$\begin{aligned} F_{\alpha , w, j} (i) \!=\! \frac{\sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l)}{\sum \nolimits _{j=1}^k \sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l)}, \quad \forall j \!=\! 1,\dots , k, \end{aligned}$$

(13)

To approximate a fading sliding histogram by a fading histogram, the older data than that within the most recent window $W = \{x_l: l=i-w+1,\ldots ,i \}$ must be taken into consideration. Therefore, for each bucket $j = 1,\ldots , k$, the proportion of weight given to old observations (with respect to $W$) in the computation of the fading histogram is defined as the bucket ballast weight:

$$\begin{aligned} B_{\alpha , w, j}(i) = \frac{\sum \nolimits _{l=w}^{i-1} \alpha ^{l}}{N_\alpha (i)},\quad \forall j = 1,\dots , k, \end{aligned}$$

(14)

where $N_\alpha (i)$ is the fading increment defined as $N_{\alpha }(i) = \sum \nolimits _{j=1}^k \sum \nolimits _{l=1}^i \alpha ^{i-l} C_j(l)$.

As with the old observations, for each bucket $j = 1,\dots , k$, the proportion of weight given to observations within the most recent window $W$ is defined by

$$\begin{aligned} B_{\alpha , w, j}'(i) = 1 - B_{\alpha , w, j}(i) = \frac{\sum \nolimits _{l=0}^{w-1} \alpha ^{l}}{N_\alpha (i)}, \quad \forall j = 1,\dots , k.\nonumber \\ \end{aligned}$$

(15)

Hence, the error of approximating the fading sliding histogram with the fading histogram, both with $k$ buckets, can be defined as

$$\begin{aligned} \Delta _{\alpha ,w}(i)&= \sum \nolimits _{j=1}^k \Delta _{\alpha ,w, j}(i) \nonumber \\&= \sum \nolimits _{j=1}^k \left\| F_{\alpha , w, j} (i) \!-\! F_{\alpha , j} (i) \right\| . \end{aligned}$$

(16)

Theorem 1

Let $\varepsilon <1$ be an admissible ballast weight for the fading histogram. Then, $\Delta _{\alpha ,w}(i) \le 2\varepsilon$.

Proof

From the respective histogram frequency definitions comes that the approximation error in each bucket is

$$\begin{aligned}&\Delta _{\alpha ,w, j}(i) = \left\| \ \frac{\sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l)}{\sum \nolimits _{j=1}^k \sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l)} - \frac{\sum \nolimits _{l=1}^i \alpha ^{i-l} C_j(l)}{\sum \nolimits _{j=1}^k \sum \nolimits _{l=1}^i \alpha ^{i-l} C_j(l)} \right\| ,\\&\quad \quad \qquad \qquad \qquad \forall j = 1,\ldots , k \end{aligned}$$

Splitting each of these errors considering the frequencies inside and outside the most recent window of size $w$

$$\begin{aligned} \Delta _{\alpha ,w, j}(i) = \left\| \Delta _{\alpha ,w, j}(i)^\mathrm{in} - \Delta _{\alpha ,w, j}(i)^\mathrm{out}\right\| , \end{aligned}$$

where

$$\begin{aligned} \Delta _{\alpha ,w, j}(i)^\mathrm{in} = \frac{\sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l)}{\sum \nolimits _{j=1}^k \sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l)} - \frac{\sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l)}{\sum \nolimits _{j=1}^k \sum \nolimits _{l=1}^i \alpha ^{i-l} C_j(l)}, \end{aligned}$$

and

$$\begin{aligned} \Delta _{\alpha ,w, j}(i)^\mathrm{out}(i) = \frac{\sum \nolimits _{l=1}^{i-w} \alpha ^{i-l} C_j(l)}{\sum \nolimits _{j=1}^k \sum \nolimits _{l=1}^i \alpha ^{i-l} C_j(l)}. \end{aligned}$$

Looking for an upper bound on the error, the worst case scenario is that these two sources of error do not cancel out, rather adding up their effect

$$\begin{aligned} \Delta _{\alpha ,w, j}(i) \le \left\| \Delta _{\alpha ,w, j}(i)^\mathrm{in} \right\| + \left\| \Delta _{\alpha ,w, j}(i)^\mathrm{out} \right\| . \end{aligned}$$

Hence

$$\begin{aligned}&\Delta _{\alpha ,w, j}(i)\\&\quad \le \left\| \frac{\left( \sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l) \right) \left( \sum \nolimits _{j=1}^k \sum \nolimits _{l=1}^{i-w} \alpha ^{i-l} C_j(l) \right) - \left( \sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l) \right) \left( \sum \nolimits _{j=1}^k \sum \nolimits _{l=i-w+1}^{i} \alpha ^{i-l} C_j(l) \right) }{ \left( \sum \nolimits _{j=1}^k \sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l) \right) \left( \sum \nolimits _{j=1}^k \sum \nolimits _{l=1}^i \alpha ^{i-l} C_j(l) \right) } \right\| \\&\qquad + \left\| \frac{\sum \nolimits _{l=1}^{i-w} \alpha ^{i-l} C_j(l)}{\sum \nolimits _{j=1}^k \sum \nolimits _{l=1}^i \alpha ^{i-l} C_j(l)} \right\| \Leftrightarrow \\&\quad \Leftrightarrow \Delta _{\alpha ,w, j}(i)\le \left\| \frac{\left( \sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l) \right) \left( \sum \nolimits _{j=1}^k \sum \nolimits _{l=1}^{i-w} \alpha ^{i-l} C_j(l) \right) }{ \left( \sum \nolimits _{j=1}^k \sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l) \right) \left( \sum \nolimits _{j=1}^k \sum \nolimits _{l=1}^i \alpha ^{i-l} C_j(l) \right) } \right\| + \left\| \frac{\sum \nolimits _{l=1}^{i-w} \alpha ^{i-l} C_j(l)}{\sum \nolimits _{j=1}^k \sum \nolimits _{l=1}^i \alpha ^{i-l} C_j(l)} \right\| \Leftrightarrow \\&\quad \Leftrightarrow \Delta _{\alpha ,w, j}(i)\le \left\| \frac{\left( \sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l) \right) \left( \sum \nolimits _{j=1}^k \sum \nolimits _{l=1}^{i-w} \alpha ^{i-l} C_j(l) \right) }{ \left( \sum \nolimits _{j=1}^k \sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} C_j(l) \right) N_{\alpha }(i)} \right\| + \left\| \frac{\sum \nolimits _{l=1}^{i-w} \alpha ^{i-l} C_j(l)}{N_{\alpha }(i)} \right\| \end{aligned}$$

The upper bound on the error is given by considering all $C_j(l) = 1$:

$$\begin{aligned}&\Delta _{\alpha ,w, j}(i)\\&\quad \le \left\| \frac{\left( \sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} \right) \left( k \sum \nolimits _{l=1}^{i-w} \alpha ^{i-l} \right) }{ \left( k \sum \nolimits _{l=i-w+1}^i \alpha ^{i-l} \right) N_{\alpha }(i)} \right\| \\&\quad + \left\| \frac{\sum \nolimits _{l=1}^{i-w} \alpha ^{i-l} }{N_{\alpha }(i)} \right\| = 2 \left\| \frac{\sum \nolimits _{l=1}^{i-w} \alpha ^{i-l} }{ N_{\alpha }(i)} \right\| \end{aligned}$$

Then, from bucket ballast weight definition comes that

$$\begin{aligned} \Delta _{\alpha ,w, j}(i)&\le 2 \left\| B_{\alpha , w, j}(i)\right\| \end{aligned}$$

Considering in each bucket $j = 1,\dots , k$ an admissible ballast weight, at most, of $\varepsilon /k$ comes that

$$\begin{aligned} \Delta _{\alpha ,w}(i) = \sum \limits _{j=1}^k \Delta _{\alpha ,w, j}(i) \le \sum \limits _{j=1}^k 2\varepsilon /k = 2\varepsilon . \end{aligned}$$