The VLDB Journal

, Volume 24, Issue 3, pp 345–368 | Cite as

Sketching distributed sliding-window data streams

  • Odysseas PapapetrouEmail author
  • Minos Garofalakis
  • Antonios Deligiannakis
Regular Paper


While traditional data management systems focus on evaluating single, ad hoc queries over static data sets in a centralized setting, several emerging applications require (possibly, continuous) answers to queries on dynamic data that is widely distributed and constantly updated. Furthermore, such query answers often need to discount data that is “stale” and operate solely on a sliding window of recent data arrivals (e.g., data updates occurring over the last 24 h). Such distributed data streaming applications mandate novel algorithmic solutions that are both time and space efficient (to manage high-speed data streams) and also communication efficient (to deal with physical data distribution). In this paper, we consider the problem of complex query answering over distributed, high-dimensional data streams in the sliding-window model. We introduce a novel sketching technique (termed ECM-sketch) that allows effective summarization of streaming data over both time-based and count-based sliding windows with probabilistic accuracy guarantees. Our sketch structure enables point, as well as inner product, queries and can be employed to address a broad range of problems, such as maintaining frequency statistics, finding heavy hitters, and computing quantiles in the sliding-window model. Focusing on distributed environments, we demonstrate how ECM-sketches of individual, local streams can be composed to generate a (low-error) ECM-sketch summary of the order-preserving merging of all streams; furthermore, we show how ECM-sketches can be exploited for continuous monitoring of sliding-window queries over distributed streams. Our extensive experimental study with two real-life data sets validates our theoretical claims and verifies the effectiveness of our techniques. To the best of our knowledge, ours is the first work to address efficient, guaranteed-error complex query answering over distributed data streams in the sliding-window model.


Sketches Continuous queries Distributed monitoring 



This work was supported by the European Commission under ICT-FP7-LEADS-318809 (Large-Scale Elastic Architecture for Data-as-a-Service) and ICT-FP7-FERARI-619491 (Flexible Event pRocessing for big dAta aRchItectures).


  1. 1.
    Alon, N., Matias, Y., Szegedy, M.: The space complexity of approximating the frequency moments. J. Comput. Syst. Sci. 58(1), 137–147 (1999)CrossRefzbMATHMathSciNetGoogle Scholar
  2. 2.
    Arlitt, M., Jin, T.: A workload characterization study of the 1998 world cup web site. Network 14(3), 30–37 (2000)Google Scholar
  3. 3.
    Busch, C., Tirthapura, S.: A deterministic algorithm for summarizing asynchronous streams over a sliding window. In: STACS, pp. 465–476 (2007)Google Scholar
  4. 4.
    Chakrabarti, A., Cormode, G., Mcgregor, A.: A near-optimal algorithm for estimating the entropy of a stream. ACM Trans. Algorithms 6(3), 51:1–51:21 (2010)CrossRefMathSciNetGoogle Scholar
  5. 5.
    Chan, H.L., Lam, T.W., Lee, L.K., Ting, H.F.: Continuous monitoring of distributed data streams over a time-based sliding window. Algorithmica 62(3–4), 1088–1111 (2012)CrossRefzbMATHMathSciNetGoogle Scholar
  6. 6.
    Charikar, M., Chen, K., Farach-Colton, M.: Finding frequent items in data streams. In: ICALP, pp. 693–703 (2002)Google Scholar
  7. 7.
    Cohen, E., Strauss, M.J.: Maintaining time-decaying stream aggregates. J. Algorithms 59(1), 19–36 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Cormode, G., Garofalakis, M.: Approximate continuous querying over distributed streams. ACM Trans. Database Syst. 33(2) (2008)Google Scholar
  9. 9.
    Cormode, G., Garofalakis, M., Muthukrishnan, S., Rastogi, R.: Holistic aggregates in a networked world: Distributed tracking of approximate quantiles. In: SIGMOD, pp. 25–36 (2005)Google Scholar
  10. 10.
    Cormode, G., Muthukrishnan, S.: What’s hot and what’s not: Tracking most frequent items dynamically. In: PODS, pp. 296–306 (2003)Google Scholar
  11. 11.
    Cormode, G., Muthukrishnan, S.: An improved data stream summary: the count-min sketch and its applications. J. Algorithms 55(1), 58–75 (2005)CrossRefzbMATHMathSciNetGoogle Scholar
  12. 12.
    Cormode, G., Muthukrishnan, S., Yi, K., Zhang, Q.: Optimal sampling from distributed streams. In: PODS, pp. 77–86 (2010)Google Scholar
  13. 13.
    Cormode, G., Muthukrishnan, S., Yi, K., Zhang, Q.: Continuous sampling from distributed streams. J. ACM 59(2), 10:1–10:25 (2012)CrossRefMathSciNetGoogle Scholar
  14. 14.
    Cormode, G., Tirthapura, S., Xu, B.: Time-decaying sketches for robust aggregation of sensor data. SIAM J. Comput. 39(4), 1309–1339 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  15. 15.
    Cormode, G., Yi, K.: Tracking distributed aggregates over time-based sliding windows. In: SSDBM, pp. 416–430 (2012)Google Scholar
  16. 16.
    Datar, M., Gionis, A., Indyk, P., Motwani, R.: Maintaining stream statistics over sliding windows. SIAM J. Comput. 31(6), 1794–1813 (2002)CrossRefzbMATHMathSciNetGoogle Scholar
  17. 17.
    Dimitropoulos, X.A., Stoecklin, M.P., Hurley, P., Kind, A.: The eternal sunshine of the sketch data structure. Computer Netw. 52(17), 3248–3257 (2008)CrossRefzbMATHGoogle Scholar
  18. 18.
    Garofalakis, M.N., Keren, D., Samoladas, V.: Sketch-based geometric monitoring of distributed stream queries. PVLDB 6(10), 937–948 (2013)Google Scholar
  19. 19.
    Gibbons, P.B.: Distinct sampling for highly-accurate answers to distinct values queries and event reports. In: VLDB, pp. 541–550 (2001)Google Scholar
  20. 20.
    Gibbons, P.B.: Distinct-values estimation over data streams. Data Stream Management: Processing High-Speed Data Streams. Springer, New York (2007)Google Scholar
  21. 21.
    Gibbons, P.B., Tirthapura, S.: Distributed streams algorithms for sliding windows. In: SPAA, pp. 63–72 (2002)Google Scholar
  22. 22.
    Greenwald, M.B., Khanna, S.: Space-efficient online computation of quantile summaries. In: SIGMOD, pp. 58–66 (2001)Google Scholar
  23. 23.
    Huang, L., Garofalakis, M., Joseph, A., Taft, N.: Communication efficient tracking of distributed cumulative triggers. In: ICDCS (2007)Google Scholar
  24. 24.
    Huang, L., Nguyen, X., Garofalakis, M., Hellerstein, J., Jordan, M., Joseph, A., Taft, N.: Communication-efficient online detection of network-wide anomalies. In: INFOCOM, pp. 134–142 (2007)Google Scholar
  25. 25.
    Hung, R.Y.S., Ting, H.F.: Finding heavy hitters over the sliding window of a weighted data stream. In: LATIN, pp. 699–710 (2008)Google Scholar
  26. 26.
    Jain, A., Hellerstein, J.M., Ratnasamy, S., Wetherall, D.: A wakeup call for internet monitoring systems: the case for distributed triggers. In: SIGCOMM Workshop on Hot Topics in Networks (HotNets) (2004)Google Scholar
  27. 27.
    Keren, D., Sharfman, I., Schuster, A., Livne, A.: Shape sensitive geometric monitoring. TKDE 24(8), 1520–1535 (2012)Google Scholar
  28. 28.
    Mirkovic, J., Prier, G., Reiher, P.L.: Attacking DDoS at the source. In: ICNP, pp. 312–321 (2002)Google Scholar
  29. 29.
    Muthukrishnan, S.: Data Streams: Algorithms and Applications. Found. Trends Theor. Comput. Sci. 1(2), 117–236 (2005)CrossRefMathSciNetGoogle Scholar
  30. 30.
    Olston, C., Jiang, J., Widom, J.: Adaptive filters for continuous queries over distributed data streams. In: SIGMOD, pp. 563–574 (2003)Google Scholar
  31. 31.
    Papapetrou, O., Garofalakis, M.N., Deligiannakis, A.: Sketch-based querying of distributed sliding-window data streams. PVLDB 5(10), 992–1003 (2012)Google Scholar
  32. 32.
    Qiao, L., Agrawal, D., El Abbadi, A.: Supporting sliding window queries for continuous data streams. In: SSDBM, pp. 85–96 (2003)Google Scholar
  33. 33.
    Sharfman, I., Schuster, A., Keren, D.: A geometric approach to monitoring threshold functions over distributed data streams. In: SIGMOD, pp. 301–312 (2006)Google Scholar
  34. 34.
    Tirthapura, S., Xu, B., Busch, C.: Sketching asynchronous streams over a sliding window. In: PODC, pp. 82–91 (2006)Google Scholar
  35. 35.
    Xu, B., Tirthapura, S., Busch, C.: Sketching asynchronous data streams over sliding windows. Distrib. Comput. 20(5), 359–374 (2008)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Odysseas Papapetrou
    • 1
    Email author
  • Minos Garofalakis
    • 1
  • Antonios Deligiannakis
    • 1
  1. 1.Technical University of CreteChaniaGreece

Personalised recommendations