The VLDB Journal

, Volume 22, Issue 4, pp 421–446 | Cite as

Modeling the execution semantics of stream processing engines with SECRET

  • Nihal Dindar
  • Nesime Tatbul
  • Renée J. Miller
  • Laura M. Haas
  • Irina Botan
Regular Paper


There are many academic and commercial stream processing engines (SPEs) today, each of them with its own execution semantics. This variation may lead to seemingly inexplicable differences in query results. In this paper, we present SECRET, a model of the behavior of SPEs. SECRET is a descriptive model that allows users to analyze the behavior of systems and understand the results of window-based queries (with time- and tuple-based windows) for a broad range of heterogeneous SPEs. The model is the result of extensive analysis and experimentation with several commercial and academic engines. In the paper, we describe the types of heterogeneity found in existing engines and show with experiments on real systems that our model can explain the key differences in windowing behavior.


Data streams Continuous queries Stream processing engines Semantic heterogeneity 


  1. 1.
    Abadi, D. et al.: Aurora: a new model and architecture for data stream management. VLDB J. 12(2) (2003)Google Scholar
  2. 2.
    Arasu, A. et al.: The CQL continuous query language: semantic foundations and query execution. VLDB J. 15(2) (2006)Google Scholar
  3. 3.
    Botan, I. et al.: Extending XQuery with window functions. In: VLDB Conference. Vienna, Austria (Sep. 2007)Google Scholar
  4. 4.
    Botan, I. et al.: Design and Implementation of the MaxStream Federated Stream Processing Architecture. Technical Report TR-632, ETH Zurich Department of Computer Science (June 2009)Google Scholar
  5. 5.
    Botan, I. et al.: SECRET: a model for analysis of the execution semantics of stream processing systems. In: VLDB Conference, Singapore (Sep. 2010)Google Scholar
  6. 6.
    Chandrasekaran, S. et al.: TelegraphCQ: continuous dataflow processing for an uncertain world. In: CIDR Conference (2003)Google Scholar
  7. 7.
  8. 8.
    Dindar, N. et al.: DejaVu: declarative pattern matching over live and archived streams of events (demonstration). In: ACM SIGMOD Conference, Providence, RI (June 2009)Google Scholar
  9. 9.
    Dindar, N. et al.: Time-based window execution equivalence across heterogeneous stream processing engines (2012) (under conference submission)Google Scholar
  10. 10.
    Gedik, B. et al.: SPADE: the system S declarative stream processing engine. In: ACM SIGMOD Conference (2008)Google Scholar
  11. 11.
  12. 12.
    Jain, N. et al.: Towards a streaming SQL standard. In: VLDB Conference, Auckland, New Zealand (Aug. 2008)Google Scholar
  13. 13.
    Kramer, J., Seeger, B.: Semantics and implementation of continuous sliding window queries over data streams. ACM TODS 34(1) (2009)Google Scholar
  14. 14.
    Li, L. et al.: Semantics and evaluation techniques for window aggregates in data streams. In: ACM SIGMOD Conference (2005) Google Scholar
  15. 15.
    Maier, D. et al.: Semantics of data streams and operators. In: ICDT Conference. Edinburgh, Scotland (Jan. 2005)Google Scholar
  16. 16.
  17. 17.
    Motwani, R. et al.: Query Processing, approximation, and resource management in a data stream management system. In: CIDR Conference, Asilomar, CA, USA (Jan. 2003)Google Scholar
  18. 18.
  19. 19.
  20. 20.
    Patroumpas, K., Sellis, T.: Window specification over data streams. In: EDBT Workshops (2006)Google Scholar
  21. 21.
  22. 22.
    Srivastava, U., Widom, J.: Flexible time management in data stream systems. In: ACM PODS Conference (2004)Google Scholar
  23. 23.
  24. 24.
    Stream Query Repository.
  25. 25.
  26. 26.

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Nihal Dindar
    • 1
  • Nesime Tatbul
    • 1
  • Renée J. Miller
    • 2
  • Laura M. Haas
    • 3
  • Irina Botan
    • 1
  1. 1.ETH ZurichZurichSwitzerland
  2. 2.University of TorontoTorontoCanada
  3. 3.IBM Almaden Research CenterSan JoseUSA

Personalised recommendations