Open Execution Engines of Stream Analysis Operations

  • Qiming Chen
  • Meichun Hsu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7450)


In this paper we describe our massively parallel and elastic stream analysis platform; it is capable of supporting the graph-structured dataflow process with each logical operator executed by multiple physical instances running in parallel over distributed server nodes. We propose the canonical dataflow operator framework to provide automated and systematic support for executing, parallelizing and granulizing the continuous operations.

We focus on the following issues: first, how to categorize the meta-properties of stream operators such as the I/O, blocking, data grouping characteristics, for providing unified and automated system support; next, how to elastically and correctly parallelize a stateful operator that is history-sensitive, relying on the prior state and data processing results; and further, how to analyze unbounded stream granularly to ensure sound semantics (e.g. aggregation). These issues are not properly abstracted and systematically addressed in the current generation of stream processing systems, but left to user programs which can result in fragile code, disappointing performance and incorrect results.

We tackle these issues by introducing the open-executors. An open executor supports the streaming operations with specific characteristics and running pattern, but is open for the application logic to be plugged-in. We illustrate the power of this approach by showing the system support in parallelizing and granulizing dataflow operations safely and correctly. The proposed canonical operation framework can be generalized to allow us to standardize various operational patterns of stream operators, and have these patterns supported systematically and automatically. We have built this platform; our experience reveals its value in real-time, continuous, elastic data-parallel and topological stream analysis process.


Application Logic Input Stream Dataflow Graph Stream Operator Input Tuples 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Arasu, A., Babu, S., Widom, J.: The CQL Continuous Query Language: Semantic Foundations and Query Execution. VLDB Journal 15(2) (June 2006)Google Scholar
  2. 2.
    Abadi, D.J., et al.: The Design of the Borealis Stream Processing Engine. In: CIDR (2005)Google Scholar
  3. 3.
    Bryant, R.E.: Data-Intensive Supercomputing: The case for DISC. CMU-CS-07-128 (2007)Google Scholar
  4. 4.
    Chen, Q., Hsu, M., Zeller, H.: Experience in Continuous analytics as a Service (CaaaS). In: EDBT 2011 (2011)Google Scholar
  5. 5.
    Chen, Q., Hsu, M.: Experience in Extending Query Engine for Continuous Analytics. In: Bach Pedersen, T., Mohania, M.K., Tjoa, A.M. (eds.) DAWAK 2010. LNCS, vol. 6263, pp. 190–202. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  6. 6.
    Chen, Q., Hsu, M.: Continuous MapReduce for In-DB Stream Analytics. In: Meersman, R., Dillon, T., Herrero, P. (eds.) OTM 2010. LNCS, vol. 6428, pp. 16–34. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  7. 7.
    Dean, J.: Experiences with MapReduce, an abstraction for large-scale computation. In: Int. Conf. on Parallel Architecture and Compilation Techniques. ACM (2006)Google Scholar
  8. 8.
    Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: Distributed data-parallel programs from sequential building blocks. In: EuroSys 2007 (March 2007)Google Scholar
  9. 9.
    Franklin, M.J., et al.: Continuous Analytics: Rethinking Query Processing in a Network¬Effect World. In: CIDR 2009 (2009)Google Scholar
  10. 10.
    Olston, C., Reed, B., Srivastava, U., Kumar, R., Tomkins, A.: Pig Latin: A Not-So-Foreign Language for Data Processing. In: ACM SIGMOD 2008 (2008)Google Scholar
  11. 11.
    ØMQ Lightweight Messaging Kernel,
  12. 12.
    Apache ZooKeeper,
  13. 13.
    Kryo - Fast, efficient Java serialization,
  14. 14.
    Twitter’s Open Source Storm Finally Hits,

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Qiming Chen
    • 1
  • Meichun Hsu
    • 1
  1. 1.HP LabsHewlett Packard Co.Palo AltoUSA

Personalised recommendations