SQL Streaming Process in Query Engine Net
The massively growing data volume and the pressing need for low latency are pushing the traditional store-first-query-later data warehousing technologies beyond their limits. Many enterprise applications are now based on continuous analytics of data streams. While integrating stream processing with query processing takes advantage of SQL’s expressive power and DBMS’s data management capability, it raises serious challenges in dealing with complex dataflow, applying queries to unbounded stream data, and providing highly scalable, dynamically configurable, elastic infrastructure.
To solve these problems, we model the general graph-structured, continuous dataflow analytics as a SQL Streaming Process with multiple connected and stationed continuous queries; then we extend the query engine to support cyclebased query execution for processing unbounded stream data chunk-wise with sound semantics; and finally, we develop the Query Engine Net (QE-Net) over the Distributed Caching Platforms (DCP) as a dynamically configurable elastic infrastructure for parallel and distributed execution of SQL Streaming Processes.
We extended the PostgreSQL engines for building the QE-Net infrastructure. Our experience shows its merit in leveraging SQL and query processing to analyze real-time, graph-structured and unbounded streams. Integrating it with a commercial and proprietary MPP based database cluster is being investigated.
KeywordsStream Data Query Processing Query Execution Continuous Query Query Engine
Unable to display preview. Download preview PDF.
- 1.Nori, A.: Distributed Caching Platforms. In: VLDB (2010)Google Scholar
- 2.Arasu, A., Babu, S., Widom, J.: The CQL Continuous Query Language: Semantic Foundations and Query Execution. VLDB Journal 2(15) (June 2006)Google Scholar
- 3.Chen, Q., Hsu, M., Zeller, H.: Experience in Continuous analytics as a Service (CaaaS). In: EDBT (2011)Google Scholar
- 4.Chen, Q., Hsu, M.: Continuous MapReduce for In-DB Stream Analytics. In: Proc. CoopIS (2010)Google Scholar
- 5.Dean, J.: Experiences with MapReduce, an abstraction for large-scale computation. In: Int. Conf. on Parallel Architecture and Compilation Techniques. ACM (2006)Google Scholar
- 6.Franklin, M.J., et al.: Continuous Analytics: Rethinking Query Processing in a Network Effect World. In: CIDR (2009)Google Scholar
- 7.Gedik, B., Andrade, H., Wu, K.-L., Yu, P.S., Doo, M.C.: SPADE: The System S Declarative Stream Processing Engine. In: ACM SIGMOD (2008)Google Scholar
- 8.Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: Distributed data-parallel programs from sequential building blocks. In: EuroSys 2007 (March 2007)Google Scholar
- 9.Jain, N., et al.: Design, Implementation, and Evaluation of the Linear Road Benchmark on the Stream Processing Core. In: SIGMOD (2006)Google Scholar
- 10.Neumeyer, L., Bruce, R., Anish, N., Anand, K.: S4: Distributed Stream Computing Platform. In: KDCloud 2010, Sydney, Australia (December 2010)Google Scholar
- 11.Memcached (2010), http://www.memcached.org/