SQL Streaming Process in Query Engine Net

  • Qiming Chen
  • Meichun Hsu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7044)

Abstract

The massively growing data volume and the pressing need for low latency are pushing the traditional store-first-query-later data warehousing technologies beyond their limits. Many enterprise applications are now based on continuous analytics of data streams. While integrating stream processing with query processing takes advantage of SQL’s expressive power and DBMS’s data management capability, it raises serious challenges in dealing with complex dataflow, applying queries to unbounded stream data, and providing highly scalable, dynamically configurable, elastic infrastructure.

To solve these problems, we model the general graph-structured, continuous dataflow analytics as a SQL Streaming Process with multiple connected and stationed continuous queries; then we extend the query engine to support cyclebased query execution for processing unbounded stream data chunk-wise with sound semantics; and finally, we develop the Query Engine Net (QE-Net) over the Distributed Caching Platforms (DCP) as a dynamically configurable elastic infrastructure for parallel and distributed execution of SQL Streaming Processes.

We extended the PostgreSQL engines for building the QE-Net infrastructure. Our experience shows its merit in leveraging SQL and query processing to analyze real-time, graph-structured and unbounded streams. Integrating it with a commercial and proprietary MPP based database cluster is being investigated.

Keywords

Stream Data Query Processing Query Execution Continuous Query Query Engine 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Nori, A.: Distributed Caching Platforms. In: VLDB (2010)Google Scholar
  2. 2.
    Arasu, A., Babu, S., Widom, J.: The CQL Continuous Query Language: Semantic Foundations and Query Execution. VLDB Journal 2(15) (June 2006)Google Scholar
  3. 3.
    Chen, Q., Hsu, M., Zeller, H.: Experience in Continuous analytics as a Service (CaaaS). In: EDBT (2011)Google Scholar
  4. 4.
    Chen, Q., Hsu, M.: Continuous MapReduce for In-DB Stream Analytics. In: Proc. CoopIS (2010)Google Scholar
  5. 5.
    Dean, J.: Experiences with MapReduce, an abstraction for large-scale computation. In: Int. Conf. on Parallel Architecture and Compilation Techniques. ACM (2006)Google Scholar
  6. 6.
    Franklin, M.J., et al.: Continuous Analytics: Rethinking Query Processing in a Network Effect World. In: CIDR (2009)Google Scholar
  7. 7.
    Gedik, B., Andrade, H., Wu, K.-L., Yu, P.S., Doo, M.C.: SPADE: The System S Declarative Stream Processing Engine. In: ACM SIGMOD (2008)Google Scholar
  8. 8.
    Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: Distributed data-parallel programs from sequential building blocks. In: EuroSys 2007 (March 2007)Google Scholar
  9. 9.
    Jain, N., et al.: Design, Implementation, and Evaluation of the Linear Road Benchmark on the Stream Processing Core. In: SIGMOD (2006)Google Scholar
  10. 10.
    Neumeyer, L., Bruce, R., Anish, N., Anand, K.: S4: Distributed Stream Computing Platform. In: KDCloud 2010, Sydney, Australia (December 2010)Google Scholar
  11. 11.
    Memcached (2010), http://www.memcached.org/

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Qiming Chen
    • 1
  • Meichun Hsu
    • 1
  1. 1.HP LabsHewlett Packard Co.Palo AltoUSA

Personalised recommendations