Advertisement

Efficient Dynamic Operator Placement in a Locally Distributed Continuous Query System

  • Yongluan Zhou
  • Beng Chin Ooi
  • Kian-Lee Tan
  • Ji Wu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4275)

Abstract

In a distributed processing environment, the static placement of query operators may result in unsatisfactory system performance due to unpredictable factors such as changes of servers’ load, data arrival rates, etc. The problem is exacerbated for continuous (and long running) monitoring queries over data streams as any suboptimal placement will affect the system for a very long time. In this paper, we formalize and analyze the operator placement problem in the context of a locally distributed continuous query system. We also propose a solution, that is asynchronous and local, to dynamically manage the load across the system nodes. Essentially, during runtime, we migrate query operators/fragments from overloaded nodes to lightly loaded ones to achieve better performance. Heuristics are also proposed to maintain good data flow locality. Results of a performance study shows the effectiveness of our technique.

Keywords

Communication Cost Processing Node Query Operator Query Plan Continuous Query 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Arasu, A., et al.: Stream: The stanford stream data manager. IEEE Data Eng. Bull. 26(1), 19–26 (2003)MathSciNetGoogle Scholar
  2. 2.
    Ayad, A., Naughton, J.F.: Static optimization of conjunctive queries with sliding windows over infinite streams. In: SIGMOD, pp. 419–430 (2004)Google Scholar
  3. 3.
    Ahmad, Y., Çetintemel, U.: Networked query processing for distributed stream-based applications. In: VLDB, pp. 456–467 (2004)Google Scholar
  4. 4.
    Carney, D., et al.: Monitoring streams - a new class of data management applications. In: VLDB, pp. 215–226 (2002)Google Scholar
  5. 5.
    Carney, D., et al.: Operator scheduling in a data stream manager. In: VLDB, pp. 838–849 (2003)Google Scholar
  6. 6.
    Chandrasekaran, S., et al.: Telegraphcq: Continuous dataflow processing for an uncertain world. In: CIDR (2003)Google Scholar
  7. 7.
    Cherniack, M., et al.: Scalable distributed stream processing. In: CIDR (2003)Google Scholar
  8. 8.
    Pietzuch, P., et al.: Network-aware operator placement for stream-processing systems. In: ICDE, p. 49 (2006)Google Scholar
  9. 9.
    Shah, M.A., et al.: Flux: An adaptive partitioning operator for continuous query systems. In: ICDE, pp. 25–36 (2003)Google Scholar
  10. 10.
    Srivastava, U., et al.: Operator Placement for In-Network Stream Query Processing. In: PODS, pp. 250–258 (2005)Google Scholar
  11. 11.
    Tian, F., DeWitt, D.J.: Tuple routing strategies for distributed eddies. In: VLDB, pp. 333–344 (2003)Google Scholar
  12. 12.
    Viglas, S., Naughton, J.F.: Rate-based query optimization for streaming information sources. In: SIGMOD, pp. 37–48 (2002)Google Scholar
  13. 13.
    Willebeek-LeMair, M., Reeves, A.P.: Strategies for dynamic load balancing on highly parallel computers. IEEE Trans. Parallel Distrib. Syst. 4(9), 979–993 (1993)CrossRefGoogle Scholar
  14. 14.
    Xing, Y., et al.: Dynamic load distribution in the Borealis stream processor. In: ICDE, pp. 791–802 (2005)Google Scholar
  15. 15.
    Zhou, Y., et al.: Adaptive reorganization of coherency-preserving dissemination tree for streaming data. In: ICDE, p. 55 (2006)Google Scholar
  16. 16.
    Zhou, Y., et al.: Dynamic load management for distributed continuous query systems. Unpublished manuscript (2005), http://www.comp.nus.edu.sg/~zhouyong/papers/op.html

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Yongluan Zhou
    • 1
  • Beng Chin Ooi
    • 1
  • Kian-Lee Tan
    • 1
  • Ji Wu
    • 1
  1. 1.National University of Singapore

Personalised recommendations