Distributed Resource Allocation in Stream Processing Systems

  • Cathy H. Xia
  • James A. Broberg
  • Zhen Liu
  • Li Zhang
Conference paper

DOI: 10.1007/11864219_34

Part of the Lecture Notes in Computer Science book series (LNCS, volume 4167)
Cite this paper as:
Xia C.H., Broberg J.A., Liu Z., Zhang L. (2006) Distributed Resource Allocation in Stream Processing Systems. In: Dolev S. (eds) Distributed Computing. DISC 2006. Lecture Notes in Computer Science, vol 4167. Springer, Berlin, Heidelberg

Abstract

Distributed stream processing architecture has emerged as appealing solution to coping with the analysis of large amount of data from dispersed sources. A fundamental problem in such stream processing systems is how to best utilize the available resources so that the overall system performance is optimized. We consider a distributed stream processing system that consists of a network of cooperating servers, collectively providing processing services for multiple data streams. Each stream is required to complete a series of operations on various servers. We assume all servers have finite computing resources and all communication links have finite available bandwidth. The problem is to find distributed schemes to allocate the limited computing resources as well as the communication bandwidth in the system so as to achieve a maximum concurrent throughput for all output streams. We present a generalized multicommodity flow model for the above problem. We develop a distributed resource allocation algorithm that guarantees the optimality. We also provide detailed analysis on the complexity of the algorithm and demonstrate the performance using numerical experiments.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Cathy H. Xia
    • 1
  • James A. Broberg
    • 2
  • Zhen Liu
    • 1
  • Li Zhang
    • 1
  1. 1.IBM T.J. Watson Research CenterYorktown HeightsUSA
  2. 2.School of Computer Science & Information TechnologyRMIT-UniversityMelbourneAustralia

Personalised recommendations