Load Management and High Availability in the Borealis Distributed Stream Processing Engine

  • Nesime Tatbul
  • Yanif Ahmad
  • Uğur Çetintemel
  • Jeong-Hyon Hwang
  • Ying Xing
  • Stan Zdonik
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4540)


Borealis is a distributed stream processing engine that has been developed at Brandeis University, Brown University, and MIT. It extends the first generation of data stream processing systems with advanced capabilities such as distributed operation, scalability with time-varying load, high availability against failures, and dynamic data and query modifications. In this paper, we focus on aspects that are related to load management and high availability in Borealis. We describe our algorithms for balanced and resilient load distribution, scalable distributed load shedding, and cooperative and self-configuring high availability. We also present experimental results from our prototype implementation showing the effectiveness of these algorithms.


Stream Processing Load Management Query Processor Backup Server Infeasible Point 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Whitney, A.T., Shasha, D.: Lots o’ Ticks: Real-Time High Performance Time Series Queries on Billions of Trades and Quotes (Demo). In: ACM SIGMOD Conference, Santa Barbara, CA (2001)Google Scholar
  2. 2.
    Babu, S., Subramanian, L., Widom, J.: A Data Stream Management System for Network Traffic Management. In: ACM Workshop on Network-Related Data Management (NRDM), Santa Barbara, CA (2001)Google Scholar
  3. 3.
    Stefanidis, A., Nittel, S. (eds.): Geosensor Networks. CRC Press, Boca Raton (2004)Google Scholar
  4. 4.
    Leonhardt, U., Magee, J.: Multi-sensor Location Tracking. In: International Conference on Mobile Computing and Networking (MobiCom), Dallas, TX (1998)Google Scholar
  5. 5.
    Franklin, M.J., Jeffery, S.R., Krishnamurthy, S., Reiss, F., Rizvi, S., Wu, E., Cooper, O., Edakkunni, A., Hong, W.: Design Considerations for High Fan-In Systems: The HiFi Approach. In: CIDR Conference, Asilomar, CA (2005)Google Scholar
  6. 6.
    Shah, M.A., Hellerstein, J.M., Brewer, E.: Highly-Available, Fault-Tolerant, Parallel Dataflows. In: ACM SIGMOD Conference, Paris, France (2004)Google Scholar
  7. 7.
    Abadi, D., Ahmad, Y., Balazinska, M., Çetintemel, U., Cherniack, M., Hwang, J., Lindner, W., Maskey, A., Rasin, A., Ryvkina, E., Tatbul, N., Xing, Y., Zdonik, S.: The Design of the Borealis Stream Processing Engine. In: CIDR Conference, Asilomar, CA (2005)Google Scholar
  8. 8.
    Pietzuch, P., Ledlie, J., Shneidman, J., Roussopoulos, M., Welsh, M., Seltzer, M.: Network-Aware Operator Placement for Stream-Processing Systems. In: IEEE ICDE Conference, Atlanta, GA (2006)Google Scholar
  9. 9.
    Amini, L., Jain, N., Sehgal, A., Silber, J., Verscheure, O.: Adaptive Control of Extreme-scale Stream Processing Systems. In: IEEE ICDCS Conference, Lisboa, Portugal (2006)Google Scholar
  10. 10.
    Zdonik, S., Stonebraker, M., Cherniack, M., Çetintemel, U., Balazinska, M., Balakrishnan, H.: The Aurora and Medusa Projects. IEEE Data Engineering Bulletin (Special Issue on Data Stream Processing) 26 (2003)Google Scholar
  11. 11.
    Abadi, D., Lindner, W., Madden, S., Schuler, J.: An Integration Framework for Sensor Networks and Data Stream Management Systems (Demo). In: VLDB Conference, Toronto, Canada (2004)Google Scholar
  12. 12.
    Ahmad, Y., Berg, B., Çetintemel, U., Humphrey, M., Hwang, J., Jhingran, A., Maskey, A., Papaemmanouil, O., Rasin, A., Tatbul, N., Xing, W., Xing, Y., Zdonik, S.: Distributed Operation in the Borealis Stream Processing Engine (Demo). In: ACM SIGMOD Conference, Baltimore, MD (2005)Google Scholar
  13. 13.
    Abadi, D., Carney, D., Çetintemel, U., Cherniack, M., Convey, C., Lee, S., Stonebraker, M., Tatbul, N., Zdonik, S.: Aurora: A New Model and Architecture for Data Stream Management. VLDB Journal 12 (2003)Google Scholar
  14. 14.
    Xing, Y., Zdonik, S., Hwang, J.H.: Dynamic Load Distribution in the Borealis Stream Processor. In: IEEE ICDE Conference, Tokyo, Japan (2005)Google Scholar
  15. 15.
    Xing, Y., Hwang, J.H., Çetintemel, U., Zdonik, S.: Providing Resiliency to Load Variations in Distributed Stream Processing. In: VLDB Conference, Seoul, Korea (2006)Google Scholar
  16. 16.
    Tatbul, N., Çetintemel, U., Zdonik, S., Cherniack, M., Stonebraker, M.: Load Shedding in a Data Stream Manager. In: VLDB Conference, Berlin, Germany (2003)Google Scholar
  17. 17.
    Tatbul, N., Çetintemel, U., Zdonik, S.: Staying FIT: Scalable Load Shedding Techniques for Distributed Stream Processing. Technical Report CS-06-13, Brown University, Computer Science (2006)Google Scholar
  18. 18.
    Tatbul, N., Zdonik, S.: Dealing with Overload in Distributed Stream Processing Systems. In: IEEE International Workshop on Networking Meets Databases (NetDB), Atlanta, GA (2006)Google Scholar
  19. 19.
    Hwang, J.H., Balazinska, M., Rasin, A., Çetintemel, U., Stonebraker, M., Zdonik, S.: High-Availability Algorithms for Distributed Stream Processing. In: IEEE ICDE Conference, Tokyo, Japan (2005)Google Scholar
  20. 20.
    Hwang, J.H., Xing, Y., Çetintemel, U., Zdonik, S.: A Cooperative, Self-Configuring High-Availability Solution for Stream Processing. In: IEEE ICDE Conference, Istanbul, Turkey (2007)Google Scholar
  21. 21.
    Hwang, J.H., Çetintemel, U., Zdonik, S.: Fast and Reliable Stream Processing over Wide Area Networks. In: IEEE International Workshop on Scalable Stream Processing Systems (SSPS), Istanbul, Turkey (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Nesime Tatbul
    • 1
  • Yanif Ahmad
    • 2
  • Uğur Çetintemel
    • 2
  • Jeong-Hyon Hwang
    • 2
  • Ying Xing
    • 2
  • Stan Zdonik
    • 2
  1. 1.Department of Computer ScienceETH ZürichZürichSwitzerland
  2. 2.Department of Computer ScienceBrown UniversityProvidenceUSA

Personalised recommendations