Abstract
Borealis is a distributed stream processing engine that has been developed at Brandeis University, Brown University, and MIT. It extends the first generation of data stream processing systems with advanced capabilities such as distributed operation, scalability with time-varying load, high availability against failures, and dynamic data and query modifications. In this paper, we focus on aspects that are related to load management and high availability in Borealis. We describe our algorithms for balanced and resilient load distribution, scalable distributed load shedding, and cooperative and self-configuring high availability. We also present experimental results from our prototype implementation showing the effectiveness of these algorithms.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Whitney, A.T., Shasha, D.: Lots o’ Ticks: Real-Time High Performance Time Series Queries on Billions of Trades and Quotes (Demo). In: ACM SIGMOD Conference, Santa Barbara, CA (2001)
Babu, S., Subramanian, L., Widom, J.: A Data Stream Management System for Network Traffic Management. In: ACM Workshop on Network-Related Data Management (NRDM), Santa Barbara, CA (2001)
Stefanidis, A., Nittel, S. (eds.): Geosensor Networks. CRC Press, Boca Raton (2004)
Leonhardt, U., Magee, J.: Multi-sensor Location Tracking. In: International Conference on Mobile Computing and Networking (MobiCom), Dallas, TX (1998)
Franklin, M.J., Jeffery, S.R., Krishnamurthy, S., Reiss, F., Rizvi, S., Wu, E., Cooper, O., Edakkunni, A., Hong, W.: Design Considerations for High Fan-In Systems: The HiFi Approach. In: CIDR Conference, Asilomar, CA (2005)
Shah, M.A., Hellerstein, J.M., Brewer, E.: Highly-Available, Fault-Tolerant, Parallel Dataflows. In: ACM SIGMOD Conference, Paris, France (2004)
Abadi, D., Ahmad, Y., Balazinska, M., Çetintemel, U., Cherniack, M., Hwang, J., Lindner, W., Maskey, A., Rasin, A., Ryvkina, E., Tatbul, N., Xing, Y., Zdonik, S.: The Design of the Borealis Stream Processing Engine. In: CIDR Conference, Asilomar, CA (2005)
Pietzuch, P., Ledlie, J., Shneidman, J., Roussopoulos, M., Welsh, M., Seltzer, M.: Network-Aware Operator Placement for Stream-Processing Systems. In: IEEE ICDE Conference, Atlanta, GA (2006)
Amini, L., Jain, N., Sehgal, A., Silber, J., Verscheure, O.: Adaptive Control of Extreme-scale Stream Processing Systems. In: IEEE ICDCS Conference, Lisboa, Portugal (2006)
Zdonik, S., Stonebraker, M., Cherniack, M., Çetintemel, U., Balazinska, M., Balakrishnan, H.: The Aurora and Medusa Projects. IEEE Data Engineering Bulletin (Special Issue on Data Stream Processing) 26 (2003)
Abadi, D., Lindner, W., Madden, S., Schuler, J.: An Integration Framework for Sensor Networks and Data Stream Management Systems (Demo). In: VLDB Conference, Toronto, Canada (2004)
Ahmad, Y., Berg, B., Çetintemel, U., Humphrey, M., Hwang, J., Jhingran, A., Maskey, A., Papaemmanouil, O., Rasin, A., Tatbul, N., Xing, W., Xing, Y., Zdonik, S.: Distributed Operation in the Borealis Stream Processing Engine (Demo). In: ACM SIGMOD Conference, Baltimore, MD (2005)
Abadi, D., Carney, D., Çetintemel, U., Cherniack, M., Convey, C., Lee, S., Stonebraker, M., Tatbul, N., Zdonik, S.: Aurora: A New Model and Architecture for Data Stream Management. VLDB Journal 12 (2003)
Xing, Y., Zdonik, S., Hwang, J.H.: Dynamic Load Distribution in the Borealis Stream Processor. In: IEEE ICDE Conference, Tokyo, Japan (2005)
Xing, Y., Hwang, J.H., Çetintemel, U., Zdonik, S.: Providing Resiliency to Load Variations in Distributed Stream Processing. In: VLDB Conference, Seoul, Korea (2006)
Tatbul, N., Çetintemel, U., Zdonik, S., Cherniack, M., Stonebraker, M.: Load Shedding in a Data Stream Manager. In: VLDB Conference, Berlin, Germany (2003)
Tatbul, N., Çetintemel, U., Zdonik, S.: Staying FIT: Scalable Load Shedding Techniques for Distributed Stream Processing. Technical Report CS-06-13, Brown University, Computer Science (2006)
Tatbul, N., Zdonik, S.: Dealing with Overload in Distributed Stream Processing Systems. In: IEEE International Workshop on Networking Meets Databases (NetDB), Atlanta, GA (2006)
Hwang, J.H., Balazinska, M., Rasin, A., Çetintemel, U., Stonebraker, M., Zdonik, S.: High-Availability Algorithms for Distributed Stream Processing. In: IEEE ICDE Conference, Tokyo, Japan (2005)
Hwang, J.H., Xing, Y., Çetintemel, U., Zdonik, S.: A Cooperative, Self-Configuring High-Availability Solution for Stream Processing. In: IEEE ICDE Conference, Istanbul, Turkey (2007)
Hwang, J.H., Çetintemel, U., Zdonik, S.: Fast and Reliable Stream Processing over Wide Area Networks. In: IEEE International Workshop on Scalable Stream Processing Systems (SSPS), Istanbul, Turkey (2007)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Tatbul, N., Ahmad, Y., Çetintemel, U., Hwang, JH., Xing, Y., Zdonik, S. (2008). Load Management and High Availability in the Borealis Distributed Stream Processing Engine . In: Nittel, S., Labrinidis, A., Stefanidis, A. (eds) GeoSensor Networks. GSN 2006. Lecture Notes in Computer Science, vol 4540. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-79996-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-540-79996-2_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-79995-5
Online ISBN: 978-3-540-79996-2
eBook Packages: Computer ScienceComputer Science (R0)