Abstract
Stream applications are widely deployed on the cloud. While modern distributed streaming systems like Flink and Spark Streaming can schedule and execute them efficiently, streaming dataflows are often dynamically changing, which may cause computation imbalance and backpressure.
We introduce AutoFlow, an automatic, hotspot-aware dynamic load balance system for streaming dataflows. It incorporates a centralized scheduler that monitors the load balance in the entire dataflow dynamically and implements state migrations correspondingly. The scheduler achieves these two tasks using a simple asynchronous distributed control message mechanism and a hotspot-diminishing algorithm. The timing mechanism supports implicit barriers and a highly efficient state-migration without global barriers or pauses to operators. It also supports a time-window based load-balance measurement and feeds them to the hotspot-diminishing algorithm without user interference. We implemented AutoFlow on top of Ray, an actor-based distributed execution framework. Our evaluation based on various streaming benchmark datasets shows that AutoFlow achieves good load-balance and incurs a low latency overhead in a highly data-skew workload.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
gRPC. https://grpc.io/
NEXMark benchmark. http://datalab.cs.pdx.edu/niagara/NEXMark/
Akidau, T., et al.: MillWheel: fault-tolerant stream processing at internet scale. Proc. VLDB Endow. 6(11), 1033–1044 (2013)
Akidau, T., et al.: The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing (2015)
Armbrust, M., et al.: Structured streaming: a declarative API for real-time applications in apache spark. In: International Conference on Management of Data, pp. 601–613 (2018)
Carbone, P., Fóra, G., Ewen, S., Haridi, S., Tzoumas, K.: Lightweight asynchronous snapshots for distributed dataflows. arXiv preprint arXiv:1506.08603 (2015)
Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache Flink: stream and batch processing in a single engine. Bull. IEEE Comput. Soc. Tech. Committee Data Eng. 36(4) (2015)
Castro Fernandez, R., Migliavacca, M., Kalyvianaki, E., Pietzuch, P.: Integrating scale out and fault tolerance in stream processing using operator state management. In: ACM SIGMOD International Conference on Management of Data, pp. 725–736 (2013)
Dai, J., Huang, J., Huang, S., Huang, B., Liu, Y.: HiTune: dataflow-based performance analysis for big data cloud. In: ATC, pp. 87–100 (2011)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)
Floratou, A., Agrawal, A., Graham, B., Rao, S., Ramasamy, K.: Dhalion: self-regulating stream processing in heron. Proc. VLDB Endow. 10(12), 1825–1836 (2017)
Garduno, E., Kavulya, S.P., Tan, J., Gandhi, R., Narasimhan, P.: Theia: visual signatures for problem diagnosis in large Hadoop clusters. In: LISA’12, pp. 33–42 (2012)
Hoffmann, M., et al.: SnailTrail: generalizing critical paths for online analysis of distributed dataflows. In: NSDI’18, pp. 95–110 (2018)
Hoffmann, M., Lattuada, A., McSherry, F.: Megaphone: latency-conscious state migration for distributed streaming dataflows. Proc. VLDB Endow. 12(9), 1002–1015 (2019)
Kalavri, V., Liagouris, J., Hoffmann, M., Dimitrova, D., Forshaw, M., Roscoe, T.: Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows. In: OSDI’18, pp. 783–798 (2018)
Kulkarni, S., et al.: Twitter Heron: stream processing at scale. In: the 2015 ACM SIGMOD International Conference on Management of Data, pp. 239–250 (2015)
Mai, L., et al.: Chi: a scalable and programmable control plane for distributed stream processing systems. Proc. VLDB Endow. 11(10), 1303–1316 (2018)
Moritz, P., et al.: Ray: a distributed framework for emerging \(\{\)AI\(\}\) applications. In: OSDI’18, pp. 561–577 (2018)
Toshniwal, A., et al.: Storm@ twitter. In: ACM SIGMOD International Conference on Management of Data, pp. 147–156 (2014)
Wang, S., et al.: Lineage stash: fault tolerance off the critical path. In: SOSP’19, pp. 338–352 (2019)
White, T.: Hadoop: The Definitive Guide. O’Reilly Media, Inc., Newton (2012)
Zaharia, M., et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: NSDI’12, pp. 15–28 (2012)
Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., Stoica, I.: Discretized streams: fault-tolerant streaming computation at scale. In: SOSP’13, pp. 423–438 (2013)
Acknowlegement
The authors would like to thank all the reviewers for their valuable comments. This work is supported by National Key R&D Program of China under Grant No. 2016YFB0200803; the National Natural Science Foundation of China under Grant No. 61972376, No. 62072431, No. 62032023; the Science Foundation of Beijing No. L182053.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Lu, P., Yue, Y., Yuan, L., Zhang, Y. (2022). AutoFlow: Hotspot-Aware, Dynamic Load Balancing for Distributed Stream Processing. In: Lai, Y., Wang, T., Jiang, M., Xu, G., Liang, W., Castiglione, A. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2021. Lecture Notes in Computer Science(), vol 13157. Springer, Cham. https://doi.org/10.1007/978-3-030-95391-1_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-95391-1_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-95390-4
Online ISBN: 978-3-030-95391-1
eBook Packages: Computer ScienceComputer Science (R0)