Overview
In data stream processing, elasticity refers to the ability of autonomously provisioning and decommissioning resources (e.g., threads or nodes) in order to match, at each point in time, the computational power needed to run a set of user-defined continuous queries.
Elastic SPEs are able to dynamically scale up and out (or down and in, respectively) the resources employed to run the existing queries in order to match the needed computational power at each point in time. They can thus benefit of decoupled architectures such as cloud computing ones, and similar metrics for the performance evaluation can be defined (Ilyushkin et al. 2017). The variations in the needed computational power are due to the fluctuating nature of data streams (both in terms of their volume and their underlying data distribution) and the variable number of queries added and removed over time by users.
Historical Background
Stream processing...
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abadi DJ, Ahmad Y, Balazinska M, Cetintemel U, Cherniack M, Hwang JH, Lindner W, Maskey A, Rasin A, Ryvkina E et al (2005) The design of the borealis stream processing engine. In: CIDR, vol 5, pp 277–289
Arasu A, Cherniack M, Galvez E, Maier D, Maskey AS, Ryvkina E, Stonebraker M, Tibbetts R (2004) Linear road: a stream data management benchmark. In: Proceedings of the thirtieth international conference on very large data bases, VLDB Endowment, vol 30, pp 480–491
Balkesen C, Tatbul N, Özsu MT (2013) Adaptive input admission and management for parallel stream processing. In: Proceedings of the 7th ACM international conference on distributed event-based systems. ACM, pp 15–26
Carney D, Çetintemel U, Cherniack M, Convey C, Lee S, Seidman G, Stonebraker M, Tatbul N, Zdonik S (2002) Monitoring streams: a new class of data management applications. In: Proceedings of the 28th international conference on very large data bases, VLDB Endowment, pp 215–226
Castro Fernandez R, Migliavacca M, Kalyvianaki E, Pietzuch P (2013) Integrating scale out and fault tolerance in stream processing using operator state management. In: Proceedings of the 2013 ACM SIGMOD international conference on management of data. ACM, pp 725–736
De Matteis T, Mencagli G (2017) Proactive elasticity and energy awareness in data stream processing. J Syst Softw 127:302–319
Ding J, Fu TZJ, Ma RTB, Winslett M, Yang Y, Zhang Z, Chao H (2015) Optimal operator state migration for elastic data stream processing. CoRR abs/1501.03619
Gedik B, Schneider S, Hirzel M, Wu KL (2014) Elastic scaling for data stream processing. IEEE Trans Parallel Distributed Syst 25(6):1447–1463
Gibbons PB (2015) Big data: scale down, scale up, scale out. In: IPDPS, p 3
Gulisano V (2012) Streamcloud: an elastic parallel-distributed stream processing engine. PhD thesis, Universidad Politécnica de Madrid
Gulisano V, Jimenez-Peris R, Patino-Martinez M, Soriente C, Valduriez P (2012) Streamcloud: an elastic and scalable data streaming system. IEEE Trans Parallel Distrib Syst 23(12):2351–2365
Gulisano V, Nikolakopoulos Y, Papatriantafilou M, Tsigas P (2016) Scalejoin: a deterministic, disjoint-parallel and skew-resilient stream join. IEEE Trans Big Data PP(99):1–1
Gulisano V, Nikolakopoulos Y, Cederman D, Papatriantafilou M, Tsigas P (2017) Efficient data streaming multiway aggregation through concurrent algorithmic designs and new abstract data types. ACM Trans Parallel Comput 4(2):11:1–11:28
Heinze T, Ji Y, Pan Y, Grueneberger FJ, Jerzak Z, Fetzer C (2013) Elastic complex event processing under varying query load. In: BD3@ VLDB, pp 25–30
Heinze T, Jerzak Z, Hackenbroich G, Fetzer C (2014a) Latency-aware elastic scaling for distributed data stream processing systems. In: Proceedings of the 8th ACM international conference on distributed event-based systems. ACM, pp 13–22
Heinze T, Pappalardo V, Jerzak Z, Fetzer C (2014b) Auto-scaling techniques for elastic data stream processing. In: IEEE 30th international conference on data engineering workshops (ICDEW). IEEE, pp 296–302
Hirzel M, Soulé R, Schneider S, Gedik B, Grimm R (2014) A catalog of stream processing optimizations. ACM Comput Surv 46(4):46:1–46:34
Hochreiner C, Vögler M, Schulte S, Dustdar S (2016) Elastic stream processing for the internet of things. In: IEEE 9th international conference on cloud computing (CLOUD). IEEE, pp 100–107
Ilyushkin A, Ali-Eldin A, Herbst N, Papadopoulos AV, Ghit B, Epema D, Iosup A (2017) An experimental performance evaluation of autoscaling algorithms for complex workflows. In: Proceedings of the 8th ACM/SPEC on international conference on performance engineering (ICPE), ICPE’17. ACM, New York, pp 75–86. http://doi.org/10.1145/3030207.3030214
Koliousis A, Weidlich M, Castro Fernandez R, Wolf AL, Costa P, Pietzuch P (2016) Saber: window-based hybrid stream processing for heterogeneous architectures. In: Proceedings of the 2016 international conference on management of data. ACM, pp 555–569
Kumbhare AG, Simmhan Y, Frincu M, Prasanna VK (2015) Reactive resource provisioning heuristics for dynamic dataflows on cloud infrastructure. IEEE Trans Cloud Comput 3(2):105–118
Loesing S, Hentschel M, Kraska T, Kossmann D (2012) Stormy: an elastic and highly available streaming service in the cloud. In: Proceedings of the 2012 joint EDBT/ICDT workshops. ACM, pp 55–60
Martin A, Brito A, Fetzer C (2014) Scalable and elastic realtime click stream analysis using streammine3g. In: Proceedings of the 8th ACM international conference on distributed event-based systems. ACM, pp 198–205
Schneider S, Andrade H, Gedik B, Biem A, Wu KL (2009) Elastic scaling of data parallel operators in stream processing. In: IEEE international symposium on parallel & distributed processing, IPDPS 2009. IEEE, pp 1–12
Tatbul N, Çetintemel U, Zdonik S, Cherniack M, Stonebraker M (2003) Load shedding in a data stream manager. In: Proceedings of the 29th international conference on very large data bases, VLDB Endowment, vol 29, pp 309–320
Xing Y, Zdonik S, Hwang JH (2005) Dynamic load distribution in the borealis stream processor. In: Proceedings 21st international conference on data engineering, ICDE 2005. IEEE, pp 791–802
Zacheilas N, Kalogeraki V, Zygouras N, Panagiotou N, Gunopulos D (2015) Elastic complex event processing exploiting prediction. In: IEEE international conference on big data (Big Data). IEEE, pp 213–222
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this entry
Cite this entry
Gulisano, V., Papatriantafilou, M., Papadopoulos, A.V. (2019). Elasticity. In: Sakr, S., Zomaya, A.Y. (eds) Encyclopedia of Big Data Technologies. Springer, Cham. https://doi.org/10.1007/978-3-319-77525-8_191
Download citation
DOI: https://doi.org/10.1007/978-3-319-77525-8_191
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-77524-1
Online ISBN: 978-3-319-77525-8
eBook Packages: Computer ScienceReference Module Computer Science and Engineering