Encyclopedia of Big Data Technologies

Living Edition
| Editors: Sherif Sakr, Albert Zomaya

Elasticity

  • Vincenzo Gulisano
  • Marina Papatriantafilou
  • Alessandro V. Papadopoulos
Living reference work entry
DOI: https://doi.org/10.1007/978-3-319-63962-8_191-1

Synonyms

Overview

In data stream processing, elasticity refers to the ability of autonomously provisioning and decommissioning resources (e.g., threads or nodes) in order to match, at each point in time, the computational power needed to run a set of user-defined continuous queries.

Elastic SPEs are able to dynamically scale up and out (or down and in, respectively) the resources employed to run the existing queries in order to match the needed computational power at each point in time. They can thus benefit of decoupled architectures such as cloud computing ones, and similar metrics for the performance evaluation can be defined (Ilyushkin et al. 2017). The variations in the needed computational power are due to the fluctuating nature of data streams (both in terms of their volume and their underlying data distribution) and the variable number of queries added and removed over time by users.

Historical Background

Stream processing...

This is a preview of subscription content, log in to check access.

References

  1. Abadi DJ, Ahmad Y, Balazinska M, Cetintemel U, Cherniack M, Hwang JH, Lindner W, Maskey A, Rasin A, Ryvkina E et al (2005) The design of the borealis stream processing engine. In: CIDR, vol 5, pp 277–289Google Scholar
  2. Arasu A, Cherniack M, Galvez E, Maier D, Maskey AS, Ryvkina E, Stonebraker M, Tibbetts R (2004) Linear road: a stream data management benchmark. In: Proceedings of the thirtieth international conference on very large data bases, VLDB Endowment, vol 30, pp 480–491Google Scholar
  3. Balkesen C, Tatbul N, Özsu MT (2013) Adaptive input admission and management for parallel stream processing. In: Proceedings of the 7th ACM international conference on distributed event-based systems. ACM, pp 15–26Google Scholar
  4. Carney D, Çetintemel U, Cherniack M, Convey C, Lee S, Seidman G, Stonebraker M, Tatbul N, Zdonik S (2002) Monitoring streams: a new class of data management applications. In: Proceedings of the 28th international conference on very large data bases, VLDB Endowment, pp 215–226Google Scholar
  5. Castro Fernandez R, Migliavacca M, Kalyvianaki E, Pietzuch P (2013) Integrating scale out and fault tolerance in stream processing using operator state management. In: Proceedings of the 2013 ACM SIGMOD international conference on management of data. ACM, pp 725–736Google Scholar
  6. De Matteis T, Mencagli G (2017) Proactive elasticity and energy awareness in data stream processing. J Syst Softw 127:302–319Google Scholar
  7. Ding J, Fu TZJ, Ma RTB, Winslett M, Yang Y, Zhang Z, Chao H (2015) Optimal operator state migration for elastic data stream processing. CoRR abs/1501.03619Google Scholar
  8. Gedik B, Schneider S, Hirzel M, Wu KL (2014) Elastic scaling for data stream processing. IEEE Trans Parallel Distributed Syst 25(6):1447–1463Google Scholar
  9. Gibbons PB (2015) Big data: scale down, scale up, scale out. In: IPDPS, p 3Google Scholar
  10. Gulisano V (2012) Streamcloud: an elastic parallel-distributed stream processing engine. PhD thesis, Universidad Politécnica de MadridGoogle Scholar
  11. Gulisano V, Jimenez-Peris R, Patino-Martinez M, Soriente C, Valduriez P (2012) Streamcloud: an elastic and scalable data streaming system. IEEE Trans Parallel Distrib Syst 23(12):2351–2365CrossRefGoogle Scholar
  12. Gulisano V, Nikolakopoulos Y, Papatriantafilou M, Tsigas P (2016) Scalejoin: a deterministic, disjoint-parallel and skew-resilient stream join. IEEE Trans Big Data PP(99):1–1Google Scholar
  13. Gulisano V, Nikolakopoulos Y, Cederman D, Papatriantafilou M, Tsigas P (2017) Efficient data streaming multiway aggregation through concurrent algorithmic designs and new abstract data types. ACM Trans Parallel Comput 4(2):11:1–11:28Google Scholar
  14. Heinze T, Ji Y, Pan Y, Grueneberger FJ, Jerzak Z, Fetzer C (2013) Elastic complex event processing under varying query load. In: BD3@ VLDB, pp 25–30Google Scholar
  15. Heinze T, Jerzak Z, Hackenbroich G, Fetzer C (2014a) Latency-aware elastic scaling for distributed data stream processing systems. In: Proceedings of the 8th ACM international conference on distributed event-based systems. ACM, pp 13–22Google Scholar
  16. Heinze T, Pappalardo V, Jerzak Z, Fetzer C (2014b) Auto-scaling techniques for elastic data stream processing. In: IEEE 30th international conference on data engineering workshops (ICDEW). IEEE, pp 296–302Google Scholar
  17. Hirzel M, Soulé R, Schneider S, Gedik B, Grimm R (2014) A catalog of stream processing optimizations. ACM Comput Surv 46(4):46:1–46:34Google Scholar
  18. Hochreiner C, Vögler M, Schulte S, Dustdar S (2016) Elastic stream processing for the internet of things. In: IEEE 9th international conference on cloud computing (CLOUD). IEEE, pp 100–107Google Scholar
  19. Ilyushkin A, Ali-Eldin A, Herbst N, Papadopoulos AV, Ghit B, Epema D, Iosup A (2017) An experimental performance evaluation of autoscaling algorithms for complex workflows. In: Proceedings of the 8th ACM/SPEC on international conference on performance engineering (ICPE), ICPE’17. ACM, New York, pp 75–86. http://doi.org/10.1145/3030207.3030214 CrossRefGoogle Scholar
  20. Koliousis A, Weidlich M, Castro Fernandez R, Wolf AL, Costa P, Pietzuch P (2016) Saber: window-based hybrid stream processing for heterogeneous architectures. In: Proceedings of the 2016 international conference on management of data. ACM, pp 555–569Google Scholar
  21. Kumbhare AG, Simmhan Y, Frincu M, Prasanna VK (2015) Reactive resource provisioning heuristics for dynamic dataflows on cloud infrastructure. IEEE Trans Cloud Comput 3(2):105–118CrossRefGoogle Scholar
  22. Loesing S, Hentschel M, Kraska T, Kossmann D (2012) Stormy: an elastic and highly available streaming service in the cloud. In: Proceedings of the 2012 joint EDBT/ICDT workshops. ACM, pp 55–60Google Scholar
  23. Martin A, Brito A, Fetzer C (2014) Scalable and elastic realtime click stream analysis using streammine3g. In: Proceedings of the 8th ACM international conference on distributed event-based systems. ACM, pp 198–205Google Scholar
  24. Schneider S, Andrade H, Gedik B, Biem A, Wu KL (2009) Elastic scaling of data parallel operators in stream processing. In: IEEE international symposium on parallel & distributed processing, IPDPS 2009. IEEE, pp 1–12Google Scholar
  25. Tatbul N, Çetintemel U, Zdonik S, Cherniack M, Stonebraker M (2003) Load shedding in a data stream manager. In: Proceedings of the 29th international conference on very large data bases, VLDB Endowment, vol 29, pp 309–320Google Scholar
  26. Xing Y, Zdonik S, Hwang JH (2005) Dynamic load distribution in the borealis stream processor. In: Proceedings 21st international conference on data engineering, ICDE 2005. IEEE, pp 791–802Google Scholar
  27. Zacheilas N, Kalogeraki V, Zygouras N, Panagiotou N, Gunopulos D (2015) Elastic complex event processing exploiting prediction. In: IEEE international conference on big data (Big Data). IEEE, pp 213–222Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Vincenzo Gulisano
    • 1
  • Marina Papatriantafilou
    • 1
  • Alessandro V. Papadopoulos
    • 2
  1. 1.Chalmers University of TechnologyGothenburgSweden
  2. 2.Mälardalen UniversityVästeråsSweden

Section editors and affiliations

  • Alessandro Margara
    • 1
  • Tilmann Rabl
    • 2
  1. 1.Politecnico di Milano
  2. 2.Database Systems and Information Management GroupTechnische Universität BerlinBerlinGermany