Abstract
The execution of complex event processing (CEP) applications on a set of clustered homogenous computing nodes is latency-sensitive, especially when workload conditions widely change at runtime. To manage the varying workloads of nodes in a scalable and cost-effective manner, adjusting the application parallelism at runtime is critical. To tackle the scalability challenge, we have extended an existing parallelization model called PARS that only supports stateless CEP operators and runs operators in parallel regions without changing the number of computing nodes assigned to parallel regions. We have added new features to PARS in support of stateful operators by introducing local controllers and new initiator and terminator event types, making partitioning fully transparent to application developers. We have proved the correctness of this extended model, called PARS+ , with respect to its presented formal definition. We have then used PARS+ as the base parallelization model in formulation of an adaptive strategy called ACEP to auto-scale operators including the stateful operators. Scaling decisions are governed by a predictive performance model that uses a control-theoretic method for estimating the resource and latency costs of each operator at runtime. The loads of clustered compute nodes are monitored, and compute nodes in a parallel region are reconfigured at runtime to ensure a balanced load on all compute nodes, accruing minimum cost to parallelize a stateful operator. ACEP minimizes network delays because it does not force using the shared state or techniques that employ state migration. We have built an event generator to simulate event sources and experimentally evaluate the ACEP strategy in terms of response time and resource costs. Two variant implementations of ACEP have been compared with the elastic strategy presented by Xiao et al. (2018), and the findings demonstrated that our implementations had adapted themselves better in different resource and response-time-sensitive scenarios, with lower response time (6%) and lower resource cost (8%).
References
Luthra M, Koldehofe B, Steinmetz R (2018) Adaptive complex event processing over fog-cloud infrastructure supporting transitions. KuVS-Fachgespräch Fog Comput 2018:17
Gutiérrez-Madroñal L, Medina-Bulo I, Domínguez-Jiménez JJ (2018) IoT–TEG: test event generator system. J Syst Softw 137:784–803. https://doi.org/10.1016/j.jss.2017.06.037
Artikis A et al (2017) A prototype for credit card fraud management: industry paper. In: Proceedings of the 11th ACM International Conference on Distributed and Event-Based Systems, New York, NY, USA, 2017, pp 249–260, https://doi.org/10.1145/3093742.3093912
Antonić A, Marjanović M, Žarko IP (2017) Modeling aggregate input load of interoperable smart city services. In: Proceedings of the 11th ACM International Conference on Distributed and Event-Based Systems, New York, NY, USA, 2017, pp 34–43, https://doi.org/10.1145/3093742.3093928
Soffer P et al (2017) From event streams to process models and back: challenges and opportunities. Inf Syst. https://doi.org/10.1016/j.is.2017.11.002
Fardbastani MA, Allahdadi F, Sharifi M (2018) Business process monitoring via decentralized complex event processing. Enterprise Inf Syst. https://doi.org/10.1080/17517575.2018.1522453
Saleh O, Betz H, Sattler K-U (2015) “Partitioning for scalable complex event processing on data streams”, in New Trends in Database and Information Systems II. Springer, Cham, pp 185–197
Gulisano V, Jimenez-Peris R, Patino-Martinez M, Valduriez P (2010) StreamCloud: a large scale data streaming system. In: Proceedings of the 2010 IEEE 30th International Conference on Distributed Computing Systems, Washington, DC, USA, 2010, pp 126–137, https://doi.org/10.1109/icdcs.2010.72
Castro Fernandez R et al (2013) Integrating scale out and fault tolerance in stream processing using operator state management. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, New York, NY, USA, 2013, pp 725–736, https://doi.org/10.1145/2463676.2465282
Heinze T, Jerzak Z, Hackenbroich G, Fetzer C (2014) Latency-aware elastic scaling for distributed data stream processing systems. In: Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, New York, NY, USA, 2014, pp 13–22, https://doi.org/10.1145/2611286.2611294
Camacho EF, Bordons C (2007) Nonlinear model predictive control: an introductory review. In: Assessment and Future Directions of Nonlinear Model Predictive Control, R Findeisen, F Allgöwer, and LT Biegler, eds. Heidelberg: Springer, 2007, pp 1–16
Gedik B (2014) Partitioning functions for stateful data parallelism in stream processing. VLDB J 23(4):517–539. https://doi.org/10.1007/s00778-013-0335-9
Hirzel M (2012) Partition and compose: parallel complex event processing. In Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, New York, NY, USA, 2012, pp 191–200, https://doi.org/10.1145/2335484.2335506
Isoyama K et al (2012) A scalable complex event processing system and evaluations of its performance. In: Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, New York, NY, USA, 2012, pp 123–126, https://doi.org/10.1145/2335484.2335498
Wu S, Kumar V, Wu K-L, Ooi BC (2012) Parallelizing stateful operators in a distributed stream processing system: how, should you and how much? In: Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, New York, NY, USA, 2012, pp 278–289, https://doi.org/10.1145/2335484.2335515
de Almeida VP, Bhowmik S, Endler M, Rothermel K (2020) DSCEP: an infrastructure for distributed semantic complex event processing. arXiv:2002.05869 [cs], Feb. 2020, Accessed: Sep. 14, 2020. [Online]. http://arxiv.org/abs/2002.05869
Mayer R et al (2017) SPECTRE: Supporting consumption policies in window-based parallel complex event processing. In: Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference, New York, NY, USA, 2017, pp 161–173, https://doi.org/10.1145/3135974.3135983
Mei Y, Madden S (2009) ZStream: a cost-based query processor for adaptively detecting composite events. In: Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data, New York, NY, USA, 2009, pp 193–206, https://doi.org/10.1145/1559845.1559867
Fardbastani MA, Sharifi M (2019) Scalable complex event processing using adaptive load balancing. J Syst Softw 149:305–317. https://doi.org/10.1016/j.jss.2018.12.012
Wang YH, Cao K, Zhang XM (2013) Complex event processing over distributed probabilistic event streams. Comput Math Appl 66(10):1808–1821. https://doi.org/10.1016/j.camwa.2013.06.032
Xiao F, Zhan C, Lai H, Tao L, Qu Z (2017) New parallel processing strategies in complex event processing systems with data streams. Int J Distrib Sens Netw 13(8):1550147717728626. https://doi.org/10.1177/1550147717728626
Ottenwälder B, Mayer R, Koldehofe B (2014) Distributed complex event processing for mobile large-scale video applications. In: Proceedings of the Posters & Demos Session, New York, NY, USA, 2014, pp 5–6, https://doi.org/10.1145/2678508.2678511
Pathak R, Vaidehi V (2015) An efficient rule balancing for scalable complex event processing. In: 2015 IEEE 28th Canadian Conference on Electrical and Computer Engineering (CCECE), May 2015, pp 190–195, https://doi.org/10.1109/ccece.2015.7129184
Choochotkaew S et al (2017) EdgeCEP: fully-distributed complex event processing on IoT edges. In: 2017 13th International Conference on Distributed Computing in Sensor Systems (DCOSS), Jun. 2017, pp 121–129, https://doi.org/10.1109/dcoss.2017.14
Vanneste S et al (2019) Distributed uniform streaming framework: an elastic fog computing platform for event stream processing and platform transparency. Fut Internet 11(7):7. https://doi.org/10.3390/fi11070158
Matteis T, Mencagli G (2017) Parallel patterns for window-based stateful operators on data streams: an algorithmic skeleton approach. Int J Parallel Program 45(2):382–401. https://doi.org/10.1007/s10766-016-0413-x
Fernandez RC, Weidlich M, Pietzuch P, Gal A (2014) Scalable stateful stream processing for smart grids. In: Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, New York, NY, USA, 2014, pp 276–281, https://doi.org/10.1145/2611286.2611326
Xiao F, Aritsugi M (2018) An adaptive parallel processing strategy for complex event processing systems over data streams in wireless sensor networks. Sensors 18(11):3732. https://doi.org/10.3390/s18113732
Kolchinsky I, Schuster A (2018) Efficient adaptive detection of complex event patterns. Proc VLDB Endow 11(11):1346–1359. https://doi.org/10.14778/3236187.3236190
Zacheilas N et al (2015) Elastic complex event processing exploiting prediction. In: 2015 IEEE International Conference on Big Data (Big Data), Oct. 2015, pp 213–222, https://doi.org/10.1109/bigdata.2015.7363758
Weisenburger P et al (2017) Quality-aware runtime adaptation in complex event processing. In: 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) (pp 140–151). 2017. IEEE
Zhao B (2018) Complex event processing under constrained resources by state-based load shedding. In: 2018 IEEE 34th International Conference on Data Engineering (ICDE), pp 1699–1703. IEEE, 2018
“Storm, distributed and fault-tolerant realtime computation.” http://www.storm-project.net/ (Accessed Jan. 20, 2018)
“Esper,” EsperTech. http://www.espertech.com/esper/ (Accessed Mar. 25, 2018)
Talebi MM, Sharifi M (2019) PARS: a parallel model for scaled processing of complex events. J Syst Softw 155:1–16. https://doi.org/10.1016/j.jss.2019.05.014
Fülöp LJ et al (2012) Predictive complex event processing: a conceptual framework for combining complex event processing and predictive analytics. In: Proceedings of the Fifth Balkan Conference in Informatics, New York, NY, USA, 2012, pp 26–31, https://doi.org/10.1145/2371316.2371323
Kingman JF (1962) On queues in heavy traffic. J R Stat Soc Ser B Methodol 24(2):383–392
De Matteis T, Mencagli G (2017) Proactive elasticity and energy awareness in data stream processing. J Syst Softw 127:302–319. https://doi.org/10.1016/j.jss.2016.08.037
Camacho EF, Bordons C (2007) Nonlinear model predictive control: an introductory review. In: Assessment and Future Directions of Nonlinear Model Predictive Control, R Findeisen, F Allgöwer, and LT Biegler, eds. Heidelberg: Springer, 2007, pp 1–16
Mencagli G, Vanneschi M (2014) Towards a systematic approach to the dynamic adaptation of structured parallel computations using model predictive control. Cluster Comput 17(4):1443–1463. https://doi.org/10.1007/s10586-014-0346-3
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Talebi, M., Sharifi, M. & Kalantari, M. ACEP: an adaptive strategy for proactive and elastic processing of complex events. J Supercomput 77, 4718–4753 (2021). https://doi.org/10.1007/s11227-020-03454-0
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-020-03454-0