Skip to main content
Log in

ACEP: an adaptive strategy for proactive and elastic processing of complex events

The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

The execution of complex event processing (CEP) applications on a set of clustered homogenous computing nodes is latency-sensitive, especially when workload conditions widely change at runtime. To manage the varying workloads of nodes in a scalable and cost-effective manner, adjusting the application parallelism at runtime is critical. To tackle the scalability challenge, we have extended an existing parallelization model called PARS that only supports stateless CEP operators and runs operators in parallel regions without changing the number of computing nodes assigned to parallel regions. We have added new features to PARS in support of stateful operators by introducing local controllers and new initiator and terminator event types, making partitioning fully transparent to application developers. We have proved the correctness of this extended model, called PARS+ , with respect to its presented formal definition. We have then used PARS+ as the base parallelization model in formulation of an adaptive strategy called ACEP to auto-scale operators including the stateful operators. Scaling decisions are governed by a predictive performance model that uses a control-theoretic method for estimating the resource and latency costs of each operator at runtime. The loads of clustered compute nodes are monitored, and compute nodes in a parallel region are reconfigured at runtime to ensure a balanced load on all compute nodes, accruing minimum cost to parallelize a stateful operator. ACEP minimizes network delays because it does not force using the shared state or techniques that employ state migration. We have built an event generator to simulate event sources and experimentally evaluate the ACEP strategy in terms of response time and resource costs. Two variant implementations of ACEP have been compared with the elastic strategy presented by Xiao et al. (2018), and the findings demonstrated that our implementations had adapted themselves better in different resource and response-time-sensitive scenarios, with lower response time (6%) and lower resource cost (8%).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

References

  1. Luthra M, Koldehofe B, Steinmetz R (2018) Adaptive complex event processing over fog-cloud infrastructure supporting transitions. KuVS-Fachgespräch Fog Comput 2018:17

    Google Scholar 

  2. Gutiérrez-Madroñal L, Medina-Bulo I, Domínguez-Jiménez JJ (2018) IoT–TEG: test event generator system. J Syst Softw 137:784–803. https://doi.org/10.1016/j.jss.2017.06.037

    Article  Google Scholar 

  3. Artikis A et al (2017) A prototype for credit card fraud management: industry paper. In: Proceedings of the 11th ACM International Conference on Distributed and Event-Based Systems, New York, NY, USA, 2017, pp 249–260, https://doi.org/10.1145/3093742.3093912

  4. Antonić A, Marjanović M, Žarko IP (2017) Modeling aggregate input load of interoperable smart city services. In: Proceedings of the 11th ACM International Conference on Distributed and Event-Based Systems, New York, NY, USA, 2017, pp 34–43, https://doi.org/10.1145/3093742.3093928

  5. Soffer P et al (2017) From event streams to process models and back: challenges and opportunities. Inf Syst. https://doi.org/10.1016/j.is.2017.11.002

    Article  Google Scholar 

  6. Fardbastani MA, Allahdadi F, Sharifi M (2018) Business process monitoring via decentralized complex event processing. Enterprise Inf Syst. https://doi.org/10.1080/17517575.2018.1522453

    Article  Google Scholar 

  7. Saleh O, Betz H, Sattler K-U (2015) “Partitioning for scalable complex event processing on data streams”, in New Trends in Database and Information Systems II. Springer, Cham, pp 185–197

    Google Scholar 

  8. Gulisano V, Jimenez-Peris R, Patino-Martinez M, Valduriez P (2010) StreamCloud: a large scale data streaming system. In: Proceedings of the 2010 IEEE 30th International Conference on Distributed Computing Systems, Washington, DC, USA, 2010, pp 126–137, https://doi.org/10.1109/icdcs.2010.72

  9. Castro Fernandez R et al (2013) Integrating scale out and fault tolerance in stream processing using operator state management. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, New York, NY, USA, 2013, pp 725–736, https://doi.org/10.1145/2463676.2465282

  10. Heinze T, Jerzak Z, Hackenbroich G, Fetzer C (2014) Latency-aware elastic scaling for distributed data stream processing systems. In: Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, New York, NY, USA, 2014, pp 13–22, https://doi.org/10.1145/2611286.2611294

  11. Camacho EF, Bordons C (2007) Nonlinear model predictive control: an introductory review. In: Assessment and Future Directions of Nonlinear Model Predictive Control, R Findeisen, F Allgöwer, and LT Biegler, eds. Heidelberg: Springer, 2007, pp 1–16

  12. Gedik B (2014) Partitioning functions for stateful data parallelism in stream processing. VLDB J 23(4):517–539. https://doi.org/10.1007/s00778-013-0335-9

    Article  Google Scholar 

  13. Hirzel M (2012) Partition and compose: parallel complex event processing. In Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, New York, NY, USA, 2012, pp 191–200, https://doi.org/10.1145/2335484.2335506

  14. Isoyama K et al (2012) A scalable complex event processing system and evaluations of its performance. In: Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, New York, NY, USA, 2012, pp 123–126, https://doi.org/10.1145/2335484.2335498

  15. Wu S, Kumar V, Wu K-L, Ooi BC (2012) Parallelizing stateful operators in a distributed stream processing system: how, should you and how much? In: Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, New York, NY, USA, 2012, pp 278–289, https://doi.org/10.1145/2335484.2335515

  16. de Almeida VP, Bhowmik S, Endler M, Rothermel K (2020) DSCEP: an infrastructure for distributed semantic complex event processing. arXiv:2002.05869 [cs], Feb. 2020, Accessed: Sep. 14, 2020. [Online]. http://arxiv.org/abs/2002.05869

  17. Mayer R et al (2017) SPECTRE: Supporting consumption policies in window-based parallel complex event processing. In: Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference, New York, NY, USA, 2017, pp 161–173, https://doi.org/10.1145/3135974.3135983

  18. Mei Y, Madden S (2009) ZStream: a cost-based query processor for adaptively detecting composite events. In: Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data, New York, NY, USA, 2009, pp 193–206, https://doi.org/10.1145/1559845.1559867

  19. Fardbastani MA, Sharifi M (2019) Scalable complex event processing using adaptive load balancing. J Syst Softw 149:305–317. https://doi.org/10.1016/j.jss.2018.12.012

    Article  Google Scholar 

  20. Wang YH, Cao K, Zhang XM (2013) Complex event processing over distributed probabilistic event streams. Comput Math Appl 66(10):1808–1821. https://doi.org/10.1016/j.camwa.2013.06.032

    Article  MATH  Google Scholar 

  21. Xiao F, Zhan C, Lai H, Tao L, Qu Z (2017) New parallel processing strategies in complex event processing systems with data streams. Int J Distrib Sens Netw 13(8):1550147717728626. https://doi.org/10.1177/1550147717728626

    Article  Google Scholar 

  22. Ottenwälder B, Mayer R, Koldehofe B (2014) Distributed complex event processing for mobile large-scale video applications. In: Proceedings of the Posters & Demos Session, New York, NY, USA, 2014, pp 5–6, https://doi.org/10.1145/2678508.2678511

  23. Pathak R, Vaidehi V (2015) An efficient rule balancing for scalable complex event processing. In: 2015 IEEE 28th Canadian Conference on Electrical and Computer Engineering (CCECE), May 2015, pp 190–195, https://doi.org/10.1109/ccece.2015.7129184

  24. Choochotkaew S et al (2017) EdgeCEP: fully-distributed complex event processing on IoT edges. In: 2017 13th International Conference on Distributed Computing in Sensor Systems (DCOSS), Jun. 2017, pp 121–129, https://doi.org/10.1109/dcoss.2017.14

  25. Vanneste S et al (2019) Distributed uniform streaming framework: an elastic fog computing platform for event stream processing and platform transparency. Fut Internet 11(7):7. https://doi.org/10.3390/fi11070158

    Article  Google Scholar 

  26. Matteis T, Mencagli G (2017) Parallel patterns for window-based stateful operators on data streams: an algorithmic skeleton approach. Int J Parallel Program 45(2):382–401. https://doi.org/10.1007/s10766-016-0413-x

    Article  Google Scholar 

  27. Fernandez RC, Weidlich M, Pietzuch P, Gal A (2014) Scalable stateful stream processing for smart grids. In: Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, New York, NY, USA, 2014, pp 276–281, https://doi.org/10.1145/2611286.2611326

  28. Xiao F, Aritsugi M (2018) An adaptive parallel processing strategy for complex event processing systems over data streams in wireless sensor networks. Sensors 18(11):3732. https://doi.org/10.3390/s18113732

    Article  Google Scholar 

  29. Kolchinsky I, Schuster A (2018) Efficient adaptive detection of complex event patterns. Proc VLDB Endow 11(11):1346–1359. https://doi.org/10.14778/3236187.3236190

    Article  Google Scholar 

  30. Zacheilas N et al (2015) Elastic complex event processing exploiting prediction. In: 2015 IEEE International Conference on Big Data (Big Data), Oct. 2015, pp 213–222, https://doi.org/10.1109/bigdata.2015.7363758

  31. Weisenburger P et al (2017) Quality-aware runtime adaptation in complex event processing. In: 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) (pp 140–151). 2017. IEEE

  32. Zhao B (2018) Complex event processing under constrained resources by state-based load shedding. In: 2018 IEEE 34th International Conference on Data Engineering (ICDE), pp 1699–1703. IEEE, 2018

  33. “Storm, distributed and fault-tolerant realtime computation.” http://www.storm-project.net/ (Accessed Jan. 20, 2018)

  34. “Esper,” EsperTech. http://www.espertech.com/esper/ (Accessed Mar. 25, 2018)

  35. Talebi MM, Sharifi M (2019) PARS: a parallel model for scaled processing of complex events. J Syst Softw 155:1–16. https://doi.org/10.1016/j.jss.2019.05.014

    Article  Google Scholar 

  36. Fülöp LJ et al (2012) Predictive complex event processing: a conceptual framework for combining complex event processing and predictive analytics. In: Proceedings of the Fifth Balkan Conference in Informatics, New York, NY, USA, 2012, pp 26–31, https://doi.org/10.1145/2371316.2371323

  37. Kingman JF (1962) On queues in heavy traffic. J R Stat Soc Ser B Methodol 24(2):383–392

    MathSciNet  MATH  Google Scholar 

  38. De Matteis T, Mencagli G (2017) Proactive elasticity and energy awareness in data stream processing. J Syst Softw 127:302–319. https://doi.org/10.1016/j.jss.2016.08.037

    Article  Google Scholar 

  39. Camacho EF, Bordons C (2007) Nonlinear model predictive control: an introductory review. In: Assessment and Future Directions of Nonlinear Model Predictive Control, R Findeisen, F Allgöwer, and LT Biegler, eds. Heidelberg: Springer, 2007, pp 1–16

  40. Mencagli G, Vanneschi M (2014) Towards a systematic approach to the dynamic adaptation of structured parallel computations using model predictive control. Cluster Comput 17(4):1443–1463. https://doi.org/10.1007/s10586-014-0346-3

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Mohsen Sharifi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Talebi, M., Sharifi, M. & Kalantari, M. ACEP: an adaptive strategy for proactive and elastic processing of complex events. J Supercomput 77, 4718–4753 (2021). https://doi.org/10.1007/s11227-020-03454-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-020-03454-0

Keywords

Navigation