Abstract
Sequential pattern matching to detect a user-defined sequence of conditions on event streams is a key feature in modern event processing systems. However, the sequential nature of event based pattern matching has two major deficiencies. First, it is hardly possible to express complex temporal relationships between situations lasting for periods of time. Because events are equipped with a single timestamp only, the expressible temporal relations are limited to before/after/at the same time. Second, a sequential pattern is mapped to a continuous subsequence of the input stream starting with an arbitrary event, making efficient parallelization a hard problem. In this paper we present TPStream, a novel event processing operator for complex temporal pattern matching on event streams. TPStream first summarizes incoming events to situations lasting for periods of time, before it matches temporal patterns. With situations, temporal patterns can easily be defined based on Allen’s interval algebra. We also show that situation based temporal pattern matching can be efficiently executed in parallel using multiple threads on a single machine or multiple machines in a cluster. Finally, we present adaptive optimization components continuously tuning the execution strategy of TPStream towards the lowest possible result latency with respect to the overall system load. The results of our experimental evaluation show that TPStream is capable of processing high-volume event streams with both low latency and high throughput while outperforming applicable CEP solutions from academia and industry.
Similar content being viewed by others
Notes
References
Abadi, D.J., et al.: The design of the borealis stream processing engine. In: CIDR 2005, pp. 277–289 (2005)
Ali, M.H., et al.: Microsoft CEP server and online behavioral targeting. Proc. VLDB Endow. 2(2), 1558–1561 (2009)
Allen, J.F.: Maintaining knowledge about temporal intervals. Commun. ACM 26(11), 832–843 (1983)
Anicic, D., Fodor, P., Rudolph, S., Stojanovic, N.: EP-SPARQL: a unified language for event processing and stream reasoning. In: Proceedings of the 20th International Conference on World Wide Web, WWW 2011, pp. 635–644 (2011)
Appelrath, H.J., et al.: Odysseus: a highly customizable framework for creating efficient event stream management systems. In: DEBS’12, pp. 367–368 (2012)
Arasu, A., Cherniack, M., Galvez, E., Maier, D., Maskey, A.: Linear road: a stream data management benchmark. In: VLDB’04, pp. 480 – 491 (2004)
Avnur, R., Hellerstein, J.M.: Eddies: Continuously adaptive query processing. In: SIGMOD’00, pp. 261–272 (2000)
Babu, S., Motwani, R., Munagala, K., Nishizawa, I., Widom, J.: Adaptive ordering of pipelined stream filters. In: SIGMOD’04, ACM, pp. 407–418 (2004)
Babu, S., Srivastava, U., Widom, J.: Exploiting k-constraints to reduce memory overhead in continuous queries over data streams. ACM Trans. Database Syst. 29(3), 545–580 (2004)
Balkesen, C., Dindar, N., Wetter, M., Tatbul, N.: RIP: run-based intra-query parallelism for scalable complex event processing. In: DEBS’13, pp. 3–14 (2013)
Barbieri, D.F., Braga, D., Ceri, S., Della Valle, E., Grossniklaus, M.: C-sparql: Sparql for continuous querying. In: Proceedings of the 18th International Conference on World Wide Web, WWW ’09, pp. 1061–1062 (2009)
Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache flink™: stream and batch processing in a single engine. IEEE Data Eng. Bull. 38(4), 28–38 (2015)
Chandramouli, B., Goldstein, J., Maier, D.: High-performance dynamic pattern matching over disordered streams. Proc. VLDB Endow. 3(1), 220–231 (2010)
Cugola, G., Margara, A.: Low latency complex event processing on parallel hardware. J. Parallel Distrib. Comput. 72(2), 205–218 (2012)
Dayarathna, M., Perera, S.: Recent advancements in event processing. ACM Comput. Surv. 51(2), 33:1–33:36 (2018)
Demers, A.J., Gehrke, J., Panda, B., Riedewald, M., Sharma, V., White, W.M.: Cayuga: a general purpose event monitoring system. In: CIDR’07, pp. 412–422 (2007)
Diao, Y., Immerman, N., Gyllstrom, D.: Sase+: an agile language for kleene closure over event streams. Tech. rep., University of Massachusetts (2007)
Dignös, A., Böhlen, M.H., Gamper, J.: Overlap interval partition join. In: SIGMOD, 2014, 1459–1470 (2014)
Erwig, M.: Toward spatio-temporal patterns. In: Spatio-Temporal Databases: Flexible Querying and Reasoning. Springer, Berlin, pp. 29–53 (2004)
Etzion, O., Fournier, F., Skarbovsky, I., von Halle, B.: A model driven approach for event processing applications. In: DEBS’16, pp. 81–92 (2016)
Flouris, I., Giatrakos, N., Deligiannakis, A., Garofalakis, M., Kamp, M., Mock, M.: Issues in complex event processing: status and prospects in the Big Data era. J. Syst. Softw. (2016)
Gao, D., Jensen, C.S., Snodgrass, R.T., Soo, M.D.: Join operations in temporal databases. VLDB J. 14(1), 2–29 (2005)
Gao, S., Scharrenbach, T., Kietz, J.U., Bernstein, A.: Running out of bindings? Integrating facts and events in linked data stream processing. CEUR Workshop Proc. 1488, 63–74 (2015)
Gao, S., Gu, J., Zaniolo, C.: RDF-TX: a fast, user-friendly system for querying the history of RDF knowledge bases. In: EDBT 2016, pp. 269–280 (2016)
Ghanem, T.M., Aref, W.G., Elmagarmid, A.K.: Exploiting predicate-window semantics over data streams. SIGMOD Rec. 35(1), 3–8 (2006)
Golab, L., Özsu, M.T.: Processing sliding window multi-joins in continuous queries over data streams. In: VLDB’03, pp. 500–511 (2003)
Golab, L., Özsu, M.T.: Update-pattern-aware modeling and processing of continuous queries. In: SIGMOD’05, pp. 658–669 (2005)
Grossniklaus, M., Maier, D., Miller, J., Moorthy, S., Tufte, K.: Frames: data-driven windows. In: DEBS’16, pp. 13–24 (2016)
Helmer, S., Persia, F.: Iseql, an interval-based surveillance event query language. IJMDEM 7(4), 1–21 (2016)
Hinze, A., Voisard, A.: Eva: An event algebra supporting complex event specification. Inf. Syst. 48, 1–25 (2015)
Hirzel, M.: Partition and compose: parallel complex event processing. In: DEBS’12, pp. 191–200 (2012)
Hoßbach, B., Glombiewski, N., Morgen, A., Ritter, F., Seeger, B.: JEPC: the java event processing connectivity. Datenbank-Spektrum 13(3), 167–178 (2013)
Jayasekara, S., Kannangara, S., Dahanayakage, T., Ranawaka, I., Perera, S., Nanayakkara, V.: Wihidum: distributed complex event processing. J. Parallel Distrib. Comput. 79–80, 42–51 (2015)
Kaufmann, M., Manjili, A.A., Vagenas, P., Fischer, P.M., Kossmann, D., Färber, F., May, N.: Timeline index: a unified data structure for processing queries on temporal data in SAP HANA. In: SIGMOD 2013, pp. 1173–1184 (2013)
Kietz, J., Scharrenbach, T., Fischer, L., Bernstein, A., Nguyen, K.: Tef-sparql: The ddis query-language for time annotated event and fact triple-streams. Tech. rep., Technical report, University of Zurich, Department of Informatics (2013)
Körber, M., Glombiewski, N., Seeger, B.: Tpstream: low-latency temporal pattern matching on event streams. In: EDBT 2018, pp. 313–324 (2018)
Li, M., Mani, M., Rundensteiner, E.A., Lin, T.: Complex event pattern detection over streams with interval-based temporal semantics. In: DEBS’11, pp. 291–302 (2011)
Lin, Q., Ooi, B.C., Wang, Z., Yu, C.: Scalable distributed stream join processing. In: SIGMOD’15, pp. 811–825 (2015)
Liu, M., Li, M., Golovnya, D., Rundensteiner, E.A., Claypool, K.: Sequence pattern query processing over out-of-order event streams. In: ICDE’09, pp. 784–795 (2009)
Mei, Y., Madden, S.: ZStream: a cost-based query processor for adaptively detecting composite events. In: SIGMOD’09, pp. 193–206 (2009)
Persia, F., Bettini, F., Helmer, S.: An interactive framework for video surveillance event detection and modeling. In: CIKM 2017, pp. 2515–2518 (2017)
Piatov, D., Helmer, S., Dignös, A.: An interval join optimized for modern hardware. In: ICDE 2016, pp. 1098–1109 (2016)
Poppe, O., Lei, C., Rundensteiner, E.A., Dougherty, D.J.: Context-aware event stream analytics. In: EDBT 2016, pp. 413–424 (2016)
Ray, M., Lei, C., Rundensteiner, E.A.: Scalable pattern sharing on event streams. In: SIGMOD’16, pp. 495–510 (2016)
Sakr, M.A., Güting, R.H.: Spatiotemporal pattern queries. GeoInformatica 15(3), 497–540 (2011)
Schneider, S., Hirzel, M., Gedik, B., Wu, K.: Auto-parallelizing stateful distributed streaming applications. In: PACT ’12, pp. 53–64 (2012)
Schultz-Møller, N.P., Migliavacca, M., Pietzuch, P.: Distributed complex event processing with query rewriting. In: DEBS’09, pp. 1–12 (2009)
Zaharia, M., et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2012, pp. 15–28 (2012)
Zhang, H., Diao, Y., Immerman, N.: On complexity and optimization of expensive queries in complex event processing. In: SIGMOD’14, ACM Press, pp. 217–228 (2014)
Acknowledgements
This work has been supported by the German Research Foundation (DFG) under Grant No. SE 553/9-1.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Körber, M., Glombiewski, N., Morgen, A. et al. TPStream: low-latency and high-throughput temporal pattern matching on event streams. Distrib Parallel Databases 39, 361–412 (2021). https://doi.org/10.1007/s10619-019-07272-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10619-019-07272-z