Skip to main content
Log in

TPStream: low-latency and high-throughput temporal pattern matching on event streams

  • Published:
Distributed and Parallel Databases Aims and scope Submit manuscript

Abstract

Sequential pattern matching to detect a user-defined sequence of conditions on event streams is a key feature in modern event processing systems. However, the sequential nature of event based pattern matching has two major deficiencies. First, it is hardly possible to express complex temporal relationships between situations lasting for periods of time. Because events are equipped with a single timestamp only, the expressible temporal relations are limited to before/after/at the same time. Second, a sequential pattern is mapped to a continuous subsequence of the input stream starting with an arbitrary event, making efficient parallelization a hard problem. In this paper we present TPStream, a novel event processing operator for complex temporal pattern matching on event streams. TPStream first summarizes incoming events to situations lasting for periods of time, before it matches temporal patterns. With situations, temporal patterns can easily be defined based on Allen’s interval algebra. We also show that situation based temporal pattern matching can be efficiently executed in parallel using multiple threads on a single machine or multiple machines in a cluster. Finally, we present adaptive optimization components continuously tuning the execution strategy of TPStream towards the lowest possible result latency with respect to the overall system load. The results of our experimental evaluation show that TPStream is capable of processing high-volume event streams with both low latency and high throughput while outperforming applicable CEP solutions from academia and industry.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Notes

  1. http://www.iii.org/fact-statistic/aggressive-driving.

  2. https://kafka.apache.org/.

  3. Datasets and source code available at http://uni-marburg.de/oaCPk.

  4. http://www.espertech.com.

  5. https://github.com/haopeng/sase.

  6. https://zookeeper.apache.org/.

References

  1. Abadi, D.J., et al.: The design of the borealis stream processing engine. In: CIDR 2005, pp. 277–289 (2005)

  2. Ali, M.H., et al.: Microsoft CEP server and online behavioral targeting. Proc. VLDB Endow. 2(2), 1558–1561 (2009)

    Article  Google Scholar 

  3. Allen, J.F.: Maintaining knowledge about temporal intervals. Commun. ACM 26(11), 832–843 (1983)

    Article  Google Scholar 

  4. Anicic, D., Fodor, P., Rudolph, S., Stojanovic, N.: EP-SPARQL: a unified language for event processing and stream reasoning. In: Proceedings of the 20th International Conference on World Wide Web, WWW 2011, pp. 635–644 (2011)

  5. Appelrath, H.J., et al.: Odysseus: a highly customizable framework for creating efficient event stream management systems. In: DEBS’12, pp. 367–368 (2012)

  6. Arasu, A., Cherniack, M., Galvez, E., Maier, D., Maskey, A.: Linear road: a stream data management benchmark. In: VLDB’04, pp. 480 – 491 (2004)

  7. Avnur, R., Hellerstein, J.M.: Eddies: Continuously adaptive query processing. In: SIGMOD’00, pp. 261–272 (2000)

  8. Babu, S., Motwani, R., Munagala, K., Nishizawa, I., Widom, J.: Adaptive ordering of pipelined stream filters. In: SIGMOD’04, ACM, pp. 407–418 (2004)

  9. Babu, S., Srivastava, U., Widom, J.: Exploiting k-constraints to reduce memory overhead in continuous queries over data streams. ACM Trans. Database Syst. 29(3), 545–580 (2004)

    Article  Google Scholar 

  10. Balkesen, C., Dindar, N., Wetter, M., Tatbul, N.: RIP: run-based intra-query parallelism for scalable complex event processing. In: DEBS’13, pp. 3–14 (2013)

  11. Barbieri, D.F., Braga, D., Ceri, S., Della Valle, E., Grossniklaus, M.: C-sparql: Sparql for continuous querying. In: Proceedings of the 18th International Conference on World Wide Web, WWW ’09, pp. 1061–1062 (2009)

  12. Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache flink™: stream and batch processing in a single engine. IEEE Data Eng. Bull. 38(4), 28–38 (2015)

    Google Scholar 

  13. Chandramouli, B., Goldstein, J., Maier, D.: High-performance dynamic pattern matching over disordered streams. Proc. VLDB Endow. 3(1), 220–231 (2010)

    Article  Google Scholar 

  14. Cugola, G., Margara, A.: Low latency complex event processing on parallel hardware. J. Parallel Distrib. Comput. 72(2), 205–218 (2012)

    Article  Google Scholar 

  15. Dayarathna, M., Perera, S.: Recent advancements in event processing. ACM Comput. Surv. 51(2), 33:1–33:36 (2018)

    Article  Google Scholar 

  16. Demers, A.J., Gehrke, J., Panda, B., Riedewald, M., Sharma, V., White, W.M.: Cayuga: a general purpose event monitoring system. In: CIDR’07, pp. 412–422 (2007)

  17. Diao, Y., Immerman, N., Gyllstrom, D.: Sase+: an agile language for kleene closure over event streams. Tech. rep., University of Massachusetts (2007)

  18. Dignös, A., Böhlen, M.H., Gamper, J.: Overlap interval partition join. In: SIGMOD, 2014, 1459–1470 (2014)

  19. Erwig, M.: Toward spatio-temporal patterns. In: Spatio-Temporal Databases: Flexible Querying and Reasoning. Springer, Berlin, pp. 29–53 (2004)

  20. Etzion, O., Fournier, F., Skarbovsky, I., von Halle, B.: A model driven approach for event processing applications. In: DEBS’16, pp. 81–92 (2016)

  21. Flouris, I., Giatrakos, N., Deligiannakis, A., Garofalakis, M., Kamp, M., Mock, M.: Issues in complex event processing: status and prospects in the Big Data era. J. Syst. Softw. (2016)

  22. Gao, D., Jensen, C.S., Snodgrass, R.T., Soo, M.D.: Join operations in temporal databases. VLDB J. 14(1), 2–29 (2005)

    Article  Google Scholar 

  23. Gao, S., Scharrenbach, T., Kietz, J.U., Bernstein, A.: Running out of bindings? Integrating facts and events in linked data stream processing. CEUR Workshop Proc. 1488, 63–74 (2015)

    Google Scholar 

  24. Gao, S., Gu, J., Zaniolo, C.: RDF-TX: a fast, user-friendly system for querying the history of RDF knowledge bases. In: EDBT 2016, pp. 269–280 (2016)

  25. Ghanem, T.M., Aref, W.G., Elmagarmid, A.K.: Exploiting predicate-window semantics over data streams. SIGMOD Rec. 35(1), 3–8 (2006)

    Article  Google Scholar 

  26. Golab, L., Özsu, M.T.: Processing sliding window multi-joins in continuous queries over data streams. In: VLDB’03, pp. 500–511 (2003)

  27. Golab, L., Özsu, M.T.: Update-pattern-aware modeling and processing of continuous queries. In: SIGMOD’05, pp. 658–669 (2005)

  28. Grossniklaus, M., Maier, D., Miller, J., Moorthy, S., Tufte, K.: Frames: data-driven windows. In: DEBS’16, pp. 13–24 (2016)

  29. Helmer, S., Persia, F.: Iseql, an interval-based surveillance event query language. IJMDEM 7(4), 1–21 (2016)

    Google Scholar 

  30. Hinze, A., Voisard, A.: Eva: An event algebra supporting complex event specification. Inf. Syst. 48, 1–25 (2015)

    Article  Google Scholar 

  31. Hirzel, M.: Partition and compose: parallel complex event processing. In: DEBS’12, pp. 191–200 (2012)

  32. Hoßbach, B., Glombiewski, N., Morgen, A., Ritter, F., Seeger, B.: JEPC: the java event processing connectivity. Datenbank-Spektrum 13(3), 167–178 (2013)

    Article  Google Scholar 

  33. Jayasekara, S., Kannangara, S., Dahanayakage, T., Ranawaka, I., Perera, S., Nanayakkara, V.: Wihidum: distributed complex event processing. J. Parallel Distrib. Comput. 79–80, 42–51 (2015)

  34. Kaufmann, M., Manjili, A.A., Vagenas, P., Fischer, P.M., Kossmann, D., Färber, F., May, N.: Timeline index: a unified data structure for processing queries on temporal data in SAP HANA. In: SIGMOD 2013, pp. 1173–1184 (2013)

  35. Kietz, J., Scharrenbach, T., Fischer, L., Bernstein, A., Nguyen, K.: Tef-sparql: The ddis query-language for time annotated event and fact triple-streams. Tech. rep., Technical report, University of Zurich, Department of Informatics (2013)

  36. Körber, M., Glombiewski, N., Seeger, B.: Tpstream: low-latency temporal pattern matching on event streams. In: EDBT 2018, pp. 313–324 (2018)

  37. Li, M., Mani, M., Rundensteiner, E.A., Lin, T.: Complex event pattern detection over streams with interval-based temporal semantics. In: DEBS’11, pp. 291–302 (2011)

  38. Lin, Q., Ooi, B.C., Wang, Z., Yu, C.: Scalable distributed stream join processing. In: SIGMOD’15, pp. 811–825 (2015)

  39. Liu, M., Li, M., Golovnya, D., Rundensteiner, E.A., Claypool, K.: Sequence pattern query processing over out-of-order event streams. In: ICDE’09, pp. 784–795 (2009)

  40. Mei, Y., Madden, S.: ZStream: a cost-based query processor for adaptively detecting composite events. In: SIGMOD’09, pp. 193–206 (2009)

  41. Persia, F., Bettini, F., Helmer, S.: An interactive framework for video surveillance event detection and modeling. In: CIKM 2017, pp. 2515–2518 (2017)

  42. Piatov, D., Helmer, S., Dignös, A.: An interval join optimized for modern hardware. In: ICDE 2016, pp. 1098–1109 (2016)

  43. Poppe, O., Lei, C., Rundensteiner, E.A., Dougherty, D.J.: Context-aware event stream analytics. In: EDBT 2016, pp. 413–424 (2016)

  44. Ray, M., Lei, C., Rundensteiner, E.A.: Scalable pattern sharing on event streams. In: SIGMOD’16, pp. 495–510 (2016)

  45. Sakr, M.A., Güting, R.H.: Spatiotemporal pattern queries. GeoInformatica 15(3), 497–540 (2011)

    Article  Google Scholar 

  46. Schneider, S., Hirzel, M., Gedik, B., Wu, K.: Auto-parallelizing stateful distributed streaming applications. In: PACT ’12, pp. 53–64 (2012)

  47. Schultz-Møller, N.P., Migliavacca, M., Pietzuch, P.: Distributed complex event processing with query rewriting. In: DEBS’09, pp. 1–12 (2009)

  48. Zaharia, M., et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2012, pp. 15–28 (2012)

  49. Zhang, H., Diao, Y., Immerman, N.: On complexity and optimization of expensive queries in complex event processing. In: SIGMOD’14, ACM Press, pp. 217–228 (2014)

Download references

Acknowledgements

This work has been supported by the German Research Foundation (DFG) under Grant No. SE 553/9-1.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Michael Körber.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Körber, M., Glombiewski, N., Morgen, A. et al. TPStream: low-latency and high-throughput temporal pattern matching on event streams. Distrib Parallel Databases 39, 361–412 (2021). https://doi.org/10.1007/s10619-019-07272-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10619-019-07272-z

Keywords

Navigation