Abstract
Complex Event Processing (CEP) or stream data processing are becoming increasingly popular as the platform underlying event-driven solutions and applications in industries such as financial services, oil & gas, smart grids, health care, and IT monitoring. Satisfactory performance is crucial for any solution across these industries. Typically, performance of CEP engines is measured as (1) data rate, i.e., number of input events processed per second, and (2) latency, which denotes the time it takes for the result (output events) to emerge from the system after the business event (input event) happened. While data rates are typically easy to measure by capturing the numbers of input events over time, latency is less well defined. As it turns out, a definition becomes particularly challenging in the presence of data arriving out of order. That means that the order in which events arrive at the system is different from the order of their timestamps. Many important distributed scenarios need to deal with out-of-order arrival because communication delays easily introduce disorder.
With out-of-order arrival, a CEP system cannot produce final answers as events arrive. Instead, time first needs to progress enough in the overall system before correct results can be produced. This introduces additional latency beyond the time it takes the system to perform the processing of the events. We denote the former as information latency and the latter as system latency. This paper discusses both types of latency in detail and defines them formally without depending on particular semantics of the CEP query plans. In addition, the paper suggests incorporating these definitions as metrics into the benchmarks that are being used to assess and compare CEP systems.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ali, M.H., et al.: Microsoft CEP Server and Online Behavioral Targeting. PVLDB 2(2), 1558–1561 (2009)
Consortium for Electric Reliability Technology Solutions: Phasor Technology and Real-Time Dynamics Monitoring System, http://www.phasor-rtdms.com/downloads/guides/RTDMSFAQ.pdf
Microsoft Corp.: Smart Energy Reference Architecture, http://www.microsoft.com/enterprise/industry/power-utilities/solutions/smart-energy-reference-architecture.aspx
Securities Technology Analysis Center, http://www.stacresearch.com/
Chandramouli, B., Goldstein, J., Barga, R.S., Riedewald, M., Santos, I.: Accurate latency estimation in a distributed event processing system. In: ICDE 2011, pp. 255–266 (2011)
Babcock, B., Babu, S., Datar, M., Motwani, R.: Chain: Operator Scheduling for Memory Minimization in Data Stream Systems. In: SIGMOD Conference 2003, pp. 253–264 (2003)
Cammert, M., Krämer, J., Seeger, B., Vaupel, S.: A Cost-Based Approach to Adaptive Resource Management in Data Stream Systems. IEEE Trans. Knowl. Data Eng. 20(2), 230–245 (2008)
Tatbul, N., Çetintemel, U., Zdonik, S.B., Cherniack, M., Stonebraker, M.: Load Shedding in a Data Stream Manager. In: VLDB 2003, pp. 309–320 (2003)
Barga, R.S., Goldstein, J., Ali, M.H., Hong, M.: Consistent Streaming Through Time: A Vision for Event Stream Processing. In: CIDR 2007, pp. 363–374 (2007)
Progress Software: Apama, http://web.progress.com/en/apama/index.html
StreamBase, http://streambase.com/
SQLStream, http://sqlstream.com/
Options Price Reporting Authority, http://www.opradata.com/ , also see, http://en.wikipedia.org/wiki/Options_Price_Reporting_Authority
Microsoft Corp.: Microsoft CEP Overview, http://download.microsoft.com/download/F/D/5/FD5E855C-D895-45A8-9F3E-110AFADBE51A/Microsoft%20CEP%20Overview.docx
Cui, Y., Widom, J., Wiener, J.L.: Tracing the lineage of view data in a warehousing environment. ACM Trans. Database Syst. 25(2), 179–227 (2000)
Transaction Processing Performance Council: TPC-H, http://www.tpc.org/tpch/default.asp
Arasu, A., et al.: Linear Road: A Stream Data Management Benchmark. In: VLDB 2004, pp. 480–491 (2004)
Oracle Corp.: Oracle Complex Event Processing Performance, http://www.oracle.com/technetwork/middleware/complex-event-processing/overview/cepperformancewhitepaper-128060.pdf
Li, J., Tufte, K., Shkapenyuk, V., Papadimos, V., Johnson, T., Maier, D.: Out-of-order processing: a new architecture for high-performance stream systems. PVLDB 1(1), 274–288 (2008)
Sybase Inc.: Sybase Aleri Streaming Platform, http://www.sybase.com/products/financialservicessolutions/aleristreamingplatform
Fujimoto, R.M.: Distributed Simulation Systems. In: 35th Winter Simulation Conference 2003, 124–134 (2003)
Jefferson, D.R.: Virtual Time. ACM Trans. Program. Lang. Syst. 7(3), 404–425 (1985)
Grabs, T., Bauhaus, C.: Building Operational Intelligence Solutions with Microsoft SQL Server and StreamInsight. Microsoft TechEd Europe 2010, http://channel9.msdn.com/Events/TechEd/Europe/2010/DAT301-LNC
Geppert, A., Berndtsson, M., Lieuwen, D.F., Roncancio, C.: Performance Evaluation of Object-Oriented Active Database Systems Using the BEAST Benchmark. TAPOS 4(3), 135–149 (1998)
Sachs, K., Kounev, S., Bacon, J., Buchmann, A.: Performance evaluation of message-oriented middleware using the SPEC jms 2007 benchmark. Performance Evaluation 66(8) (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Grabs, T., Lu, M. (2012). Measuring Performance of Complex Event Processing Systems. In: Nambiar, R., Poess, M. (eds) Topics in Performance Evaluation, Measurement and Characterization. TPCTC 2011. Lecture Notes in Computer Science, vol 7144. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-32627-1_6
Download citation
DOI: https://doi.org/10.1007/978-3-642-32627-1_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-32626-4
Online ISBN: 978-3-642-32627-1
eBook Packages: Computer ScienceComputer Science (R0)