TPCTC 2009: Performance Evaluation and Benchmarking pp 221-236 | Cite as
A Performance Study of Event Processing Systems
Abstract
Event processing engines are used in diverse mission-critical scenarios such as fraud detection, traffic monitoring, or intensive care units. However, these scenarios have very different operational requirements in terms of, e.g., types of events, queries/patterns complexity, throughput, latency and number of sources and sinks. What are the performance bottlenecks? Will performance degrade gracefully with increasing loads? In this paper we make a first attempt to answer these questions by running several micro-benchmarks on three different engines, while we vary query parameters like window size, window expiration type, predicate selectivity, and data values. We also perform some experiments to assess engines scalability with respect to number of queries and propose ways for evaluating their ability in adapting to changes in load conditions. Lastly, we show that similar queries have widely different performances on the same or different engines and that no engine dominates the other two in all scenarios.
Keywords
Benchmarking Complex Event Processing Micro-benchmarksPreview
Unable to display preview. Download preview PDF.
References
- 1.Abadi, D.J., et al.: Aurora. A New Model and Architecture for Data Stream Management. VLDB Journal 12, 120–139 (2003)CrossRefGoogle Scholar
- 2.Arasu, A., et al.: STREAM: The Stanford Stream Data Manager. In: Proc. SIGMOD 2003 (2003)Google Scholar
- 3.Arasu, A., et al.: Linear Road: A Stream Data Management Benchmark. In: Proc. of VLDB 2004 (2004)Google Scholar
- 4.Babcock, B., et al.: Models and Issues in Data Stream Systems. In: Proc. of SIGMOD 2002 (2002)Google Scholar
- 5.Berndtsson, M., et al.: Performance Evaluation of Object-Oriented Active Database Management Systems Using the BEAST Benchmark. Theory and Practice of Object Systems 4(3), 135–149 (1998)CrossRefGoogle Scholar
- 6.Bizarro, P., et al.: Event Processing Use Cases. In: Tutorial, DEBS 2009, Nashville USA (2009)Google Scholar
- 7.Chandrasekaran, S., et al.: TelegraphCQ: Continuous dataflow processing for an uncertain world. In: Proc. of CIDR 2003 (2003)Google Scholar
- 8.Dekkers, P.: Master Thesis Computer Science. Complex Event Processing. Radboud University Nijmegen, Thesis number 574 (October 2007)Google Scholar
- 9.DSAL Real-Time Event Processing Benchmark, http://www.datastreamanalysis.com/images/Real-Time%20EP%20Benchmark.pdf
- 10.Chakravarthy, S., Mishra, D.: Snoop: An Expressive Event Specification Language for Active Databases. Data Knowl. Eng. (DKE) 14(1), 1–26 (1994)CrossRefGoogle Scholar
- 11.Esper, http://esper.codehaus.org/
- 12.Golab, L., Özsu, M.T.: Issues in data stream management. SIGMOD Record 32(2), 5–14 (2003)CrossRefGoogle Scholar
- 13.Gray, J. (ed.): The Benchmark Handbook for Database and Transaction Processing Systems, 2nd edn. Morgan Kaufmann, San Francisco (1993)MATHGoogle Scholar
- 14.Gray, J., et al.: Data Cube: A Relational Aggregation Operator Generalizing Group-by, Cross-Tab, and Sub Totals. Data Min. Knowl. Discov. 1(1), 29–53 (1997)CrossRefGoogle Scholar
- 15.Mendes, M.R.N., Bizarro, P., Marques, P.: A Framework for Performance Evaluation of Complex Event Processing Systems. In: Proc. of DEBS 2008 (2008)Google Scholar
- 16.Motwani, R., et al.: Query Processing, Resource Management, and Approximation in a Data Stream Management System. In: Proc. of CIDR 2003 (2003)Google Scholar
- 17.NEXMark Benchmark, http://datalab.cs.pdx.edu/niagara/NEXMark/
- 18.Sachs, K., Kounev, S., Bacon, J.M., Buchmann, A.: Workload Characterization of the SPECjms2007 Benchmark. In: Wolter, K. (ed.) EPEW 2007. LNCS, vol. 4748, pp. 228–244. Springer, Heidelberg (2007)CrossRefGoogle Scholar
- 19.STAC-A1 Benchmark, http://www.stacresearch.com/council
- 20.STAC Report: Aleri Order Book Consolidation on Intel Tigertown and Solaris 10, http://www.stacresearch.com/node/3844
- 21.Stream Query Repository, http://infolab.stanford.edu/stream/sqr/
- 22.White, S., Alves, A., Rorke, D.: WebLogic event server: a lightweight, modular application server for event processing. In: Proc. of DEBS 2008 (2008)Google Scholar
- 23.Wu, E., Diao, Y., Rizvi, S.: High Performance Complex Event Processing over Streams. In: Proc. of SIGMOD 2006 (2006)Google Scholar