Seven Commandments for Benchmarking Semantic Flow Processing Systems

  • Thomas Scharrenbach
  • Jacopo Urbani
  • Alessandro Margara
  • Emanuele Della Valle
  • Abraham Bernstein
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7882)


Over the last few years, the processing of dynamic data has gained increasing attention in the Semantic Web community. This led to the development of several stream reasoning systems that enable on-the-fly processing of semantically annotated data that changes over time. Due to their streaming nature, analyzing such systems is extremely difficult. Currently, their evaluation is conducted under heterogeneous scenarios, hampering their comparison and an understanding of their benefits and limitations. In this paper, we strive for a better understanding of the key challenges that these systems must face and define a generic methodology to evaluate their performance. Specifically, we identify three Key Performance Indicators and seven commandments that specify how to design the stress tests for system evaluation.


Expressive Power Background Data Query Model Annotate Data Complex Event Processing 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Abadi, D., Carney, D., Çetintemel, U.U.G., Cherniack, M., Convey, C., Lee, S., Stonebraker, M., Tatbul, N., Zdonik, S.: Aurora: a new model and architecture for data stream management. VLDB J. 12(2), 120–139 (2003)CrossRefGoogle Scholar
  2. 2.
    Agrawal, J., Diao, Y., Gyllstrom, D., Immerman, N.: Efficient pattern matching over event streams. In: Wang, J.T.L. (ed.) Proc. SIGMOD 2008, p. 147. ACM (2008)Google Scholar
  3. 3.
    Allen, J.F.: Maintaining knowledge about temporal intervals. Commun. of the ACM 26(11), 832–843 (1983)zbMATHCrossRefGoogle Scholar
  4. 4.
    Anicic, D., Fodor, P., Rudolph, S., Stojanovic, N.: EP-SPARQL: a unified language for event processing and stream reasoning. In: Srinivasan, S., Ramamritham, K., Kumar, A., Ravindra, M.P., Bertino, E., Kumar, R. (eds.) Proc. WWW 2011, pp. 635–644. ACM (2011)Google Scholar
  5. 5.
    Arasu, A., Babcock, B., Babu, S., Datar, M., Ito, K., Nishizawa, I., Rosenstein, J., Widom, J.: STREAM: The Stanford Stream Data Manager. IEEE Data Eng. Bull., 19–26 (2003)Google Scholar
  6. 6.
    Arasu, A., Cherniack, M., Galvez, E., Maier, D., Maskey, A.S., Ryvkina, E., Stonebraker, M., Tibbetts, R.: Linear Road: A Stream Data Management Benchmark. VLDB J. (2004)Google Scholar
  7. 7.
    Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and issues in data stream systems. In: Popa, L., Abiteboul, S., Kolaitis, P.G. (eds.) Proc. PODS 2002, pp. 1–16. ACM (2002)Google Scholar
  8. 8.
    Bai, Y., Thakkar, H., Wang, H., Luo, C., Zaniolo, C.: A data stream language and system designed for power and extensibility. In: Yu, P.S., Tsotras, V.J., Fox, E.A., Liu, B. (eds.) Proc. CIKM 2006, pp. 337–346. ACM (2006)Google Scholar
  9. 9.
    Barbieri, D.F., Braga, D., Ceri, S., Della Valle, E., Grossniklaus, M.: Incremental reasoning on streams and rich background knowledge. In: Aroyo, L., Antoniou, G., Hyvönen, E., ten Teije, A., Stuckenschmidt, H., Cabral, L., Tudorache, T. (eds.) ESWC 2010, Part I. LNCS, vol. 6088, pp. 1–15. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  10. 10.
    Barbieri, D.F., Braga, D., Ceri, S., Della Valle, E., Grossniklaus, M.: C-SPARQL: A Continuous Query Language for RDF Data Streams. Int. J. of Semantic Computing 4(1), 3–25 (2010)zbMATHCrossRefGoogle Scholar
  11. 11.
    Barbieri, D., Braga, D., Ceri, S., Della Valle, E., Grossniklaus, M.: C-SPARQL: SPARQL for continuous querying. In: Quemada, J., León, G., Maarek, Y.S., Nejdl, W. (eds.) Proc. WWW 2009, pp. 1061–1062. ACM (2009)Google Scholar
  12. 12.
    Brenna, L., Demers, A., Gehrke, J., Hong, M., Ossher, J., Panda, B., Riedewald, M., Thatte, M., White, W.: Cayuga: A High-Performance Event Processing Engine. In: Chan, C.Y., Ooi, B.C., Zhou, A. (eds.) Proc. SIGMOD 2007, pp. 1100–1102. ACM (2007)Google Scholar
  13. 13.
    Chandrasekaran, S., Cooper, O., Deshpande, A., Franklin, M.J., Hellerstein, J.M., Hong, W., Krishnamurthy, S., Madden, S., Reiss, F., Shah, M.A.: TelegraphCQ: Continuous Dataflow Processing. In: Halevy, A.Y., Ives, Z.G., Doan, A. (eds.) Proc. SIGMOD 2003, p. 668. ACM (2003)Google Scholar
  14. 14.
    Cugola, G., Margara, A.: Complex event processing with T-REX. J. Syst. Softw. 85(8), 1709–1728 (2012)CrossRefGoogle Scholar
  15. 15.
    Cugola, G., Margara, A.: Processing Flows of Information: from Data Stream to Complex Event Processing. ACM Comput. Surv. 44(3), 1–62 (2012)CrossRefGoogle Scholar
  16. 16.
    Della Valle, E., Ceri, S., Milano, P., Van Harmelen, F.: It’s a Streaming World! Reasoning upon Rapidly Changing Information. J. Intell. Syst., IEEE (2009)Google Scholar
  17. 17.
    Etzion, O., Niblett, P.: Event Processing In Action. Manning Publications Co., Greenwich (2010)Google Scholar
  18. 18.
    Gray, J.: The Benchmark Handbook for Database and Transaction Systems, 2nd edn. Morgan Kaufmann (1993)Google Scholar
  19. 19.
    Hellerstein, J.M., Haas, P.J., Wang, H.J.: Online Aggregation. In: Peckham, J. (ed.) Proc. SIGMOD 1997, pp. 171–182. ACM (1997)Google Scholar
  20. 20.
    Le-Phuoc, D., Dao-Tran, M., Parreira, J.X., Hauswirth, M.: A Native and Adaptive Approach for Unified Processing of Linked Streams and Linked Data. In: Aroyo, L., Welty, C., Alani, H., Taylor, J., Bernstein, A., Kagal, L., Noy, N., Blomqvist, E. (eds.) ISWC 2011, Part I. LNCS, vol. 7031, pp. 370–388. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  21. 21.
    Le-Phuoc, D., Dao-Tran, M., Pham, M.-D., Boncz, P., Eiter, T., Fink, M.: Linked Stream Data Processing Engines: Facts and Figures. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012, Part II. LNCS, vol. 7650, pp. 300–312. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  22. 22.
    Luckham, D.: The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems. Addison-Wesley (2002)Google Scholar
  23. 23.
    Tichy, W.F., Lukowicz, P., Prechelt, L., Heinz, E.A.: A Quantitative Evaluation Study in Computer Science. J. Syst. and Softw. 28(1), 9–18 (1995)CrossRefGoogle Scholar
  24. 24.
    Wainer, J., Novoa Barsottini, C.G., Lacerda, D., Magalhães de Marco, L.R.: Empirical evaluation in Computer Science research published by ACM. J. Inform. and Softw. Tech. 51(6), 1081–1085 (2009)CrossRefGoogle Scholar
  25. 25.
    White, W., Riedewald, M., Gehrke, J., Demers, A.: What is ”next” in event processing? In: Libkin, L. (ed.) Proc. PODS 2007, pp. 263–272. ACM (2007)Google Scholar
  26. 26.
    Zhang, Y., Duc, P.M., Corcho, O., Calbimonte, J.-P.: SRBench: A Streaming RDF/SPARQL Benchmark. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012, Part I. LNCS, vol. 7649, pp. 641–657. Springer, Heidelberg (2012)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Thomas Scharrenbach
    • 1
  • Jacopo Urbani
    • 2
  • Alessandro Margara
    • 2
  • Emanuele Della Valle
    • 3
  • Abraham Bernstein
    • 1
  1. 1.University of ZurichSwitzerland
  2. 2.Vrije Universiteit AmsterdamThe Netherlands
  3. 3.Politecnico di MilanoItaly

Personalised recommendations