On Correctness in RDF Stream Processor Benchmarking

  • Daniele Dell’Aglio
  • Jean-Paul Calbimonte
  • Marco Balduini
  • Oscar Corcho
  • Emanuele Della Valle
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8219)


Two complementary benchmarks have been proposed so far for the evaluation and continuous improvement of RDF stream processors: SRBench and LSBench. They put a special focus on different features of the evaluated systems, including coverage of the streaming extensions of SPARQL supported by each processor, query processing throughput, and an early analysis of query evaluation correctness, based on comparing the results obtained by different processors for a set of queries. However, none of them has analysed the operational semantics of these processors in order to assess the correctness of query evaluation results. In this paper, we propose a characterization of the operational semantics of RDF stream processors, adapting well-known models used in the stream processing engine community: CQL and SECRET. Through this formalization, we address correctness in RDF stream processor benchmarks, allowing to determine the multiple answers that systems should provide. Finally, we present CSRBench, an extension of SRBench to address query result correctness verification using an automatic method.


Operational Semantic Input Stream SPARQL Query Continuous Query Window Slide 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Barbieri, D.F., Braga, D., Ceri, S., Della Valle, E., Grossniklaus, M.: C-SPARQL: A continuous query language for RDF data streams. IJSC 4(1), 3–25 (2010)zbMATHGoogle Scholar
  2. 2.
    Calbimonte, J.P., Jeung, H., Corcho, O., Aberer, K.: Enabling Query Technologies for the Semantic Sensor Web. IJSWIS 8(1), 43–63 (2012)Google Scholar
  3. 3.
    Le-Phuoc, D., Dao-Tran, M., Xavier Parreira, J., Hauswirth, M.: A native and adaptive approach for unified processing of linked streams and linked data. In: Aroyo, L., Welty, C., Alani, H., Taylor, J., Bernstein, A., Kagal, L., Noy, N., Blomqvist, E. (eds.) ISWC 2011, Part I. LNCS, vol. 7031, pp. 370–388. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  4. 4.
    Anicic, D., Fodor, P., Rudolph, S., Stojanovic, N.: EP-SPARQL: a unified language for event processing and stream reasoning. In: WWW, pp. 635–644 (2011)Google Scholar
  5. 5.
    Guo, Y., Pan, Z., Heflin, J.: LUBM: A benchmark for OWL knowledge base systems. Journal of Web Semantics 3(2-3), 158–182 (2005)CrossRefGoogle Scholar
  6. 6.
    Bizer, C., Schultz, A.: The berlin SPARQL benchmark. IJSWIS 5(2), 1–24 (2009)Google Scholar
  7. 7.
    Arasu, A., Cherniack, M., Galvez, E.F., Maier, D., Maskey, A., Ryvkina, E., Stonebraker, M., Tibbetts, R.: Linear Road: A Stream Data Management Benchmark. In: VLDB, pp. 480–491 (2004)Google Scholar
  8. 8.
    Le-Phuoc, D., Dao-Tran, M., Pham, M.-D., Boncz, P., Eiter, T., Fink, M.: Linked stream data processing engines: Facts and figures. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012, Part II. LNCS, vol. 7650, pp. 300–312. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  9. 9.
    Zhang, Y., Duc, P.M., Corcho, O., Calbimonte, J.-P.: SRBench: A Streaming RDF/SPARQL Benchmark. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012, Part I. LNCS, vol. 7649, pp. 641–657. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  10. 10.
    Arasu, A., Babu, S., Widom, J.: The CQL continuous query language: semantic foundations. The VLDB Journal 15(2), 121–142 (2006)CrossRefGoogle Scholar
  11. 11.
    Botan, I., Derakhshan, R., Dindar, N., Haas, L., Miller, R.J., Tatbul, N.: Secret: A model for analysis of the execution semantics of stream processing systems. PVLDB 3(1), 232–243 (2010)Google Scholar
  12. 12.
    Dell’Aglio, D., Balduini, M., Della Valle, E.: On the need to include functional testing in RDF stream engine benchmarks. In: BeRSys 2013 (2013)Google Scholar
  13. 13.
    Srivastava, U., Widom, J.: Flexible time management in data stream systems. In: PODS, New York, USA, p. 263 (2004)Google Scholar
  14. 14.
    Calbimonte, J.-P., Corcho, O., Gray, A.J.G.: Enabling ontology-based access to streaming data sources. In: Patel-Schneider, P.F., Pan, Y., Hitzler, P., Mika, P., Zhang, L., Pan, J.Z., Horrocks, I., Glimm, B. (eds.) ISWC 2010, Part I. LNCS, vol. 6496, pp. 96–111. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  15. 15.
    Gutierrez, C., Hurtado, C., Vaisman, A.: Introducing time into RDF. IEEE Transactions on Knowledge and Data Engineering 19(2), 207–218 (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Daniele Dell’Aglio
    • 1
  • Jean-Paul Calbimonte
    • 2
  • Marco Balduini
    • 1
  • Oscar Corcho
    • 2
  • Emanuele Della Valle
    • 1
  1. 1.Dipartimento di Elettronica, Informazione e BioingegneriaPolitecnico of MilanoItaly
  2. 2.Ontology Engineering GroupUniversidad Politécnica de MadridSpain

Personalised recommendations