Skip to main content
Book cover

European Semantic Web Conference

ESWC 2021: The Semantic Web pp 664–679Cite as

KOBE: Cloud-Native Open Benchmarking Engine for Federated Query Processors

Part of the Lecture Notes in Computer Science book series (LNISA,volume 12731)

Abstract

In the SPARQL query processing community, as well as in the wider databases community, benchmark reproducibility is based on releasing datasets and query workloads. However, this paradigm breaks down for federated query processors, as these systems do not manage the data they serve to their clients but provide a data-integration abstraction over the actual query processors that are in direct contact with the data. As a consequence, benchmark results can be greatly affected by the performance and characteristics of the underlying data services. This is further aggravated when one considers benchmarking in more realistic conditions, where internet latency and throughput between the federator and the federated data sources is also a key factor. In this paper we present KOBE, a benchmarking system that leverages modern containerization and Cloud computing technologies in order to reproduce collections of data sources. In KOBE, data sources are formally described in more detail than what is conventionally provided, covering not only the data served but also the specific software that serves it and its configuration as well as the characteristics of the network that connects them. KOBE provides a specification formalism and a command-line interface that completely hides from the user the mechanics of provisioning and orchestrating the benchmarking process on Kubernetes-based infrastructures; and of simulating network latency. Finally, KOBE automates the process of collecting and comprehending logs, and extracting and visualizing evaluation metrics from these logs.

Keywords

  • Benchmarking
  • Federated query processing
  • Cloud-native

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-77385-4_40
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   109.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-77385-4
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   139.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.

Notes

  1. 1.

    Previously demonstrated in ISWC 2020, with extended abstract proceedings [6].

  2. 2.

    See https://github.com/semagrow/kobe.

  3. 3.

    cf. https://kubernetes.io.

  4. 4.

    cf. https://istio.io.

  5. 5.

    cf. https://www.elastic.co/elasticsearch.

  6. 6.

    cf. https://www.fluentd.org.

  7. 7.

    cf. https://www.elastic.co/kibana.

  8. 8.

    https://semagrow.github.io/kobe/references/api.

  9. 9.

    cf. https://virtuoso.openlinksw.com.

  10. 10.

    cf. http://strabon.di.uoa.gr.

  11. 11.

    https://semagrow.github.io/kobe/use/create_benchmark.

  12. 12.

    https://semagrow.github.io/kobe/use/tune_network.

  13. 13.

    https://semagrow.github.io/kobe/use/create_experiment.

  14. 14.

    https://semagrow.github.io/kobe/extend/add_evaluator.

  15. 15.

    https://semagrow.github.io/kobe/extend/add_metrics.

  16. 16.

    https://semagrow.github.io/kobe/extend/add_dataset_server.

  17. 17.

    https://semagrow.github.io/kobe/extend/add_federator.

  18. 18.

    Specifically, see the first step of the walk-through for adding a new federator. See also details about collecting logs to compute evaluation metrics https://semagrow.github.io/kobe/extend/support_metrics.

References

  1. Acosta, M., Vidal, M.-E., Sure-Vetter, Y.: Diefficiency metrics: measuring the continuous efficiency of query processing approaches. In: d’Amato, C., et al. (eds.) ISWC 2017, Part II. LNCS, vol. 10588, pp. 3–19. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68204-4_1

    CrossRef  Google Scholar 

  2. Charalambidis, A., Troumpoukis, A., Konstantopoulos, S.: SemaGrow: optimizing federated SPARQL queries. In: Proceedings of the 11th International Conference on Semantic Systems (SEMANTiCS 2015), Vienna, Austria, Sept 2015 (2015)

    Google Scholar 

  3. Garbis, G., Kyzirakos, K., Koubarakis, M.: Geographica: a benchmark for geospatial RDF stores (long version). In: Alani, H., et al. (eds.) ISWC 2013, Part II. LNCS, vol. 8219, pp. 343–359. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-41338-4_22

    CrossRef  Google Scholar 

  4. Görlitz, O., Staab, S.: SPLENDID: SPARQL endpoint federation exploiting VOID descriptions. In: Proceedings of the 2nd International Workshop on Consuming Linked Data (COLD 2011), vol. 782, Bonn, Germany, Oct 2011. CEUR (2011)

    Google Scholar 

  5. Guo, Y., Pan, Z., Heflin, J.: LUBM: a benchmark for OWL knowledge base systems. Web Semant. 3(2) (2005). https://doi.org/10.1016/j.websem.2005.06.005

  6. Kostopoulos, C., Mouchakis, G., Prokopaki-Kostopoulou, N., Troumpoukis, A., Charalambidis, A., Konstantopoulos, S.: KOBE: Cloud-native open benchmarking engine for federated query processors. Posters & Demos Session, ISWC 2020 (2020)

    Google Scholar 

  7. Ngonga Ngomo, A.C., Röder, M.: HOBBIT: Holistic benchmarking for big linked data. In: Processings of the ESWC 2016 EU Networking Session (2016)

    Google Scholar 

  8. Saleem, M., Hasnain, A., Ngonga Ngomo, A.C.: BigRDFBench: A billion triples benchmark for SPARQL endpoint federation

    Google Scholar 

  9. Schmidt, M., Görlitz, O., Haase, P., Ladwig, G., Schwarte, A., Tran, T.: FedBench: a benchmark suite for federated semantic data query processing. In: Aroyo, L., et al. (eds.) ISWC 2011, Part I. LNCS, vol. 7031, pp. 585–600. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25073-6_37

    CrossRef  Google Scholar 

  10. Schwarte, A., Haase, P., Hose, K., Schenkel, R., Schmidt, M.: FedX: a federation layer for distributed query processing on linked open data. In: Antoniou, G., et al. (eds.) ESWC 2011, Part II. LNCS, vol. 6644, pp. 481–486. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21064-8_39

    CrossRef  Google Scholar 

  11. Troumpoukis, A., et al.: Developing a benchmark suite for semantic web data from existing workflows. In: Proceedings of the Benchmarking Linked Data Workshop (BLINK), (ISWC 2016), Kobe, Japan, Oct 2016 (2016)

    Google Scholar 

  12. Troumpoukis, A., et al.: GeoFedBench: a benchmark for federated GeoSPARQL query processors. In: Proceedings Posters & Demos Session of ISWC 2020 (2020)

    Google Scholar 

  13. Verborgh, R., et al.: Triple pattern fragments: a low-cost knowledge graph interface for the web. J. Web Semant. 37–38, 184–206 (2016)

    Google Scholar 

Download references

Acknowledgments

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 825258. Please see http://earthanalytics.eu for more details.

Author information

Authors and Affiliations

  1. Institute and Informatics and Telecommunications, NCSR “Demokritos”, Agia Paraskevi, Greece

    Charalampos Kostopoulos, Giannis Mouchakis, Antonis Troumpoukis, Nefeli Prokopaki-Kostopoulou, Angelos Charalambidis & Stasinos Konstantopoulos

Authors
  1. Charalampos Kostopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giannis Mouchakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Antonis Troumpoukis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nefeli Prokopaki-Kostopoulou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Angelos Charalambidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Stasinos Konstantopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stasinos Konstantopoulos .

Editor information

Editors and Affiliations

  1. Ghent University, Ghent, Belgium

    Ruben Verborgh

  2. Aalborg University, Aalborg, Denmark

    Katja Hose

  3. University of Mannheim, Mannheim, Germany

    Heiko Paulheim

  4. ERCIM, Sophia Antipolis, France

    Pierre-Antoine Champin

  5. University of Siegen, Siegen, Germany

    Maria Maleshkova

  6. Universidad Politécnica de Madrid, Boadilla del Monte, Spain

    Oscar Corcho

  7. eBay Inc., San Jose, CA, USA

    Petar Ristoski

  8. FIZ Karlsruhe - Leibniz Institute for Information Infrastructure, Eggenstein-Leopoldshafen, Germany

    Dr. Mehwish Alam

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Kostopoulos, C., Mouchakis, G., Troumpoukis, A., Prokopaki-Kostopoulou, N., Charalambidis, A., Konstantopoulos, S. (2021). KOBE: Cloud-Native Open Benchmarking Engine for Federated Query Processors. In: , et al. The Semantic Web. ESWC 2021. Lecture Notes in Computer Science(), vol 12731. Springer, Cham. https://doi.org/10.1007/978-3-030-77385-4_40

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77385-4_40

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77384-7

  • Online ISBN: 978-3-030-77385-4

  • eBook Packages: Computer ScienceComputer Science (R0)