Introducing Statistical Design of Experiments to SPARQL Endpoint Evaluation

  • Kjetil Kjernsmo
  • John S. Tyssedal
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8219)


This paper argues that the common practice of benchmarking is inadequate as a scientific evaluation methodology. It further attempts to introduce the empirical tradition of the physical sciences by using techniques from Statistical Design of Experiments applied to the example of SPARQL endpoint performance evaluation. It does so by studying full as well as fractional factorial experiments designed to evaluate an assertion that some change introduced in a system has improved performance. This paper does not present a finished experimental design, rather its main focus is didactical, to shift the focus of the community away from benchmarking towards higher scientific rigor.


Orthogonal Array SPARQL Query Query Engine Normal Plot Full Factorial Experiment 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds.): Experimental Methods for the Analysis of Optimization Algorithms. Springer, Heidelberg (2010)zbMATHGoogle Scholar
  2. 2.
    Duan, S., Kementsietsidis, A., Srinivas, K., Udrea, O.: Apples and oranges: a comparison of RDF benchmarks and real RDF datasets. In: Proc. of the 2011 Int. Conf. on Management of Data, SIGMOD 2011, pp. 145–156. ACM, New York (2011), Google Scholar
  3. 3.
    Groemping, U.: DoE.base: Full factorials, orthogonal arrays and base utilities for DoE packages, R package version 0.23-4 (2013),
  4. 4.
    Groemping, U.: FrF2: Fractional Factorial designs with 2-level factors, R package version 1.6-6 (2013),
  5. 5.
    Görlitz, O., Thimm, M., Staab, S.: SPLODGE: Systematic Generation of SPARQL Benchmark Queries for Linked Open Data. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012, Part I. LNCS, vol. 7649, pp. 116–132. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  6. 6.
    Harris, S., Lamb, N., Shadbolt, N.: 4store: The design and implementation of a clustered RDF store. In: 5th International Workshop on Scalable Semantic Web Knowledge Base Systems, SSWS 2009 (2009)Google Scholar
  7. 7.
    Harris, S., Seaborne, A., Prud’hommeaux, E.: SPARQL 1.1 Query Language (2013),
  8. 8.
    Jain, R.K.: The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, 1st edn. Wiley (April 1991)Google Scholar
  9. 9.
    Lang, D.T.: R as a Web Client–the RCurl package. Journal of Statistical Software (2007)Google Scholar
  10. 10.
    Lenth, R.V.: Quick and Easy Analysis of Unreplicated Factorials. Technometrics 31(4), 469–473 (1989)Google Scholar
  11. 11.
    Montoya, G., Vidal, M.-E., Corcho, O., Ruckhaus, E., Buil-Aranda, C.: Benchmarking Federated SPARQL Query Engines: Are Existing Testbeds Enough? In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012, Part II. LNCS, vol. 7650, pp. 313–324. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  12. 12.
    Morsey, M., Lehmann, J., Auer, S., Ngonga Ngomo, A.-C.: DBpedia SPARQL Benchmark – Performance Assessment with Real Queries on Real Data. In: Aroyo, L., Welty, C., Alani, H., Taylor, J., Bernstein, A., Kagal, L., Noy, N., Blomqvist, E. (eds.) ISWC 2011, Part I. LNCS, vol. 7031, pp. 454–469. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  13. 13.
    R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2012) ISBN 3-900051-07-0,
  14. 14.
    Wu, C., Hamada, M.: Experiments: planning, analysis, ana optimization, 2nd edn. Wiley, New York (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Kjetil Kjernsmo
    • 1
  • John S. Tyssedal
    • 2
  1. 1.Department of InformaticsOsloNorway
  2. 2.Department of Mathematical SciencesNorwegian University of Science and TechnologyTrondheimNorway

Personalised recommendations