Introducing Statistical Design of Experiments to SPARQL Endpoint Evaluation
This paper argues that the common practice of benchmarking is inadequate as a scientific evaluation methodology. It further attempts to introduce the empirical tradition of the physical sciences by using techniques from Statistical Design of Experiments applied to the example of SPARQL endpoint performance evaluation. It does so by studying full as well as fractional factorial experiments designed to evaluate an assertion that some change introduced in a system has improved performance. This paper does not present a finished experimental design, rather its main focus is didactical, to shift the focus of the community away from benchmarking towards higher scientific rigor.
Unable to display preview. Download preview PDF.
- 3.Groemping, U.: DoE.base: Full factorials, orthogonal arrays and base utilities for DoE packages, R package version 0.23-4 (2013), http://CRAN.R-project.org/package=DoE.base
- 4.Groemping, U.: FrF2: Fractional Factorial designs with 2-level factors, R package version 1.6-6 (2013), http://CRAN.R-project.org/package=FrF2
- 6.Harris, S., Lamb, N., Shadbolt, N.: 4store: The design and implementation of a clustered RDF store. In: 5th International Workshop on Scalable Semantic Web Knowledge Base Systems, SSWS 2009 (2009)Google Scholar
- 7.Harris, S., Seaborne, A., Prud’hommeaux, E.: SPARQL 1.1 Query Language (2013), http://www.w3.org/TR/2013/REC-sparql11-query-20130321/
- 8.Jain, R.K.: The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, 1st edn. Wiley (April 1991)Google Scholar
- 9.Lang, D.T.: R as a Web Client–the RCurl package. Journal of Statistical Software (2007)Google Scholar
- 10.Lenth, R.V.: Quick and Easy Analysis of Unreplicated Factorials. Technometrics 31(4), 469–473 (1989)Google Scholar
- 12.Morsey, M., Lehmann, J., Auer, S., Ngonga Ngomo, A.-C.: DBpedia SPARQL Benchmark – Performance Assessment with Real Queries on Real Data. In: Aroyo, L., Welty, C., Alani, H., Taylor, J., Bernstein, A., Kagal, L., Noy, N., Blomqvist, E. (eds.) ISWC 2011, Part I. LNCS, vol. 7031, pp. 454–469. Springer, Heidelberg (2011)CrossRefGoogle Scholar
- 13.R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2012) ISBN 3-900051-07-0, http://www.R-project.org/
- 14.Wu, C., Hamada, M.: Experiments: planning, analysis, ana optimization, 2nd edn. Wiley, New York (2009)Google Scholar