Issues in Benchmark Metric Selection
It is true that a metric can influence a benchmark but will esoteric metrics create more problems than they will solve? We answer this question affirmatively by examining the case of the TPC-D metric which used the much debated geometric mean for the single-stream test. We will show how a simple choice influenced the benchmark and its conduct and, to some extent, DBMS development. After examining other alternatives our conclusion is that the “real” measure for a decision-support benchmark is the arithmetic mean.
Unable to display preview. Download preview PDF.
- 1.TPC BENCHMARK D (Decision Support), Transaction Processing Council, http://www.tpc.org
- 3.Licea-Kane, B.: http://www.spec.org/gwpg/gpc.static/geometric.html
- 4.Crolotte, A.: Issues in Metric Selection and the TPC-D Single Stream Power, http://www.tpc.org
- 5.Smith, J.: Characterizing Computer Performance with a Single Number. Communications of of the ACM 31(10) (October 1988)Google Scholar
- 6.Kurian John, L.: More on Finding a Single Number to indicate Overall Performance of a Benchmark Suite. ACM SIGARCH Computer Architecture News 32 (March 2004)Google Scholar
- 7.Citron, D., Hurani, D., Gnadrey, A.: The Harmonic or Geometric Mean: Does it Really Matter? ACM SIGARCH Computer Architecture News 34(4) (September 2006)Google Scholar
- 8.Mashey, J.: War of the benchmark Means: Time for a Truce. ACM SIGARCH Computer Architecture News 32(4) (September 2004)Google Scholar
- 9.Calot, G.: Cours de Statistique Descriptive, Dunod, Paris (1964)Google Scholar