Issues in Benchmark Metric Selection

  • Alain Crolotte
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5895)


It is true that a metric can influence a benchmark but will esoteric metrics create more problems than they will solve? We answer this question affirmatively by examining the case of the TPC-D metric which used the much debated geometric mean for the single-stream test. We will show how a simple choice influenced the benchmark and its conduct and, to some extent, DBMS development. After examining other alternatives our conclusion is that the “real” measure for a decision-support benchmark is the arithmetic mean.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    TPC BENCHMARK D (Decision Support), Transaction Processing Council,
  2. 2.
    Fleming, P., Wallace, J.: How Not to Lie With Statistics: The Correct Way to Summarize Benchmarks. Comm. ACM 29(3), 218–221 (1986)CrossRefGoogle Scholar
  3. 3.
  4. 4.
    Crolotte, A.: Issues in Metric Selection and the TPC-D Single Stream Power,
  5. 5.
    Smith, J.: Characterizing Computer Performance with a Single Number. Communications of of the ACM 31(10) (October 1988)Google Scholar
  6. 6.
    Kurian John, L.: More on Finding a Single Number to indicate Overall Performance of a Benchmark Suite. ACM SIGARCH Computer Architecture News 32 (March 2004)Google Scholar
  7. 7.
    Citron, D., Hurani, D., Gnadrey, A.: The Harmonic or Geometric Mean: Does it Really Matter? ACM SIGARCH Computer Architecture News 34(4) (September 2006)Google Scholar
  8. 8.
    Mashey, J.: War of the benchmark Means: Time for a Truce. ACM SIGARCH Computer Architecture News 32(4) (September 2004)Google Scholar
  9. 9.
    Calot, G.: Cours de Statistique Descriptive, Dunod, Paris (1964)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Alain Crolotte
    • 1
  1. 1.Teradata Corporation

Personalised recommendations