Encyclopedia of Big Data Technologies

Living Edition
| Editors: Sherif Sakr, Albert Zomaya

Metrics for Big Data Benchmarks

  • Alain Crolotte
Living reference work entry
DOI: https://doi.org/10.1007/978-3-319-63962-8_122-1



A big data (BD) benchmark metric is a standard measure indicating the adequacy and cost-effectiveness of the system under test (SUT) to perform a particular big data task or set of tasks.


The following section provides a survey of the big data benchmark and associated metrics evolution over the years until present day. In the foundations section we list all the main metrics while the following sections present the definitions and properties for each metrics category namely, response time and throughput, availability and reliability, price-performance and system-level metrics. We then go over the various methods to aggregate these individual metrics while the criticisms section reviews the metrics surveyed. We then list the key applications of these benchmarks. Finally cross-references and references are provided.

Historical Background

The need for computer performance metrics was identified as early as 1985 (Anon 1985...

This is a preview of subscription content, log in to check access.


  1. Anon et al (1985) A measure of transaction processing power. Datamation, 31 pp. 112–118, 1 April 1985Google Scholar
  2. Crolotte A (2009) Issues in benchmark metric selection. In: TPCTC, Lyon, pp 146–152Google Scholar
  3. Fleming P, Wallace J (1986) How to not lie with statistics: the correct way to summarize benchmark results. Commun ACM 29:218–221CrossRefGoogle Scholar
  4. Ghazal A, Rabl T, Hu M, Raab F, Poess M, Crolotte A, Jacobsen HA (2013) BigBench: towards an industry standard benchmark for big data analytics. In: SIGMODGoogle Scholar
  5. Ghazal, A., Ivanov, T., Kostamaa, P., Crolotte, A., Voong, R., Al-Kateb, M., Ghazal, W., Zicari, R. (2017) BigBench V2 – the new and improved BigBench. In: SIGMODGoogle Scholar
  6. Han R, Kurian L, Zhan J (2017) Benchmarking big data systems: a review. IEEE Trans Serv Comput, (99):1–18Google Scholar
  7. Huang S, Huang J, Dai J, Xie T and Huang B (2010) The HiBench benchmark suite: characterization of the mapreduce-based data analysis. In: ICDEW, Mar 2010Google Scholar
  8. Huppler K (2009) The art of building a good benchmark. In: Performance evaluation and benchmarking, vol 5895. Springer, Berlin/Heidellberg, pp 18–30Google Scholar
  9. Ivanov T, Rabl T, Poess M, Queralt A, Poelman J, Poggi N, Buell J (2015) Big data benchmark compendium. In: 7th TPC technology conference. (TPCTC)Google Scholar
  10. Laney D (2012) Deja VVVu: others claiming Gartner’s construct for big data. Gartner Blog Netw. 14 Jan 2012. https://blogs.gartner.com/doug-laney/deja-vvvue-others-claiming-gartners-volume-velocity-variety-construct-for-big-data/

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Teradata CorporationEl SegundoUSA

Section editors and affiliations

  • Meikel Poess
    • 1
  • Tilmann Rabl
    • 2
  1. 1.Server TechnologiesOracleRedwood ShoresUnited States
  2. 2.Database Systems and Information Management GroupTechnische Universität BerlinBerlinGermany