Skip to main content

Revisiting Benchmarking Principles and Methodologies for Big Data Benchmarking

  • Conference paper
  • First Online:
Big Data Benchmarks, Performance Optimization, and Emerging Hardware (BPOE 2015)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9495))

Included in the following conference series:

  • 908 Accesses


Benchmarking as yardsticks for system design and evaluation, has developed a long period and plays a pivotal role in many domains, such as database systems and high performance computing. Through prolonged and unremitting efforts, benchmarks on these domains have been reaching their maturity gradually. However, in terms of emerging scenarios of big data, its different properties in data volume, data types, data processing requirements and techniques, make that existing benchmarks are rarely appropriate for big data systems and further make us wonder how to define a good big data benchmark. In this paper, we revisit successful benchmarks in other domains from two perspectives: benchmarking principles which define fundamental rules, and methodologies which guide the benchmark constructions. Further, we conclude the benchmarking principle and methodology on big data benchmarking from a recent open-source effort – BigDataBench.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 34.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 44.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions


  1. Bdbc.

  2. Spec.

  3. Tpc.

  4. Angles, R.: Benchmark principles and methods. In: Linked Data Benchmark Council (LDBC). Project No 317548, European Community’s Seventh Framework Programme FP7 (2012–2014)

    Google Scholar 

  5. Bienia, C., Li, K.: Benchmarking Modern Multiprocessors. Princeton University, New York (2011)

    Google Scholar 

  6. Chen, Y., Raab, F., Katz, R.: From TPC-C to big data benchmarks: a functional workload model. In: Rabl, T., Poess, M., Baru, C., Jacobsen, H.-A. (eds.) WBDB 2012. LNCS, vol. 8163, pp. 28–43. Springer, Heidelberg (2014)

    Chapter  Google Scholar 

  7. Gao, W., Luo, C., Zhan, J., Ye, H., He, X., Wang, L., Zhu, Y., Tian, X.: Identifying dwarfs workloads in big data analytics (2015). arXiv preprint arXiv:1505.06872

  8. Gray, J.: Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., San Francisco (1992)

    Google Scholar 

  9. Levine, C.: TPC benchmarks. In: SIGMOD International Conference on Managementof Data - Industrial Session (1997)

    Google Scholar 

  10. Luo, C., Gao, W., Jia, Z., Han, R., Li, J., Lin, X., Wang, L., Zhu, Y., Zhan, J.: Handbook of bigdatabench (version 3.1) - a big data benchmark suite

    Google Scholar 

  11. Ming, Z., Luo, C., Gao, W., Han, R., Yang, Q., Wang, L., Zhan, J.: BDGS: A scalable big data generator suite in big data benchmarking. In: Rabl, T., Raghunath, N., Poess, M., Bhandarkar, M., Jacobsen, H.A., Baru, C. (eds.) Advancing Big Data Benchmarks. LNCS, vol. 8585. Springer, Heidelberg (2014)

    Google Scholar 

  12. Seltzer, M., Krinsky, D., Smith, K., Zhang, X.: The case for application-specific benchmarking. In: Proceedings of the Seventh Workshop on Hot Topics in Operating Systems, pp. 102–107. IEEE (1999)

    Google Scholar 

  13. Wang, L., Zhan, J., Luo, C., Zhu, Y., Yang, Q., He, Y., Gao, W., Jia, Z., Shi, Y., Zhang, S., et al.: BigDataBench: A big data benchmark suite from internet services. In: 2014 IEEE 20th International Symposium on High Performance Computer Architecture (HPCA), pp. 488–499. IEEE (2014)

    Google Scholar 

  14. Zhu, Y., Zhan, J., Weng, C., Nambiar, R., Zhang, J., Chen, X., Wang, L.: BigOP: generating comprehensive big data workloads as a benchmarking framework. In: Bhowmick, S.S., Dyreson, C.E., Jensen, C.S., Lee, M.L., Muliantara, A., Thalheim, B. (eds.) DASFAA 2014, Part II. LNCS, vol. 8422, pp. 483–492. Springer, Heidelberg (2014)

    Chapter  Google Scholar 

Download references


This work is supported by the National High Technology Research and Development Program of China (Grant No. 2015AA015308), the Major Program of National Natural Science Foundation of China (Grant No. 61432006), and the Key Technology Research and Development Programs of Guangdong Province, China (Grant No. 2015B010108006).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Liutao Zhao .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Zhao, L., Gao, W., Jin, Y. (2016). Revisiting Benchmarking Principles and Methodologies for Big Data Benchmarking. In: Zhan, J., Han, R., Zicari, R. (eds) Big Data Benchmarks, Performance Optimization, and Emerging Hardware. BPOE 2015. Lecture Notes in Computer Science(), vol 9495. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-29005-8

  • Online ISBN: 978-3-319-29006-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics