TPCx-HS v2: Transforming with Technology Changes
The TPCx-HS Hadoop benchmark has helped drive competition in the Big Data marketplace and has proven to be a successful industry standard benchmark for Hadoop systems. However, the Big Data landscape has rapidly changed since its initial release in 2014. Key technologies have matured, while new ones have risen to prominence in an effort to keep pace with the exponential expansion of datasets. For example, Hadoop has undergone a much-needed upgrade to the way that scheduling, resource management, and execution occur in Hadoop, while Apache Spark has risen to be the de facto standard for in-memory cluster compute for ETL, Machine Learning, and Data Science Workloads. Moreover, enterprises are increasingly considering cloud infrastructure for Big Data processing. What has not changed since TPCx-HS was first released is the need for a straightforward, industry standard way in which these current technologies and architectures can be evaluated. In this paper, we introduce TPCx-HS v2 that is designed to address these changes in the Big Data technology landscape and stress both the hardware and software stacks including the execution engine (MapReduce or Spark) and Hadoop Filesystem API compatible layers for both on-premise and cloud deployments.
KeywordsTPC Big Data Benchmark Hadoop Spark Cloud Performance
Developing a TPC benchmark for a new environment requires a huge effort to conceptualize, research, specify, review, prototype, and verify the benchmark. The authors acknowledge the work and contributions made by Da Qi Ren, David Grimes, Jamie Reding, John Poelman, Karthik Kulkarni, Matthew Emmerton, Meikel Poess, Mike Brey, Paul Cao, and Reza Taheri.
- 1.Nambiar, R., Poess, M., Dey, A., Cao, P., Magdon-Ismail, T., Ren, D.Q., Bond, A.: Introducing TPCx-HS: the first industry standard for benchmarking Big Data systems. In: Nambiar, R., Poess, M. (eds.) TPCTC 2014. LNCS, vol. 8904, pp. 1–12. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-15350-6_1 Google Scholar
- 2.TPCx-HS Results. http://www.tpc.org/tpcx-hs/results/tpcxhs_results.asp. Accessed 20 June 2017
- 3.Taneja Group Spark Market Adoption Report. https://www.cloudera.com/content/dam/www/marketing/resources/analyst-reports/taneja-group-spark-survey-exec-summary-Oct-2016.pdf.landing.html. Accessed 20 June 2017
- 4.TPC Specifications, http://www.tpc.org//tpc_documents_current_versions/current_specifications.asp. Accessed 20 June 2017
- 5.Apache Hadoop Project Page, 02 July 2008. http://web.archive.org/web/20080702015052/http://hadoop.apache.org. Accessed 20 June 2017
- 6.Apache Hadoop Ecosystem and Open Source Big Data Projects. https://hortonworks.com/ecosystems/. Accessed 20 June 2017
- 7.Getting MapReduce 2 Up to Speed. http://blog.cloudera.com/blog/2014/02/getting-mapreduce-2-up-to-speed/. Accessed 21 June 2017
- 8.Sort Benchmark Home Page. http://sortbenchmark.org. Accessed 21 June 2017
- 9.Zaharia, M. et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the Ninth USENIX NSDI Symposium on Networked Systems Design and Implementation, San Jose, CA (2012)Google Scholar
- 10.Guo, Z., Fox, G., Zhou, M., Ruan, Y.: Improving resource utilization in MapReduce. In: 2013 IEEE International Conference on Cluster Computing (CLUSTER) (2013)Google Scholar
- 11.Vavilapalli, V.K. et al.: Apache Hadoop YARN: yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing (SOCC), Santa Clara, CA (2013). Article No. 5Google Scholar