TPCx-HS on the Cloud!
The introduction of web scale operations needed for social media coupled with ease of access to the internet by mobile devices has exponentially increased the amount of data being generated every day. By conservative estimates the world generates close to 50,000 GB of data every second, 90% of which is unstructured, and this growth is accelerating. From its origins as a web log processing system at Yahoo, the open source nature and efficient processing of Apache Hadoop has made it the industry standard for Big Data processing.
TPCx-HS was the first benchmark standard by a major Industry-Standard performance consortium for the Big Data space. TPCx-HS is a derivative of Apache Hadoop Workloads; Teragen, Terasort and Teravalidate. Ever since its release by the TPC in August 2014, all the 18 results published (as of August 2016) have been based on on-premise, Bare-metal hardware configurations.
This paper will show how Hadoop can be deployed on an OpenStack cloud using the OpenStack Sahara project and how TPCx-HS can be used to measure and evaluate the performance of the Cloud under Test (CuT). It will also show how an OpenStack cloud can be optimized to get the performance of TPCx-HS on the Cloud to match as closely as possible that on a Bare-metal configuration. Lastly, it will share results and experiences based on a Hadoop on Cloud Proof-of-Concept (POC), a study that was undertaken by the Dell Open Source Solutions team.
KeywordsApache Hadoop OpenStack Big data Cloud TPCx-HS Benchmark
The authors would like to thank John Terpstra, Michael Pittaro, Randy Perryman, Michael Tondee and David Grimes for participating in the technical review meetings of the POC. Their input, feedback and guidance helped shape this investigation. Mr. Ashok Malani is recognized for his technical leadership of the xFlow Research team that did such a tremendous job performing the tests and drafting this paper.
- 1.Navint: Why is big data important? (2012). www.navint.com/images/Navint.BigData.FINAL.pdf
- 2.TPC: Tpcx-hs (2016). http://www.tpc.org/tpcx-hs/
- 3.VMware: Virtualized hadoop performance with vmware vsphere 6 on highperformance servers (2015). http://www.vmware.com/files/pdf/techpaper/Virtualized-Hadoop-Performance-with-VMware-vSphere6.pdf
- 4.Stata, R.: Understanding hadoop-as-a-service offerings (2014). http://www.datacenterknowledge.com/archives/2014/05/14/understanding-hadoop-service-offerings/
- 5.Hurtgen, A.: Using apache hadoop on rackspace private cloud (2013). https://support.rackspace.com/how-to/apache-hadoop-on-rackspace-private-cloud/
- 6.Wendt, M.E.: Cloud-based hadoop deployments: benefits and considerations (2014). https://goo.gl/re0Ov5
- 7.OpenStack: Openstack sahara user documentation (2016). http://docs.openstack.org/developer/sahara/userdoc/overview.html
- 8.Mirantis: Openstack sahara kilo images (2016). http://sahara-files.mirantis.com/images/upstream/kilo/
- 9.Cloudera, I.: Cloudera manager free edition user guide (2012)Google Scholar
- 10.TPC: Dell poweredge r720xd with vmware vsphere 6.0 (2015). http://www.tpc.org/5504
- 11.OpenStack: Install and configure a storage node - openstack kilo (2015). http://docs.openstack.org/kilo/install-guide/install/yum/content/cinder-install-storage-node.html
- 12.RedHat: Cpu pinning and numa topology awareness in openstack compute (2015). http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/
- 13.OpenStack: Openstack cinder multi-backend (2015). https://wiki.openstack.org/wiki/Cinder-multi-backend