High Performance Computing and Big Data

Abstract

High Performance Computing (HPC) has traditionally been characterized by low-latency, high throughput, massive parallelism and massively distributed systems. Big Data or analytics platforms share some of the same characteristics but as of today are limited somewhat in their guarantees on latency and throughput. The application of Big Data platforms has been in solving problems where data that is being operated upon is in motion while HPC has traditionally been applied to performing scientific computations where data is at rest. The programing paradigms that are in use in Big Data platforms for example Map-Reduce (Google Research Publication: MapReduce. Retrieved November 29, 2016, from http://research.google.com/archive/mapreduce.html) and Spark streaming (Spark Streaming/Apache Spark. Retrieved November 29, 2016, from https://spark.apache.org/streaming/) have their genesis in HPC but they need to address some of the distinct characteristics of Big Data platforms. So bringing High Performance to Big Data platforms means addressing the following:

• 1. Ingesting Data at high volume with low latency

• 2. Processing streaming data at high volume with low latency

• 3. Storing Data in a distributed data store

• 4. Indexing and searching the stored data for Real–Time processing

In order to achieve 1, 2, 3, 4 mentioned above, the right hardware and software components need to be chosen. With the plethora of software stacks and different kinds of hardware infrastructure–including public/private cloud, on­ premise and co–located hardware there are many criteria, characteristics and metrics to be evaluated in order to make the right choices. We show that it is of the utmost importance to have the right tools to make this kind of evaluation as accurate as possible and then have the appropriate software to maintain performance of such systems as they scale. We then identify the different types of hardware infrastructure in the cloud including Amazon Web Services (AWS) (Amazon Web Services. What is AWS?. Retrieved November 29, 2016, from https://aws.amazon.com/what-is-aws), and different types of on-premise hardware infrastructure including converged hyperscale infrastructure from vendors such as Nutanix (Nutanix-The Enterprise Cloud Company. Retrieved November 29, 2016, from http://www.nutanix.com/) and traditional vendors such as Dell and HP. We also explore high-performance offerings from emerging open network switch device makers such as Cumulus (Better, Faster, Easier Networks. Retrieved November 29, 2016, from (https://cumulusnetworks.com/) and from traditional vendors such as Cisco (Cisco. Retrieved November 29, 2016, from (http://www.cisco.com/) as well as explore various storage architectures and their relative merits in the context of Big Data.