Cluster Computing

, Volume 13, Issue 3, pp 243–256

Harnessing parallelism in multicore clusters with the All-Pairs, Wavefront, and Makeflow abstractions

  • Li Yu
  • Christopher Moretti
  • Andrew Thrasher
  • Scott Emrich
  • Kenneth Judd
  • Douglas Thain
Article

Abstract

Both distributed systems and multicore systems are difficult programming environments. Although the expert programmer may be able to carefully tune these systems to achieve high performance, the non-expert may struggle. We argue that high level abstractions are an effective way of making parallel computing accessible to the non-expert. An abstraction is a regularly structured framework into which a user may plug in simple sequential programs to create very large parallel programs. By virtue of a regular structure and declarative specification, abstractions may be materialized on distributed, multicore, and distributed multicore systems with robust performance across a wide range of problem sizes. In previous work, we presented the All-Pairs abstraction for computing on distributed systems of single CPUs. In this paper, we extend All-Pairs to multicore systems, and introduce the Wavefront and Makeflow abstractions, which represent a number of problems in economics and bioinformatics. We demonstrate good scaling of both abstractions up to 32 cores on one machine and hundreds of cores in a distributed system.

Keywords

Abstractions Multicore Distributed systems Bioinformatics Economics 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Li Yu
    • 1
  • Christopher Moretti
    • 1
  • Andrew Thrasher
    • 1
  • Scott Emrich
    • 1
  • Kenneth Judd
    • 2
  • Douglas Thain
    • 1
  1. 1.Department of Computer Science and EngineeringUniversity of Notre DameSouth BendUSA
  2. 2.Hoover InstitutionStanford UniversityStanfordUSA

Personalised recommendations