Cluster Computing

, Volume 13, Issue 3, pp 243–256

Harnessing parallelism in multicore clusters with the All-Pairs, Wavefront, and Makeflow abstractions

  • Li Yu
  • Christopher Moretti
  • Andrew Thrasher
  • Scott Emrich
  • Kenneth Judd
  • Douglas Thain
Article

DOI: 10.1007/s10586-010-0134-7

Cite this article as:
Yu, L., Moretti, C., Thrasher, A. et al. Cluster Comput (2010) 13: 243. doi:10.1007/s10586-010-0134-7

Abstract

Both distributed systems and multicore systems are difficult programming environments. Although the expert programmer may be able to carefully tune these systems to achieve high performance, the non-expert may struggle. We argue that high level abstractions are an effective way of making parallel computing accessible to the non-expert. An abstraction is a regularly structured framework into which a user may plug in simple sequential programs to create very large parallel programs. By virtue of a regular structure and declarative specification, abstractions may be materialized on distributed, multicore, and distributed multicore systems with robust performance across a wide range of problem sizes. In previous work, we presented the All-Pairs abstraction for computing on distributed systems of single CPUs. In this paper, we extend All-Pairs to multicore systems, and introduce the Wavefront and Makeflow abstractions, which represent a number of problems in economics and bioinformatics. We demonstrate good scaling of both abstractions up to 32 cores on one machine and hundreds of cores in a distributed system.

Keywords

AbstractionsMulticoreDistributed systemsBioinformaticsEconomics

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Li Yu
    • 1
  • Christopher Moretti
    • 1
  • Andrew Thrasher
    • 1
  • Scott Emrich
    • 1
  • Kenneth Judd
    • 2
  • Douglas Thain
    • 1
  1. 1.Department of Computer Science and EngineeringUniversity of Notre DameSouth BendUSA
  2. 2.Hoover InstitutionStanford UniversityStanfordUSA