Pipelined Multi-GPU MapReduce for Big-Data Processing

  • Yi Chen
  • Zhi Qiao
  • Spencer Davis
  • Hai Jiang
  • Kuan-Ching Li
Conference paper

DOI: 10.1007/978-3-319-00804-2_17

Volume 493 of the book series Studies in Computational Intelligence (SCI)
Cite this paper as:
Chen Y., Qiao Z., Davis S., Jiang H., Li KC. (2013) Pipelined Multi-GPU MapReduce for Big-Data Processing. In: Lee R. (eds) Computer and Information Science. Studies in Computational Intelligence, vol 493. Springer, Heidelberg

Abstract

MapReduce is a popular large-scale data-parallel processing model. Its success has stimulated several studies of implementing MapReduce on Graphic Processing Unit (GPU). However, these studies focus most of their efforts on single-GPU algorithms and cannot handle large data sets which exceed GPU memory capacity. This paper describes an upgrade version of MGMR, a pipelined multi-GPU MapReduce system (PMGMR), which addresses the challenge of big data. PMGMR employs the power of multiple GPUs, improves GPU utilization using new GPU features such as streams and Hyper-Q, and handles large data sets which exceeds GPU and even CPU memory. Compared to MGMR, the newly proposed scheme achieves a 2.5-fold performance improvement and increases system scalability, while allowing users to write straight forward MapReduce code.

Keywords

MapReduce big-data multi-GPU stream concurrency Hyper-Q 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer International Publishing Switzerland 2013

Authors and Affiliations

  • Yi Chen
    • 1
  • Zhi Qiao
    • 1
  • Spencer Davis
    • 1
  • Hai Jiang
    • 1
  • Kuan-Ching Li
    • 2
  1. 1.Dept. of Computer ScienceArkansas State UniversityJonesboroUSA
  2. 2.Dept. of Computer Science & Information Eng.Providence UniversityTaichungTaiwan