The development of Mellanox/NVIDIA GPUDirect over InfiniBand—a new model for GPU to GPU communications

  • Gilad Shainer
  • Ali Ayoub
  • Pak Lui
  • Tong Liu
  • Michael Kagan
  • Christian R. Trott
  • Greg Scantlen
  • Paul S. Crozier
Special Issue Paper

Abstract

The usage and adoption of General Purpose GPUs (GPGPU) in HPC systems is increasing due to the unparalleled performance advantage of the GPUs and the ability to fulfill the ever-increasing demands for floating points operations. While the GPU can offload many of the application parallel computations, the system architecture of a GPU-CPU-InfiniBand server does require the CPU to initiate and manage memory transfers between remote GPUs via the high speed InfiniBand network. In this paper we introduce for the first time a new innovative technology—GPUDirect that enables Tesla GPUs to transfer data via InfiniBand without the involvement of the CPU or buffer copies, hence dramatically reducing the GPU communication time and increasing overall system performance and efficiency. We also explore for the first time the performance benefits of GPUDirect using Amber and LAMMPS applications.

Keywords

GPUDirect InfiniBand RDMA 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Kindratenko V, Enos J, Shi G, Showerman M, Arnold G, Stone J, Phillips J, Hwu W-m (2009) GPU clusters for high-performance computing. In: Cluster computing and workshops Google Scholar
  2. 2.
    Wu E, Liu Y (2008) Emerging technology about GPGPU. In: Circuits and systems Google Scholar
  3. 3.
    Chen G, Li G, Pei S, Wu B (2009) High performance computing via a GPU. In: Information science and engineering (ICISE) Google Scholar
  4. 4.
    Garland M (2010) Parallel computing with CUDA. In: Parallel & distributed processing (IPDPS) Google Scholar
  5. 5.
    The TOP500 list—www.top500.org
  6. 6.
    InfiniBand Trade Association—www.infinibandta.org/
  7. 7.
    Mellanox Technologies—www.mellanox.com
  8. 8.

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Gilad Shainer
    • 1
  • Ali Ayoub
    • 2
  • Pak Lui
    • 2
  • Tong Liu
    • 2
  • Michael Kagan
    • 2
  • Christian R. Trott
    • 3
  • Greg Scantlen
    • 4
  • Paul S. Crozier
    • 5
  1. 1.HPC Advisory CouncilSunnyvaleUSA
  2. 2.Mellanox TechnologiesSunnyvaleUSA
  3. 3.Institut für PhysikTechnische Universität at IlmenauIlmenauGermany
  4. 4.Creative ConsultantsAlbuquerqueUSA
  5. 5.Sandia National LaboratoriesAlbuquerqueUSA

Personalised recommendations