Scalability Pipelined Algorithm of the Conjugate Gradient Method on Heterogeneous Platforms

Abstract

This paper presents a parallelized iterative solver for large sparse linear systems implemented on a heterogeneous platform. Traditionally, these problems do not scale well on multi-CPU/multi-GPUs clusters. We consider the standard preconditioned Conjugate Gradient (PCG) algorithm, and as an alternative the pipelined variant, a formulation that is potentially better suited for hybrid CPU/GPU computing since it requires only one synchronization point per iteration, instead of two for standard CG. On heterogeneous cluster, the PCG iteration needs the vector entries generated by current GPU and other GPUs, so the communication between GPUs becomes a major performance bottleneck. In this paper, we study the implementation of the pipeline PCG on multi-CPU/multi-GPU platform. This paper presents an approach to reduce the communications between cluster compute nodes for these solvers. Additionally, computation and communication are overlapped to reduce the impact of data exchange. To achieve scalability, we adopt pipelined version of the conjugate gradient method with one synchronization point, the possibility of asynchronous calculations, load balancing between the CPU and GPU for parallel solving the large linear systems. The algorithm is implemented with the combined use of technologies: MPI, OpenMP and CUDA. We show that almost optimum speed up on 8-CPU/2GPU may be reached (relatively to a one GPU execution). The parallelized solver achieves a speedup of up to 5.49 times on 16 NVIDIA Tesla GPUs, as compared to a one GPU.