Sparsifying Synchronization for High-Performance Shared-Memory Sparse Triangular Solver
The last decade has seen rapid growth of single-chip multiprocessors (CMPs), which have been leveraging Moore’s law to deliver high concurrency via increases in the number of cores and vector width. Modern CMPs execute from several hundreds to several thousands concurrent operations per second, while their memory subsystem delivers from tens to hundreds Giga-bytes per second bandwidth.
Taking advantage of these parallel resources requires highly tuned parallel implementations of key computational kernels, which form the back-bone of modern HPC. Sparse triangular solver is one such kernel and is the focus of this paper. It is widely used in several types of sparse linear solvers, and it is commonly considered challenging to parallelize and scale even on a moderate number of cores. This challenge is due to the fact that triangular solver typically has limited task-level parallelism and relies on fine-grain synchronization to exploit this parallelism, compared to data-parallel operations such as sparse matrix-vector multiplication.
This paper presents synchronization sparsification technique that significantly reduces the overhead of synchronization in sparse triangular solver and improves its scalability. We discover that a majority of task dependencies are redundant in task dependency graphs which are used to model the flow of computation in sparse triangular solver. We propose a fast and approximate sparsification algorithm, which eliminates more than 90% of these dependencies, substantially reducing synchronization overhead. As a result, on a 12-core Intel® Xeon® processor, our approach improves the performance of sparse triangular solver by 1.6x, compared to the conventional level-scheduling with barrier synchronization. This, in turn, leads to a 1.4x speedup in a pre-conditioned conjugate gradient solver.
KeywordsCritical Path Barrier Synchronization Synchronization Overhead Dependency Edge Redundant Edge
Unable to display preview. Download preview PDF.
- 1.Agullo, E., Demmel, J., Dongarra, J., Hadri, B., Kurzak, J., Langou, J., Ltaief, H., Luszczek, P., Tomov, S.: Numerical linear algebra on emerging architectures: the PLASMA and MAGMA projects. Journal of Physics: Conference Series 180 (2009)Google Scholar
- 2.Anderson, E., Saad, Y.: Solving Sparse Triangular Linear Systems on Parallel Computers. International Journal of High Speed Computing 1(1) (1989)Google Scholar
- 3.Chan, E., Quintana-Orti, E.S., Quintana-Orti, G., van de Geijn, R.: SuperMatrix Out-of-Order Scheduling of Matrix Operations for SMP and Multi-Core Architectures. In: Symposium on Parallelism in Algorithms and Architectures (SPAA) (2007)Google Scholar
- 4.Chhugani, J., Satish, N., Kim, C., Sewall, J., Dubey, P.: Fast and Efficient Graph Traversal Algorithm for CPUs: Maximizing Single-Node Efficiency. In: International Symposium on Parallel and Distributed Processing (IPDPS) (2012)Google Scholar
- 5.Molka, R.S.D., Hackenberg, D., Müller, M.S.: Memory Performance and Cache Coherency Effects on an Intel Nehalem Multiprocessor System. In: International Conference on Parallel Architectures and Compilation Techniques (PACT) (2009)Google Scholar
- 6.Davis, T.A., Hu, Y.: The University of Florida Sparse Matrix Collection. ACM Transactions on Mathematical Software 15(1) (2011), http://www.cise.ufl.edu/research/sparse/matrices
- 7.Dongarra, J., Heroux, M.A.: Toward a New Metric for Ranking High Performance Computing Systems. Technical Report 4744, Sandia National Laboratories (2013)Google Scholar
- 8.Graham, R.L.: Bounds on Multiprocessing Timing Anomalies. SIAM Journal on Applied Mathematics 17(2) (1969)Google Scholar
- 9.Hensgen, D., Finkel, R., Manber, U.: Two Algorithms for Barrier Synchronization. International Journal of Parallel Programming 17(1) (1988)Google Scholar
- 11.Hestenes, M.R., Stiefel, E.: Methods of Conjugate Gradients for Solving Linear Systems. Journal of Research of the National Bureau of Standards 49(6) (1952)Google Scholar
- 12.Hsu, H.T.: An Algorithm for Finding a Minimal Equivalent Graph of a Digraph. Journal of the ACM (JACM) 22(1) (1975)Google Scholar
- 13.Hu, T.C.: Parallel Sequencing and Assembly Line Problems. Operations Research 19(6) (1961)Google Scholar
- 14.Iwashita, T., Nakashima, H., Takahashi, Y.: Algebraic Block Multi-Color Ordering Method for Parallel Multi-Threaded Sparse Triangular Solver in ICCG Method. In: International Symposium on Parallel and Distributed Processing (IPDPS) (2012)Google Scholar
- 15.Kepner, J., Gilbert, J.: Graph Algorithms in the Language of Linear Algebra. Society for Industrial & Applied Mathematics (2011)Google Scholar
- 16.Kim, K., Eijkhout, V.: A Parallel Sparse Direct Solver via Hierarchical DAG Scheduling. Technical Report 5, The Texas Advanced Computing Center (2012)Google Scholar
- 17.Mayer, J.: Parallel algorithms for solving linear systems with sparse triangular matrices. Computing 86(4) (2009)Google Scholar
- 18.Meijerink, J.A., van der Vorst, H.A.: An Iterative Solution Method for Linear Systems of Which the Coefficient Matrix is a Symmetric M-Matrix. Mathematics of Computation 31(137) (1977)Google Scholar
- 19.Naumov, M.: Parallel Solution of Sparse Triangular Linear Systems in the Preconditioned Iterative Methods on the GPU. Technical Report 001, NVIDIA Corporation (2011)Google Scholar
- 20.Park, J., Dally, W.J.: Buffer-space Efficient and Deadlock-free Scheduling of Stream Applications on Multi-core Architectures. In: Symposium on Parallelism in Algorithms and Architectures (SPAA) (2010)Google Scholar
- 21.Petitet, A., Whaley, R.C., Dongarra, J., Cleary, A.: HPL - A Portable Implementation of the High-Performance Linpack Benchmark for Distributed-Memory Computers, http://www.netlib.org/benchmark/hpl/
- 22.Poole, E.L., Ortega, J.M.: Multicolor ICCG Methods for Vector Computers. SIAM Journal on Numerical Analysis 24(6) (1987)Google Scholar
- 23.Rothberg, E., Gupta, A.: Parallel ICCG on a Hierarchical Memory Multiprocessor - Addressing the Triangular Solve Bottleneck. Parallel Computing 18(7) (1992)Google Scholar
- 24.Saltz, J.H.: Aggregation Methods for Solving Sparse Triangular Systems on Multiprocessors. SIAM Journal of Scientific and Statistical Computing 11(1) (1990)Google Scholar
- 25.Saltz, J.H., Mirchandaney, R., Baxter, D.: Run-Time Parallelization and Scheduling of Loops. In: Symposium on Parallelism in Algorithms and Architectures (SPAA) (1989)Google Scholar