# High Performance Computing for Mathematical Optimization Problem

## Abstract

The semidefinite programming (SDP) problem is one of the central problems in mathematical optimization. The primal-dual interior-point method (PDIPM) is one of the most powerful algorithms for solving SDP problems, and many research groups have employed it for developing software packages. However, two well-known major bottlenecks, i.e., the generation of the Schur complement matrix (SCM) and its Cholesky factorization, exist in the algorithmic framework of the PDIPM. We have developed a new version of the semidefinite programming algorithm parallel version (SDPARA), which is a parallel implementation on multiple CPUs and GPUs for solving extremely large-scale SDP problems with over a million constraints. SDPARA can automatically extract the unique characteristics from an SDP problem and identify the bottleneck. When the generation of the SCM becomes a bottleneck, SDPARA can attain high scalability using a large quantity of CPU cores and some processor affinity and memory interleaving techniques. SDPARA can also perform parallel Cholesky factorization using thousands of GPUs and techniques for overlapping computation and communication if an SDP problem has over 2 million constraints and Cholesky factorization constitutes a bottleneck. We demonstrate that SDPARA is a high-performance general solver for SDPs in various application fields through numerical experiments conducted on the TSUBAME 2.5 supercomputer, and we solved the largest SDP problem (which has over 2.33 million constraints), thereby creating a new world record. Our implementation also achieved 1.713 PFlops in double precision for large-scale Cholesky factorization using 2,720 CPUs and 4,080 GPUs.

## Keywords

Mathematical optimization Semidefinite programming Interior-point method Parallel computing High performance computing## Notes

### Acknowledgments

This research was supported by the Japan Science and Technology Agency (JST), the Core Research of Evolutionary Science and Technology (CREST), and the TSUBAME 2.0 and 2.5 Supercomputer Grand Challenge Program at the Tokyo Institute of Technology.

## References

- 1.H. Wolkowicz, R. Saigal, L. Vandenberghe (eds.),
*Handbook of Semidefinite Programming (International Series in Operations Research and Management Science)*, (Kluwer Academic Publishers, Massachusetts, 2000)Google Scholar - 2.M.F. Anjos, J.B. Lasserre (eds.),
*Handbook on Semidefinite, Conic and Polynomial Optimization (International Series in Operations Research and Management Science)*. (Springer Science+Business Media, Dordrecht, 2011)Google Scholar - 3.M. Yamashita, K. Fujisawa, M. Fukuda, K. Kobayashi, K. Nakata, M. Nakata, in
*Latest Developments in the SDPA Family for Solving Large-Scale SDPs*, ed. by M.F. Anjos and J.B. Lasserre. Handbook on Semidefinite, Cone and Polynomial Optimization: Theory, Algorithms, Software and Applications. (Springer Press, New York, 2011), pp 687–713 (Chapter 24)Google Scholar - 4.K. Fujisawa, M. Kojima, K. Nakata, Exploiting sparsity in primal-dual interior-point methods for semidefinite programming. Math. Prog.
**79**, 235–253 (1997)MathSciNetMATHGoogle Scholar - 5.K. Fujisawa, K. Nakata, M. Yamashita, M. Fukuda, SDPA project: solving large-scale semidefinite programs. J. Oper. Res. Soc. Japan
**50**, 278–298 (2007)MathSciNetMATHGoogle Scholar - 6.B. Borchers, CSDP, a C library for semidefinite programming. Optim. Meth. Softw.
**11**,**12**, 613–623 (1999)Google Scholar - 7.J.F. Sturm, SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones. Optim. Meth. Softw.
**11**,**12**, 625–653 (1999)Google Scholar - 8.K.-C. Toh, M.J. Todd, R.H. Tütüncü, SDPT3—MATLAB software package for semidefinite programming, version 1.3. Optim. Meth. Softw.
**11**,**12**, 545–581 (1999)Google Scholar - 9.M. Yamashita, K. Fujisawa, M. Kojima, SDPARA: semidefinite programming algorithm parallel version. Parallel Comput.
**29**, 1053–1067 (2003)MathSciNetCrossRefGoogle Scholar - 10.M. Yamashita, K. Fujisawa, M. Fukuda, K. Nakata, M. Nakata, Parallel solver for semidefinite programming problem having sparse Schur complement matrix. ACM Trans. Math. Softw.
**39**(12), (2012)Google Scholar - 11.K. Fujisawa, T. Endo, H. Sato, M. Yamashita, S. Matsuoka, M. Nakata, in
*Proceedings of the 2012 ACM/IEEE Conference on Supercomputing, SC’12*. High-Performance General Solver for Extremely Large-Scale Semidefinite Programming Problems (2012)Google Scholar - 12.K. Fujisawa, T. Endo, Y. Yasui, H. Sato, N. Matsuzawa, S. Matsuoka, H.Waki, in
*The 28th IEEE International Parallel and Distributed Processing Symposium (IPDPS2014)*. Peta-scale General Solver for Semidefinite Programming Problems with over Two Million Constraints (2014)Google Scholar - 13.I.D. Ivanov, E. de Klerk, Parallel implementation of a semidefinite programming solver based on CSDP in a distributed memory cluster. Optim. Meth. Softw.
**25**(3), 405–420 (2010)CrossRefMATHGoogle Scholar - 14.S. Benson, Parallel computing on semidefinite programs. (Mathematics and Computer Science Division Argonne Science Laboratory, IL, 2002) (Preprint ANL/MCS-P939-0302)Google Scholar
- 15.K. Kobayashi, S. Kim, M. Kojima, Correlative sparsity in primal-dual interior-point methods for LP, SDP and SOCP. Appl. Math. Optim.
**58**, 69–88 (2008)MathSciNetCrossRefMATHGoogle Scholar - 16.P.R. Amestoy, I.S. Duff, J.-Y. L’Excellent, Multifrontal parallel distributed symmetric and unsymmetric solvers. Comput. Methods in Appl. Mech. Eng.
**184**, 501–520 (2000)CrossRefMATHGoogle Scholar - 17.S. Burer, On the copositive representation of binary and continuous nonconvex quadratic programs. Math. Prog.
**120**, 479–495 (2009)MathSciNetCrossRefMATHGoogle Scholar - 18.Q. Zhao, S.E. Karisch, F. Rendl, H. Wolkowicz, Semidefinite programming relaxations for the quadratic assignment problem. J. Comb. Optim.
**2**, 71–109 (1998)MathSciNetCrossRefMATHGoogle Scholar - 19.P.A.M. Dirac, The quantum theory of the electron. Roy. Soc. A
**123**, 714 (1929) (London)Google Scholar - 20.A.J. Coleman, Structure of fermion density matrices. Rev. Mod. Phys.
**35**, 668–687 (1963)CrossRefGoogle Scholar - 21.C. Garrod, J.K. Percus, Reduction of the \(N\)-particle variational problem. J. Math. Phys.
**5**, 1756–1776 (1964)MathSciNetCrossRefMATHGoogle Scholar - 22.M. Nakata, H. Nakatsuji, M. Ehara, M. Fukuda, K. Nakata, K. Fujisawa, Variational calculations of fermion second-order reduced density matrices by semidefinite programming algorithm. J. Chem. Phys.
**114**, 8282–8292 (2001)CrossRefGoogle Scholar - 23.Z. Zhao, B.J. Braams, M. Fukuda, M.L. Overton, J.K. Percus, The reduced density matrix method for electronic structure calculations and the role of three-index representability conditions. J. Chem. Phys.
**120**, 2095–2104 (2004)CrossRefGoogle Scholar - 24.F. Alizadeh, J.-P.A. Haeberly, M.L. Overton, Primal-dual interior-point methods for semidefinite programming: convergence rates, stability and numerical results. SIAM J. Optim.
**8**, 746–768 (1998)MathSciNetCrossRefMATHGoogle Scholar - 25.C. Helmberg, F. Rendl, R.J. Vanderbei, H. Wolkowicz, An interior-point method for semidefinite programming. SIAM J. Optim.
**6**, 342–361 (1996)MathSciNetCrossRefMATHGoogle Scholar - 26.M. Kojima, S. Shindoh, S. Hara, Interior-point methods for the monotone semidefinite linear complementarity problems. SIAM J. Optim.
**7**, 86–125 (1997)MathSciNetCrossRefMATHGoogle Scholar - 27.R.D.C Monteiro, Primal-dual path following algorithms for semidefinite programming. SIAM J. Optim.
**7**, 663–678 (1997)Google Scholar - 28.Y.E. Nesterov, M.J. Todd, Primal-dual interior-point methods for self-scaled cones. SIAM J. Optim
**8**, 324–364 (1998)MathSciNetCrossRefMATHGoogle Scholar - 29.S. Matsuoka, T. Endo, N. Maruyama, H. Sato, S. Takizawa: The Total Picture of TSUBAME2.0. TSUBAME e-Science Journal, GSIC, Tokyo Institute of Technology, Vol. 1, 2–4 (2010).Google Scholar
- 30.T. Endo, A. Nukada, S. Matsuoka, N. Maruyama, in
*Proceedings of IEEE IPDPS10*. Linpack Evaluation on a Supercomputer with Heterogeneous Accelerators, pp. 1–8 (2010)Google Scholar - 31.A. Petitet, R.C. Whaley, J. Dongarra, A. Cleary, HPL—A Portable Implementation of the High-Performance Linpack Benchmark for Distributed-Memory Computers, http://www.netlib.org/benchmark/hpl/