Up to 700k GPU Cores, Kepler, and the Exascale Future for Simulations of Star Clusters Around Black Holes

  • Peter Berczik
  • Rainer Spurzem
  • Shiyan Zhong
  • Long Wang
  • Keigo Nitadori
  • Tsuyoshi Hamada
  • Alexander Veles
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7905)


We present benchmarks on high precision direct astrophysical N-body simulations using up to several 100k GPU cores; their soft and strong scaling behaves very well at that scale and allows further increase of the core number in the future path to Exascale computing. Our simulations use large GPU clusters both in China (Chinese Academy of Sciences) as well as in Germany (Judge/Milkyway cluster at FZ Jülich). Also we present first results on the performance gain by the new Kepler K20 GPU technology, which we have tested in two small experimental systems, and which also runs in the titan supercomputer in the United States, currently the fastest computer in the world. Our high resolution astrophysical N-body simulations are used for simulations of star clusters and galactic nuclei with central black holes. Some key issues in theoretical physics and astrophysics are addressed with them, such as galaxy formation and evolution, massive black hole formation, gravitational wave emission. The models have to cover thousands or more orbital time scales for the order of several million bodies. The total numerical effort is comparable if not higher than for the more widely known cosmological N-body simulations. Due to a complex structure in time (hierarchical blocked time steps) our codes are not considered “brute force”.


Black Hole Graphical Processing Unit Star Cluster Astrophysical Journal Strong Scaling 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aarseth, S.J.: From NBODY1 to NBODY6: The Growth of an Industry. Publications of the Astronomical Society of the Pacific 111, 1333–1346 (1999)CrossRefGoogle Scholar
  2. 2.
    Aarseth, S.J.: Star Cluster Simulations: the State of the Art. Celestial Mechanics and Dynamical Astronomy 73, 127–137 (1999)zbMATHCrossRefGoogle Scholar
  3. 3.
    Aarseth, S.J.: Gravitational N-Body Simulations (November 2003)Google Scholar
  4. 4.
    Ahmad, A., Cohen, L.: A numerical integration scheme for the N-body gravitational problem. Journal of Computational Physics 12, 389–402 (1973)zbMATHCrossRefGoogle Scholar
  5. 5.
    Akeley, K., Nguyen, H.: GPU Gems 3 (2007)Google Scholar
  6. 6.
    Berczik, P., Merritt, D., Spurzem, R.: Long-Term Evolution of Massive Black Hole Binaries. II. Binary Evolution in Low-Density Galaxies. The Astrophysical Journal 633, 680–687 (2005)CrossRefGoogle Scholar
  7. 7.
    Berczik, P., Merritt, D., Spurzem, R., Bischof, H.: Efficient Merger of Binary Supermassive Black Holes in Nonaxisymmetric Galaxies. The Astrophysical Journal Letters 642, L21–L24 (2006)Google Scholar
  8. 8.
    Berczik, P., Nitadori, K., Hamada, T., Spurzem, R.: The Parallel GPU N-Body Code ϕGPUİn: New Astronomy (2013) in preparationGoogle Scholar
  9. 9.
    Dorband, E.N., Hemsendorf, M., Merritt, D.: Systolic and hyper-systolic algorithms for the gravitational N-body problem, with an application to Brownian motion. Journal of Computational Physics 185, 484–511 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Egri, G., Fodor, Z., Hoelbling, C., Katz, S., Nogradi, D., Szabo, K.: Lattice QCD as a video game. Computer Physics Communications 177, 631–639 (2007)CrossRefGoogle Scholar
  11. 11.
    Greengard, L., Rokhlin, V.: A fast algorithm for particle simulations. Journal of Computational Physics 73, 325–348 (1987)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Gualandris, A., Merritt, D.: Ejection of Supermassive Black Holes from Galaxy Cores. The Astrophysical Journal 678, 780–797 (2008)CrossRefGoogle Scholar
  13. 13.
    Hamada, T., Iitaka, T.: The Chamomile Scheme: An Optimized Algorithm for N-body simulations on Programmable Graphics Processing Units. ArXiv Astrophysics e-prints (March 2007)Google Scholar
  14. 14.
    Harfst, S., Gualandris, A., Merritt, D., Spurzem, R., Portegies Zwart, S., Berczik, P.: Performance analysis of direct N-body algorithms on special-purpose supercomputers. New Astronomy 12, 357–377 (2007)CrossRefGoogle Scholar
  15. 15.
    Hwu, W.-M.-W.: GPU Computing Gems (2011)Google Scholar
  16. 16.
    Khan, F.M., Preto, M., Berczik, P., Berentzen, I., Just, A., Spurzem, R.: Mergers of Unequal-mass Galaxies: Supermassive Black Hole Binary Evolution and Structure of Merger Remnants. The Astrophysical Journal 749, 147 (2012)CrossRefGoogle Scholar
  17. 17.
    Li, S., Liu, F.K., Berczik, P., Chen, X., Spurzem, R.: Interaction of Recoiling Supermassive Black Holes with Stars in Galactic Nuclei. The Astrophysical Journal 748, 65 (2012)CrossRefGoogle Scholar
  18. 18.
    Makino, J.: A Modified Aarseth Code for GRAPE and Vector Processors. Proceedings of Astronomical Society of Japan 43, 859–876 (1991)Google Scholar
  19. 19.
    Makino, J.: Optimal order and time-step criterion for Aarseth-type N-body integrators. The Astrophysical Journal 369, 200–212 (1991)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Makino, J., Aarseth, S.J.: On a Hermite integrator with Ahmad-Cohen scheme for gravitational many-body problems. Publications of the Astronomical Society of Japan 44, 141–151 (1992)Google Scholar
  21. 21.
    Makino, J., Hut, P.: Performance analysis of direct N-body calculations. The Astrophysical Journal Supplement Series 68, 833–856 (1988)CrossRefGoogle Scholar
  22. 22.
    Nitadori, K., Makino, J.: Sixth- and eighth-order Hermite integrator for N-body simulations. New Astronomy 13, 498–507 (2008)CrossRefGoogle Scholar
  23. 23.
    Spurzem, R.: Direct N-body Simulations. Journal of Computational and Applied Mathematics 109, 407–432 (1999)MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    Yang, J., Wang, Y., Chen, Y.: Journal of Computational Physics 221, 799 (2007)zbMATHCrossRefGoogle Scholar
  25. 25.
    Yasuda, K.: Journal of Computational Chemistry 29, 334 (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Peter Berczik
    • 1
    • 2
    • 3
  • Rainer Spurzem
    • 1
    • 3
    • 4
  • Shiyan Zhong
    • 1
  • Long Wang
    • 4
    • 1
  • Keigo Nitadori
    • 5
  • Tsuyoshi Hamada
    • 6
  • Alexander Veles
    • 2
  1. 1.National Astronomical Observatories of China, CASBeijingP.R. China
  2. 2.Main Astronomical Observatory, NASUKyivUkraine
  3. 3.Astronomisches Rechen-Institut, ZAHUniv. of HeidelbergGermany
  4. 4.Kavli Institute for Astronomy and AstrophysicsPeking UniversityBeijingP.R. China
  5. 5.RIKEN InstituteTokyoJapan
  6. 6.Nagasaki Advanced Computing CenterNagasakiJapan

Personalised recommendations