Hybrid Supercomputer Desmos with Torus Angara Interconnect: Efficiency Analysis and Optimization

  • Nikolay Kondratyuk
  • Grigory Smirnov
  • Ekaterina Dlinnova
  • Sergey Biryukov
  • Vladimir StegailovEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 910)


The paper describes the first experience of practical deployment of the hybrid supercomputer Desmos at the Joint Institute for High Temperatures of the Russian Academy of Sciences (JIHT RAS). We consider job scheduling statistics, energy efficiency, case studies of GPU acceleration efficiency and benchmarks of the distributed storage with a parallel file system.


Job accounting and statistics Energy efficiency GPU acceleration Parallel I/O 


  1. 1.
    Stegailov, V., et al.: Early performance evaluation of the hybrid cluster with torus interconnect aimed at molecular-dynamics simulations. In: Wyrzykowski, R., Dongarra, J., Deelman, E., Karczewski, K. (eds.) PPAM 2017 Part I. LNCS, vol. 10777, pp. 327–336. Springer, Cham (2018). Scholar
  2. 2.
    Vecher, V.S., Kondratyuk, N.D., Smirnov, G.S., Stegailov, V.V.: Angara-based hybrid supercomputer for efficient acceleration of computational materials science studies. In: Proceeding of International Conference Russian Supercomputing Days 2017, pp. 557–571 (2017)Google Scholar
  3. 3.
    Neuwirth, S., Frey, D., Nuessle, M., Bruening, U.: Scalable communication architecture for network-attached accelerators. In: 2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA), pp. 627–638 (2015).
  4. 4.
    Puente, V., Beivide, R., Gregorio, J.A., Prellezo, J.M., Duato, J., Izu, C.: Adaptive bubble router: a design to improve performance in torus networks. In: Proceedings of the 1999 International Conference on Parallel Processing, pp. 58–67 (1999).
  5. 5.
    Scott, S.L., Thorson, G.M.: The Cray T3E network: adaptive routing in a high performance 3D torus. In: HOT Interconnects IV. Stanford University, 15–16 August 1996 (1996)Google Scholar
  6. 6.
    Adiga, N.R., et al.: Blue Gene/L torus interconnection network. IBM J. Res. Dev. 49(2), 265–276 (2005). Scholar
  7. 7.
    Gómez-Martín, C., Vega-Rodríguez, M.A., González-Sánchez, J.L.: Fattened backfilling: an improved strategy for job scheduling in parallel systems. J. Parallel Distrib. Comput. 97(Suppl. C), 69–77 (2016). Scholar
  8. 8.
    Kraemer, A., Maziero, C., Richard, O., Trystram, D.: Reducing the number of response time SLO violations by a Cloud-HPC convergence scheduler. In: 2016 2nd International Conference on Cloud Computing Technologies and Applications (CloudTech), pp. 293–300 (2016).
  9. 9.
    Mamaeva, A.A., Voevodin, V.V.: Methods for statistical analysis of large supercomputer job flow. In: Proceeding of International Conference Russian Supercomputing Days 2017, pp. 788–799 (2017)Google Scholar
  10. 10.
    Hoefler, T., Belli, R.: Scientific benchmarking of parallel computing systems: twelve ways to tell the masses when reporting performance results. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015, pp. 73:1–73:12. ACM, New York (2015).
  11. 11.
    Scogland, T., Azose, J., Rohr, D., Rivoire, S., Bates, N., Hackenberg, D.: Node variability in large-scale power measurements: perspectives from the Green500, Top500 and EEHPCWG. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015, pp. 74:1–74:11. ACM, New York (2015).
  12. 12.
    Höhnerbach, M., Ismail, A.E., Bientinesi, P.: The vectorization of the Tersoff multi-body potential: an exercise in performance portability. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2016, pp. 7:1–7:13. IEEE Press, Piscataway (2016).
  13. 13.
    Kutzner, C., Pall, S., Fechner, M., Esztermann, A., de Groot, B.L., Grubmuller, H.: Best bang for your buck: GPU nodes for gromacs biomolecular simulations. J. Comput. Chemis. 36(26), 1990–2008 (2015). Scholar
  14. 14.
    Luehr, N., Ufimtsev, I.S., Martínez, T.J.: Dynamic precision for electron repulsion integral evaluation on graphical processing units (GPUs). J. Chem. Theory Comput. 7(4), 949–954 (2011). Scholar
  15. 15.
    Mills, N., Alex Feltus, F., Ligon III, W.B.: Maximizing the performance of scientific data transfer by optimizing the interface between parallel file systems and advanced research networks. Futur. Gener. Comput. Syst. 79(Part 1), 190–198 (2018). Scholar
  16. 16.
    Yoo, A.B., Jette, M.A., Grondona, M.: SLURM: simple linux utility for resource management. In: Feitelson, D., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2003. LNCS, vol. 2862, pp. 44–60. Springer, Heidelberg (2003). Scholar
  17. 17.
    Vecher, V., Nikolskii, V., Stegailov, V.: GPU-accelerated molecular dynamics: energy consumption and performance. In: Voevodin, V., Sobolev, S. (eds.) RuSCDays 2016. CCIS, vol. 687, pp. 78–90. Springer, Cham (2016). Scholar
  18. 18.
    Stegailov, V., Vecher, V.: Efficiency analysis of intel and AMD x86\(\_\)64 architectures for Ab initio calculations: a case study of VASP. In: Voevodin, V., Sobolev, S. (eds.) RuSCDays 2017. CCIS, vol. 793, pp. 430–441. Springer, Cham (2017). Scholar
  19. 19.
    Stegailov, V., Vecher, V.: Efficiency analysis of Intel, AMD and Nvidia 64-Bit hardware for memory-bound problems: a case study of Ab Initio calculations with VASP. In: Wyrzykowski, R., Dongarra, J., Deelman, E., Karczewski, K. (eds.) PPAM 2017 Part II. LNCS, vol. 10778, pp. 81–90. Springer, Cham (2018). Scholar
  20. 20.
    Smirnov, G.S., Stegailov, V.V.: Anomalous diffusion of guest molecules in hydrogen gas hydrates. High Temp. 53(6), 829–836 (2015). Scholar
  21. 21.
    Orekhov, N.D., Stegailov, V.V.: Simulation of the adhesion properties of the Polyethylene/Carbon nanotube interface. Polym. Sci. Ser. A 58(3), 476–486 (2016). Scholar
  22. 22.
    Pavlov, S.V., Kislenko, S.A.: Effects of carbon surface topography on the electrode/electrolyte interface structure and relevance to li-air batteries. Phys. Chem. Chem. Phys. 18, 30830–30836 (2016). Scholar
  23. 23.
    Antropov, A.S., Fidanyan, K.S., Stegailov, V.V.: Phonon density of states for solid uranium: accuracy of the embedded atom model classical interatomic potential. J. Phys.: Conf. Ser. 946(012094), 94 (2018). Scholar
  24. 24.
    Logunov, M.A., Orekhov, N.D.: Molecular dynamics study of cavitation in carbon nanotube reinforced polyethylene nanocomposite. J. Phys.: Conf. Ser. 946(1), 2044 (2018). Scholar
  25. 25.
    Stegailov, V.V., Orekhov, N.D., Smirnov, G.S.: HPC hardware efficiency for quantum and classical molecular dynamics. In: Malyshkin, V. (ed.) PaCT 2015. LNCS, vol. 9251, pp. 469–473. Springer, Cham (2015). Scholar
  26. 26.
    Aristova, N.M., Belov, G.V.: Refining the thermodynamic functions of scandium triflouride SCF3 in the condensed state. Russ. J. Phys. Chemis. A 90(3), 700–703 (2016). Scholar
  27. 27.
    Kochikov, I.V., Kovtun, D.M., Tarasov, Y.I.: Electron diffraction analysis for the molecules with degenerate large amplitude motions: intramolecular dynamics in arsenic pentafluoride. J. Mol. Struct. 1132, 139–148 (2017). Scholar
  28. 28.
    Stegailov, V.V., Zhilyaev, P.A.: Warm dense gold: effective ionioninteraction and ionisation. Mol. Phys. 114(3–4), 509–518 (2016). Scholar
  29. 29.
    Minakov, D.V., Levashov, P.R.: Melting curves of metals with excited electrons in the quasiharmonic approximation. Phys. Rev. B 92, 224102 (2015). Scholar
  30. 30.
    Minakov, D., Levashov, P.: Thermodynamic properties of LiD under compression with different pseudopotentials for lithium. Comput Mat. Sci. 114, 128–134 (2016). Scholar
  31. 31.
    Eckhardt, W., et al.: 591 TFLOPS multi-trillion particles simulation on SuperMUC. In: Kunkel, J.M., Ludwig, T., Meuer, H.W. (eds.) ISC 2013. LNCS, vol. 7905, pp. 1–12. Springer, Heidelberg (2013). Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Nikolay Kondratyuk
    • 1
    • 2
  • Grigory Smirnov
    • 1
  • Ekaterina Dlinnova
    • 3
  • Sergey Biryukov
    • 4
  • Vladimir Stegailov
    • 1
    Email author
  1. 1.Joint Institute for High Temperatures of the RASMoscowRussia
  2. 2.Moscow Institute of Physics and TechnologyDolgoprudnyRussia
  3. 3.National Research University Higher School of EconomicsMoscowRussia
  4. 4.JSC NICEVTMoscowRussia

Personalised recommendations