Advertisement

Blue Matter: Strong Scaling of Molecular Dynamics on Blue Gene/L

  • Blake G. Fitch
  • Aleksandr Rayshubskiy
  • Maria Eleftheriou
  • T. J. Christopher Ward
  • Mark Giampapa
  • Yuri Zhestkov
  • Michael C. Pitman
  • Frank Suits
  • Alan Grossfield
  • Jed Pitera
  • William Swope
  • Ruhong Zhou
  • Scott Feller
  • Robert S. Germain
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3992)

Abstract

This paper presents strong scaling performance data for the Blue Matter molecular dynamics framework using a novel n-body spatial decomposition and a collective communications technique implemented on both MPI and low level hardware interfaces. Using Blue Matter on Blue Gene/L, we have measured scalability through 16,384 nodes with measured time per time-step of under 2.3 milliseconds for a 43,222 atom protein/lipid system. This is equivalent to a simulation rate of over 76 nanoseconds per day and represents an unprecedented time-to-solution for biomolecular simulation as well as continued speed-up to fewer than three atoms per node. On a smaller, solvated lipid system with 13,758 atoms, we have achieved continued speedups through fewer than one atom per node and less than 2 milliseconds/time-step. On a 92,224 atom system, we have achieved floating point performance of over 1.8 TeraFlops/second on 16,384 nodes. Strong scaling of fixed-size classical molecular dynamics of biological systems to large numbers of nodes is necessary to extend the simulation time to the scale required to make contact with experimental data and derive biologically relevant insights.

Keywords

Classical Molecular Dynamic Strong Scaling Biomolecular Simulation Range Electrostatic Interaction Float Point Performance 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Fitch, B., Germain, R., Mendell, M., Pitera, J., Pitman, M., Rayshubskiy, A., Sham, Y., Suits, F., Swope, W., Ward, T., Zhestkov, Y., Zhou, R.: Blue Matter, an application framework for molecular simulation on Blue Gene. Journal of Parallel and Distributed Computing 63, 759–773 (2003)CrossRefGoogle Scholar
  2. 2.
    Germain, R., Zhestkov, Y., Eleftheriou, M., Rayshubskiy, A., Suits, F., Ward, T., Fitch, B.: Early performance data on the Blue Matter molecular simulation framework. IBM Journal of Research and Development 49(2/3), 447–456 (2005)CrossRefGoogle Scholar
  3. 3.
    Germain, R.S., Fitch, B., Rayshubskiy, A., Eleftheriou, M., Pitman, M.C., Suits, F., Giampapa, M., Ward, T.C.: Blue Matter on Blue Gene/L: massively parallel computation for biomolecular simulation. In: CODES+ISSS 2005: Proceedings of the 3rd IEEE/ACM/IFIP international conference on Hardware/software codesign and system synthesis, pp. 207–212. ACM Press, New York (2005)CrossRefGoogle Scholar
  4. 4.
    Allen, F., et al.: Blue Gene: a vision for protein science using a petaflop supercomputer. IBM Systems Journal 40(2), 310–327 (2001)CrossRefGoogle Scholar
  5. 5.
    Pitman, M.C., Grossfield, A., Suits, F., Feller, S.E.: Role of cholesterol and polyunsaturated chains in lipid-protein interactions: Molecular dynamics simulation of rhodopsin in a realistic membrane environment. Journal of the American Chemical Society 127(13), 4576–4577 (2005)CrossRefGoogle Scholar
  6. 6.
    Gara, A., et al.: Overview of the Blue Gene/L system architecture. IBM Journal of Research and Development 49(2/3), 195–212 (2005)CrossRefGoogle Scholar
  7. 7.
    Adiga, N., et al.: Blue Gene/L torus interconnection network. IBM Journal of Research and Development 49(2/3), 265–276 (2005)CrossRefGoogle Scholar
  8. 8.
    Bader, J., Chandler, D.: Computer simulation study of the mean forces between ferrous and ferric ions in water. The Journal of Physical Chemistry 96(15) (1992)Google Scholar
  9. 9.
    De Leeuw, S., Perram, J., Smith, E.: Simulation of electrostatic systems in periodic boundary conditions I. lattice sums and dielectric constants. Proc. Roy. Soc. Lond. A 373, 27–56 and references therein (1980)Google Scholar
  10. 10.
    Deserno, M., Holm, C.: How to mesh up ewald sums. J. Chem. Phys. 109(18), 7678–7693 (1998)CrossRefGoogle Scholar
  11. 11.
    Plimpton, S., Hendrickson, B.: A new parallel method for molecular dynamics simulation of macromolecular systems. Journal of Computational Chemistry 17(3), 326–337 (1996)CrossRefGoogle Scholar
  12. 12.
    Kale, L., Skeel, R., Bhandarkar, M., Brunner, R., Gursoy, A., Krawetz, N., Phillips, J., Shinozaki, A., Varadarajan, K., Schulten, K.: NAMD2: Greater scalability for parallel molecular dynamics. Journal of Computational Physics 151, 283–312 (1999)MATHCrossRefGoogle Scholar
  13. 13.
    Phillips, J., Zheng, G., Kumar, S., Kale, L.: NAMD: biomolecular simulation on thousands of processors. In: Supercomputing 2002 Proceedings (2002), http://www.sc2002.org/paperpdfs/pap.pap277.pdf
  14. 14.
    Kumar, S., Huang, C., Almasi, G., Kale, L.V.: Achieving strong scaling with NAMD on Blue Gene/l. In: 20th IEEE International Parallel and Distributed Processing Symposium. IEEE, Los Alamitos (2006), http://charm.cs.uiuc.edu/papers/NAMDIDPDS06.pdf Google Scholar
  15. 15.
    Snir, M.: A note on n-body computations with cutoffs. Theory of Computing Systems 37, 295–318 (2004), doi:10.1007/s00224-003-1071-0.MathSciNetMATHCrossRefGoogle Scholar
  16. 16.
    Shaw, D.E.: A fast, scalable method for the parallel evaluation of distance-limited pairwise particle interactions. Journal of Computational Chemistry 26(13), 1318–1328 (2005)CrossRefGoogle Scholar
  17. 17.
    Kale, L., Kumar, S., Varadarajan, K.: A framework for collective personalized communication. In: Parallel and Distributed Processing Symposium, 2003. Proceedings International. IEEE, Los Alamitos (2003), http://dx.doi.org/10.1109/IPDPS.2003.1213166 Google Scholar
  18. 18.
    Swope, W., Andersen, H., Berens, P., Wilson, K.: A computer simulation method for the calculation of equilibrium constants for the formation of physical clusters of molecules: Application to small water clusters. Journal of Chemical Physics 76, 637–649 (1982)CrossRefGoogle Scholar
  19. 19.
    Tuckerman, M., Berne, B., Martyna, G.: Reversible multiple time scale molecular dynamics. J. Chem. Phys. 97(3), 1990–2001 (1992)CrossRefGoogle Scholar
  20. 20.
    Almasi, G., et al.: Design and implementation of message-passing services for the Blue Gene/L supercomputer. IBM Journal of Research and Development 49(2/3), 393–406 (2005)MATHCrossRefGoogle Scholar
  21. 21.
    Giampapa, M., et al.: Blue Gene/L advanced diagnostics environment. IBM Journal of Research and Development 49(2/3), 319–332 (2005)CrossRefGoogle Scholar
  22. 22.
    Eleftheriou, M., Fitch, B., Rayshubskiy, A., Ward, T., Germain, R.: Performance measurements of the 3d FFT on the Blue Gene/L supercomputer. In: Cunha, J.C., Medeiros, P.D. (eds.) Euro-Par 2005. LNCS, vol. 3648, pp. 795–803. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  23. 23.
    Suits, F., Pitman, M.C., Feller, S.E.: Molecular dynamics investigation of the structural properties of phosphatidylethanolamine lipid bilayers. Journal of Chemical Physics 122(24) (2005)Google Scholar
  24. 24.
    Pitman, M.C., Suits, F., Gawrisch, K., Feller, S.E.: Molecular dynamics investigation of dynamical properties of phosphatidylethanolamine lipid bilayers. Journal of Chemical Physics 122(24) (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Blake G. Fitch
    • 1
  • Aleksandr Rayshubskiy
    • 1
  • Maria Eleftheriou
    • 1
  • T. J. Christopher Ward
    • 2
  • Mark Giampapa
    • 1
  • Yuri Zhestkov
    • 1
  • Michael C. Pitman
    • 1
  • Frank Suits
    • 1
  • Alan Grossfield
    • 1
  • Jed Pitera
    • 3
  • William Swope
    • 3
  • Ruhong Zhou
    • 1
  • Scott Feller
    • 4
  • Robert S. Germain
    • 1
  1. 1.IBM Thomas J. Watson Research CenterYorktown HeightsUSA
  2. 2.IBM Hursley Park, HursleyHursleyUnited Kingdom
  3. 3.IBM Almaden Research CenterSan Jose
  4. 4.Department of ChemistryWabash CollegeCrawfordsville

Personalised recommendations