Progress in Scaling Biomolecular Simulations to Petaflop Scale Platforms

  • Blake G. Fitch
  • Aleksandr Rayshubskiy
  • Maria Eleftheriou
  • T. J. Christopher Ward
  • Mark Giampapa
  • Michael C. Pitman
  • Robert S. Germain
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4375)


This paper describes some of the issues involved with scaling biomolecular simulations onto massively parallel machines drawing on the Blue Matter application team’s experiences with Blue Gene/L. Our experiences in scaling biomolecular simulation to one atom/node on BG/L should be relevant to scaling biomolecular simulations onto larger peta-scale platforms because the path to increased performance is through the exploitation of increased concurrency so that even larger systems will have to operate in the extreme strong scaling regime. Petascale platforms also present challenges with regard to the correctness of biomolecular simulations since longer time-scale simulations are more likely to encounter significant energy drift. Total energy drift data for a microsecond-scale simulation is presented along with the measured scalability of various components of a molecular dynamics time-step.


Molecular Dynamic Molecular Simulation Replica Exchange Replica Exchange Molecular Dynamic Strong Scaling 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Allen, F., et al.: Blue Gene: a vision for protein science using a petaflop supercomputer. IBM Journal of Research and Development 40(2), 310–327 (2001)Google Scholar
  2. 2.
    Benettin, G., Giorgilli, A.: On the hamiltonian interpolation of near-to-the-identity symplectic mappings with application to symplectic integration algorithms. J. Statist. Phys. 74, 1117–1143 (1994)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Deserno, M., Holm, C.: How to mesh up Ewald sums. ii. an accurate error estimate for the particle-particle-particle-mesh algorithm. J. Chem. Phys. 109(18), 7694–7701 (1998)CrossRefGoogle Scholar
  4. 4.
    Eleftheriou, M., Fitch, B., Rayshubskiy, A., Ward, T.J.C., Germain, R.S.: Performance measurements of the 3d FFT on the Blue Gene/L supercomputer. In: Cunha, J.C., Medeiros, P.D. (eds.) Euro-Par 2005. LNCS, vol. 3648, pp. 795–803. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  5. 5.
    Eleftheriou, M., Germain, R., Royyuru, A., Zhou, R.: Thermal denaturing of mutant lysozyme with both oplsaa and charmm force fields. To appear in J. Am. Chem. Soc. (2006)Google Scholar
  6. 6.
    Eleftheriou, M., Rayshubskiy, A., Pitera, J.W., Fitch, B.G., Zhou, R., Germain, R.S.: Parallel implementation of the replica exchange molecular dynamics algorithm on Blue Gene/L. In: Fifth IEEE International Workshop on High Performance Computational Biology, April 2006, IEEE Computer Society Press, Los Alamitos (2006)Google Scholar
  7. 7.
    Engle, R.D., Skeel, R.D., Drees, M.: Monitoring energy drift with shadow hamiltonians. Journal of Computational Physics 206(2), 432–452 (2005)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Fitch, B.G., Germain, R.S., Mendell, M., Pitera, J., Pitman, M., Rayshubskiy, A., Sham, Y., Suits, F., Swope, W., Ward, T.J.C., Zhestkov, Y., Zhou, R.: Blue Matter, an application framework for molecular simulation on Blue Gene. Journal of Parallel and Distributed Computing 63, 759–773 (2003)CrossRefGoogle Scholar
  9. 9.
    Fitch, B.G., Rayshubskiy, A., Eleftheriou, M., Ward, T.J.C., Giampapa, M., Pitman, M.C., Germain, R.S.: Blue matter: Approaching the limits of concurrency for molecular dynamics. Research Report RC 23956, IBM Research Division (April 2006). To appear in the Proceedings of the 2006 ACM/IEEE conference on SupercomputingGoogle Scholar
  10. 10.
    Fitch, B.G., Rayshubskiy, A., Eleftheriou, M., Ward, T.J.C., Giampapa, M., Zhestkov, Y., Pitman, M.C., Suits, F., Grossfield, A., Pitera, J., Swope, W., Zhou, R., Feller, S., Germain, R.S.: Blue Matter: Strong scaling of molecular dynamics on Blue Gene/L. In: Alexandrov, V.N., van Albada, G.D., Sloot, P.M.A., Dongarra, J.J. (eds.) ICCS 2006. LNCS, vol. 3992, pp. 846–854. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  11. 11.
    Fitch, B.G., Rayshubskiy, A., Eleftheriou, M., Ward, T.J.C., Giampapa, M., Zhestkov, Y., Pitman, M.C., Suits, F., Grossfield, A., Pitera, J., Swope, W., Zhou, R., Germain, R.S., Feller, S.: Blue matter: Strong scaling of molecular dynamics on Blue Gene/L. Research Report RC23688, IBM Research Division (August 2005)Google Scholar
  12. 12.
    Frenkel, D., Smit, B.: Understanding Molecular Simulation. Academic Press, San Diego (1996)zbMATHGoogle Scholar
  13. 13.
    Gara, A., et al.: Overview of the Blue Gene/L system architecture. IBM Journal of Research and Development 49(2/3), 195–212 (2005)CrossRefGoogle Scholar
  14. 14.
    Germain, R.S., Fitch, B., Rayshubskiy, A., Eleftheriou, M., Pitman, M.C., Suits, F., Giampapa, M., Ward, T.J.C.: Blue Matter on Blue Gene/L: massively parallel computation for biomolecular simulation. In: CODES+ISSS ’05: Proceedings of the 3rd IEEE/ACM/IFIP international conference on Hardware/software codesign and system synthesis, pp. 207–212. ACM Press, New York (2005)Google Scholar
  15. 15.
    Germain, R.S., Zhestkov, Y., Eleftheriou, M., Rayshubskiy, A., Suits, F., Ward, T.J.C., Fitch, B.G.: Early performance data on the Blue Matter molecular simulation framework. IBM Journal of Research and Development 49(2/3), 447–456 (2005)CrossRefGoogle Scholar
  16. 16.
    Grossfield, A., Feller, S.E., Pitman, M.C.: A role for direct interactions in the modulation of rhodopsin by omega-3 polyunsaturated lipids. PNAS 103(13), 4888–4893 (2006)CrossRefGoogle Scholar
  17. 17.
    Leimkuhler, B., Reich, S.: Simulating Hamiltonian Dynamics. Cambridge Monographs in Applied and Computational Mathematics, vol. 14. Cambridge University Press, Cambridge (2004)zbMATHGoogle Scholar
  18. 18.
    Pitman, M.C., Grossfield, A., Suits, F., Feller, S.E.: Role of cholesterol and polyunsaturated chains in lipid-protein interactions: Molecular dynamics simulation of rhodopsin in a realistic membrane environment. Journal of the American Chemical Society 127(13), 4576–4577 (2005)CrossRefGoogle Scholar
  19. 19.
    Reich, S.: Backward error analysis for numerical integrators. SIAM Journal on Numerical Analysis 36(5), 1549–1570 (1999)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Sagui, C., Darden, T.: Multigrid methods for classical molecular dynamics simulations of biomolecules. Journal of Chemical Physics 114(15), 6578–6591 (2001)CrossRefGoogle Scholar
  21. 21.
    Sexton, J.C., Weingarten, D.H.: Hamiltonian evolution for the hybrid Monte Carlo algorithm. Nuclear Physics B 380, 665–677 (1992)CrossRefGoogle Scholar
  22. 22.
    Sugita, Y., Okamoto, Y.: Replica-exchange molecular dynamics method for protein folding. Chem. Phys. Lett. 314, 141–151 (1999)CrossRefGoogle Scholar
  23. 23.
    Swope, W.C, Andersen, H.C., Berens, P.H., Wilson, K.R.: A computer simulation method for the calculation of equilibrium constants for the formation of physical clusters of molecules: Application to small water clusters. Journal of Chemical Physics 76, 637–649 (1982)CrossRefGoogle Scholar
  24. 24.
    Toxvaerd, S.: Hamiltonians for discrete dynamics. Phys. Rev. E 50(3), 2271–2274 (1994)CrossRefGoogle Scholar
  25. 25.
    Tuckerman, M., Berne, B.J., Martyna, G.J.: Reversible multiple time scale molecular dynamics. J. Chem. Phys. 97(3), 1990–2001 (1992)CrossRefGoogle Scholar
  26. 26.
    Zhou, R., Harder, E., Xu, H., Berne, B.J.: Efficient multiple time step method for use with Ewald and particle mesh Ewald for large biomolecular systems. Journal of Chemical Physics 115(5), 2348–2358 (2001)CrossRefGoogle Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Blake G. Fitch
    • 1
  • Aleksandr Rayshubskiy
    • 1
  • Maria Eleftheriou
    • 1
  • T. J. Christopher Ward
    • 2
  • Mark Giampapa
    • 1
  • Michael C. Pitman
    • 1
  • Robert S. Germain
    • 1
  1. 1.IBM Thomas J. Watson Research Center, 1101 Kitchawan Road/Route 134, Yorktown Heights, NY 10598USA
  2. 2.IBM Hursley Park, Hursley, Hursley SO212JNUK

Personalised recommendations