Skip to main content
View expanded cover

International Conference on Exascale Applications and Software

EASC 2014: Solving Software Challenges for Exascale pp 3–27Cite as

Tackling Exascale Software Challenges in Molecular Dynamics Simulations with GROMACS

Part of the Lecture Notes in Computer Science book series (LNTCS,volume 8759)

Abstract

GROMACS is a widely used package for biomolecular simulation, and over the last two decades it has evolved from small-scale efficiency to advanced heterogeneous acceleration and multi-level parallelism targeting some of the largest supercomputers in the world. Here, we describe some of the ways we have been able to realize this through the use of parallelization on all levels, combined with a constant focus on absolute performance. Release 4.6 of GROMACS uses SIMD acceleration on a wide range of architectures, GPU offloading acceleration, and both OpenMP and MPI parallelism within and between nodes, respectively. The recent work on acceleration made it necessary to revisit the fundamental algorithms of molecular simulation, including the concept of neighborsearching, and we discuss the present and future challenges we see for exascale simulation - in particular a very fine-grained task parallelism. We also discuss the software management, code peer review and continuous integration testing required for a project of this complexity.

Keywords

  • Particle Mesh Ewald
  • Strong Scaling
  • Remote Direct Memory Access
  • OpenMP Thread
  • Hardware Thread

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The authors ‘S. Páll and M.J. Abraham’ contributed equally.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-15976-8_1
  • Chapter length: 25 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   39.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-15976-8
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   49.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.

Notes

  1. 1.

    Amdahl’s law gives a model for the expected (and maximum) speedup of a program when parallelized over multiple processors with respect to the serial version. It states that the achievable speedup is limited by the sequential part of the program.

  2. 2.

    At the time of that decision, sharing a GPU among multiple MPI ranks was inefficient, so the only efficient way to use multiple cores in a node was with OpenMP within a rank. This constraint has since been relaxed.

  3. 3.

    http://redmine.gromacs.org.

  4. 4.

    http://gerrit.gromacs.org.

  5. 5.

    http://jenkins.gromacs.org.

  6. 6.

    http://www.bsc.es/computer-sciences/extrae.

References

  1. Intel Thread Building Blocks. https://www.threadingbuildingblocks.org

  2. Abraham, M.J., Gready, J.E.: Optimization of parameters for molecular dynamics simulation using smooth particle-mesh Ewald in GROMACS 4.5. J. Comput. Chem. 32(9), 2031–2040 (2011)

    CrossRef  Google Scholar 

  3. Amdahl, G.M.: Validity of the single processor approach to achieving large scale computing capabilities. In: Proceedings of the Spring Joint Computer Conference, AFIPS 1967 (Spring), pp. 483–485. ACM, New York, NY, USA (1967). http://doi.acm.org/10.1145/1465482.1465560

  4. Anderson, J.A., Lorenz, C.D., Travesset, A.: General purpose molecular dynamics simulations fully implemented on graphics processing units. J. Comput. Phys. 227, 5324–5329 (2008)

    CrossRef  Google Scholar 

  5. Andoh, Y., Yoshii, N., Fujimoto, K., Mizutani, K., Kojima, H., Yamada, A., Okazaki, S., Kawaguchi, K., Nagao, H., Iwahashi, K., Mizutani, F., Minami, K., Ichikawa, S.I., Komatsu, H., Ishizuki, S., Takeda, Y., Fukushima, M.: MODYLAS: a highly parallelized general-purpose molecular dynamics simulation program for large-scale systems with long-range forces calculated by Fast Multipole Method (FMM) and highly scalable fine-grained new parallel processing algorithms. J. Chem. Theory Comput. 9(7), 3201–3209 (2013). http://pubs.acs.org/doi/abs/10.1021/ct400203a

    CrossRef  Google Scholar 

  6. Arnold, A., Fahrenberger, F., Holm, C., Lenz, O., Bolten, M., Dachsel, H., Halver, R., Kabadshow, I., Gähler, F., Heber, F., Iseringhausen, J., Hofmann, M., Pippig, M., Potts, D., Sutmann, G.: Comparison of scalable fast methods for long-range interactions. Phys. Rev. E 88, 063308 (2013). http://link.aps.org/doi/10.1103/PhysRevE.88.063308

    CrossRef  Google Scholar 

  7. Bowers, K.J., Dror, R.O., Shaw, D.E.: Overview of neutral territory methods for the parallel evaluation of pairwise particle interactions. J. Phys. Conf. Ser. 16(1), 300 (2005). http://stacks.iop.org/1742-6596/16/i=1/a=041

    CrossRef  Google Scholar 

  8. Bowers, K.J., Dror, R.O., Shaw, D.E.: Zonal methods for the parallel execution of range-limited n-body simulations. J. Comput. Phys. 221(1), 303–329 (2007). http://dx.doi.org/10.1016/j.jcp.2006.06.014

    CrossRef  MATH  MathSciNet  Google Scholar 

  9. Brown, W.M., Wang, P., Plimpton, S.J., Tharrington, A.N.: Implementing molecular dynamics on hybrid high performance computers - short range forces. Comp. Phys. Comm. 182, 898–911 (2011)

    CrossRef  MATH  Google Scholar 

  10. Eastman, P., Pande, V.S.: Efficient nonbonded interactions for molecular dynamics on a graphics processing unit. J. Comput. Chem. 31, 1268–1272 (2010)

    Google Scholar 

  11. Eleftheriou, M., Moreira, J.E., Fitch, B.G., Germain, R.S.: A volumetric FFT for BlueGene/L. In: Pinkston, T.M., Prasanna, V.K. (eds.) HiPC 2003. LNCS (LNAI), vol. 2913, pp. 194–203. Springer, Heidelberg (2003)

    CrossRef  Google Scholar 

  12. Essmann, U., Perera, L., Berkowitz, M.L., Darden, T., Lee, H., Pedersen, L.G.: A smooth particle mesh Ewald method. J. Chem. Phys. 103(19), 8577–8593 (1995)

    CrossRef  Google Scholar 

  13. Faradjian, A., Elber, R.: Computing time scales from reaction coordinates by milestoning. J. Chem. Phys. 120, 10880–10889 (2004)

    CrossRef  Google Scholar 

  14. Hess, B., Kutzner, C., van der Spoel, D., Lindahl, E.: GROMACS 4: algorithms for highly efficient, load-balanced, and scalable molecular simulation. J. Chem. Theor. Comput. 4(3), 435–447 (2008)

    CrossRef  Google Scholar 

  15. Humphrey, W., Dalke, A., Schulten, K.: VMD: visual molecular dynamics. J. Mol. Graph. 14(1), 33–38 (1996)

    CrossRef  Google Scholar 

  16. Jagode, H.: Fourier transforms for the BlueGene/L communication network. Ph.D. thesis, The University of Edinburgh, Edinburgh, UK (2005)

    Google Scholar 

  17. Páll, S., Hess, B.: A flexible algorithm for calculating pair interactions on SIMD architectures. Comput. Phys. Commun. 184(12), 2641–2650 (2013). http://www.sciencedirect.com/science/article/pii/S0010465513001975

    CrossRef  Google Scholar 

  18. Phillips, J.C., Braun, R., Wang, W., Gumbart, J., Tajkhorshid, E., Villa, E., Chipot, C., Skeel, R.D., Kale, L., Schulten, K.: Scalable molecular dynamics with NAMD. J. Comput. Chem. 26, 1781–1802 (2005)

    CrossRef  Google Scholar 

  19. Pronk, S., Larsson, P., Pouya, I., Bowman, G.R., Haque, I.S., Beauchamp, K., Hess, B., Pande, V.S., Kasson, P.M., Lindahl, E.: Copernicus: A new paradigm for parallel adaptive molecular dynamics. In: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2011, pp. 60:1–60:10. ACM, New York, NY, USA (2011) http://doi.acm.org/10.1145/2063384.2063465

  20. Pronk, S., Páll, S., Schulz, R., Larsson, P., Bjelkmar, P., Apostolov, R., Shirts, M.R., Smith, J.C., Kasson, P.M., van der Spoel, D., Hess, B., Lindahl, E.: GROMACS 4.5: a high-throughput and highly parallel open source molecular simulation toolkit. Bioinformatics 29(7), 845–854 (2013). http://bioinformatics.oxfordjournals.org/content/29/7/845.abstract

    CrossRef  Google Scholar 

  21. Reyes, R., Turner, A., Hess, B.: Introducing SHMEM into the GROMACS molecular dynamics application: experience and results. In: Weiland, M., Jackson, A., Johnson, N. (eds.) Proceedings of the 7th International Conference on PGAS Programming Models. The University of Edinburgh, October 2013. http://www.pgas2013.org.uk/sites/default/files/pgas2013proceedings.pdf

  22. Schütte, C., Winkelmann, S., Hartmann, C.: Optimal control of molecular dynamics using Markov state models. Math. Program. (Series B) 134, 259–282 (2012)

    CrossRef  MATH  Google Scholar 

  23. Shirts, M., Pande, V.S.: Screen savers of the world unite!. Science 290(5498), 1903–1904 (2000). http://www.sciencemag.org/content/290/5498/1903.short

    CrossRef  Google Scholar 

  24. Sugita, Y., Okamoto, Y.: Replica-exchange molecular dynamics method for protein folding. Chem. Phys. Lett. 314, 141–151 (1999)

    CrossRef  Google Scholar 

  25. Verlet, L.: Computer “Experiments” on classical fluids. I. Thermodynamical properties of Lennard-Jones molecules. Phys. Rev. 159, 98–103 (1967). http://link.aps.org/doi/10.1103/PhysRev.159.98

    CrossRef  Google Scholar 

  26. Wilson, G., Aruliah, D.A., Brown, C.T., Chue Hong, N.P., Davis, M., Guy, R.T., Haddock, S.H.D., Huff, K.D., Mitchell, I.M., Plumbley, M.D., Waugh, B., White, E.P., Wilson, P.: Best practices for scientific computing. PLoS Biol 12(1), e1001745 (2014). http://dx.doi.org/10.1371/journal.pbio.1001745

    CrossRef  Google Scholar 

  27. Yokota, R., Barba, L.A.: A tuned and scalable fast multipole method as a preeminent algorithm for exascale systems. Int. J. High Perform. Comput. Appl. 26(4), 337–346 (2012). http://hpc.sagepub.com/content/26/4/337.abstract

    CrossRef  Google Scholar 

Download references

Acknowledgments

This work was supported by the European research Council (258980, BH), the Swedish e-Science research center, and the EU FP7 CRESTA project (287703). Computational resources were provided by the Swedish National Infrastructure for computing (grants SNIC 025/12-32 & 2013-26/24) and the Leibniz Supercomputing Center.

Author information

Authors and Affiliations

  1. Department of Theoretical Biophysics, Science for Life Laboratory, KTH Royal Institute of Technology, 17121, Solna, Sweden

    Szilárd Páll, Mark James Abraham, Berk Hess & Erik Lindahl

  2. Theoretical and Computational Biophysics Department, Max Planck Institute for Biophysical Chemistry, Am Fassberg 11, 37077, Göttingen, Germany

    Carsten Kutzner

  3. Department of Biochemistry & Biophysics, Center for Biomembrane Research, Stockholm University, 10691, Stockholm, Sweden

    Erik Lindahl

Authors
  1. Szilárd Páll
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark James Abraham
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carsten Kutzner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Berk Hess
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Erik Lindahl
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Erik Lindahl .

Editor information

Editors and Affiliations

  1. KTH Royal Institute of Technology, Stockholm, Sweden

    Stefano Markidis

  2. KTH Royal Institute of Technology, Stockholm, Sweden

    Erwin Laure

Rights and permissions

Reprints and Permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Páll, S., Abraham, M.J., Kutzner, C., Hess, B., Lindahl, E. (2015). Tackling Exascale Software Challenges in Molecular Dynamics Simulations with GROMACS. In: Markidis, S., Laure, E. (eds) Solving Software Challenges for Exascale. EASC 2014. Lecture Notes in Computer Science(), vol 8759. Springer, Cham. https://doi.org/10.1007/978-3-319-15976-8_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-15976-8_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-15975-1

  • Online ISBN: 978-3-319-15976-8

  • eBook Packages: Computer ScienceComputer Science (R0)