Advertisement

The European Physical Journal Special Topics

, Volume 227, Issue 14, pp 1757–1778 | Cite as

Adaptive grid implementation for parallel continuum mechanics methods in particle simulations

  • Miriam MehlEmail author
  • Michael Lahnert
Review
  • 21 Downloads
Part of the following topical collections:
  1. Particle Methods in Natural Science and Engineering

Abstract

In this tutorial review paper, we present our minimally invasive approach for integrating dynamically adaptive tree-structured grids into existing simulation software that has been developed for regular Cartesian grids. We introduce different physical models that we target and that span a wide range of typical simulation characteristics – from grid-based Lattice-Boltzmann, finite volume and finite difference discretized models to particle-based molecular dynamics models. We derive the respective typical data access requirements and extensions of the algorithms to adaptively refined grids along with possible grid adaptivity criteria. In addition, after introducing basics of tree-structured adaptively refined grids, we present the adaptive grid framework p4est and our enhancement of p4est in order to provide a grid and partitioning infrastructure that can easily be used in existing simulation codes. Finally, we explain how such a grid infrastructure can be integrated into regular grid codes in general in three major steps and how we integrated p4est in the soft matter simulation package ESPResSo in particular. A summary of results from previously published performance and scalability studies together with new results for more realistic coupled simulation scenarios shows the efficiency and validity of the resulting new version of ESPResSo.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    T. Weinzierl, M. Mehl, SIAM J. Sci. Comput. 33, 2732 (2011)MathSciNetCrossRefGoogle Scholar
  2. 2.
    R.S. Sampath, S.S. Adavani, H. Sundar, I. Lashuk, G. Biros, Dendro: parallel algorithms for multigrid and AMR methods on 2:1 balanced octrees, in SC ‘08: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing (2008), pp. 1–12Google Scholar
  3. 3.
    C. Burstedde, L.C. Wilcox, O. Ghattas, SIAM J. Sci. Comput. 33, 1103 (2011)MathSciNetCrossRefGoogle Scholar
  4. 4.
    H. Klimach, K. Jain, S.P. Roller, End-to-end parallel simulations with APES, in Advances in parallel computing, (IOS Press, 2014), Vol. 25, pp. 703–711Google Scholar
  5. 5.
    A. Lintermann, S. Schlimpert, J.H. Grimmen, C. Günther, M. Meinke, W. Schrüder, Comput. Methods Appl. Mech. Eng. (2014) 277, 131ADSCrossRefGoogle Scholar
  6. 6.
    T. Tu, D.R. O’Hallaron, O. Ghattas, Scalable parallel octree meshing for teraScale applications, in SC ‘05: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (ACM/IEEE, 2005)Google Scholar
  7. 7.
    F. Schornbaum, U. Rüde, SIAM J. Sci. Comput. 38, C96 (2016)CrossRefGoogle Scholar
  8. 8.
    F. Schornbaum, U. Rüde, SIAM J. Sci. Comput. 40, C358 (2018)CrossRefGoogle Scholar
  9. 9.
    S.G. Parker, Future Gener. Comput. Syst. 22, 204 (2006)MathSciNetCrossRefGoogle Scholar
  10. 10.
    M. Wahib, N. Maruyama, T. Aoki, Daino: A high-level framework for parallel and effcient AMR on GPUs, in SC16: International Conference for High Performance Computing, Networking, Storage and Analysis (IEEE, 2016), pp. 621–632Google Scholar
  11. 11.
    A. Dubey, A. Almgren, J. Bell, M. Berzins, S. Brandt, G. Bryan, P. Colella, D. Graves, M. Lijewski, F. Lüffler, B. O’Shea, J. Parallel Distrib. Comput. 74, 3217 (2014)CrossRefGoogle Scholar
  12. 12.
    H.J. Limbach, A. Arnold, B.A. Mann, C. Holm, Comput. Phys. Commun. 174, 704 (2006)ADSCrossRefGoogle Scholar
  13. 13.
    A. Arnold, O. Lenz, S. Kesselheim, R. Weeber, F. Fahrenberger, D. Roehm, P. Košovan, C. Holm, ESPResSo 3.1: Molecular dynamics software for coarse-grained models, in Meshfree methods for partial differential equations VI, Lecture Notes in Computational Science and Engineering, edited by M. Griebel, M.A. Schweitzer (Springer, Berlin, Heidelberg, 2012), Vol. 89, pp. 1–23Google Scholar
  14. 14.
    M. Bader, C. Böck, J. Schwaiger, C. Vigh, SIAM J. Sci. Comput. 32, 212 (2010)MathSciNetCrossRefGoogle Scholar
  15. 15.
    M. Bader, Space-filling curves: an introduction with applications in scientific computing (Springer, Heidelberg, New York, 2013)Google Scholar
  16. 16.
    G. Inci, A. Arnold, A. Kronenburg, R. Weeber, Aerosol Sci. Technol. 48, 842 (2014)ADSCrossRefGoogle Scholar
  17. 17.
    J. Hüpfner, T. Richter, P. Košovan, C. Holm, M. Wilhelm, Progr. Colloid. Polym. Sci. 140, 140 (2013)Google Scholar
  18. 18.
    B.J. Reynwar, G. Illya, V.A. Harmandaris, M.M. Müller, K. Kremer, M. Deserno, Nature 447, 461 (2007)ADSCrossRefGoogle Scholar
  19. 19.
    M. Kuron, A. Arnold, Eur. Phys. J. E 38, 20 (2015)CrossRefGoogle Scholar
  20. 20.
    S. Kesselheim, M. Sega, C. Holm, Soft Matter 8, 9480 (2012)ADSCrossRefGoogle Scholar
  21. 21.
    K. Breitsprecher, P. Košovan, C. Holm, J. Phys.: Condens. Matter 26, 284108 (2014)Google Scholar
  22. 22.
    K. Breitsprecher, P. Košovan, C. Holm, J. Phys.: Condens. Matter 26, 284114 (2014)Google Scholar
  23. 23.
    S. Succi, The Lattice Boltzmann equation for fluid dynamics and beyond (Clarendon Press, 2001)Google Scholar
  24. 24.
    D.A. Wolf-Gladrow, Lattice-gas cellular automata and Lattice Boltzmann models – an introduction (Springer, 2000)Google Scholar
  25. 25.
    S. Succi, The Lattice Boltzmann equation: for complex states of flowing matter (Oxford University Press, 2018)Google Scholar
  26. 26.
    P.L. Bhatnagar, E.P. Gross, M. Krook, Phys. Rev. 94, 511 (1954)ADSCrossRefGoogle Scholar
  27. 27.
    D. d’Humières, S. Succi, D. d’Humières, I. Ginzburg, M. Krafczyk, P. Lallemand, L.-S. Luo, Philos. Trans. R. Soc. A: Math. Phys. Eng. Sci. 360, 437 (2002)ADSCrossRefGoogle Scholar
  28. 28.
    L.-S. Luo, W. Liao, X. Chen, Y. Peng, W. Zhang, Phys. Rev. E 83, 056710 (2011)ADSCrossRefGoogle Scholar
  29. 29.
    I. Ginzburg, F. Verhaeghe, D. d’Humières, Commun. Comput. Phys. 3, 427 (2008)MathSciNetGoogle Scholar
  30. 30.
    I. Ginzburg, F. Verhaeghe, D. d’Humieres. Commun. Comput. Phys. 3, 519 (2008)Google Scholar
  31. 31.
    S. Hou, J. Sterling, S. Chen, G.D. Doolen, A Lattice Boltzmann subgrid model for high Reynolds number flow, in Pattern formation and lattice gas automata (American Mathematical Society, 1996), Vol. 6, pp. 151–166Google Scholar
  32. 32.
    R. Brownlee, A. Gorban, J. Levesley, Physica A 387, 385 (2008)ADSMathSciNetCrossRefGoogle Scholar
  33. 33.
    X. Shan, X.-F. Yuan, H. Chen, J. Fuid Mech. 550, 413 (2006)ADSCrossRefGoogle Scholar
  34. 34.
    R. Deiterding, S.L. Wood, J. Phys.: Conf. Ser. 753, 082005 (2016)Google Scholar
  35. 35.
    M. Geier, A. Greiner, J.G. Korvink, Phys. Rev. E 73, 066705 (2006)ADSCrossRefGoogle Scholar
  36. 36.
    P. Asinari, Phys. Rev. E 78, 016701 (2008)ADSCrossRefGoogle Scholar
  37. 37.
    S. Seeger, H. Hoffmann, Continuum Mech. Thermodyn. 12, 403 (2000)ADSMathSciNetCrossRefGoogle Scholar
  38. 38.
    M. Geier, M. Schönherr, A. Pasquali, M. Krafczyk, Comput. Math. Appl. 70, 507 (2015)MathSciNetCrossRefGoogle Scholar
  39. 39.
    E.K. Far, M. Geier, K. Kutscher, M. Krafczyk, Comput. Fluids 140, 222 (2016)MathSciNetCrossRefGoogle Scholar
  40. 40.
    K. Kutscher, M. Geier, M. Krafczyk, Comput. Fluids, in pressGoogle Scholar
  41. 41.
    M. Geier, M. Schönherr, Computation 5, 15 (2017)CrossRefGoogle Scholar
  42. 42.
    M. Wittmann, T. Zeiser, G. Hager, G. Wellein, Modeling and analyzing performance for highly optimized propagation steps of the Lattice Boltzmann method on sparse lattices, arXiv:1410.0412 (2014)Google Scholar
  43. 43.
    U.D. Schiller, Thermal uctuations and boundary conditions in the Lattice Boltzmann method, Ph.D. thesis, Johannes Gutenberg-Universität, Mainz, 2008Google Scholar
  44. 44.
    L. Li, R. Mei, J.F. Klausner, J. Comput. Phys. 237, 366 (2013)ADSMathSciNetCrossRefGoogle Scholar
  45. 45.
    A. Pasquali, M. Geier, M. Krafczyk, Comput. Math. Appl., in pressGoogle Scholar
  46. 46.
    A. Fakhari, T. Lee, Phys. Rev. E 89, 033310 (2014)ADSCrossRefGoogle Scholar
  47. 47.
    A. Fakhari, T. Lee, Comput. Fluids 107, 205 (2015)MathSciNetCrossRefGoogle Scholar
  48. 48.
    O. Filippova, D. Hänel, J. Comput. Phys. 147, 219 (1998)ADSCrossRefGoogle Scholar
  49. 49.
    D. Yu, R. Mei, W. Shyy, Int. J. Numer. Methods Fluids 39, 99 (2002)ADSCrossRefGoogle Scholar
  50. 50.
    J. Tölke, S. Freudiger, M. Krafczyk, Comput. Fluids 35, 820 (2006)CrossRefGoogle Scholar
  51. 51.
    M. Geier, A. Greiner, J.G. Korvink, Eur. Phys. J. Special Topics 171, 173 (2009)ADSCrossRefGoogle Scholar
  52. 52.
    H. Chen, Phys. Rev. E 58, 3955 (1998)ADSCrossRefGoogle Scholar
  53. 53.
    H. Chen, O. Filippova, J. Hoch, K. Molvig, R. Shock, C. Teixeira, R. Zhang, Physica A 362, 158 (2006)ADSCrossRefGoogle Scholar
  54. 54.
    M. Rohde, D. Kandhai, J.J. Derksen, H. Van den Akker, Int. J. Numer. Methods Fluids 51, 439 (2006)ADSCrossRefGoogle Scholar
  55. 55.
    P. Neumann, Hybrid multiscale simulation approaches for micro- and nano ows, Ph.D. thesis, Technische Universität München, 2013Google Scholar
  56. 56.
    M. Lahnert, C. Burstedde, F. Weik, Towards Lattice-Boltzmann on dynamically adaptive grids – minimally-invasive grid exchange in ESPResSo. Englisch, in ECCOMAS Congress 2016, VII European Congress on Computational Methods in Applied Sciences and Engineering, edited by M. Papadrakakis, et al.(ECCOMAS, 2016)Google Scholar
  57. 57.
    M. Mehl, T. Neckel, P. Neumann, Int. J. Numer. Methods Fluids 65, 67 (2010)CrossRefGoogle Scholar
  58. 58.
    M. Lahnert, C. Burstedde, M. Mehl, Scalable Lattice-Boltzmann Simulation on Dynamically Adaptive Grids, submittedGoogle Scholar
  59. 59.
    F. Capuani, I. Pagonabarraga, D. Frenkel, J. Chem. Phys. 121, 973 (2004)ADSCrossRefGoogle Scholar
  60. 60.
    G. Rempfer, Electrokinetic transport phenomena in soft-matter systems, Ph.D. thesis, University of Stuttgart, 2018Google Scholar
  61. 61.
    G. Rempfer, G.B. Davies, C. Holm, J. de Graaf, J. Chem. Phys. 145, 044901 (2016)ADSCrossRefGoogle Scholar
  62. 62.
    P.P. Ewald, Ann. Phys. 369, 253 (1921)CrossRefGoogle Scholar
  63. 63.
    S.W. de Leeuw, J.W. Perram, E.R. Smith, Proc. R. Soc. London A: Math. Phys. Eng. Sci. 373, 27 (1980)ADSCrossRefGoogle Scholar
  64. 64.
    R.W. Hockney, J.W. Eastwood, Computer simulation using particles (Taylor & Francis, Inc, Bristol, PA, USA, 1988)Google Scholar
  65. 65.
    M. Deserno, C. Holm, J. Chem. Phys. 109, 7678 (1998)ADSCrossRefGoogle Scholar
  66. 66.
    M. Deserno, C. Holm, J. Chem. Phys. 109, 7694 (1998)ADSCrossRefGoogle Scholar
  67. 67.
    A. Arnold, K. Breitsprecher, F. Fahrenberger, S. Kesselheim, O. Lenz, C. Holm, Entropy 15, 4569 (2013)ADSCrossRefGoogle Scholar
  68. 68.
    J.E. Jones, Proc. Roy. Soc. London A: Math. Phys. Eng. Sci. 106, 463 (1924)ADSCrossRefGoogle Scholar
  69. 69.
    J.E. Lennard-Jones, Proc. Phys. Soc. 43, 461 (1931)ADSCrossRefGoogle Scholar
  70. 70.
    M. Buchholz, Framework zur Parallelisierung vonMolekulardynamiksimulationen in verfahrenstechnischen Anwendungen, Dissertation, München, Institut für Informatik, Technische Universität MünchenGoogle Scholar
  71. 71.
    T. Isaac, C. Burstedde, L.C. Wilcox, O. Ghattas, SIAM J. Sci. Comput. 37, C497 (2015)CrossRefGoogle Scholar
  72. 72.
    M. Lahnert, T. Aoki, C. Burstedde, M. Mehl, Minimally-invasive integration of p4est in ESPResSo for adaptive Lattice-Boltzmann, in The 30th Computational Fluid Dynamics Symposium (Japan Society of Fluid Mechanics, 2016)Google Scholar
  73. 73.
    C. Burstedde, O. Ghattas, L.C. Wilcox, Towards adaptive mesh PDE simulations on petascale computers, in Proceedings of Teragrid (2008), Vol. 8Google Scholar
  74. 74.
    I. Tischler, Implementing adaptive electrokinetics in ESPResSo, MA thesis, University of Stuttgart, 2018Google Scholar
  75. 75.
    M. Brunn, Coupling of particle simulation and Lattice Boltzmann background flow on adaptive grids, MA thesis, Universität Stuttgart, 2017Google Scholar
  76. 76.
    S. Hirschmann, M. Brunn, M. Lahnert, C.W. Glass, D. Pflüger, Load Balancing with p4est for Short-Range Molecular Dynamics with ESPResSo, in Advances in parallel computing, edited by S. Bassini, et al. (IOS Press, 2017), Vol. 32, pp. 455–464Google Scholar
  77. 77.
    P. Ahlrichs, B. Dünweg, J. Chem. Phys. 111, 8225 (1999)ADSCrossRefGoogle Scholar
  78. 78.
    B. Dünweg, A.J.C. Ladd, Adv. Polym. Sci. 221, 89 (2009)Google Scholar
  79. 79.
    S. Hirschmann, et al., Load-balancing and spatial adaptivity for coarse-grained molecular dynamics applications, in High Performance Computing in Science and Engineering ‘18 (Springer, 2018), forthcomingGoogle Scholar
  80. 80.
    F. Schornbaum, Block-structured adaptive mesh refinement for simulations on extreme-scale supercomputers, Doctoral thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), 2018, p. 152Google Scholar
  81. 81.
    G. Wellein, T. Zeiser, G. Hager, S. Donath, Comput. Fluids 35, 910 (2006)CrossRefGoogle Scholar
  82. 82.
    R.M.M. Smeets, U.F. Keyser, D. Krapf, M.Y. Wu, N.H. Dekker, C. Dekker, Nano Lett. 6, 89 (2006)ADSCrossRefGoogle Scholar
  83. 83.
    G. Rempfer, S. Ehrhardt, C. Holm, J. de Graaf, Macromol. Theor. Simul. 26, 160051 (2016)Google Scholar
  84. 84.
    B. Kurz, Lattice-Boltzmann simulationen auf mehreren GPUs, Bachelor’s thesis, University of Stuttgart, 2018Google Scholar

Copyright information

© EDP Sciences, Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Institute for Parallel and Distributed SystemsStuttgartGermany

Personalised recommendations