A New Matrix-Free Approach for Large-Scale Geodynamic Simulations and its Performance

  • Simon Bauer
  • Markus Huber
  • Marcus MohrEmail author
  • Ulrich Rüde
  • Barbara Wohlmuth
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10861)


We report on a two-scale approach for efficient matrix-free finite element simulations. The proposed method is based on surrogate element matrices constructed by low-order polynomial approximations. It is applied to a Stokes-type PDE system with variable viscosity as is a key component in mantle convection models. We set the ground for a rigorous performance analysis inspired by the concept of parallel textbook multigrid efficiency and study the weak scaling behavior on SuperMUC, a peta-scale supercomputer system. For a complex geodynamical model, we achieve a parallel efficiency of 95% on up to 47 250 compute cores. Our largest simulation uses a trillion (\(\mathcal {O}(10^{12})\)) degrees of freedom for a global mesh resolution of 1.7 km.


Two-scale PDE discretization Massively parallel multigrid Matrix-free on-the-fly assembly Large scale geophysical application 



This work was partly supported by the German Research Foundation through the Priority Programme 1648 “Software for Exascale Computing” (SPPEXA) and WO671/11-1. The authors gratefully acknowledge the Gauss Centre for Supercomputing (GCS) for providing computing time on the supercomputer SuperMUC at LRZ. Special thanks go to the members of LRZ for the organization and their assistance at the “LRZ scaling workshop: Emergent applications”. Most scaling results where obtained during this workshop.


  1. 1.
    Bauer, S., et al.: Hybrid parallel multigrid methods for geodynamical simulations. In: Bungartz, H.-J., Neumann, P., Nagel, W.E. (eds.) Software for Exascale Computing - SPPEXA 2013–2015. LNCSE, vol. 113, pp. 211–235. Springer, Cham (2016). Scholar
  2. 2.
    Bauer, S., Mohr, M., Rüde, U., Weismüller, J., Wittmann, M., Wohlmuth, B.: A two-scale approach for efficient on-the-fly operator assembly in massively parallel high performance multigrid codes. Appl. Numer. Math. 122, 14–38 (2017)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bergen, B., Gradl, T., Rüde, U., Hülsemann, F.: A massively parallel multigrid method for finite elements. Comput. Sci. Eng. 8(6), 56–62 (2006)CrossRefGoogle Scholar
  4. 4.
    Bergen, B., Hülsemann, F.: Hierarchical hybrid grids: data structures and core algorithms for multigrid. Numer. Linear Algebra Appl. 11, 279–291 (2004)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Brandt, A.: Barriers to achieving textbook multigrid efficiency (TME) in CFD. Institute for Computer Applications in Science and Engineering, NASA Langley Research Center (1998)Google Scholar
  6. 6.
    Brezzi, F., Douglas, J.: Stabilized mixed methods for the Stokes problem. Numer. Math. 53(1), 225–235 (1988)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Colli, L., Ghelichkhan, S., Bunge, H.P., Oeser, J.: Retrodictions of Mid Paleogene mantle flow and dynamic topography in the Atlantic region from compressible high resolution adjoint mantle convection models: sensitivity to deep mantle viscosity and tomographic input model. Gondwana Res. 53, 252–272 (2018)CrossRefGoogle Scholar
  8. 8.
    Davies, D.R., Davies, J.H., Bollada, P.C., Hassan, O., Morgan, K., Nithiarasu, P.: A hierarchical mesh refinement technique for global 3-D spherical mantle convection modelling. Geosci. Model Dev. 6(4), 1095–1107 (2013)CrossRefGoogle Scholar
  9. 9.
    Davies, D.R., Goes, S., Davies, J., Schuberth, B., Bunge, H.P., Ritsema, J.: Reconciling dynamic and seismic models of earth’s lower mantle: the dominant role of thermal heterogeneity. Earth Planet. Sci. Lett. 353–354, 253–269 (2012)CrossRefGoogle Scholar
  10. 10.
    Durnov, D., Steyer, M.: Intel MPI Memory Consumption. The Parallel Universe 21 (2015)Google Scholar
  11. 11.
    Gmeiner, B., Rüde, U., Stengel, H., Waluga, C., Wohlmuth, B.: Performance and scalability of hierarchical hybrid multigrid solvers for Stokes systems. SIAM J. Sci. Comput. 37(2), C143–C168 (2015)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Gmeiner, B., Huber, M., John, L., Rüde, U., Wohlmuth, B.: A quantitative performance study for Stokes solvers at the extreme scale. J. Comput. Sci. 17(Part 3), 509–521 (2016)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Gmeiner, B., Köstler, H., Stürmer, M., Rüde, U.: Parallel multigrid on hierarchical hybrid grids: a performance study on current high performance computing clusters. Concurr. Comput.: Pract. Exp. 26(1), 217–240 (2014)CrossRefGoogle Scholar
  14. 14.
    Gmeiner, B., Rüde, U., Stengel, H., Waluga, C., Wohlmuth, B.: Towards textbook efficiency for parallel multigrid. Numer. Math. Theor. Meth. Appl. 8(01), 22–46 (2015)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Heister, T., Dannberg, J., Gassmöller, R., Bangerth, W.: High accuracy mantle convection simulation through modern numerical methods - II: realistic models and problems. Geophys. J. Int. 210(2), 833–851 (2017)CrossRefGoogle Scholar
  16. 16.
    Intel Corp.: MPI Performance Snapshot, version: 2017.0.4 (2017).
  17. 17.
    Kronbichler, M., Kormann, K.: A generic interface for parallel cell-based finite element operator application. Comput. Fluids 63, 135–147 (2012)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Logg, A., Ølgaard, K.B., Rognes, M.E., Wells, G.N.: FFC: the FEniCS form compiler. In: Logg, A., Mardal, K.A., Wells, G. (eds.) Automated solution of differential equations by the finite element method. LNCSE, vol. 84, pp. 227–238. Springer, Heidelberg (2012). Scholar
  19. 19.
    May, D.A., Brown, J., Pourhiet, L.L.: A scalable, matrix-free multigrid preconditioner for finite element discretizations of heterogeneous Stokes flow. Comput. Methods Appl. Mech. Eng. 290, 496–523 (2015)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Müller, R.D., Sdrolias, M., Gaina, C., Roest, W.R.: Age, spreading rates, and spreading asymmetry of the world’s ocean crust. Geochem. Geophys. Geosyst. 9(4), 1525–2027 (2008)CrossRefGoogle Scholar
  21. 21.
    Rudi, J., Malossi, A.C.I., Isaac, T., Stadler, G., Gurnis, M., Staar, P.W.J., Ineichen, Y., Bekas, C., Curioni, A., Ghattas, O.: An extreme-scale implicit solver for complex PDEs: highly heterogeneous flow in earth’s mantle. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015, pp. 5:1–5:12. ACM (2015)Google Scholar
  22. 22.
    Weismüller, J., Gmeiner, B., Ghelichkhan, S., Huber, M., John, L., Wohlmuth, B., Rüde, U., Bunge, H.P.: Fast asthenosphere motion in high-resolution global mantle flow models. Geophys. Res. Lett. 42(18), 7429–7435 (2015). Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Earth and Environmental SciencesLudwig-Maximilians-Universität MünchenMunichGermany
  2. 2.Institute for Numerical Mathematics (M2)Technische Universität MünchenMunichGermany
  3. 3.Department of Computer Science 10FAU Erlangen-NürnbergErlangenGermany
  4. 4.Parallel Algorithms ProjectCERFACSToulouseFrance

Personalised recommendations