Hybrid Parallel Multigrid Methods for Geodynamical Simulations

  • Simon Bauer
  • Hans-Peter Bunge
  • Daniel Drzisga
  • Björn Gmeiner
  • Markus Huber
  • Lorenz JohnEmail author
  • Marcus Mohr
  • Ulrich Rüde
  • Holger Stengel
  • Christian Waluga
  • Jens Weismüller
  • Gerhard Wellein
  • Markus Wittmann
  • Barbara Wohlmuth
Conference paper
Part of the Lecture Notes in Computational Science and Engineering book series (LNCSE, volume 113)


Even on modern supercomputer architectures, Earth mantle simulations are so compute intensive that they are considered grand challenge applications. The dominating roadblocks in this branch of Geophysics are model complexity and uncertainty in parameters and data, e.g., rheology and seismically imaged mantle heterogeneity, as well as the enormous space and time scales that must be resolved in the computational models. This article reports on a massively parallel all-at-once multigrid solver for the Stokes system as it arises in mantle convection models. The solver employs the hierarchical hybrid grids framework and demonstrates that a system with coupled velocity components and with more than a trillion (1. 7 ⋅ 1012) degrees of freedom can be solved in about 1,000 s using 40,960 compute cores of JUQUEEN. The simulation framework is used to investigate the influence of asthenosphere thickness and viscosity on upper mantle velocities in a static scenario. Additionally, results for a time-dependent simulation with a time-variable temperature-dependent viscosity model are presented.


Coarse Grid Multigrid Method Work Unit Stokes System Mantle Convection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was supported (in part) by the German Research Foundation (DFG) through the Priority Programme 1648 “Software for Exascale Computing” (SPPEXA) and grant WO 671/11-1. The authors gratefully acknowledge the Gauss Centre for Supercomputing (GCS) for providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS share of the supercomputer JUQUEEN at Jülich Supercomputing Centre (JSC).


  1. 1.
    Bangerth, W., Burstedde, C., Heister, T., Kronbichler, M.: Algorithms and data structures for massively parallel generic adaptive finite element codes. ACM Trans. Math. Soft. 38 (2), 14:1–14:28 (2011)Google Scholar
  2. 2.
    Bank, R.E., Welfert, B.D., Yserentant, H.: A class of iterative methods for solving saddle point problems. Numer. Math. 56 (7), 645–666 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Baumgardner, J.R.: Three-dimensional treatment of convective flow in the Earth’s mantle. J. Stat. Phys. 39 (5/6), 501–511 (1985)CrossRefGoogle Scholar
  4. 4.
    Becker, T.W., Boschi, L.: A comparison of tomographic and geodynamic mantle models. Geochem. Geophy. Geosy. 3, 1525–2027 (2002)CrossRefGoogle Scholar
  5. 5.
    Bergen, B., Gradl, T., Rüde, U., Hülsemann, F.: A massively parallel multigrid method for finite elements. Comput. Sci. Eng. 8 (6), 56–62 (2006)CrossRefGoogle Scholar
  6. 6.
    Bergen, B., Wellein, G., Hülsemann, F., Rüde, U.: Hierarchical hybrid grids: achieving TERAFLOP performance on large scale finite element simulations. Int. J. Parallel Emergent Distrib. Syst. 22 (4), 311–329 (2007)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Brandt, A.: Guide to multigrid development. In: Multigrid methods, pp. 220–312. Springer, Berlin/Heidelberg (1982). Republished as: Multigrid Techniques: 1984 guide with applications to fluid dynamics, revised edition, SIAM, 2011Google Scholar
  8. 8.
    Brandt, A., Dinar, N.: Multigrid solutions to elliptic flow problems. In: Numerical methods for partial differential equations (Proc. Adv. Sem., Math. Res. Center, Univ. Wisconsin, Madison, Wis., 1978), Publ. Math. Res. Center Univ. Wisconsin, vol. 42, pp. 53–147. Academic Press, New York/London (1979)Google Scholar
  9. 9.
    Brezzi, F., Douglas, Jr., J.: Stabilized mixed methods for the Stokes problem. Numer. Math. 53 (1–2), 225–235 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Brezzi, F., Fortin, M.: Mixed and Hybrid Finite Element Methods. Springer, New York (1991)CrossRefzbMATHGoogle Scholar
  11. 11.
    Bunge, H.P., Baumgardner, J.R.: Mantle convection modeling on parallel virtual machines. Comput. Phys. 9 (2), 207–215 (1995)CrossRefGoogle Scholar
  12. 12.
    Bunge, H.P., Hagelberg, C.R., Travis, B.J.: Mantle circulation models with variational data assimilation: inferring past mantle flow and structure from plate motion histories and seismic tomography. Geophys. J. Int. 152 (2), 280–301 (2003). CrossRefGoogle Scholar
  13. 13.
    Bunge, H.P., Richards, M.A., Baumgardner, J.R.: A sensitivity study of three-dimensional spherical mantle convection at 108 Rayleigh number: effects of depth-dependent viscosity, heating mode, and an endothermic phase change. J. Geophys. Res. 102, 11991–12007 (1997)CrossRefGoogle Scholar
  14. 14.
    Bunge, H.P., Richards, M., Lithgow-Bertelloni, C., Baumgardner, J.R., Grand, S., Romanowicz, B.: Time scales and heterogeneous structure in geodynamic earth models. Science 280, 91–95 (1998). CrossRefGoogle Scholar
  15. 15.
    Burstedde, C., Stadler, G., Alisic, L., Wilcox, L.C., Tan, E., Gurnis, M., Ghattas, O.: Large-scale adaptive mantle convection simulation. Geophys. J. Internat. 192 (3), 889–906 (2013)CrossRefGoogle Scholar
  16. 16.
    Burstedde, C., Wilcox, L.C., Ghattas, O.: p4est: Scalable algorithms for parallel adaptive mesh refinement on forests of octrees. SIAM J. Sci. Comp. 33 (3), 1103–1133 (2011)Google Scholar
  17. 17.
    CIG – Computational Infrastructure for Geodynamics: ASPECT: Advanced Solver for Problems in Earth’s ConvecTion, User Manual (2015), version 1.3Google Scholar
  18. 18.
    Council, N.R.: Origin and Evolution of Earth: Research Questions for a Changing Planet. The National Academies Press, Washington, DC (2008). Google Scholar
  19. 19.
    Davies, D.R., Davies, J.H., Bollada, P.C., Hassan, O., Morgan, K., Nithiarasu, P.: A hierarchical mesh refinement technique for global 3-D spherical mantle convection modelling. Geosci. Model Dev. 6 (4), 1095–1107 (2013)CrossRefGoogle Scholar
  20. 20.
    Davies, D.R., Goes, S., Davies, J.H., Schuberth, B.S.A., Bunge, H.P., Ritsema, J.: Reconciling dynamic and seismic models of Earth’s lower mantle: the dominant role of thermal heterogeneity. Earth Planet. Sci. Lett. 353–354 (1), 253–269 (2012)CrossRefGoogle Scholar
  21. 21.
    Dziewonski, A.M., Anderson, D.L.: Preliminary reference Earth model. Phys. Earth Plan. Int. 25, 297–356 (1981)CrossRefGoogle Scholar
  22. 22.
    Elman, H.C., Silvester, D.J., Wathen, A.J.: Finite Elements and Fast Iterative Solvers: With Applications in Incompressible Fluid Dynamics. Oxford University Press, New York (2005)zbMATHGoogle Scholar
  23. 23.
    Engelman, M.S., Sani, R.L., Gresho, P.M.: The implementation of normal and/or tangential boundary conditions in finite element codes for incompressible fluid flow. Int. J. Numer. Methods Fluids 2 (3), 225–238 (1982)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Fichtner, A., Kennett, B.L.N., Igel, H., Bunge, H.P.: Full seismic waveform tomography for upper-mantle structure in the Australasian region using adjoint methods. Geophys. J. Int. 179 (3), 1703–1725 (2009)CrossRefGoogle Scholar
  25. 25.
    Gaspar, F.J., Notay, Y., Oosterlee, C.W., Rodrigo, C.: A simple and efficient segregated smoother for the discrete Stokes equations. SIAM J. Sci. Comput. 36 (3), A1187–A1206 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Girault, V., Raviart, P.A.: Finite Element Methods for Navier-Stokes Equations. Springer, New York (1986)CrossRefzbMATHGoogle Scholar
  27. 27.
    Gmeiner, B., Huber, M., John, L., Rüde, U., Waluga, C., Wohlmuth, B.: Massively parallel large scale stokes flow simulation. In: Binder, K., Müller, M., Kremer, M., Schnurpfeil, A. (eds.) NIC Symposium 2016. Schriften des Forschungszentrums Jülich, NIC Series, vol. 48, pp. 333–341. ISBN:978-3-95806-109-5Google Scholar
  28. 28.
    Gmeiner, B., Huber, M., John, L., Rüde, U., Wohlmuth, B.: A quantitative performance analysis for Stokes solvers at the extreme scale (submitted, arXiv:1511.02134)Google Scholar
  29. 29.
    Gmeiner, B., Rüde, U., Stengel, H., Waluga, C., Wohlmuth, B.: Performance and scalability of hierarchical hybrid multigrid solvers for stokes systems. SIAM J. Sci. Comput. 37 (2), C143–C168 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Gmeiner, B., Rüde, U., Stengel, H., Waluga, C., Wohlmuth, B.: Towards textbook efficiency for parallel multigrid. Numer. Math. Theory Methods Appl. 8, 22–46 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Gmeiner, B., Waluga, C., Wohlmuth, B.: Local mass-corrections for continuous pressure approximations of incompressible flow. SIAM J. Numer. Anal. 52 (6), 2931–2956 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Grand, S.P., van der Hilst, R.D., Widiyantoro, S.: Global seismic tomography: a snapshot of convection in the earth. GSA Today 7, 1–7 (1997)Google Scholar
  33. 33.
    Hager, G., Treibig, J., Habich, J., Wellein, G.: Exploring performance and power properties of modern multi-core chips via simple machine models. Concurr. Comput. 28, 1–2 (2014)Google Scholar
  34. 34.
    Hartley, R.A., Roberts, G.G., White, N., Richardson, C.: Transient convective uplift of an ancient buried landscape. Nat. Geosci. 4, 562–565 (2011)CrossRefGoogle Scholar
  35. 35.
    Haskell, N.A.: The motion of a fluid under a surface load. Physics 6, 265–269 (1935)CrossRefzbMATHGoogle Scholar
  36. 36.
    Höink, T., Lenardic, A.: Three-dimensional mantle convection simulations with a low-viscosity asthenosphere and the relationship between heat flow and the horizontal length scale of convection. Geophys. Res. Lett. 35, L10304 (2008)CrossRefGoogle Scholar
  37. 37.
    Huber, M., Gmeiner, B., Rüde, U., Wohlmuth, B.: Resilience for multigrid software at the extreme scale (preprint, arXiv:1506.06185)Google Scholar
  38. 38.
    Huber, M., John, L., Pustejovska, P., Rüde, U., Waluga, C., Wohlmuth, B.: Solution Techniques for the Stokes System: a priori and a posteriori modifications, resilient algorithms. In Proceedings of the ICIAM, Beijing (2015). arXiv:151105759Google Scholar
  39. 39.
    Mitrovica, J.X.: Haskell [1935] revisited. J. Geophys. Res. 101, 555–569 (1996)CrossRefGoogle Scholar
  40. 40.
    Müller, R.D., Sdrolias, M., Gaina, C., Roest, W.R.: Age, spreading rates, and spreading asymmetry of the world’s ocean crust. Geochem. Geophy. Geosy. 9, 1525–2027 (2008)CrossRefGoogle Scholar
  41. 41.
    Oeser, J., Bunge, H.P., Mohr, M.: Cluster Design in the Earth Sciences: TETHYS. In: Gerndt, M., Kranzlmüller, D. (eds.) High Performance Computing and Communications – Second International Conference, HPCC 2006, Munich. Lecture Notes in Computer Science, vol. 4208, pp. 31–40. Springer (2006).
  42. 42.
    Parnell-Turner, R., White, N., Henstock, T., Murton, B., Maclennan, J., Jones, S.M.: A continuous 55 million year record of transient mantle plume activity beneath Iceland. Nat. Geosci. 7, 914–919 (2014)CrossRefGoogle Scholar
  43. 43.
    Resovsky, J., Trampert, J.: Using probabilistic seismic tomography to test mantle velocity–density relationships. Earth Planet. Sci. Lett. 215 (1), 121–134 (2003)CrossRefGoogle Scholar
  44. 44.
    Ricard, Y.: Physics of mantle convection. In: Schubert, G. (ed.) Treatise on Geophysics, vol. 7. Elsevier, Amsterdam (2007)Google Scholar
  45. 45.
    Ritsema, J., von Heijst, H.J., Woodhouse, J.H.: Global transition zone tomography. J. Geophys. Res. 109, B02302 (2004)CrossRefGoogle Scholar
  46. 46.
    Rudi, J., Malossi, A.C.I., Isaac, T., Stadler, G., Gurnis, M., Staar, P.W.J., Ineichen, Y., Bekas, C., Curioni, A., Ghattas, O.: An Extreme-scale Implicit Solver for Complex PDEs: Highly Heterogeneous Flow in Earth’s Mantle. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’15), pp. 5:1–5:12. ACM, New York (2015).
  47. 47.
    Schöberl, J., Zulehner, W.: On Schwarz-type smoothers for saddle point problems. Numer. Math. 95 (2), 377–399 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  48. 48.
    Seton, M., Müller, R.D., Zahirovic, S., Gaina, C., Torsvik, T.H., Shephard, G., Talsma, A., Gurnis, M., Turner, M., Maus, S., Chandler, M.: Global continental and ocean basin reconstructions since 200 ma. Earth-Sci. Rev. 113, 212–270 (2012)CrossRefGoogle Scholar
  49. 49.
    Stixrude, L., Lithgow-Bertelloni, C.: Thermodynamics of mantle minerals – I. Physical properties. Geophys. J. Int. 162, 610–632 (2005)CrossRefGoogle Scholar
  50. 50.
    Sundar, H., Stadler, G., Biros, G.: Comparison of multigrid algorithms for high-order continuous finite element discretizations. Numer. Linear Algebra Appl. 22 (4), 664–680 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  51. 51.
    Tackley, P.J.: Effects of strongly variable viscosity on three-dimensional compressible convection in planetary mantles. J. Geophys. Res. 101, 3311–3332 (1996)CrossRefGoogle Scholar
  52. 52.
    Tackley, P.J.: Mantle convection and plate tectonics: toward an integrated physical and chemical theory. Science 16, 2002–2007 (2000)CrossRefGoogle Scholar
  53. 53.
    Tackley, P.J., Stevenson, D.J., Glatzmaier, G.A., Schubert, G.: Effects of multiple phase transitions in a three-dimensional spherical model of convection in earth’s mantle. J. Geophys. Res. 99 (B8), 15877–15901 (1994)CrossRefGoogle Scholar
  54. 54.
    Tan, E., Choi, E., Thoutireddy, P., Gurnis, M., Aivazis, M.: GeoFramework: coupling multiple models of mantle convection within a computational framework. Geochem. Geophy. Geosy. 7 (6), Q06001 (2006)CrossRefGoogle Scholar
  55. 55.
    Urquiza, J.M., Garon, A., Farinas, M.I.: Weak imposition of the slip boundary condition on curved boundaries for Stokes flow. J. Comput. Phys. 256, 748–767 (2014)MathSciNetCrossRefGoogle Scholar
  56. 56.
    Verfürth, R.: Finite element approximation of incompressible Navier-Stokes equations with slip boundary condition. Numer. Math. 50 (6), 697–721 (1987)MathSciNetCrossRefzbMATHGoogle Scholar
  57. 57.
    Vogt, P.R.: Asthenosphere motion recorded by the ocean floor south of Iceland. Earth Planet. Sci. Lett. 13, 153–160 (1971), CrossRefGoogle Scholar
  58. 58.
    Waluga, C., Wohlmuth, B., Rüde, U.: Mass-corrections for the conservative coupling of flow and transport on collocated meshes. J. Comp. Phys. 305, 319–332 (2016)MathSciNetCrossRefGoogle Scholar
  59. 59.
    Weismüller, J., Gmeiner, B., Ghelichkhan, S., Huber, M., John, L., Wohlmuth, B., Rüde, U., Bunge, H.P.: Fast asthenosphere motion in high-resolution global mantle flow models. Geophys. Res. Lett. 42 (18), 7429–7435 (2015)CrossRefGoogle Scholar
  60. 60.
    Williams, S.W., Waterman, A., Patterson, D.A.: Roofline: an insightful visual performance model for floating-point programs and multicore architectures. Tech. Rep. UCB/EECS-2008-134, EECS Department, University of California, Berkeley (Oct 2008)Google Scholar
  61. 61.
    Zhong, S., McNamara, A., Tan, E., Moresi, L., Gurnis, M.: A benchmark study on mantle convection in a 3-D spherical shell using CitcomS. Geochem. Geophy. Geosy. 9, Q10017 (2008)CrossRefGoogle Scholar
  62. 62.
    Zhong, S., Zuber, M.T., Moresi, L., Gurnis, M.: The role of temperature-dependent viscosity and surface plates in spherical shell models of mantle convection. J. Geophys. Res. 105 (B5), 11063–11082 (2000)CrossRefGoogle Scholar
  63. 63.
    Zulehner, W.: Analysis of iterative methods for saddle point problems: a unified approach. Math. Comput. 71 (238), 479–505 (2002)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Simon Bauer
    • 1
  • Hans-Peter Bunge
    • 1
  • Daniel Drzisga
    • 2
  • Björn Gmeiner
    • 3
  • Markus Huber
    • 2
  • Lorenz John
    • 2
    Email author
  • Marcus Mohr
    • 1
  • Ulrich Rüde
    • 3
  • Holger Stengel
    • 4
  • Christian Waluga
    • 2
  • Jens Weismüller
    • 1
  • Gerhard Wellein
    • 4
  • Markus Wittmann
    • 4
  • Barbara Wohlmuth
    • 2
  1. 1.Department of Earth and Environmental SciencesLudwig-Maximilians-Universität MünchenMünchenGermany
  2. 2.Institute for Numerical MathematicsTechnische Universität MünchenMünchenGermany
  3. 3.Department of Computer Science 10Friedrich-Alexander-University Erlangen-NurembergErlangenGermany
  4. 4.Erlangen Regional Computing Center (RRZE)Friedrich-Alexander-University Erlangen-NurembergErlangenGermany

Personalised recommendations