Developing a Geodynamics Simulator with PETSc

  • Matthew G. Knepley
  • Richard F. Katz
  • Barry Smith
Part of the Lecture Notes in Computational Science and Engineering book series (LNCSE, volume 51)


Most high-performance simulation codes are not written from scratch but begin as desktop experiments and are subsequently migrated to a scalable, parallel paradigm. This transition can be painful, however, because the restructuring required in conversion forces most authors to abandon their serial code and begin an entirely new parallel code. Starting a parallel code from scratch has many disadvantages, such as the loss of the original test suite and the introduction of new bugs. We present a disciplined, incremental approach to parallelization of existing scientific code using the PETSc framework. In addition to the parallelization, it allows the addition of more physics (in this case strong nonlinearities) without the user having to program anything beyond the new pieces of discretization code. Our approach permits users to easily develop and experiment on the desktop with the same code that scales efficiently to large clusters with excellent parallel performance. As a motivating example, we present work integrating PETSc into an existing plate tectonic subduction code.


Subduction Zone Mantle Wedge Linear Solver User Code Nonlinear Solver 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    M. F. Adams, H. H. Bayraktar, T.M. Keaveny, and P. Papadopoulos. Ultrascalable implicit finite element analyses in solid mechanics with over a half a billion degrees of freedom. In Proceedings of SC04, 2004. Winner of Gordon Bell Special Prize at SC2004: Large scale trabecular bone finite element modeling.Google Scholar
  2. 2.
    V. Akcelik, J. Bielak, G. Biros, I. Epanomeritakis, A. F. O. Ghattas, E. J. Kim, D. O’Hallaron, and T. Tu. High resolution forward and inverse earthquake modeling on terascale computers. In Proceedings of SC2003, 2003. A winner of the Gordon Bell Prize for special achievement at SC2003.Google Scholar
  3. 3.
    P. R. Amestoy, I. S. Duff, J.-Y. L’Excellent, and J. Koster. A fully asynchronous multifrontal solver using distributed dynamic scheduling. SIAM Journal on Matrix Analysis and Applications, 23(1):15–41, 2001.MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    W. K. Anderson, W. D. Gropp, D. K. Kaushik, D. E. Keyes, and B. F. Smith. Achieving high sustained performance in an unstructured mesh CFD application. In Proceedings of SC 99, 1999.Google Scholar
  5. 5.
    S. Balay, K. Buschelman, V. Eijkhout, W. D. Gropp, D. Kaushik, M. G. Knepley, L. C. McInnes, B. F. Smith, and H. Zhang. PETSc Web page. Scholar
  6. 6.
    S. Balay, W. D. Gropp, L. C. McInnes, and B. F. Smith. Efficient management of parallelism in object oriented numerical software libraries. In E. Arge, A. M. Bruaset, and H. P. Langtangen, editors, Modern Software Tools in Scientific Computing, pages 163–202. Birkhäuser Press, 1997.Google Scholar
  7. 7.
    S. Carbotte, C. Small, and K. Donnelly. The influence of ridge migration on the magmatic segmentation of mid-ocean ridges. Nature, 429:743–746, 2004.CrossRefGoogle Scholar
  8. 8.
    J. Conder, D. Wiens, and J. Morris. On the decompression melting structure at volcanic arcs and back-arc spreading centers. Geophys. Res. Letts., 29, 2002.Google Scholar
  9. 9.
    J. W. Demmel, J. R. Gilbert, and X. S. Li. SuperLU user’s guide. Technical Report LBNL-44289, Lawrence Berkeley National Laboratory, October 2003.Google Scholar
  10. 10.
    M. Eberle, O. Grasset, and C. Sotin. A numerical study of the interaction of the mantle wedge, subducting slab, and overriding plate. Phys. Earth Planet. In., 134:191–202, 2002.CrossRefGoogle Scholar
  11. 11.
    P. England, R. Engdahl, and W. Thatcher. Systematic variation in the depth of slabs beneath arc volcanos. Geophys. J. Int., 156(2):377–408, 2003.CrossRefGoogle Scholar
  12. 12.
    Y. Furukawa. Depth of the decoupling plate interface and thermal structure under arcs. J. Geophys. Res., 98:20005–20013, 1993.CrossRefGoogle Scholar
  13. 13.
    V. E. Henson and U. M. Yang. BoomerAMG: A parallel algebraic multigrid solver and preconditioner. Technical Report UCRL-JC-133948, Lawrence Livermore National Laboratory, 2000.Google Scholar
  14. 14.
    G. Hirth and D. Kohlstedt. Rheology of the upper mantle and the mantle wedge: A view from the experimentalists. In Inside the Subduction Factory, volume 138 of Geophysical Monograph. American Geophysical Union, 2003.Google Scholar
  15. 15.
    P. Hovland, B. Norris, and B. Smith. Making automatic differentiation truly automatic: Coupling PETSc with ADIC. In Proceedings of ICCS2002, 2002.Google Scholar
  16. 16.
    D. Hysom and A. Pothen. A scalable parallel algorithm for incomplete factor preconditioning. SIAM Journal on Scientific Computing, 22:2194–2215, 2001.MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    R. Katz, M. Spiegelman, and S. Carbotte. Ridge migration, asthenospheric flow and the origin of magmatic segmentation in the global mid-ocean ridge system. Geophys. Res. Letts., 31, 2004.Google Scholar
  18. 18.
    P. Kelemen, J. Rilling, E. Parmentier, L. Mehl, and B. Hacker. Thermal structure due to solid-state flow in the mantle wedge beneath arcs. In Inside the Subduction Factory, volume 138 of Geophysical Monograph. American Geophysical Union, 2003.Google Scholar
  19. 19.
    S. Peacock and K. Wang. Seismic consequences of warm versus cool subduction metamorphism: Examples from southwest and northeast Japan. Science, 286:937–939, 1999. sau]20._PETSc SNES Example 30. Scholar
  20. 21.
    PETSc SNES Example 5. Scholar
  21. 22.
    PETSc Solvers. Scholar
  22. 23.
    B. Smith et al. Scientific Applications Using PETSc. Scholar
  23. 24.
    R. Stern. Subduction zones. Rev. Geophys., 40(4), 2002.Google Scholar
  24. 25.
    P. van Keken, B. Kiefer, and S. Peacock. High-resolution models of subduction zones: Implications for mineral dehydration reactions and the transport of water into the deep mantle. Geochem. Geophys. Geosys., 3(10), 2003.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Matthew G. Knepley
    • 1
  • Richard F. Katz
    • 2
  • Barry Smith
    • 1
  1. 1.Mathematics and Computer Science DivisionArgonne National LaboratoryArgonneUSA
  2. 2.Department of Earth and Environmental SciencesLamont Doherty Earth ObservatoryPalisadesUSA

Personalised recommendations