Engineering with Computers

, Volume 22, Issue 3–4, pp 237–254 | Cite as

libMesh : a C++ library for parallel adaptive mesh refinement/coarsening simulations

  • Benjamin S. KirkEmail author
  • John W. Peterson
  • Roy H. Stogner
  • Graham F. Carey
Original Article


In this paper we describe the libMesh ( framework for parallel adaptive finite element applications. libMesh is an open-source software library that has been developed to facilitate serial and parallel simulation of multiscale, multiphysics applications using adaptive mesh refinement and coarsening strategies. The main software development is being carried out in the CFDLab ( at the University of Texas, but as with other open-source software projects; contributions are being made elsewhere in the US and abroad. The main goals of this article are: (1) to provide a basic reference source that describes libMesh and the underlying philosophy and software design approach; (2) to give sufficient detail and references on the adaptive mesh refinement and coarsening (AMR/C) scheme for applications analysts and developers; and (3) to describe the parallel implementation and data structures with supporting discussion of domain decomposition, message passing, and details related to dynamic repartitioning for parallel AMR/C. Other aspects related to C++ programming paradigms, reusability for diverse applications, adaptive modeling, physics-independent error indicators, and similar concepts are briefly discussed. Finally, results from some applications using the library are presented and areas of future research are discussed.


Parallel computing Adaptive mesh refinement Finite elements 



The student authors of libMesh have been partially supported by a Department of Energy Computational Science Graduate Fellowship, Institute for Computational and Engineering Sciences (ICES) fellowships, NASA Graduate Student Research Grant NGT5-139, and DARPA Grant No. HR0011-06-1-0005. David Knezevic performed the tumor angiogenesis simulation and implemented support for 1D problems in the library. Varis Carey provided a patch recovery error indicator implementation. Infinite elements, support for complex-valued systems, and eigenvalue problems were provided by Daniel Dreyer and Steffen Petersen from Technische Universität Hamburg-Harburg. Additionally, we are grateful to the ( deal.II project for inspiring libMesh , and Wolfgang Bangerth in particular for many useful discussions.


  1. 1.
    Gropp W, Lusk E, Doss N, Skjellum A (1996) MPICH: A high-performance, portable implementation of the MPI message passing interface standard. Parallel Comput 22(6):789–828zbMATHCrossRefGoogle Scholar
  2. 2.
    Carey GF (1997) Computational grids: generation, adaptation, and solution strategies. Taylor & Francis, LondonGoogle Scholar
  3. 3.
    Barth W, Kirk B (2003) PC cluster construction: lessons learned and friendly advice. Shortcourse on cluster computing and adaptive finite element methods, The University of Western Australia, PerthGoogle Scholar
  4. 4.
    Becker EB, Carey GF, Oden JT (1981) Finite elements—an introduction, vol 1. Prentice Hall, Englewood cliffsGoogle Scholar
  5. 5.
    Flaherty JE, Paslow PJ, Shephard MS, Vasilakis JD (eds) (1989) Adaptive methods for partial differential equations. SIAM, PhiladelphiaGoogle Scholar
  6. 6.
    Babuska I, Zienkiewicz OC, Gago J, Oliviera ERA (1986) Accuracy estimates and adaptive refinements in finite element computations. Wiley, LondonzbMATHGoogle Scholar
  7. 7.
    Babuska I, Rheinboldt WC (1982) Computational error estimates and adaptive processes for some nonlinear structural problems. Comput Methods Appl Mech Eng 34:895–937zbMATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Carey GF, Anderson M, Carnes B, Kirk B (2004) Some aspects of adaptive grid technology related to boundary and interior layers. J Comput Appl Math 166(1):55–86, ISSN 0377-0427, Google Scholar
  9. 9.
    Barth W, Carey GF, Kirk B, McLay R (2000) Parallel distributed solution of viscous flow with heat transfer on workstation clusters. in high performance computing 2000 proceedings, WashingtonGoogle Scholar
  10. 10.
    Gamma E, Helm R, Johnson R, Vlissides J (1995) Design patterns: elements of reusable object-oriented software. Addison-Wesley, ReadingGoogle Scholar
  11. 11.
    Carey GF, McLay R (1995) Multi-level preconditioners for the parallel PCG package, copper Mtn. Conference on multigrid methods. In: Proceedings of the copper mountain conference on multigrid methods, Copper Mountain, COGoogle Scholar
  12. 12.
    Carey GF (1976) A mesh refinement scheme for finite element computations. Comput Methods Appl Mech Eng 7:93–105CrossRefMathSciNetGoogle Scholar
  13. 13.
    Carey GF (1976) An analysis of finite element equations and mesh subdivision. Comput Methods Appl Mech Eng 9:165–179CrossRefMathSciNetGoogle Scholar
  14. 14.
    Carey GF, Finlayson BA (1975) Orthogonal collocation on finite elements. J Chem Eng Sci 30:587–596CrossRefGoogle Scholar
  15. 15.
    Bank RE (1998) PLTMG: A software package for solving elliptic partial differential equations, Users’ Guide 8.0, Software, environments and tools, vol 5. SIAM, PhiladelphiaGoogle Scholar
  16. 16.
    Plaza A, Carey GF (2000) Local refinement of simplicial grids based on the skeleton. Appl Numerical Math 32:195–218zbMATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    Plaza A, Padrón MA, Carey GF (2000) A 3D refinement/derefinement algorithm for solving evolution problems. Appl Numerical Math 32:401–418zbMATHCrossRefGoogle Scholar
  18. 18.
    Carey GF, Kabaila A, Utku M (1982) On penalty methods for interelement constraints. Comput Methods Appl Mech Eng 30:151–171zbMATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Bangerth W, Rannacher R (2003) Adaptive finite element methods for differential equations. Birkhäuser Verlag, BaselzbMATHGoogle Scholar
  20. 20.
    Eriksson K, Estep D, Hansbo P, Johnson C (1996) Adaptive finite elements. Springer, Berlin Heidelberg New YorkGoogle Scholar
  21. 21.
    Estep D, Larson M, Williams R (2000) Estimating the error of numerical solutions of systems of nonlinear reaction–diffusion equations. Mem Am Math Soc 696:1–109MathSciNetGoogle Scholar
  22. 22.
    Estep D, Holst M, Mikulencak D (2002) Accounting for stability: a posteriori estimates based on residuals and variational analysis. Commun Numerical Methods Eng 8:15–30zbMATHCrossRefMathSciNetGoogle Scholar
  23. 23.
    Estep D, Holst M, Larson M (2005) Generalized Green’s functions and the effective domain of influence. SIAM J Sci Comput 26:1314–1339zbMATHCrossRefMathSciNetGoogle Scholar
  24. 24.
    Kelly DW, Gago JP, Zienkiewicz OC, Babuska I (1983) A posteriori error analysis and adaptive processes in the finite element method: part I. Error analysis. Int J Num Meth Eng 19:1593–1619zbMATHCrossRefMathSciNetGoogle Scholar
  25. 25.
    Zienkiewicz OC, Zhu JZ (1987) A simple error estimator and adaptive procedure for practical engineering analysis. Int J Numerical Methods Eng 24:337–357zbMATHCrossRefMathSciNetGoogle Scholar
  26. 26.
    Wahlbin LB (1995) Superconvergence in Galerkin finite element methods. Springer, Berlin Heidelberg New YorkzbMATHGoogle Scholar
  27. 27.
    Carey V (2005) A posteriori error estimation for the finite element method via local averaging. Ph.D. thesis, Center for Applied Mathematics Deptartment, Cornell UniversityGoogle Scholar
  28. 28.
    Stewart JR, Edwards HC (2004) A framework approach for developing parallel adaptive multiphysics applications. Finite Elem Anal Des 40(12):1599–1617, ISSN 0168-874X,
  29. 29.
    Bangerth W (2000) Using modern features of C++ for adaptive finite element methods: dimension-independent programming in deal.II. In: Deville M, Owens R (eds) Proceedings of the 16th IMACS World Congress 2000, Lausanne, Switzerland, 2000, Document Sessions/118-1Google Scholar
  30. 30.
    Budge K, Peery J (1996) Experiences developing ALEGRA: a C++ coupled Physics Framework. In: Henderson ME, Anderson CR, Lyons SL (eds) Object oriented methods for interoperable scientific and engineering computingGoogle Scholar
  31. 31.
    Barton JJ, Nackman LR (1994) Scientific and engineering C++: an introduction with advanced techniques and examples. Addison-Wesley, ReadingGoogle Scholar
  32. 32.
    Bastian P, et al (1997) UG – A flexible software toolbox for solving partial differential equations. Comput Vis Sci 1:27–40zbMATHCrossRefGoogle Scholar
  33. 33.
    Devloo PRB, Longhin GC (2002) Object oriented design philosophy for scientific computing. Math Model Numerical Anal 36:793–807zbMATHCrossRefGoogle Scholar
  34. 34.
    Demkowicz L, Rachowicz W, Devloo PRB (2002) A fully automatic hp-adaptivity. J Sci Comput17:127–155CrossRefMathSciNetGoogle Scholar
  35. 35.
    Free software foundation (1999) GNU lesser general public license.
  36. 36.
    Imamura M (2002) Using Doxygen: A quick guide to getting started and using the Doxygen inline documentation system for documenting source code. Tech. rep., Linux Users Group at Georgia TechGoogle Scholar
  37. 37.
    Meyers RJ, Tautges TJ, Tuchinsky PM (1998) The hex-tet hex-dominant meshing algorithm as implemented in CUBIT. In: Proceedings of the 7th International Meshing Roundtable, 151–158Google Scholar
  38. 38.
    Shewchuk JR (1996) Triangle: Engineering a 2D Quality Mesh Generator and Delaunay Triangulator. In: Lin MC, Manocha D (eds) Applied computational geometry: towards geometric engineering, vol. 1148 of lecture notes in computer science, 203–222, Springer, from the First ACM Workshop on Applied Computational GeometryGoogle Scholar
  39. 39.
    Si H (2005) TetGen—a quality tetrahedral mesh generator and three-dimensional Delaunay triangulator. Weierstrass Institute for Applied Analysis and Stochastics, BerlinGoogle Scholar
  40. 40.
    Karypis G, Kumar V (1995) METIS unstructured graph partitioning and sparse matrix order. Technical report, University of Minnesota, Department of Computer ScienceGoogle Scholar
  41. 41.
    Karypis G, Kumar V (1998) A parallel algorithm for multilevel graph partitioning and sparse matrix reordering. Parallel Distrib Comput 48:71–95CrossRefGoogle Scholar
  42. 42.
    Devine K, Boman E, Heaphy R, Hendrickson B, Vaughan C (2002) Zoltan data management services for parallel dynamic applications. Comput Sci Eng 4(2):90–97CrossRefGoogle Scholar
  43. 43.
    Skalický T (1996) LASPack Reference Manual. Dresden University of TechnologyGoogle Scholar
  44. 44.
    Balay S, Eijkhout V, Gropp WD, McInnes LC, Smith BF (1997) Efficient management of parallelism in object oriented numerical software libraries. In: Arge E ,Bruaset AM, Langtangen HP (eds) Modern software tools in scientific computing, 163–202, Birkhäuser PressGoogle Scholar
  45. 45.
    Hernandez V, Roman JE, Vidal V (2005) SLEPc: a scalable and flexible toolkit for the solution of eigenvalue problems. ACM Trans Math Softw 31(3):351–362CrossRefMathSciNetGoogle Scholar
  46. 46.
    Iqbal S, Carey GF (2005) Performance analysis of dynamic load balancing algorithms with variable number of processors. J Parallel Distrib Comput 65(8):934–948zbMATHCrossRefGoogle Scholar
  47. 47.
    Balay S, Buschelman K, Eijkhout V, Gropp W, Kaushik D, Knepley M, McInnes LC, Smith B, Zhang H (2005) PETSc users manual. Argonne National Laboratory, Mathematics and Computer Science Division, 2.3.0 ednGoogle Scholar
  48. 48.
    Gresho PM, Sani RL (1998) Incompressible flow and the finite element method. Wiley, LondonzbMATHGoogle Scholar
  49. 49.
    Clough R, Tocher J (1965) Finite element stiffness matrices for analysis of plates in blending. In: Proceedings of conference on matrix methods in structural analysisGoogle Scholar
  50. 50.
    Ciarlet PJ (1978) The finite element method for elliptic problems. North-Holland, AmsterdamzbMATHGoogle Scholar
  51. 51.
    Dreyer D, Petersen S, von Estorff O (2006) Effectiveness and robustness of improved infinite elements for exterior acoustics. Comput Methods Appl Mech Eng 195(29–32):3591–3607CrossRefGoogle Scholar
  52. 52.
    Dreyer D, von Estorff O (2003) Improved conditioning of infinite elements for exterior acoustics. Int J Numerical Methods Eng 58(6):933–953zbMATHCrossRefGoogle Scholar
  53. 53.
    Marichalar JJ, Rochelle WC, Kirk BS, Campbell CH (2006) BLIMPK/Streamline Surface Catalytic Heating Predictions on the Space Shuttle Orbiter, 44th AIAA Aerospace Sciences Meeting and Exhibit, AIAA Paper 2006-180Google Scholar
  54. 54.
    Valenciano J, Chaplain MAJ (2004) An explicit subparametric spectral element method of lines applied to a tumor angiogenesis system of partial differential equations. Math Model Methods Appl Sci 14:165–187zbMATHCrossRefMathSciNetGoogle Scholar
  55. 55.
    Carey G, Barth W, Woods JA, Kirk BS, Anderson ML, Chow S, Bangerth W (2004) Modelling error and constitutive relations in simulation of flow and transport. Int J Numerical Methods Fluids 46:1211–1236zbMATHCrossRefMathSciNetGoogle Scholar
  56. 56.
    Peterson JW, Carey GF, Knezevic DJ, Murray BT (accepted Mar. 2006) Adaptive finite element methodology for tumor angiogenesis modeling. Int J Numer Meth EngGoogle Scholar
  57. 57.
    Simedrea P, Antiga L, Steinman DA (2006) Towards a new framework for simulating magnetic resonance imaging. In: First Canadian student conference on biomedical computing,
  58. 58.
    Schindler M, Talkner P, Hänggi P (2005) Computing stationary free-surface shapes in microfluidics. submitted to J Fluid Mech
  59. 59.
    Petersen S, Dreyer D, von Estorff O (2006) Assessment of finite and spectral element shape functions for efficient iterative simulations of interior acoustics. Comput Methods Appl Mech Eng (in press)Google Scholar
  60. 60.
    Brinkmeier M, Nackenhorst U, Petersen S, von Estorff O (2006) A numerical model for the simulation of tire rolling noise. J Sound Vib (submitted)Google Scholar
  61. 61.
    Lu S, Holst MJ, Bank RE, McCulloch AD, Michailova A (2005) 3D model of synchronous calcium signaling in ventricular myocyte, biophysical society annual meeting abstract issueGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2006

Authors and Affiliations

  • Benjamin S. Kirk
    • 1
    Email author
  • John W. Peterson
    • 1
  • Roy H. Stogner
    • 1
  • Graham F. Carey
    • 1
  1. 1.CFDLab, Department of Aerospace Engineering and Engineering MechanicsThe University of Texas at AustinAustinUSA

Personalised recommendations