Engineering with Computers

, Volume 28, Issue 3, pp 287–301 | Cite as

Tools to support mesh adaptation on massively parallel computers

  • Min Zhou
  • Ting Xie
  • Seegyoung Seol
  • Mark S. ShephardEmail author
  • Onkar Sahni
  • Kenneth E. Jansen
Original Article


The scalable execution of parallel adaptive analyses requires the application of dynamic load balancing to repartition the mesh into a set of parts with balanced work load and minimal communication. As the adaptive meshes being generated reach billions of elements and the analyses are performed on massively parallel computers with 100,000’s of computing cores, a number of complexities arise that need to be addressed. This paper presents procedures developed to deal with two of them. The first is a procedure to support multiple parts per processor which is used as the mesh increases in size and it is desirable to partition the mesh to a larger number of computing cores than are currently being used. The second is a predictive load balancing method that sets entity weights before dynamic load balancing steps so that the mesh is well balanced after the mesh adaptation step thus avoiding excessive memory spikes that would otherwise occur during mesh adaptation.


Mesh adaptation Parallel computation Dynamic load balancing 



We gratefully acknowledge the support of this work by NSF’s PetaApps program under grant OCI-0749152 and by the DOE Office of Science’s SciDAC/ITAPS program under grant DE-FC02-06ER25769. This research used computing resources provided by: (a) Rensselaer’s Computational Center for Nanotechnology Innovations that is funded by the State of New York, IBM and Rensselaer Polytechnic Institute, (b) Argonne Leadership Computing Facility at Argonne National Laboratory, which is supported by the Office of Science of the DOE under contracts DE-AC02-06CH11357, and (c) NSF through TeraGrid resources provided by National Institute for Computational Sciences (NICS); Kraken system was used. We would like to acknowledge that results presented in this article made use of software components provided by ACUSIM Software Inc. and Simmetrix Inc.


  1. 1.
    Flaherty J, Loy R, Özturan C, Shephard M, Szymanski B, Teresco J, Ziantz L (1996) Parallel structure and dynamic load balancing for adaptive finite element computation. Appl Numer Math 26:241–263CrossRefGoogle Scholar
  2. 2.
    Shephard M, Flaherty J, Bottasso C, de Cougny H, Ozturan C, Simone M (1997) Parallel automated adaptive analysis. Parallel Comput 23:1327–1347MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Sahni O, Zhou M, Shephard M, Jansen K (2009) Scalable implicit finite element solver for massively parallel processing with demonstration to 160k cores. In: Proceedings of the SC09. Springer, BerlinGoogle Scholar
  4. 4.
    Shephard M, Jansen K, Sahni O, Diachin L (2007) Parallel adaptive simulations on unstructured meshes. J Phys Conf Ser 78:012053CrossRefGoogle Scholar
  5. 5.
    Zhou M, Sahni O, Kim H, Figueroa C, Taylor C, Shephard M, Jansen (2010) Cardiovascular flow simulation at extreme scale. Comput Mech 46:71–82 doi: 10.1007/s00466-009-0450-z MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Zhou M, Sahni O, Shephard M, Devine K, Jansen K (2010) Controlling unstructured mesh partitions for massively parallel simulations. SIAM J Sci Comput 32(6):3201–3227Google Scholar
  7. 7.
    Beall MW, Shephard MS (1997) A general topology-based mesh data structure. Int J Numer Methods Eng 40:1573–1596MathSciNetCrossRefGoogle Scholar
  8. 8.
    Celes ERW, Paulino GH (2005) A compact adjacency-based topological data structure for finite element mesh representation. Int J Numer Methods Eng 64:1529–1565zbMATHCrossRefGoogle Scholar
  9. 9.
    Garimella R (2002), Mesh data structure selection for mesh generation and fea applications. Int J Numer Methods Eng 55:451–478zbMATHCrossRefGoogle Scholar
  10. 10.
    Seol E, Shephard M (2006) Efficient distributed mesh data structure for parallel automated adaptive analysis. Eng Comput 22(3–4):197–213CrossRefGoogle Scholar
  11. 11.
    Chand K, Diachin L, Li X, Ollivier-Gooch C, Seol E, Shephard M, Tautges T, Trease H (2008) Toward interoperable mesh, geometry and field components for PDE simulation development. Eng Comput 24(2):165–182CrossRefGoogle Scholar
  12. 12.
    Devine K, Diachin L, Kraftcheck J, Jansen KE, Leung V, Luo X, Miller M, Ollivier-Gooch C, Ovcharenko A, Sahni O, Shephard M, Tautges T, Xie T, Zhou M (2009) Interoperable mesh components for large-scale, distributed-memory simulations. J Phys Conf Ser 180:012011CrossRefGoogle Scholar
  13. 13.
    Boman E, Devine K, Fisk L, Heaphy R, Hendrickson B, Leung V, Vaughan C, Catalyurek U, Bozdag D, Mitchell W (1999) Zoltan home page.
  14. 14.
    Devine K, Boman E, Heaphy R, Hendrickson B, Vaughan C (2002) Zoltan data management services for parallel dynamic applications. Comput Sci Eng 4(2):90–97CrossRefGoogle Scholar
  15. 15.
    Devine K, Boman E, Heaphy R, Bisseling R, Catalyurek U (2006) Parallel hypergraph partitioning for scientific computing. In: Proceedings of 20th international parallel and distributed processing symposium (IPDPS06), IEEEGoogle Scholar
  16. 16.
    Karypis G, Kumar V (1999) Parallel multilevel k-way partitioning scheme for irregular graphs. SIAM Rev 41(2):278–300MathSciNetzbMATHCrossRefGoogle Scholar
  17. 17.
    Pellegrini F (2007) Scotch 5.0 user’s guide. Tech. rep., LaBRIGoogle Scholar
  18. 18.
    Walshaw C, Cross M (2007) JOSTLE: parallel multilevel graph-partitioning software—an overview. In: Mesh partitioning techniques and domain decomposition techniques, pp 27–58Google Scholar
  19. 19.
    Teresco J, Devine K, Flaherty J (2005) Partitioning and dynamic load balancing for the numerical solution of partial differential equations. In: Numerical solution of partial differential equations on parallel computers. Springer, BerlinGoogle Scholar
  20. 20.
    Karypis G, Kumar V (1996) A parallel algorithm for multilevel graph partitioning and sparse matrix ordering. In: 10th intlernational parallel processing symposium, pp 314–319Google Scholar
  21. 21.
    Karypis G, Kumar V (1998) Multilevel algorithms for multi-constraint graph partitioning. In: Proceedings of the 1998 ACM/IEEE conference on supercomputing, pp 1–13Google Scholar
  22. 22.
    Womersley J (1955) Method for the calculation of velocity, rate of flow and viscous drag in arteries when the pressure gradient is known. J Physiol 127:553–563Google Scholar
  23. 23.
    Vignon-Clementel I, Figueroa C, Jansen K, Taylor C (2006) Outflow boundary conditions for three-dimensional finite element modeling of blood flow and pressure in arteries. Comput Methods Appl Mech Eng 195(29–32):3776–3796MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    Flaherty J, Loy R, Shephard M, Szymanski B, Teresco J, Ziantz L (1997) Predictive load balancing for parallel adaptive finite element computation. In: Proceedings of the international conference on parallel and distributed processing techniques and applications, PDPTA’97, vol 1, pp 460–469Google Scholar
  25. 25.
    Oliker L, Biswas R, Strawn R (1996) Parallel implementation of an adaptive scheme for 3D unstructured grids on the SP2. In: Proceedings of the 3rd international workshop on parallel algorithm for irregularly structured problemsGoogle Scholar
  26. 26.
    Li X, Shephard M, Beall M (2005) 3D anisotropic mesh adaptation by mesh modification. Comput Methods Appl Mech Eng 194:4915–4950MathSciNetzbMATHCrossRefGoogle Scholar
  27. 27.
    Sahni O, Müller Y, Jansen K, Shephard M, Taylor C (2006) Efficient anisotropic adaptive discretization of the cardiovascular system. Comput Methods Appl Mech Eng 195:5634–5655zbMATHCrossRefGoogle Scholar
  28. 28.
    Borouchaki H, George P, Hecht F, Laug P, Saltel E (1997) Delaunay mesh generation governed by metric specifications. part i. algorithms. Finite Elem Anal Des 25(1–2):61–83MathSciNetzbMATHCrossRefGoogle Scholar
  29. 29.
    Frey P, George PL, Hecht F (2008) Mesh generation: application to finite elements, 2nd edn. ISTE, LondonzbMATHGoogle Scholar
  30. 30.
    Labbe P, Dompierre J, Vallet M-G, Guibault F, Trpanier J-Y (2004) A universal measure of the conformity of a mesh with respect to an anisotropic metric field. Int J Numer Methods Eng 61(16):2675–2695zbMATHCrossRefGoogle Scholar
  31. 31.
    Li X (2003) Mesh modification procedures for general 3D non-manifold domains. Ph.D. thesis, Rensselaer Polytechnic InstituteGoogle Scholar
  32. 32.
    Pain C, Umpleby A, de Oliveira C, Goddard A (2001) Tetrahedron mesh optimization and adaptivity for steady-state and transient finite element calculations. Comput Methods Appl Mech Eng 190:3771–3796zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Min Zhou
    • 1
  • Ting Xie
    • 1
  • Seegyoung Seol
    • 1
  • Mark S. Shephard
    • 1
    Email author
  • Onkar Sahni
    • 1
  • Kenneth E. Jansen
    • 2
  1. 1.Scientific Computation Research CenterRensselaer Polytechnic InstituteTroyUSA
  2. 2.Department of Aerospace Engineering SciencesUniversity of Colorado at BoulderBoulderUSA

Personalised recommendations