The Journal of Supercomputing

, Volume 59, Issue 3, pp 1218–1228 | Cite as

Unstructured mesh partition improvement for implicit finite element at extreme scale

  • Min ZhouEmail author
  • Onkar Sahni
  • Ting Xie
  • Mark S. Shephard
  • Kenneth E. Jansen


Parallel simulations at extreme scale require that the mesh is distributed across a large number of processors with equal work load and minimum inter-part communications. A number of algorithms have been developed to meet these goals and graph/hypergraph-based methods are by far the most powerful ones. However, the global implementation of current approaches can fail on very large core counts and the vertex imbalance is not optimal where individual cores are lightly loaded. Those issues are resolved by combination of global and local partitioning and an iterative improvement algorithm, LIIPBMod, developed in the previous study (Zhou et al. in SIAM J. Sci. Comput. 32:3201–3227, 2010). In the current work, this combined partition strategy is applied to the simulations at extreme scale with up to O(1010) elements and up to O(300K) cores. Strong scaling studies on IBM BlueGene/P and Cray XT5 systems demonstrate the effectiveness of this combined partition algorithm.


Partition improvement Extreme scale Unstructured mesh Finite element 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Boman E, Devine K, Fisk LA, Heaphy R, Hendrickson B, Leung V, Vaughan C, Catalyurek U, Bozdag  D, Mitchell W (1999) Zoltan home page.
  2. 2.
    Bui T, Jones C (1993) A heuristic for reducing fill in sparse matrix factorization. In: Proceedings of the 6th SIAM conference on parallel processing for scientific computing. SIAM, Philadelphia, pp 445–452 Google Scholar
  3. 3.
    Çatalyürek ÜV, Aykanat C (1999) PaToH: a multilevel hypergraph partitioning tool, version 3.0. Bilkent University, Department of Computer Engineering, Ankara, 06533 Turkey. PaToH is available at
  4. 4.
    Devine KD, Boman EG, Heaphy RT, Bisseling RH, Catalyurek UV (2006) Parallel hypergraph partitioning for scientific computing. In: Proceedings of 20th international parallel and distributed processing symposium (IPDPS’06). IEEE, New York Google Scholar
  5. 5.
    Hendrickson B, Leland R (1995) A multilevel algorithm for partitioning graphs. In: Proceedings of supercomputing ’95, December 1995. ACM, New York Google Scholar
  6. 6.
    Jansen KE, Whiting CH, Hulbert GM (1999) A generalized-α method for integrating the filered Navier–Stokes equations with a stabilized finite element method. Comput Methods Appl Mech Eng 190:305–319 MathSciNetCrossRefGoogle Scholar
  7. 7.
    Karypis G, Kumar V (1996) A parallel algorithm for multilevel graph partitioning and sparse matrix ordering. In: 10th international parallel processing symposium, pp 314–319 Google Scholar
  8. 8.
    Karypis G, Kumar V (1998) A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM J Sci Comput 20(1):359–392 MathSciNetCrossRefGoogle Scholar
  9. 9.
    Karypis G, Kumar V (1998) Multilevel algorithms for multi-constraint graph partitioning. In: Proceedings of the 1998 ACM/IEEE conference on supercomputing, pp 1–13 Google Scholar
  10. 10.
    Sahni O, Müller Y, Jansen KE, Shephard MS, Taylor CA (2006) Efficient anisotropic adaptive discretization of the cardiovascular system. Comput Methods Appl Mech Eng 195:5634–5655 zbMATHCrossRefGoogle Scholar
  11. 11.
    Sahni O, Zhou M, Shephard MS, Jansen KE (2009) Scalable implicit finite element solver for massively parallel processing with demonstration to 160k cores. In: Proceedings of IEEE/ACM SC’09, Portland, OR, USA, November 2009. Finalist paper for the Gordon Bell prize Google Scholar
  12. 12.
    Schloegel K, Karypis G, Kumar V (2002) Parallel static and dynamic multiconstraint graph partitioning. Concurr Comput, Pract Exp 14(3):219–240 zbMATHCrossRefGoogle Scholar
  13. 13.
    Teresco JD, Devine KD, Flaherty JE (2005) Partitioning and dynamic load balancing for the numerical solution of partial differential equations. In: Numerical solution of partial differential equations on parallel computers. Springer, Berlin Google Scholar
  14. 14.
    Trifunovic A, Knottenbelt WJ (2004) Parkway 2.0: a parallel multilevel hypergraph partitioning tool. In: Proceedings of 19th international symposium on computer and information sciences (ISCIS 2004). LNCS, vol 3280. Springer, Berlin, pp 789–800 Google Scholar
  15. 15.
    Vignon-Clementel IE, Figueroa CA, Jansen KE, Taylor CA (2006) Outflow boundary conditions for three-dimensional finite element modeling of blood flow and pressure in arteries. Comput Methods Appl Mech Eng 195(29–32):3776–3796 MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Whiting CH, Jansen KE (2001) A stabilized finite element method for the incompressible Navier–Stokes equations using a hierarchical basis. Int J Numer Methods Fluids 35:93–116 zbMATHCrossRefGoogle Scholar
  17. 17.
    Womersley J (1955) Method for the calculation of velocity, rate of flow and viscous drag in arteries when the pressure gradient is known. J Physiol 127:553–563 Google Scholar
  18. 18.
    Zhou M (2009) Petascale adaptive computational fluid dynamics. PhD thesis, Rensselaer Polytechnic Institute, August 2009 Google Scholar
  19. 19.
    Zhou M, Sahni O, Devine KD, Shephard MS, Jansen KE (2010) Controlling unstructured mesh partitions for massively parallel simulations. SIAM J Sci Comput 32:3201–3227 MathSciNetzbMATHCrossRefGoogle Scholar
  20. 20.
    Zhou M, Sahni O, Shephard MS, Carothers CD, Jansen KE (2010) Adjacency-based data reordering algorithm for acceleration of finite element computations. Sci Program 18:107–123 Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Min Zhou
    • 1
    Email author
  • Onkar Sahni
    • 1
  • Ting Xie
    • 1
  • Mark S. Shephard
    • 1
  • Kenneth E. Jansen
    • 2
  1. 1.Scientific Computational Research CenterRensselaer Polytechnic InstituteTroyUSA
  2. 2.Aerospace Engineering SciencesUniversity of Colorado at BoulderBoulderUSA

Personalised recommendations