Weighted Decomposition in High-Performance Lattice-Boltzmann Simulations: Are Some Lattice Sites More Equal than Others?

  • Derek GroenEmail author
  • David Abou Chacra
  • Rupert W. Nash
  • Jiri Jaros
  • Miguel O. Bernabeu
  • Peter V. Coveney
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8759)


Obtaining a good load balance is a significant challenge in scaling up lattice-Boltzmann simulations of realistic sparse problems to the exascale. Here we analyze the effect of weighted decomposition on the performance of the HemeLB lattice-Boltzmann simulation environment, when applied to sparse domains. Prior to domain decomposition, we assign wall and in/outlet sites with increased weights which reflect their increased computational cost. We combine our weighted decomposition with a second optimization, which is to sort the lattice sites according to a space filling curve. We tested these strategies on a sparse bifurcation and very sparse aneurysm geometry, and find that using weights reduces calculation load imbalance by up to 85 %, although the overall communication overhead is higher than some of our runs.


High performance computing Lattice-Boltzmann Domain decomposition 



We thank Timm Krueger for his valuable input. This work has received funding from the CRESTA and MAPPER projects within the EC-FP7 (ICT-2011.9.13) under Grant Agreements nos. 287703 and 261507, and from EPSRC Grants EP/I017909/1 ( and EP/I034602/1. This work made use of the HECToR supercomputer at EPCC in Edinburgh, funded by the Office of Science and Technology through EPSRC’s High End Computing Programme.


  1. 1.
    Palabos LBM Wiki (2011).
  2. 2.
    Cresta case study: application soars above petascale after tools collaboration (2014).
  3. 3.
  4. 4.
    Axner, L., Bernsdorf, J., Zeiser, T., Lammers, P., Linxweiler, J., Hoekstra, A.G.: Performance evaluation of a parallel sparse lattice Boltzmann solver. J. Computat. Phys. 227(10), 4895–4911 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  5. 5.
    Barad, M.F., Colella, P., Schladow, S.G.: An adaptive cut-cell method for environmental fluid mechanics. Int. J. Numer. Methods Fluids 60(5), 473–514 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  6. 6.
    Bhatnagar, P.L., Gross, E.P., Krook, M.: A model for collision processes in gases. i. small amplitude processes in charged and neutral one-component systems. Phys. Rev. 94, 511–525 (1954)CrossRefzbMATHGoogle Scholar
  7. 7.
    Bouzidi, M., Firdaouss, M., Lallemand, P.: Momentum transfer of a Boltzmann-lattice fluid with boundaries. Phys. Fluids 13(11), 3452–3459 (2001)CrossRefGoogle Scholar
  8. 8.
    Carver, H.B., Groen, D., Hetherington, J., Nash, R.W., Bernabeu, M.O., Coveney, P.V.: Coalesced communication: a design pattern for complex parallel scientific software. Advances in Engineering Software (2015, in press)Google Scholar
  9. 9.
    Catalyurek, U.V., Boman, E.G., Devine, K.D., Bozdag, D., Heaphy, R., Riesen, L.A.: Hypergraph-based dynamic load balancing for adaptive scientific computations. In: IEEE International Parallel and Distributed Processing Symposium, 2007, IPDPS 2007, pp. 1–11. IEEE (2007)Google Scholar
  10. 10.
    Godenschwager, C., Schornbaum, F., Bauer, M., Köstler, H., Rüde, U.: A framework for hybrid parallel flow simulations with a trillion cells in complex geometries. In: Proceedings of SC13: International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013, pp. 35:1–35:12. ACM, New York (2013)Google Scholar
  11. 11.
    Groen, D., Borgdorff, J., Bona-Casas, C., Hetherington, J., Nash, R.W., Zasada, S.J., Saverchenko, I., Mamonski, M., Kurowski, K., Bernabeu, M.O., Hoekstra, A.G., Coveney, P.V.: Flexible composition and execution of high performance, high fidelity multiscale biomedical simulations. Interface Focus 3(2), 20120087 (2013)CrossRefGoogle Scholar
  12. 12.
    Groen, D., Henrich, O., Janoschek, F., Coveney, P.V., Harting, J.: Lattice-boltzmann methods in fluid dynamics: turbulence and complex colloidal fluids. In: Jülich Blue Gene/P Extreme Scaling Workshop (2011)Google Scholar
  13. 13.
    Groen, D., Hetherington, J., Carver, H.B., Nash, R.W., Bernabeu, M.O., Coveney, P.V.: Analyzing and modeling the performance of the HemeLB lattice-Boltzmann simulation environment. J. Comput. Sci. 4(5), 412–422 (2013)CrossRefGoogle Scholar
  14. 14.
    Hasert, M., Masilamani, K., Zimny, S., Klimach, H., Qi, J., Bernsdorf, J., Roller, S.: Complex fluid simulations with the parallel tree-based lattice boltzmannsolver musubi. J. Comput. Sci. 5, 784–794 (2013)CrossRefMathSciNetGoogle Scholar
  15. 15.
    Hendrickson, B., Kolda, T.G.: Graph partitioning models for parallel computing. Parallel Comput. 26(12), 1519–1534 (2000)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Mazzeo, M.D., Coveney, P.V.: HemeLB: a high performance parallel lattice-Boltzmann code for large scale fluid flow in complex geometries. Comput. Phys. Commun. 178(12), 894–914 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  17. 17.
    Mazzeo, M.D., Manos, S., Coveney, P.V.: In situ ray tracing and computational steering for interactive blood flow simulation. Comput. Phys. Commun. 181, 355–370 (2010)CrossRefzbMATHGoogle Scholar
  18. 18.
    Nash, R.W., Carver, H.B., Bernabeu, M.O., Hetherington, J., Groen, D., Krüger, T., Coveney, P.V.: Choice of boundary condition for lattice-Boltzmann simulation of moderate Reynolds number flow in complex domains. Phys. Rev. E 89, 023303 (2014)CrossRefGoogle Scholar
  19. 19.
    Peters, A., Melchionna, S., Kaxiras, E., Lätt, J., Sircar, J., Bernaschi, M., Bison, M., Succi, S.: Multiscale simulation of cardiovascular flows on the ibm bluegene/p: full heart-circulation system at red-blood cell resolution. In: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2010, Washington, DC, USA, pp. 1–10. IEEE Computer Society (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Derek Groen
    • 1
    Email author
  • David Abou Chacra
    • 1
  • Rupert W. Nash
    • 1
  • Jiri Jaros
    • 2
  • Miguel O. Bernabeu
    • 1
  • Peter V. Coveney
    • 1
  1. 1.Centre for Computational ScienceUniversity College LondonLondonUK
  2. 2.Faculty of Information TechnologyBrno University of TechnologyBrnoCzech Republic

Personalised recommendations