nSharma: Numerical Simulation Heterogeneity Aware Runtime Manager for OpenFOAM

  • Roberto Ribeiro
  • Luís Paulo Santos
  • João Miguel Nóbrega
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10860)


CFD simulations are a fundamental engineering application, implying huge workloads, often with dynamic behaviour due to runtime mesh refinement. Parallel processing over heterogeneous distributed memory clusters is often used to process such workloads. The execution of dynamic workloads over a set of heterogeneous resources leads to load imbalances that severely impacts execution time, when static uniform load distribution is used. This paper proposes applying dynamic, heterogeneity aware, load balancing techniques within CFD simulations. nSharma, a software package that fully integrates with OpenFOAM, is presented and assessed. Performance gains are demonstrated, achieved by reducing busy times standard deviation among resources, i.e., heterogeneous computing resources are kept busy with useful work due to an effective workload distribution. To best of authors’ knowledge, nSharma is the first implementation and integration of heterogeneity aware load balancing in OpenFOAM and will be made publicly available in order to foster its adoption by the large community of OpenFOAM users.


Computational Fluid Dynamics OpenFOAM Heterogeneous systems Dynamic Load Balancing 



The authors would like to thank the financial funding by FEDER through the COMPETE 2020 Program, the National Funds through FCT under the projects UID/CTM/50025/2013. The first author was partially funded by the PT-FLAD Chair on Smart Cities & Smart Governance and also by the School of Engineering, University of Minho within project Performance Portability on Scalable Heterogeneous Computing Systems. The authors also wish to thank Kyle Mooney for making available his code supporting migration of dynamically refined meshes, as well as acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC resources.


  1. 1.
    ANSYS: ANSYS CFX Users’ Guide (2017)Google Scholar
  2. 2.
    ANSYS: ANSYS Fluent User’s Guide (2017)Google Scholar
  3. 3.
    Barker, K., et al.: A load balancing framework for adaptive and asynchronous applications. IEEE Trans. Parallel Distrib. Syst. 15, 183–192 (2004)CrossRefGoogle Scholar
  4. 4.
    Basermann, A., et al.: Dynamic load-balancing of finite element applications with the DRAMA library. Appl. Math. Model. 25, 83–98 (2000)CrossRefGoogle Scholar
  5. 5.
    Brodtkorb, A.R., et al.: State-of-the-art in heterogeneous computing. Sci. Program. 18, 1–33 (2010)Google Scholar
  6. 6.
    CD-adapco: STAR-CCM+ (2017)Google Scholar
  7. 7.
    Chevalier, C., Pellegrini, F.: PT-Scotch: a tool for efficient parallel graph ordering. Parallel Comput. 34, 318–331 (2008)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Clarke, D., Lastovetsky, A., Rychkov, V.: Dynamic load balancing of parallel computational iterative routines on highly heterogeneous HPC platforms. Parallel Process. Lett. 21, 195–217 (2011)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Clarke, D., et al.: FuPerMod: a software tool for the optimization of data-parallel applications on heterogeneous platforms. J. Supercomput. 69, 61–69 (2014)CrossRefGoogle Scholar
  10. 10.
    Da Costa, G., et al.: Exascale machines require new programming paradigms and runtimes. Supercomput. Front. Innov. 2, 6–27 (2015)Google Scholar
  11. 11.
    Devine, K., et al.: Design of dynamic load-balancing tools for parallel applications. In: Proceedings of the 14th International Conference on Supercomputing - ICS 2000 (2000)Google Scholar
  12. 12.
    Faik, J., Teresco, J.D., Devine, K.D., Flaherty, J.E., Gervasio, L.G.: A model for resource-aware load balancing on heterogeneous clusters. CS-05-01. Williams College Department of Computer Science (2005)Google Scholar
  13. 13.
    Martínez, J.A., Garzón, E.M., Plaza, A., García, I.: Automatic tuning of iterative computation on heterogeneous multiprocessors with ADITHE. J. Supercomput. 58, 151–159 (2011)CrossRefGoogle Scholar
  14. 14.
    OpenFOAM Foundation: OpenFOAM Users’ Guide. Technical report (2018)Google Scholar
  15. 15.
    Schloegel, K., Karypis, G., Kumar, V.: Multilevel diffusion schemes for repartitioning of adaptive meshes. J. Parallel Distrib. Comput. 47, 109–124 (1997)CrossRefGoogle Scholar
  16. 16.
    Top500: TOP500 Supercomputer Site (2017)Google Scholar
  17. 17.
    Zhong, Z.: Optimization of data-parallel scientific applications on highly heterogeneous modern HPC platforms. Ph.D. thesis (2014)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Informatics, Algoritmi Research CenterUniversity of MinhoBragaPortugal
  2. 2.Department of Informatics, INESC-TECUniversity of MinhoBragaPortugal
  3. 3.Institute for Polymers and Composites/I3NUniversity of MinhoGuimarãesPortugal

Personalised recommendations