nSharma: Numerical Simulation Heterogeneity Aware Runtime Manager for OpenFOAM
CFD simulations are a fundamental engineering application, implying huge workloads, often with dynamic behaviour due to runtime mesh refinement. Parallel processing over heterogeneous distributed memory clusters is often used to process such workloads. The execution of dynamic workloads over a set of heterogeneous resources leads to load imbalances that severely impacts execution time, when static uniform load distribution is used. This paper proposes applying dynamic, heterogeneity aware, load balancing techniques within CFD simulations. nSharma, a software package that fully integrates with OpenFOAM, is presented and assessed. Performance gains are demonstrated, achieved by reducing busy times standard deviation among resources, i.e., heterogeneous computing resources are kept busy with useful work due to an effective workload distribution. To best of authors’ knowledge, nSharma is the first implementation and integration of heterogeneity aware load balancing in OpenFOAM and will be made publicly available in order to foster its adoption by the large community of OpenFOAM users.
KeywordsComputational Fluid Dynamics OpenFOAM Heterogeneous systems Dynamic Load Balancing
The authors would like to thank the financial funding by FEDER through the COMPETE 2020 Program, the National Funds through FCT under the projects UID/CTM/50025/2013. The first author was partially funded by the PT-FLAD Chair on Smart Cities & Smart Governance and also by the School of Engineering, University of Minho within project Performance Portability on Scalable Heterogeneous Computing Systems. The authors also wish to thank Kyle Mooney for making available his code supporting migration of dynamically refined meshes, as well as acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC resources.
- 1.ANSYS: ANSYS CFX Users’ Guide (2017)Google Scholar
- 2.ANSYS: ANSYS Fluent User’s Guide (2017)Google Scholar
- 5.Brodtkorb, A.R., et al.: State-of-the-art in heterogeneous computing. Sci. Program. 18, 1–33 (2010)Google Scholar
- 6.CD-adapco: STAR-CCM+ (2017)Google Scholar
- 10.Da Costa, G., et al.: Exascale machines require new programming paradigms and runtimes. Supercomput. Front. Innov. 2, 6–27 (2015)Google Scholar
- 11.Devine, K., et al.: Design of dynamic load-balancing tools for parallel applications. In: Proceedings of the 14th International Conference on Supercomputing - ICS 2000 (2000)Google Scholar
- 12.Faik, J., Teresco, J.D., Devine, K.D., Flaherty, J.E., Gervasio, L.G.: A model for resource-aware load balancing on heterogeneous clusters. CS-05-01. Williams College Department of Computer Science (2005)Google Scholar
- 14.OpenFOAM Foundation: OpenFOAM Users’ Guide. Technical report (2018)Google Scholar
- 16.Top500: TOP500 Supercomputer Site (2017)Google Scholar
- 17.Zhong, Z.: Optimization of data-parallel scientific applications on highly heterogeneous modern HPC platforms. Ph.D. thesis (2014)Google Scholar