Abstract
We consider the communication costs of gyrokinetic plasma physics simulations running at large scale. For this we apply virtual decompositions of the toroidal domain in three dimensions and additional domain cloning to existing simulations done with the ORB5 code. The communication volume and the number of communication partners per timestep for every virtual task (node) are evaluated for the particles and the structured mesh. Thus the scaling properties of a code with the new domain decompositions are derived for simple models of a modern computer network and corresponding processing units. The effectiveness of the suggested decomposition has been shown. For a typical simulation with \(2\cdot 10^9\) particles and a mesh of \(256\times 1024\times 512\) grid points scaling to 2, 800 nodes should be achieved.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adams, M.F., Ku, S.H., Worley, P., D’Azevedo, E., Cummings, J.C., Chang, C.S.: Scaling to 150K cores: recent algorithm and performance engineering developments enabling XGC1 to run at scale. In: Journal of Physics: Conference Series, vol. 180, p. 012036. IOP Publishing (2009). https://doi.org/10.1088/1742-6596/180/1/012036
Bruck, J., Ho, C.T., Kipnis, S., Upfal, E., Weathersby, D.: Efficient algorithms for all-to-all communications in multiport message-passing systems. IEEE Trans. Parallel Distrib. 8(11), 1143–1156 (1997). https://doi.org/10.1109/71.642949
Burau, H., Widera, R., Honig, W., Juckeland, G., Debus, A., Kluge, T., Schramm, U., Cowan, T.E., Sauerbrey, R., Bussmann, M.: PIConGPU: a fully relativistic particle-in-cell code for a GPU cluster. IEEE Trans. Plasma Sci. 38(10), 2831–2839 (2010). https://doi.org/10.1109/TPS.2010.2064310
Chen, G., Chacón, L., Barnes, D.C.: An efficient mixed-precision, hybrid CPU-GPU implementation of a nonlinearly implicit one-dimensional particle-in-cell algorithm. J. Comput. Phys. 231(16), 5374–5388 (2012). https://doi.org/10.1016/j.jcp.2012.04.040
Decyk, V.K.: Skeleton particle-in-cell codes on emerging computer architectures. Comput. Sci. Eng. 17(2), 47–52 (2015). https://doi.org/10.1109/MCSE.2014.131
Ethier, S., Tang, W.M., Lin, Z.: Gyrokinetic particle-in-cell simulations of plasma microturbulence on advanced computing platforms. In: Journal of Physics: Conference Series, vol. 16, p. 1. IOP Publishing (2005). https://doi.org/10.1088/1742-6596/16/1/001
Hariri, F., Tran, T.M., Jocksch, A., Lanti, E., Progsch, J., Messmer, P., Brunner, S., Gheller, C., Villard, L.: A portable platform for accelerated PIC codes and its application to GPUs using OpenACC. Comput. Phys. Commun. 207, 69–82 (2016). https://doi.org/10.1016/j.cpc.2016.05.008
Hockney, R.W., Eastwood, J.W.: Computer Simulation Using Particles. CRC Press, Cambridge (1988)
Ishiguro, S.: Large scale Particle-In-Cell plasma simulation. In: Resch, M., Roller, S., Benkert, K., Galle, M., Bez, W., Kobayashi, H., Hirayama, T. (eds.) High Performance Computing on Vector Systems, pp. 139–144. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-540-85869-0_13
Jolliet, S., Bottino, A., Angelino, P., Hatzky, R., Tran, T.M., McMillan, B.F., Sauter, O., Appert, K., Idomura, Y., Villard, L.: A global collisionless PIC code in magnetic coordinates. Comput. Phys. Commun. 177(5), 409–425 (2007). https://doi.org/10.1016/j.cpc.2007.04.006
McMillan, B.F., Jolliet, S., Bottino, A., Angelino, P., Tran, T.M., Villard, L.: Rapid Fourier space solution of linear partial integro-differential equations in toroidal magnetic confinement geometries. Comput. Phys. Commun. 181(4), 715–719 (2010). https://doi.org/10.1016/j.cpc.2009.12.001
Naitou, H., Hashimoto, H., Yamada, Y., Tokuda, S., Yagi, M.: Parallelization of gyrokinetic PIC code for MHD simulation. Progress in Nuclear Science and Technology 2, 657–662 (2011). https://doi.org/10.15669/pnst.2.657
Ohana, N., Jocksch, A., Lanti, E., Tran, T.M., Brunner, S., Gheller, C., Hariri, F., Villard, L.: Towards the optimization of a gyrokinetic Particle-In-Cell (PIC) code on large-scale hybrid architectures. In: Journal of Physics: Conference Series, vol. 775, p. 012010. IOP Publishing (2016). https://doi.org/10.1088/1742-6596/775/1/012010
Wang, B., Ethier, S., Tang, W., Ibrahim, K., Madduri, K., Williams, S., Oliker, L.: Modern gyrokinetic particle-in-cell simulation of fusion plasmas on top supercomputers. Int. J. High Perform. Comput. Appl. (2017). https://doi.org/10.1177/1094342017712059
Wei, Y., Wang, Y., Cai, L., Tang, W., Wang, B., Ethier, S., See, S., Lin, J.: Performance and portability studies with OpenACC accelerated version of GTC-P. In: 17th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT) (2016). https://doi.org/10.1109/PDCAT.2016.019
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Jocksch, A. et al. (2018). Prediction of the Inter-Node Communication Costs of a New Gyrokinetic Code with Toroidal Domain. In: Wyrzykowski, R., Dongarra, J., Deelman, E., Karczewski, K. (eds) Parallel Processing and Applied Mathematics. PPAM 2017. Lecture Notes in Computer Science(), vol 10777. Springer, Cham. https://doi.org/10.1007/978-3-319-78024-5_33
Download citation
DOI: https://doi.org/10.1007/978-3-319-78024-5_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-78023-8
Online ISBN: 978-3-319-78024-5
eBook Packages: Computer ScienceComputer Science (R0)