Using Simulation to Evaluate and Tune the Performance of Dynamic Load Balancing of an Over-Decomposed Geophysics Application

  • Rafael Keller TesserEmail author
  • Lucas Mello Schnorr
  • Arnaud Legrand
  • Fabrice Dupros
  • Philippe Olivier Alexandre Navaux
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10417)


Finite difference methods are commonplace in scientific computing. Despite their apparent regularity, they often exhibit load imbalance that damages their efficiency. We characterize the spatial and temporal load imbalance of Ondes3D, a seismic wave propagation simulator. We reveal that this imbalance originates from the nature of the input data and from low-level CPU optimizations. Such dynamic imbalance should therefore be quite common and is intractable by any static approach or classical code reorganization. An effective solution, with few code modifications, combines domain over-decomposition and dynamic load balancing (e.g., with AMPI), migrating data and computation at the granularity of an MPI rank. It generally requires a careful tuning of the over-decomposition level, the load balancing heuristic and frequency. These choices are quite dependent on application and platform characteristics. In this paper, we propose a methodology that leverages the capabilities of the SimGrid framework to conduct such study at low experimental cost. It combines emulation, simulation, and application modeling that requires minimal code modification and yet manages to capture both spatial and temporal load imbalance, faithfully predicting its overall performance. We compare simulation and real executions results and show how our strategy can be used to determine the best load balancing configuration for a given application/hardware configuration.


Load balancing and over-decomposition Performance prediction Simulation Geophysics FDM application 



We thank CAPES/Cofecub 764-13, FAPERGS/Inria ExaSE, FAPERGS Green-Cloud, CNPq 447311/2014-0, CNRS/LICIA Intl. Lab, the EU H2020 Programme and from MCTI/RNP-Brazil under the HPC4E Project, grant 689772. Some experiments were carried out at the Grid’5000 platform (, with support from Inria, CNRS, RENATER and several other organizations.


  1. 1.
    Aochi, H., Ducellier, A., Dupros, F., Delatre, M., Ulrich, T., Martin, F., Yoshimi, M.: Finite difference simulations of seismic wave propagation for the 2007 mw 6.6 Niigata-ken Chuetsu-Oki earthquake: Validity of models and reliable input ground motion in the near-field. Pure Appl. Geophys. 170(1–2), 43–64 (2013)CrossRefGoogle Scholar
  2. 2.
    Aochi, H., Ducellier, A., Dupros, F., Terrier, M., Lambert, J.: Investigation of historical earthquake by seismic wave propagation simulation: source parameters of the 1887 M6.3 Ligurian, north-western Italy, earthquake. In: 8ème colloque AFPS, Vers une maitrise durable du risque sismique. p. 6, September 2011Google Scholar
  3. 3.
    Balouek, D., et al.: Adding virtualization capabilities to the Grid’5000 testbed. In: Ivanov, I.I., Sinderen, M., Leymann, F., Shan, T. (eds.) CLOSER 2012. CCIS, vol. 367, pp. 3–20. Springer, Cham (2013). doi: 10.1007/978-3-319-04519-1_1 CrossRefGoogle Scholar
  4. 4.
    Bédaride, P., et al.: Toward better simulation of MPI applications on Ethernet/TCP networks. In: Jarvis, S.A., Wright, S.A., Hammond, S.D. (eds.) PMBS 2013. LNCS, vol. 8551, pp. 158–181. Springer, Cham (2013). doi: 10.1007/978-3-319-10214-6_8 Google Scholar
  5. 5.
    Casanova, H., Giersch, A., Legrand, A., Quinson, M., Suter, F.: Versatile, scalable, and accurate simulation of distributed applications and platforms. Parallel Distrib. Comput. 74(10), 2899–2917 (2014)CrossRefGoogle Scholar
  6. 6.
    Dupros, F., Do, H.T., Aochi, H.: On scalability issues of the elastodynamics equations on multicore platforms. In: International Conference on Computer Science, Procedia Computer Science, p. 9. Elsevier, Barcelone, June 2013Google Scholar
  7. 7.
    Dupros, F., Martin, F.D., Foerster, E., Komatitsch, D., Roman, J.: High-performance finite-element simulations of seismic wave propagation in three-dimensional nonlinear inelastic geological media. Parallel Comput. 36(5–6), 308–325 (2010)CrossRefzbMATHGoogle Scholar
  8. 8.
    Engelmann, C.: Scaling to a million cores and beyond: using light-weight simulation to understand the challenges ahead on the road to exascale. Future Gener. Comput. Syst. 30, 59–65 (2014)CrossRefGoogle Scholar
  9. 9.
    Engelmann, C., Naughton, T.: A network contention model for the extreme-scale simulator. In: Press, A. (ed.) 34th IASTED International Conference on Modelling, Identification and Control (MIC) (2015)Google Scholar
  10. 10.
    Huang, C., Lawlor, O., Kalé, L.V.: Adaptive MPI. In: Rauchwerger, L. (ed.) LCPC 2003. LNCS, vol. 2958, pp. 306–322. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-24644-2_20 CrossRefGoogle Scholar
  11. 11.
    Kalé, L., Krishnan, S.: CHARM++: a portable concurrent object oriented system based on C++. In: Proceedings of OOPSLA 1993, pp. 91–108. ACM Press (1993)Google Scholar
  12. 12.
    Keller Tesser, R., Lima Pilla, L., Dupros, F., Navaux, P., Mehaut, J.F., Mendes, C.: Improving the performance of seismic wave simulations with dynamic load balancing. In: International Conference Parallel, Distributed and Network-Based Processing (2014)Google Scholar
  13. 13.
    Martinez, V., Michéa, D., Dupros, F., Aumage, O., Thibault, S., Aochi, H., Navaux, P.O.A.: Towards seismic wave modeling on heterogeneous many-core architectures using task-based runtime system. In: SBAC-PAD. IEEE Computer Society (2015)Google Scholar
  14. 14.
    Mucci, P.J., Browne, S., Deane, C., Ho, G.: PAPI: a portable interface to hardware performance counters. In: Proceedings of the Department of Defense HPCMP Users Group Conference, pp. 7–10 (1999)Google Scholar
  15. 15.
    Rodrigues, A.F., Hemmert, K.S., Barrett, B.W., Kersey, C., Oldfield, R., Weston, M., Risen, R., Cook, J., Rosenfeld, P., CooperBalls, E., et al.: The structural simulation toolkit. ACM SIGMETRICS Perform. Eval. Rev. 38(4), 37–42 (2011)CrossRefGoogle Scholar
  16. 16.
    Velho, P., Schnorr, L.M., Casanova, H., Legrand, A.: On the validity of flow-level TCP network models for grid and cloud simulations. ACM Trans. Model. Comput. Simul. 23(4), 23:1–23:26 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Zheng, G., Kakulapati, G., Kale, L.: Bigsim: a parallel simulator for performance prediction of extremely large parallel machines. In: Parallel and Distributed Processing Symposium, Proceedings, 18th International, p. 78, April 2004Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Rafael Keller Tesser
    • 1
    Email author
  • Lucas Mello Schnorr
    • 1
  • Arnaud Legrand
    • 2
  • Fabrice Dupros
    • 3
  • Philippe Olivier Alexandre Navaux
    • 1
  1. 1.Informatics InstituteFederal University of Rio Grande do SulPorto AlegreBrazil
  2. 2.Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIGGrenobleFrance
  3. 3.BRGMOrléansFrance

Personalised recommendations