Modeling the propagation of elastic waves using spectral elements on a cluster of 192 GPUs

  • Dimitri Komatitsch
  • Dominik Göddeke
  • Gordon Erlebacher
  • David Michéa
Special Issue Paper


We implement a high-order finite-element application, which performs the numerical simulation of seismic wave propagation resulting for instance from earthquakes at the scale of a continent or from active seismic acquisition experiments in the oil industry, on a large GPU-enhanced cluster. Mesh coloring enables an efficient accumulation of degrees of freedom in the assembly process over an unstructured mesh. We use non-blocking MPI and show that computations and communications over the network and between the CPUs and the GPUs are almost fully overlapped. The GPU solver scales excellently up to 192 GPUs and achieves significant speedup over a carefully tuned equivalent CPU code.


GPU computing Finite elements Spectral elements Seismic modeling CUDA MPI 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abdelkhalek R, Calandra H, Coulaud O, Roman J, Latu G (2009) Fast seismic modeling and reverse time migration on a GPU cluster. In: Smari WW, McIntire JP (eds) High performance computing & simulation 2009, pp 36–44.
  2. 2.
    Carrington L, Komatitsch D, Laurenzano M, Tikir M, Michéa D, Le Goff N, Snavely A, Tromp J (2008) High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE on 62k processors. In: SC ’08: proceedings of the 2008 ACM/IEEE conference on supercomputing, article #60, Gordon Bell Prize finalist article Google Scholar
  3. 3.
    Chaljub E, Komatitsch D, Vilotte JP, Capdeville Y, Valette B, Festa G (2007) Spectral element analysis in seismology. In: Wu RS, Maupin V (eds) Advances in wave propagation in heterogeneous media. Advances in geophysics, vol 48. Elsevier/Academic Press, Amsterdam/San Diego, pp 365–419 CrossRefGoogle Scholar
  4. 4.
    Che S, Boyer M, Meng J, Tarjan D, Sheaffer JW, Skadron K (2008) A performance study of general-purpose applications on graphics processors using CUDA. J Parall Distrib Comput 68(10):1370–1380. doi: 10.1016/j.jpdc.2008.05.014 CrossRefGoogle Scholar
  5. 5.
    Corporation NVIDIA (2009) NVIDIA CUDA programming guide version 2.3. Santa Clara, California, USA, URL
  6. 6.
    De Basabe JD, Sen MK (2007) Grid dispersion and stability criteria of some common finite-element methods for acoustic and elastic wave equations. Geophysics 72(6):T81–T95. doi: 10.1190/1.2785046 CrossRefGoogle Scholar
  7. 7.
    Dziewoński AM, Anderson DL (1981) Preliminary reference Earth model. Phys Earth Planet Inter 25(4):297–356. doi: 10.1016/0031-9201(81)90046-7 CrossRefGoogle Scholar
  8. 8.
    Fan Z, Qiu F, Kaufman AE, Yoakum-Stover S (2004) GPU cluster for high performance computing. In: SC ’04: proceedings of the 2004 ACM/IEEE conference on supercomputing, p 47. doi: 10.1109/SC.2004.26
  9. 9.
    Fatahalian K, Houston M (2008) A closer look at GPUs. Commun ACM 51(10):50–57. doi: 10.1145/1400181.1400197 CrossRefGoogle Scholar
  10. 10.
    Garland M, Grand SL, Nickolls J, Anderson JA, Hardwick J, Morton S, Phillips EH, Zhang Y, Volkov V (2008) Parallel computing experiences with CUDA. IEEE Micro 28(4):13–27. doi: 10.1109/MM.2008.57 CrossRefGoogle Scholar
  11. 11.
    Göddeke D, Strzodka R, Mohd-Yusof J, McCormick P, Buijssen SHM, Grajewski M, Turek S (2007) Exploring weak scalability for FEM calculations on a GPU-enhanced cluster. Parall Comput 33(10–11):685–699. doi: 10.1016/j.parco.2007.09.002 Google Scholar
  12. 12.
    Hughes TJR (1987) The finite element method, linear static and dynamic finite element analysis. Prentice Hall International, Englewood Cliffs zbMATHGoogle Scholar
  13. 13.
    Kindratenko VV, Enos JJ, Shi G, Showerman MT, Arnold GW, Stone JE, Phillips JC, Hwu W (2009) GPU clusters for high-performance computing. In: Proceedings on the IEEE cluster’2009 workshop on parallel programming on accelerator clusters (PPAC’09), pp 1–8 Google Scholar
  14. 14.
    Komatitsch D, Coutel F, Mora P (1996) Tensorial formulation of the wave equation for modelling curved interfaces. Geophys J Int 127(1):156–168. doi: 10.1111/j.1365-246X.1996.tb01541.x CrossRefGoogle Scholar
  15. 15.
    Komatitsch D, Tsuboi S, Ji C, Tromp J (2003) A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator. In: SC ’03: proceedings of the 2003 ACM/IEEE conference on supercomputing, pp 4–11, Gordon Bell Prize winner article. doi: 10.1109/SC.2003.10023
  16. 16.
    Komatitsch D, Labarta J, Michéa D (2008) A simulation of seismic wave propagation at high resolution in the inner core of the Earth on 2166 processors of MareNostrum. In: High performance computing for computational science—VECPAR 2008. Lecture notes in computer science, vol 5336. Springer, Berlin, pp 364–377. doi: 10.1007/978-3-540-92859-1_33 CrossRefGoogle Scholar
  17. 17.
    Komatitsch D, Michéa D, Erlebacher G (2009) Porting a high-order finite-element earthquake modeling application to NVIDIA graphics cards using CUDA. J Parall Distrib Comput 69(5):451–460. doi: 10.1016/j.jpdc.2009.01.006 CrossRefGoogle Scholar
  18. 18.
    Liu Q, Polet J, Komatitsch D, Tromp J (2004) Spectral-element moment tensor inversions for earthquakes in Southern California. Bull Seismol Soc Am 94(5):1748–1761. doi: 10.1785/012004038 CrossRefGoogle Scholar
  19. 19.
    Michéa D, Komatitsch D (2010) Accelerating a 3D finite-difference wave propagation code using graphics cards. Geophys J Int (in press) Google Scholar
  20. 20.
    Micikevicius P (2009) 3D finite-difference computation on GPUs using CUDA. In: GPGPU-2: proceedings of the 2nd workshop on general purpose processing on graphics processing units, pp 79–84. doi: 10.1145/1513895.1513905
  21. 21.
    Nissen-Meyer T, Fournier A, Dahlen FA (2008) A 2-D spectral-element method for computing spherical-earth seismograms—II. Waves in solid-fluid media. Geophys J Int 174(3):873–888. doi: 10.1111/j.1365-246X.2008.03813.x CrossRefGoogle Scholar
  22. 22.
    Owens JD, Houston M, Luebke DP, Green S, Stone JE, Phillips JC (2008) GPU computing. Proc IEEE 96(5):879–899. doi: 10.1109/JPROC.2008.917757 CrossRefGoogle Scholar
  23. 23.
    Phillips JC, Stone JE, Schulten K (2008) Adapting a message-driven parallel application to GPU-accelerated clusters. In: SC ’08: proceedings of the 2008 ACM/IEEE conference on supercomputing, article no 8. doi: 10.1145/1413370.1413379
  24. 24.
    Phillips EH, Zhang Y, Davis RL, Owens JD (2009) Rapid aerodynamic performance prediction on a cluster of graphics processing units. In: Proceedings of the 47th AIAA aerospace sciences meeting, aIAA 2009-565 Google Scholar
  25. 25.
    Seriani G, Priolo E (1994) A spectral element method for acoustic wave simulation in heterogeneous media. Finite Elem Anal Des 16(3–4):337–348. doi: 10.1016/0168-874X(94)90076-0 zbMATHCrossRefMathSciNetGoogle Scholar
  26. 26.
    Tromp J, Komatitsch D, Liu Q (2008) Spectral-element and adjoint methods in seismology. Commun Comput Phys 3(1):1–32 zbMATHGoogle Scholar
  27. 27.
    van Wijk K, Komatitsch D, Scales JA, Tromp J (2004) Analysis of strong scattering at the micro-scale. J Acoust Soc Am 115(3):1006–1011. doi: 10.1121/1.1647480 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2010

Authors and Affiliations

  • Dimitri Komatitsch
    • 1
  • Dominik Göddeke
    • 2
  • Gordon Erlebacher
    • 3
  • David Michéa
    • 4
  1. 1.CNRS & INRIA Magique-3D, Laboratoire de Modélisation et d’Imagerie en Géosciences UMR 5212Université de PauPauFrance
  2. 2.Institut für Angewandte MathematikTU DortmundGermany
  3. 3.Department of Scientific ComputingFlorida State UniversityTallahasseeUSA
  4. 4.Bureau de Recherches Géologiques et MinièresOrléansFrance

Personalised recommendations