EDGE: Extreme Scale Fused Seismic Simulations with the Discontinuous Galerkin Method

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10266)


This article introduces EDGE, a solver package for fused seismic simulations. Fused seismic simulations are a novel technique addressing one of the grand challenges of computational seismology: large ensemble runs of geometrically similar forward simulations. Application fields include, but are not limited to: uncertainty quantification in the context of seismic hazard analysis or the accurate derivation of velocity models through tomographic inversion. For efficient and accurate handling of complex model geometries (topography, fault geometries, material heterogeneities), EDGE utilizes the Discontinuous Galerkin (DG) method for spatial and Arbitrary high order DERivatives (ADER) for time discretization, implemented for unstructured tetrahedral meshes. EDGE’s ADER-DG scheme requires sparse and dense matrix-matrix multiplications at the kernel level. By choosing a sufficient memory layout and relying on runtime code generation and specialization, both, sparse and dense operations, can be efficiently vectorized on wide-SIMD machines. We present a convergence study of single and fused seismic simulations, code verification in an established benchmark, as well as a detailed performance assessment for different discretization orders. As target architecture we select the recently released Intel Xeon Phi processor, which powers the Theta and Cori-II supercomputers. For a single sixth order seismic forward simulation we achieved 10.4 PFLOPS of hardware performance and 5.0 PFLOPS for fused simulations in fourth order, both occupying 9,000 nodes of Cori-II. From a throughput perspective, fused seismic simulations can outperform a single forward simulation by 1.8\(\times \) to 4.6\(\times \), depending on the chosen order of the method.


Discontinuous Galerkin Finite Difference Method Spectral Element Method Seismic Wave Propagation Forward Simulation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



Only the great support of experts at NERSC and ALCF made our extreme-scale results possible. In particular, we thank J. Deslippe, S. Dosanjh, R. Gerber, and K. Kumaran. This work was supported by the Southern California Earthquake Center (SCEC) through contribution #16247. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. This research used resources of the Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC02-06CH11357. This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1053575.

Optimization Notice: Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to Intel, Xeon, and Intel Xeon Phi are trademarks of Intel Corporation in the U.S. and/or other countries.


  1. 1.
    Reinders, J., Jeffers, J., Sodani, A. (eds.) Intel Xeon Phi Processor High Performance Programming Knights Landing Edition (2016). Ch. 4 and Ch. 6Google Scholar
  2. 2.
    Aoi, S., Fujiwara, H.: 3D finite-difference method using discontinuous grids. Bull. Seismol. Soc. Am. 89(4), 918–930 (1999)Google Scholar
  3. 3.
    Appelö, D., Petersson, N.A.: A stable finite difference method for the elastic wave equation on complex geometries with free surfaces. Commun. Comput. Phys. 5(1), 84–107 (2009)MathSciNetGoogle Scholar
  4. 4.
    Bastian, P., et al.: Hardware-based efficiency advances in the EXA-DUNE project. In: Bungartz, H.-J., Neumann, P., Nagel, W.E. (eds.) Software for Exascale Computing - SPPEXA 2013-2015. LNCSE, vol. 113, pp. 3–23. Springer, Cham (2016). doi: 10.1007/978-3-319-40528-5_1 CrossRefGoogle Scholar
  5. 5.
    Benjemaa, M., Glinsky-Olivier, N., Cruz-Atienza, V.M., Virieux, J.: 3D dynamic rupture simulations by a finite volume method. Geophys. J. Int. 178, 541–560 (2009)CrossRefGoogle Scholar
  6. 6.
    Breuer, A., Heinecke, A., Rettenberger, S., Bader, M., Gabriel, A.-A., Pelties, C.: Sustained petascale performance of seismic simulations with SeisSol on SuperMUC. In: Kunkel, J.M., Ludwig, T., Meuer, H.W. (eds.) ISC 2014. LNCS, vol. 8488, pp. 1–18. Springer, Cham (2014). doi: 10.1007/978-3-319-07518-1_1 Google Scholar
  7. 7.
    Chaljub, E., Komatitsch, D., Vilotte, J.-P., Capdeville, Y., Valette, B., Festa, G.: Spectral-element analysis in seismology. Adv. Geophys. 48, 365–419 (2007). Advances in Wave Propagation in Heterogenous Earth, CrossRefGoogle Scholar
  8. 8.
    Chaljub, E., Maufroy, E., Moczo, P., Kristek, J., Hollender, F., Bard, P.-Y., Priolo, E., Klin, P., de Martin, F., Zhang, Z., Zhang, W., Chen, X.: 3-D numerical simulations of earthquake ground motion in sedimentary basins: testing accuracy through stringent models. Geophys. J. Int. 201(1), 90–111 (2015)CrossRefGoogle Scholar
  9. 9.
    Dumbser, M., Käser, M.: An arbitrary high-order discontinuous galerkin method for elastic waves on unstructured meshes - II. The three-dimensional isotropic case. Geophys. J. Int. 167(1), 319–336 (2006)CrossRefGoogle Scholar
  10. 10.
    Duru, K., Dunham, E.M.: Dynamic earthquake rupture simulations on nonplanar faults embedded in 3D geometrically complex, heterogeneous elastic solids. J. Comput. Phys. 305, 185–207 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Etienne, V., Chaljub, E., Virieux, J.: An hp-adaptive discontinuous Galerkin finite-element method for 3-D elastic wave modelling. Geophys. J. Int. 183(2), 941–962 (2010)CrossRefGoogle Scholar
  12. 12.
    Geuzaine, C., Remacle, J.-F.: Gmsh: A 3-D finite element mesh generator with built-in pre-and post-processing facilities. Int. J. Numer. Methods Eng. 79(11), 1309–1331 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Heinecke, A., Breuer, A., Bader, M., Dubey, P.: High order seismic simulations on the intel Xeon Phi processor (Knights Landing). In: Kunkel, J.M., Balaji, P., Dongarra, J. (eds.) ISC High Performance 2016. LNCS, vol. 9697, pp. 343–362. Springer, Cham (2016). doi: 10.1007/978-3-319-41321-1_18 Google Scholar
  14. 14.
    Heinecke, A., Breuer, A., Rettenberger, S., Bader, M., Gabriel, A.-A., Pelties, C., Bode, A., Barth, W., Liao, X.-K., Vaidyanathan, K., et al.: Petascale high order dynamic rupture earthquake simulations on heterogeneous supercomputers. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (2014)Google Scholar
  15. 15.
    Heinecke, A., Henry, G., Hutchinson, M., Pabst, H.: LIBXSMM: accelerating small matrix multiplications by runtime code generation. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (2016)Google Scholar
  16. 16.
    Ichimura, T., Fujita, K., Quinay, P., Maddegedara, L., Hori, M., Tanaka, S., Shizawa, Y., Kobayashi, H., Minami, K.: Implicit nonlinear wave simulation with 1.08T DOF and 0.270T unstructured finite elements to enhance comprehensive earthquake simulation (2015)Google Scholar
  17. 17.
    Ichimura, T., Hori, M., Bielak, J.: A hybrid multiresolution meshing technique for finite element three-dimensional earthquake ground motion modelling in basins including topography. Geophys. J. Int. 177(3), 1221–1232 (2009)CrossRefGoogle Scholar
  18. 18.
    George, K., Vipin, K.: MeTis: Unstructured Graph Partitioning and Sparse Matrix Ordering System, version 4.0 (2009)Google Scholar
  19. 19.
    Kang, T.-S., Baag, C.-E.: An efficient finite-difference method for simulating 3D seismic response of localized basin structures. Bull. Seismol. Soc. Am. 94(5), 1690–1705 (2004)CrossRefGoogle Scholar
  20. 20.
    Käser, M., Dumbser, M., Puente, J., Igel, H.: An arbitrary high-order discontinuous galerkin method for elastic waves on unstructured meshes - III. viscoelastic attenuation. Geophys. J. Int. 168(1), 224–242 (2007)CrossRefGoogle Scholar
  21. 21.
    Komatitsch, D., Tromp, J.: Spectral-element simulations of global seismic wave propagation-II. Three-dimensional models, oceans, rotation and self-gravitation. Geophys. J. Int. 150(1), 303–318 (2002)CrossRefGoogle Scholar
  22. 22.
    Moczo, P., Ampuero, J.P., Kristek, J., Day, S.M., Kristekova, M., Pazak, P., Galis, M., Igel, H.: Comparison of numerical methods for seismic wave propagation and source dynamics - the SPICE code validation. In: Third International Symposium on the Effects of Surface Geology on Seismic Motion. Actes des journées scientifiques du LCPC. Laboratoire central des ponts et chaussées, Paris, France, pp. 1–10 (2006). ISBN 9782720824654Google Scholar
  23. 23.
    Kristeková, M., Kristek, J., Moczo, P.: Time-frequency misfit and goodness-of-fit criteria for quantitative comparison of time signals. Geophys. J. Int. 178(2), 813–825 (2009)CrossRefGoogle Scholar
  24. 24.
    Kristeková, M., Kristek, J., Moczo, P., Day, S.M.: Misfit criteria for quantitative comparison of seismograms. Bull. Seismol. Soc. Am. 96(5), 1836–1850 (2006)CrossRefGoogle Scholar
  25. 25.
    Liu, J., Chandrasekaran, B., Wu, J., Jiang, W., Kini, S., Yu, W., Buntinas, D., Wyckoff, P., Panda, D.K.: Performance comparison of MPI implementations over InfiniBand, Myrinet and Quadrics. In: Proceedings of the 2003 ACM/IEEE Conference on Supercomputing (2003)Google Scholar
  26. 26.
    Ma, S., Liu, P.: Modeling of the perfectly matched layer absorbing boundaries and intrinsic attenuation in explicit finite-element methods. Bull. Seismol. Soc. Am. 96(5), 1779–1794 (2006)CrossRefGoogle Scholar
  27. 27.
    Malas, T., Kurth, T., Deslippe, J.: Optimization of the sparse matrix-vector products of an IDR Krylov iterative solver in EMGeo for the Intel KNL manycore processor. In: Taufer, M., Mohr, B., Kunkel, J.M. (eds.) ISC High Performance 2016. LNCS, vol. 9945, pp. 378–389. Springer, Cham (2016). doi: 10.1007/978-3-319-46079-6_27 CrossRefGoogle Scholar
  28. 28.
    Moczo, P., Kristek, J., Vavryčuk, V.: 3D heterogeneous staggered-grid finite-difference modeling of seismic motion with volume harmonic and arithmetic averaging of elastic moduli and densities. Bull. Seismol. Soc. Am. 92(8), 3042–3066 (2002)CrossRefGoogle Scholar
  29. 29.
    Moczo, P., Robertsson, J.O.A., Eisner, L.: The finite-difference time-domain method for modeling of seismic wave propagation. Adv. Geophys. 48, 421–516 (2007). Advances in Wave Propagation in Heterogenous Earth, CrossRefGoogle Scholar
  30. 30.
    Modave, A., St-Cyr, A., Warburton, T.: GPU performance analysis of a nodal discontinuous Galerkin method for acoustic and elastic models. Comput. Geosci. 91, 64–76 (2016)CrossRefGoogle Scholar
  31. 31.
    Peter, D., Komatitsch, D., Luo, Y., Martin, R., Goff, N., Casarotti, E., Loher, P., Magnoni, F., Liu, Q., Blitz, C., Nissen-Meyer, T., Basini, P., Tromp, J.: Forward and adjoint simulations of seismic wave propagation on fully unstructured hexahedral meshes. Geophys. J. Int. 186(2), 721–739 (2011)CrossRefGoogle Scholar
  32. 32.
    Pitarka, A.: 3D elastic finite-difference modeling of seismic motion using staggered grids with nonuniform spacing. Bull. Seismol. Soc. Am. 89(1), 54–68 (1999)MathSciNetGoogle Scholar
  33. 33.
    Shepherd, J.F., Johnson, C.R.: Hexahedral mesh generation constraints. Eng. Comput. 24(3), 195–213 (2008)CrossRefGoogle Scholar
  34. 34.
    Symes, W.W., Vdovina, T.: Interface error analysis for numerical wave propagation. Comput. Geosci. 13(3), 363–371 (2009)CrossRefzbMATHGoogle Scholar
  35. 35.
    Taborda, R., Bielak, J.: Large-Scale earthquake simulation: computational seismology and complex engineering systems. Comput. Sci. Eng. 13(4), 14–27 (2011)CrossRefGoogle Scholar
  36. 36.
    Cruz-Atienza, V.M., Virieux, J., Aochi, H.: 3D finite-difference dynamic-rupture modeling along nonplanar faults. Geophysics 72(5), 123–137 (2007)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.University of California, San DiegoLa JollaUSA
  2. 2.Intel CorporationSanta ClaraUSA

Personalised recommendations