1 Introduction

To successfully model earthquakes and perform seismic simulations, accurate models for the source dynamics and the propagation of seismic waves are needed. For seismic wave propagation, acoustic, isotropic and anisotropic elastic, attenuating and poroelastic materials are the most important rheologies [21]. Seismic anisotropy affects speed and scattering of seismic waves depending on the direction of propagation and can be found on all scales in the solid Earth. Anisotropy can stem from finely layered or cracked materials, the internal crystal structure of minerals or the alignment of ice crystals in glaciers. Anisotropic material behavior is observed in fault zones [16, 17] and accounted for on global scale in refinements of the Preliminary Reference Earth Model [8]. Effective anisotropy on the scales of seismic wavelengths can be modeled by assuming homogeneous materials with directional dependent properties.

Anisotropy is one of the key seismic properties next to velocity and intrinsic attenuation. Locally, at the scale of earthquake fault zones, large variations in anisotropy reflect the strong material contrasts, extreme strains, and high dissipation of damaged rock. At the global scale, variations in anisotropy provide snapshots of our planet’s interior that inform our understanding of plate tectonics. Imaging of anisotropy is also crucial in industry contexts such as exploration or geothermal reservoir development and maintenance. All these applications require efficient forward solvers, ideally accounting for physical anisotropy together with the geometrical complexity of the geological subsurface. High-order accuracy is crucial to resolve small variations of anisotropy, which are often within only a few percent variation of isotropic material, depending on tectonic context.

Anisotropic material behavior has been successfully included in Finite Difference schemes [11, 22], pseudo-spectral methods [5], Spectral Element codes [13] and Discontinuous Galerkin (DG) schemes for the velocity–stress formulation [20] and for the velocity–strain formulation [26]. Only few open-source codes exist which are able to simulate seismic wave propagation in anisotropic materials and which are also tailored to run efficiently on supercomputers. The DG ansatz on unstructured grids allows us to include full physical anisotropy as we do not encounter geometrical restrictions. The DG software SeisSol has undergone end-to-end performance optimization over the last years [3, 10, 25]. However, anisotropic effects have been neglected thus far.

In this paper, we present a novel implementation of fully anisotropic wave propagation that exploits SeisSol’s high-performance implementation of element-local matrix operations and supports local time stepping. We first lay out the physical model and state the governing equations (Sect. 2). In Sect. 3 these equations are discretized using the DG method combined with arbitrary high-order derivative time stepping (ADER-DG). Our main numerics contribution is to introduce a two-sided numerical flux of the Godunov type in conjunction with a free-surface boundary condition based on solving an inverse Riemann problem. We highlight details of how we implemented theses features into the existing code base, and extended it to make use of local time stepping. Here, the key novelty is a general approach to integrate a numerical eigenvalue solver in SeisSol that replaces analytically derived formulas in the respective precomputation steps. In Sect. 5 we verify our implementation against various analytical solutions and community benchmark problems. We also present an updated reference solution for the AHSP1 benchmark [18], since our implementation revealed physical inconsistencies in the previous community reference solution. To demonstrate the capability of our code to solve real-world geophysical problems we model anisotropically scattering seismic waves radiating from a point source under the strong topography contrasts of Mount Zugspitze in the Bavarian Alps.

2 Physical Model

Linear elastic materials are characterized by a stress–strain relation in the form

(1)

with stress and strain tensors denoted by \(\sigma , \epsilon \in \mathbb {R}^{3\times 3}\). Symmetry considerations reduce the 81 parameters \(c_{ijkl}\) to only 21 independent values [2]. Employing Voigt notation we can write the relation in a matrix–vector manner:

(2)

This constitutive relation can be combined with the equations of motion of continuum mechanics which we write in the velocity–strain formulation, where the vector \(Q = \begin{pmatrix} \sigma _{11}, &{} \sigma _{22}, &{} \sigma _{33}, &{} \sigma _{13}, &{} \sigma _{23}, &{} \sigma _{13}, &{} u_1, &{} u_2, &{} u_3 \\ \end{pmatrix}^T\) defines the quantities of interest. The combined equation reads:

$$\begin{aligned} { \frac{\partial Q_p}{\partial t} } + A^1_{pq} { \frac{\partial Q_q}{\partial x_1} } + A^2_{pq} { \frac{\partial Q_q}{\partial x_2} } + A^3_{pq} { \frac{\partial Q_q}{\partial x_3} } = 0. \end{aligned}$$
(3)

The Jacobian matrices \(A^d, d=1,2,3,\) can be deduced from the stress–strain relation and have the form

(4)

We observe that the second index is constant for each column. To construct the matrices \(\mathcal {C}^2\) and \(\mathcal {C}^3\) we replace these by 6, 2, 4 and 5, 4, 3 respectively. The blocks \(\mathcal {R}^d\) are the same as for the isotropic case and are detailed in [20]. The material parameters can vary in space. For better readability the space dependence has been dropped in Eq. (3). Isotropic material behavior can be seen as a specialization of anisotropy, where \(c_{11} = c_{22} = c_{33} = \lambda + 2\mu \), \(c_{12} = c_{13} = c_{23} = \lambda \), \(c_{44} = c_{55} = c_{66} = \mu \) and all other parameters are zero.

3 Numerical Approximation

De la Puente et al. [20] presented the numerics of including anisotropic material effects into ADER-DG seismic wave propagation simulations. We here improve the numerical scheme by a two-sided Godunov flux and a free-surface boundary condition, as well as adaptions necessary for local time stepping with anisotropy. A two-sided flux is physically more accurate and allows for coupling between different rheologies. Local time stepping improves performance drastically.

3.1 Spatial Discretization

To solve Eq. (3), we follow a DG ansatz [7]. The underlying geometry is approximated by a mesh of tetrahedral elements \(\tau ^m\). For the discretization polynomial ansatz functions \(\varPhi _l\) are defined on a reference element \(\tau _\text {ref}\). On each element the numerical solution \(Q_p^m\) is expanded in terms of the basis functions:

$$\begin{aligned} Q_p^m(x, t) = \hat{Q}_{pl}^m(t) \varPsi ^m_l(x) = \hat{Q}_{pl}^m(t) \varPhi _l(\varXi ^m(x)). \end{aligned}$$
(5)

Here the function \(\varXi ^m: \tau ^m \rightarrow \tau _\text {ref}\) is an affine linear coordinate transformation. By \(\varPsi _l^m\) we denote the \(l^\text {th}\) basis function transformed to the \(m^\text {th}\) element.

On each element Eq. (3) is multiplied by a test function and integration by parts is applied leading to a semi-discrete formulation:

$$\begin{aligned} \int _{\tau ^m} \varPsi ^m_k { \frac{\partial \hat{Q}_{pl}^m}{\partial t} } \varPsi ^m_l dV + \int _{\partial \tau ^m} \varPsi ^m_k (n_d A^d_{pq} Q_q)^* dS - \int _{\tau ^m} { \frac{\partial \varPsi ^m_k}{\partial x_d} } A^d_{pq} \hat{Q}^m_{ql} \varPsi ^m_l dV = 0. \end{aligned}$$
(6)

The Jacobians \(A^d_{pq}\) are element-wise constant. Also \(\hat{Q}\) and its time derivative are constant on each cell. This allows us to pull these quantities out of the integrals. Applying a change of variables to the reference element the integrals can be precomputed. Together with an appropriate flux formulation this leads to a quadrature-free numerical scheme.

3.2 Flux and Boundary Conditions

In DG schemes continuity across element boundaries is only enforced in a weak sense, via the flux term \((n_d A^d_{pq} Q_q)^*\) in Eq. (6). Hence, a proper numerical flux, which also takes the underlying physics into account, is essential. De la Puente et al. [20] demonstrated anisotropy with one-sided Rusanov flux and discussed an extension to Godunov fluxes for ADER-DG. Two-sided fluxes capture the correct jump conditions of the Rankine-Hugoniot condition on both sides of the inter-element boundaries. They have been introduced to SeisSol in [23] for acoustic and (visco)elastic materials. In the following we give an overview over the most important aspects of using two-sided flux formulations and on generalizing the isotropic flux to a two-sided formulation for the anisotropic case.

In Eq. (6), the surface integral over \(\partial \tau ^m\) can be dispatched into four integrals over the four triangular faces of each element. We evaluate the flux for each face individually, so we need to transform the quantities Q as well as the stress–strain relation into face-aligned coordinate systems. For anisotropic materials the stress–strain relation is represented by the matrix \(\mathcal {H}\), see Eq. (2). We can express the constitutive behavior in the face-aligned coordinate system via the matrix \(\tilde{\mathcal {H}} = N \cdot \mathcal {H} \cdot N^T\) (cf. [4]), where N is the so-called Bond matrix (cf. [20]). We define the matrix \(\tilde{A}\) to have the same structure as the matrix \(A^1\) but with entries \(\tilde{c}_{ij}\) from the matrix \(\tilde{\mathcal {H}}\). At each face we have the Jacobians and approximations on the inside \(\tilde{A}^-, Q^-\) and on the outside \(\tilde{A}^+, Q^+\).

The Godunov flux approximates the solution at the element boundary by solving a Riemann problem across the element interfaces. First the equations are transformed to a face-aligned coordinate system. The Rankine-Hugoniot condition states that discontinuities travel with wave speeds given via the eigenvalues of the Jacobian. The differences between the quantities are the corresponding eigenvectors. A detailed derivation can be found in [23, 26].

We observe that the eigenvectors of the Jacobian \(\tilde{A}\) have the form

(7)

The eigenvectors and corresponding eigenvalues resemble three incoming and outgoing waves. To take different material values on the inside and outside into account an eigendecomposition of both Jacobians \(\tilde{A}^-\) and \(\tilde{A}^+\) is performed and the matrix R is constructed taking the first three columns from \(R^-\) and the last three columns from \(R^+\). With indicator matrices \(I^- = \mathop {\mathrm {diag}}(1, 1, 1, 0, 0, 0, 0, 0, 0)\) and \(I^+ = \mathop {\mathrm {diag}}(0, 0, 0, 0, 0, 0, 1, 1, 1)\), we can then compute the flux as:

$$\begin{aligned} F = \frac{1}{2} \underbrace{T \tilde{A}^- (R I^+ R^{-1}) T^{-1}}_{G^+} Q^- + \frac{1}{2} \underbrace{T \tilde{A}^- (R I^- R^{-1}) T^{-1}}_{G^-}Q^+. \end{aligned}$$
(8)

Here T is a matrix that rotates Q from the global coordinate system to the face-aligned coordinate system. We take into account that the first six components of \(Q \in \mathbb {R}^9\) represent a symmetric tensor and the last three components represent a vector. Both parts can be rotated independently, so T combines the rotation of tensorial and vectorial quantities.

Analogous to inter-element boundaries, we also impose boundary conditions via a specialized flux. For a free surface boundary we want to impose \(s = \sigma n = 0\). In Eq. (8) the term \(R I^+ R^{-1} T^{-1}Q^-\) is identified with the state at the inside of the inter-element boundary. We can use this fact to construct a flux which will yield the free surface boundary. To do so we set the traction s to zero and compute the velocity u consistently:

(9)

Superscripts b denote values at the boundary. The values \(s^-\) and \(u^-\) are the traction and velocity in the face-aligned coordinate system. The matrices \(R_{11}\) and \(R_{21}\) slice out the first three columns of R and the rows corresponding to the traction respectively the velocity components. The flux is obtained as \(F = T \tilde{A}^- Q^b\). Note that we did not specify the non-traction components of \(\sigma \). As these lie in the null space of the Jacobian \(\tilde{A}\) the flux is not altered by their value.

3.3 Time Discretization

To integrate Eq. (6) in time SeisSol employs the ADER method [6]. ADER time stepping expands each element solution locally in time as a Taylor series up to a certain order. The time derivatives are replaced by the discretized spatial derivatives following the Cauchy–Kowalewski theorem. To update one element we therefore only need the values of the element itself and its four neighbors, which fosters efficient parallelization.

ADER time stepping inherently supports local time stepping [7]: Smaller elements or elements with high wave speeds will be updated more often than large elements or elements with low wave speeds. Local time stepping is thus crucial for performant applications that use strong adaptive mesh refinement or where meshes suffer from badly shaped elements. Setups with a heterogeneous material can also benefit substantially from local time stepping.

Each element has to satisfy the stability criterion \(\varDelta t^m < \frac{1}{2N+1} \frac{l^m}{v^m}\) for the time step size \(\varDelta t^m\), where \(l^m\) and \(v^m\) denote the in-sphere diameter and maximum wave speed of element \(\tau ^m\), N is the order of the method.

In anisotropic materials the wave speeds depend on the direction of propagation. This has not been considered in previous work (e.g. [7]). For a fixed direction d we define the matrix \(M(d)_{ij} = d_k c_{iklj} d_l\). We calculate the wave speeds in direction d from the eigenvalues \(\lambda _i\) of the matrix M(d) via \(v_i = \sqrt{\lambda _i / \rho }\) resulting in a primary and two secondary waves (cf. [4]). The element-wise maximum wave speed is the maximum of these speeds over all directions d.

4 Implementation

SeisSol’s ADER-DG discretization is implemented via element-local matrix chain multiplications, which allows for high performance on modern CPUs [10, 25]. All required matrices are precomputed in the initialization phase and optimized kernels are generated for the matrix chain operations [24]. In the following we present the most important choices we made for our implementation of anisotropy. Concerning the matrices, the compute kernels of the isotropic case can be reused, just the assembly of the Jacobians and flux matrices differs.

4.1 Storage Requirements

For each element we store the material values. In comparison to isotropic materials 22 instead of 3 values have to be stored. This overhead is negligible compared to the storage required for the degrees of freedoms. For example, a discretization of order 6 requires 504 degrees of freedom per element.

Concerning the storage of the precomputed matrices, only the Jacobians A and the flux matrices \(G^+\) and \(G^-\) change between the isotropic and the anisotropic case. Based on the sparsity pattern of a matrix, the biggest rectangular non-zero block of the matrix is stored. We store the matrices A, \(G^+\) and \(G^-\) as full \(9\times 9\) matrices for the isotropic as well as for the anisotropic case, thus no overhead is produced. The underlying data structures are not changed.

4.2 Calculation of the Flux Term

Unlike as with isotropic materials the eigenstructure of the Jacobians given in Eq. (7) is hard to express analytically. In the applied scheme the eigendecomposition has to be calculated once for every element. We use the open source software package Eigen3 [9] to obtain the eigenvectors numerically. We chose this approach for three reasons:

  1. (i)

    Even if an analytic expression for the eigenvalues is available it is lengthy and hence the implementation is error-prone.

  2. (ii)

    From a software engineering point of view the use of the numerical eigenvalue solver replaces a lot of code that was previously needed for each material model individually. We unified the formulation of the Riemann solver for all material models (isotropic, anisotropic and viscoelastic). We use templating to distinguish the assembly of the Jacobians for each model. From then on we can use the same code to precalculate \(G^+\) and \(G^-\). We expect that these software engineering choices make it easy to include additional material models into SeisSol in the future. Also coupling between different material models within the same simulation can be obtained with little overhead.

  3. (iii)

    The question of accuracy and stability of the numerical solver may arise. But stability of an analytically derived formula is also not guaranteed. Round-off errors and cancellation could drastically influence the accuracy of the derived eigenvectors. With our choice for using a stable numerical solver instead, we circumvent this problem.

4.3 Maximal Wave Speeds for Local Time Stepping

Local time stepping is implemented with a clustered scheme to meet the requirements of modern supercomputers [3]. To cluster the elements the required time step for each element has to be known in advance. To obtain the maximum wave speed for one element, we would have to find the maximum wave speed over all directions. This boils down to solving an optimization problem which involves the calculation of eigenvalues of arbitrary matrices. Solving this optimization problem analytically results in lengthy calculations. In practice, the time step is relaxed by a security factor to meet the CFL condition, so the maximum wave speed does not have to be computed exactly. We sample the wave speeds for several directions d and take their maximum as the maximum wave speed \(v^m\).

5 Validation and Performance

5.1 Convergence Analysis

Planar wave analytic descriptions are widely used in wave propagation problems. Here we present a numerical convergence study to analyze the correct implementation to confirm its expected convergence properties. To this end, we verify our implementation solving the 3-D, anisotropic, seismic wave equations in the form of periodic, sinusoidal waves in a unit-cube as explained in [19]. The computational domain is the unit cube \([-1, 1]^3\) with periodic boundary conditions. The ansatz for our plane-wave solution is

$$\begin{aligned} Q_p(x, t) = \text {Re} \left( Q_p^0 e^{ i \left( \omega t - k \cdot x \right) } \right) \end{aligned}$$
(10)

where \(\omega \) denotes the frequency and k is the vector of wave numbers. When we combine this with Eq. (3) we see that the initial condition \(Q^0\) has to be a solution of the eigenvalue problem

$$\begin{aligned} A_{pq}^d k_d Q_q^0 = \omega Q_p^0. \end{aligned}$$
(11)

In the case of linear elasticity there is a zero eigenvalue with multiplicity 3. The other eigenvalues appear pairwise with different signs and correspond to the P wave and two S waves. For isotropic materials the two S waves coincide, whereas for anisotropic media a slow and a fast S wave can be distinguished.

For linear PDEs a linear combination of several solutions is a solution again. To take the directional dependence of anisotropic materials into account we superimpose three planar waves with wave number vectors \(k^1 = \begin{pmatrix} \pi ,&0,&0 \end{pmatrix}\), \(k^2 = \begin{pmatrix} 0,&\pi ,&0 \end{pmatrix}\) and \(k^3 = \begin{pmatrix} 0,&0,&\pi \end{pmatrix}\). For each direction a P wave traveling in the direction of \(k^l\) and an S wave traveling in the opposite direction has been chosen. When we denote the eigenvectors of the matrix \(A^dk_d^l\) with \(R^l\) and the corresponding eigenvalues with \(\omega ^l\), the analytic solution can be written as

$$\begin{aligned} Q_p(x, t) = \sum _{l=1}^3 \text {Re} \left( R_{p2}^l e^{i \left( \omega ^l_2 t - k^l \cdot x) \right) } + R_{p9}^l e^{ i\left( \omega ^l_9 t - k^l \cdot x \right) } \right) . \end{aligned}$$
(12)

The computational domain is discretized into cubes of edge length \(h = \frac{1}{2}, \frac{1}{4}, \frac{1}{8}, \frac{1}{16}\). Each cube is split up into five tetrahedrons. The material is given by density \(\rho = 1.00\,\frac{\mathrm{k}g}{\mathrm{m}^3}\) and the elastic tensor \(c_{ij} = 0\) except for

(13)

We compare the numerical solution to the analytic solution at time \(t=0.02\). Figure 1 shows the convergence behavior for the stress component \(\sigma _{11}\) in the \(L^2\)-norm. We clearly observe the expected convergence orders. The plots for the \(L^1\)- and \(L^\infty \)-norm are comparable. All other quantities also show the expected convergence rates.

Fig. 1.
figure 1

3D unit cube plane wave test case. Visualization of convergence behavior of the stress component \(\sigma _{11}\). The \(L^2\) error is shown versus the mesh spacing h.

Full size image

5.2 Isotropy via Anisotropy: Layer over Halfspace (LOH1)

The community benchmark LOH1 [18] is designed for isotropic elastic materials. We here use it to validate backwards compatibility, as isotropic elasticity is a special case of anisotropic elasticity. The setup consists of a layered half space with a free surface boundary on top. The top layer is \(1\,\mathrm{km}\) thick with material parameters \(\rho = 2600\,\frac{\mathrm{kg}}{\mathrm{m}^3}\), \(\mu = 10.4\,\mathrm{GPa} \) and \(\lambda = 20.8\,\mathrm{GPa} \). The half space below has material parameters \(\rho = 2700\,\frac{\mathrm{kg}}{\mathrm{m}^3} \), \(\mu = \lambda = 32.4\,\mathrm{GPa} \). The source is a double couple point source with the only non-zero moment \(M_{xy}\) located in \(2\,\mathrm{km}\) depth. The moment rate time history is given by the function \(M_{xy}(t) = M_0 \frac{t}{T^2} \exp \left( -\frac{t}{T^2}\right) \) with maximal moment \(M_0 = 10^{18}\,\mathrm{Nm}\) and onset time \(t = 0.1\,\mathrm{s}\).

We compare the velocities at the free surface to the given reference solutions. There are nine receivers located along three different lines through the origin. The domain is the cuboid \([-40000,40000] \times [-40000,40000] \times [0, 32500]\,\mathrm{m}^3\). On all other surfaces than the free surface (\(x_3 = 0\)) we impose absorbing boundary conditions. The mesh is refined around the source and coarsened away from it. The characteristic length is approximately \(300\,\mathrm{m}\) in the vicinity of the source and grows up to \(2000\,\mathrm{m}\) towards the boundary. In total the mesh consists of 2.57 million cells. The mesh is large enough that waves do not leave the domain during the computational time and we do not observe problems with artificial reflections. At the same time we keep the computational effort reasonable due to the coarsening towards the absorbing boundaries.

We ran the simulation with convergence order 6 up to a final time of \(5\,\mathrm{s}\) and compared our solutions to the reference solution using envelope and pulse misfit [14]. We recorded the velocities at all nine receivers. In Table 1 we present the results for the third, sixth and ninth receiver, which are farthest away from the source (the longer the wave is propagated, the less accurately it is typically resolved due to numerical dissipation and dispersion). With a maximal envelope misfit of \(1.32\%\) and a maximal phase misfit of \(0.20\%\) we are well within the highest level of accuracy defined in the LOH1 benchmark description. To investigate the computational overhead of incorporating anisotropic effects, we compare the execution times for the setup executed with the isotropic and the anisotropic implementation. We ran each simulation 5 times on 100 nodes of SuperMUC-NG (Intel Skylake Xeon Platinum 8174, 48 cores per node) and averaged the total wall time. The average runtime of the anisotropic implementation was \(185.0\,\mathrm{s}\), which is only a little slower than the isotropic implementation, which averaged at \(184.6\,\mathrm{s}\). The difference between the runtimes may be explained by the more complicated initialization phase for the anisotropic case. We also point out that the deviation in run times in between different runs was larger than the difference between the averages. The computations achieved an average of 990.3 GFlop/s in the isotropic case and 994.7 GFlop/s in the anisotropic case.

Table 1. Envelope and pulse misfits for the different velocity components
Full size table

5.3 Anisotropic Homogeneous Space (AHSP, SISMOWINE)

The SISMOWINE test suite [18] proposes the following test case for seismic wave propagation in anisotropic materials: The geometry is a homogeneous full space. The homogeneous material has a density of \(\rho = 2700\,\frac{\mathrm{kg}}{\mathrm{m}^3}\). The elastic response is characterized by the elastic tensor \(c_{ij} = 0\) expect for

(14)

The source is identical to the LOH1 benchmark. We again refine the mesh around the source with a characteristic edge length of about \(300\,\mathrm{m}\) and coarsen away from the source. This results in a total of 3.98 million cells. To simulate a full space, absorbing boundary conditions are imposed on all six surfaces. Just as for the LOH1 test case we do not encounter artificial reflections from the boundaries.

Figure 2 shows an exemplary comparison between our new implementation and the reference solution of SISMOWINE. One can clearly see that the reference solution does not feature a second shear wave whereas our implementation does. Since shear wave splitting is a well-known physical feature of wave propagation in anisotropic media [2] we assume an error in the proposed reference solution. The correctness of our calculation is confirmed in comparison to an analytical reference: We used the open source tool christoffel [12] to compute the wave speeds for the given material depending on the direction of wave propagation. The shown receiver 6 is located at (7348, 7348, 0), \(10392\,\mathrm{m}\) away from the source. The calculated arrival time for the P wave is \(1.73\,\mathrm{s}\), the slow and the fast S wave arrive after \(2.59\,\mathrm{s}\) and \(3.00\,\mathrm{s}\) respectively. We observe that the arrival times align very well with the results calculated by the new SeisSol implementation. In agreement with the original authors and the maintainers of the SISMOWINE project, our here presented solution has been accepted as the new reference solution due to its physical plausibility.

Fig. 2.
figure 2

Comparison of velocity component in x direction for receiver 6

Full size image

5.4 Tilted Transversally Isotropic Medium

For transversally isotropic materials analytical solutions can be found along the axis of symmetry [5]. As mentioned earlier the representation of the elastic tensor depends on the chosen coordinate system. By tilting the symmetry axis of the transversally isotropic material we can generate an almost densely filled elastic tensor. We take the material characterized by the tensor \(c_{ij} = 0\) except for

(15)

and density \(\rho = 2590\,\frac{\mathrm{kg}}{\mathrm{m}^3}\) and tilt it around the x axis about \(30^\circ \). We consider the computational domain \(\varOmega = [0, 2500]^3\). The source is placed at (1250, 1562.5, 937.5) and a receiver is placed at (1250, 1198.05, 1568.75), which is along the symmetry axis of the tilted material \(728.9\,\mathrm{m}\) away from the source. The source either acts along the axis of symmetry or orthogonal to the axis. The time history is a Ricker wavelet with dominant frequency \(f_0 = 16.0\,\mathrm{Hz}\) and onset time \(t_0 = 0.07\,\mathrm{s}\). The whole simulation was run for \(0.6\,\mathrm{s}\) on a mesh which is refined in a sphere around the source and in a cylinder along the axis of symmetry. In the most refined region the characteristic length is \(5\,\mathrm{m}\) and grows towards the boundary. In total the mesh consisted of 6.10 million elements.

We compare our solution obtained at the receiver with the analytic solution using envelope and pulse misfit. For the horizontal source we obtained a maximal envelope misfit of \(2.09\%\) and a pulse misfit of \(0.49\%\). For the vertical source the misfits were \(2.16\%\) and \(0.28\%\) respectively. For both source types the numerical solution fits the analytic solution very well.

5.5 Application Example: Mount Zugspitze

Accurate numerical simulation of scattering seismic waves by complex geometries are critical for assessing and quantifying seismic hazards. In the context of regional scale seismology wave propagation simulations, many numerical methods are challenged by geometric restrictions or low-order accuracy. We here spatially discretize the surface topography around Mount Zugspitze [1] in an unstructured tetrahedral computational mesh of size \(90\,\mathrm{km} \times 90\,\mathrm{km}\) up to \(70\,\mathrm{km}\) depth with a resolution of \(600\,\mathrm{m}\) at the surface. The mesh contains 1.47 million cells. We chose a discretization of order 6 which results in 740 million degrees of freedom. This means we can resolve frequencies up to \(4.2\,\mathrm{Hz}\) with an estimated envelope misfit smaller than \(1\%\) [15]. A kinematic point source with the same parameters as for the LOH1 test case is placed in the center of the domain at \(10\,\mathrm{km}\) depth. We visually compare the wave field scattered by topography in an isotropic material with parameters \(\rho = 2670\,\frac{\mathrm{kg}}{\mathrm{m}^3}, \lambda = 36.4\,\mathrm{GPa}, \mu = 29.8\,\mathrm{GPa}\) with an anisotropic material. The elastic tensor is chosen such that in EW-direction the P wave speed is \(6000\,\frac{\mathrm{m}}{\mathrm{s}}\) for both materials. To illustrate the effects of anisotropy the P wave speed in NS-direction of the anisotropic material is \(5\%\) lower.

Fig. 3.
figure 3

Isotropic (left) vs. anisotropic (right) seismic wave field scattered at the strong topography free surface of Mount Zugspitze at \(t=3\,\mathrm{s}\) simulation time.

Full size image

In Fig. 3 snapshots of the vertical velocity field on the free surface are plotted. A circular shape for the isotropic example and an elliptic shape for the anisotropic part illustrate the effects of anisotropic materials on wave propagation under strong velocity contrast. We compare this simulation with and without local time stepping: moving from global to local time stepping drastically reduced the execution time from \(5990\,\mathrm{s}\) to \(210\,\mathrm{s}\). The simulation was run on 50 nodes of SuperMUC-NG. The computational more intense version with global time stepping achieved 1.414 TFlop/s, the version with local time stepping achieved 1.015 TFlop/s. This shows that local time stepping is crucial to obtain fast simulations when the element sizes vary a lot, such as in the case of surface topography.

6 Conclusion

The earthquake simulation code SeisSol has been successfully extended to take general anisotropic materials into account. A two-sided Godunov flux for anisotropic media has been derived and implemented. Together with the formulation of the free-surface boundary condition as the solution of an inverse Riemann problem it fits well with the other rheological models. Necessary changes to include local time stepping have been described and implemented.

The scheme has been validated against various benchmarks. The expected convergence rates are demonstrated in comparison to analytic solutions. The mismatch between our results on a community benchmark have been discussed with the maintainers and led to an update of the reference solution.

As anisotropy is non-neglectable to describe the Earth’s subsurface structure we expect a wide range of applications. Besides the importance of seismic anisotropy for exploration purposes, earthquake fault zones may be characterised by pervasive anisotropy. Earthquake monitoring and forecasting can be built upon this observation.