Parallelization Strategy for Wavefield Simulation with an Elastic Iterative Solver

  • Mikhail Belonosov
  • Vladimir Cheverda
  • Victor Kostin
  • Dmitry Neklyudov
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 965)


We present a parallelization strategy for our novel iterative method to simulate elastic waves in 3D land inhomogeneous isotropic media via MPI and OpenMP. The unique features of the solver are the preconditioner developed to assure fast convergence of the Krylov-type iteration method at low time frequencies and the way to calculate how the forward modeling operator acts on a vector. We successfully benchmark the accuracy of our solver against the exact solution and compare it to another iterative solver. The quality of the parallelization is justified by weak and strong scaling analysis. Our modification allows simulation in big models including a modified 2.5D Marmousi model comprising 90 million cells.


Elastic equation MPI OpenMP Preconditioner Krylov iterations 



We are grateful to Vincent Etienne and Michael Jervis for reviewing of our manuscript. Special thanks to Maxim Dmitriev for useful discussions and advice on this topic. Two of the authors (Victor Kostin and Vladimir Tcheverda) have been sponsored by the Russian Science Foundation grant 17-17-01128.


  1. 1.
    Albanese, C., Gilblom, K.: This oil major has a supercomputer the size of a soccer field. Bloomberg, 18 January 2018Google Scholar
  2. 2.
    Aminzadeh, F., Brac, J., Kuntz, T.: 3-D Salt and Overthrust Models. SEG/EAGE Modelling Series, no. 1. SEG Book Series, Tulsa, Oklahoma (1997)Google Scholar
  3. 3.
    Aki, K., Richards, P.G.: Quantitative Seismology, Theory and Methods, vol. 1. W.H. Freeman and Co., San Francisco (1980)Google Scholar
  4. 4.
    Belonosov, M.A., Kostov, C., Reshetova, G.V., Soloviev, S.A., Tcheverda, V.A.: Parallel numerical simulation of seismic waves propagation with intel math kernel library. In: Manninen, P., Öster, P. (eds.) PARA 2012. LNCS, vol. 7782, pp. 153–167. Springer, Heidelberg (2013). Scholar
  5. 5.
    Belonosov, M., Dmitriev, M., Kostin, V., Neklyudov, D., Tcheverda, V.: An iterative solver for the 3D Helmholtz equation. J. Comput. Phys. 345, 330–344 (2017)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Berenger, J.P.: Three-dimensional perfectly matched layer for the absorption of electromagnetic waves. J. Comput. Phys. 127, 363–379 (1996)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Darbas, M., Louer, F.: Analytic preconditioners for the iterative solution of elastic scattering problems. HAL, hal-00839653, pp. 1–32 (2013)Google Scholar
  8. 8.
    Erlangga, Y.A., Nabben, R.: On a multilevel Krylov method for the Helmholtz equation preconditioned by shifted Laplacian. Electron. Trans. Numer. Anal. 31, 403–424 (2008)MathSciNetzbMATHGoogle Scholar
  9. 9.
    Etienne, V., Tonellot, T., Thierry, P., Berthoumieux, V., Andreolli, C.: Optimization of the seismic modeling with the time-domain finite-difference method. In: 84th Annual International Meeting, SEG, Expanded Abstracts, pp. 3536–3540 (2014)Google Scholar
  10. 10.
    Intel, 2018, Intel®Math Kernel Library (Intel®MKL).
  11. 11.
    Khajdukov, V.G., et al.: Modelling of seismic waves propagation for 2D media (direct and inverse problems). In: Malyshkin, V. (ed.) PaCT 1997. LNCS, vol. 1277, pp. 350–357. Springer, Heidelberg (1997). Scholar
  12. 12.
    Kostin, V., Lisitsa, V., Reshetova, G., Tcheverda, V.: Local time-space mesh refinement for simulation of elastic wave propagation in multi-scale media. J. Comput. Phys. 281, 669–689 (2015)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Kostin, V., Neklyudov, D., Tcheverda, V., Belonosov, M., Dmitriev, M.: 3D elastic frequency-domain iterative solver for full-waveform inversion. In: 86th Annual International Meeting, SEG, Expanded Abstracts, pp. 3825–3829 (2016)Google Scholar
  14. 14.
    Kostin, V., Solovyev, S., Liu, H., Bakulin, A.: HSS cluster-based direct solver for acoustic wave equation. In: 87th Annual International Meeting, SEG, Expanded Abstracts, pp. 4017–4021 (2017)Google Scholar
  15. 15.
    Li, Y., Metivier, L., Brossier, R., Han, B., Virieux, J.: 2D and 3D frequency-domain elastic wave modeling in complex media with a parallel iterative solver. Geophysics 80, T101–T118 (2015)CrossRefGoogle Scholar
  16. 16.
    Lisitsa, V., Tcheverda, V., Botter, C.: Combination of discontinuous Galerkin method with finite differences for simulation of elastic wave. J. Comput. Phys. 311, 142–157 (2016)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Operto, S., Virieux, J., Amestoy, P., L’Excellent, J., Giraud, L., Hadj, H.: 3D finite-difference frequency-domain modeling of visco-acoustic wave propagation using a massively parallel direct solver: a feasibility study. Geophysics 72, SM195–SM211 (2007)CrossRefGoogle Scholar
  18. 18.
    Pissarenko, D., Reshetova, G., Tcheverda, V.: 3D finite-difference synthetic acoustic log in cylindrical coordinates: parallel implementation. J. Comput. Appl. Math. 234(6), 1766–1772 (2010)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Pratt, R.G.: Seismic waveform inversion in the frequency domain, Part 1: theory and verification in a physical scale model. Geophysics 64, 888–901 (1999)CrossRefGoogle Scholar
  20. 20.
    Rizzuti, G., Mulder, W.A.: A multigrid-based iterative solver for the frequency-domain elastic wave equation. In: 77th EAGE Conference and Exhibition, Expanded Abstracts, pp. 1–4 (2015)Google Scholar
  21. 21.
    Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd edn. SIAM, Philadelphia (2003)CrossRefGoogle Scholar
  22. 22.
    Sirgue, L., Etgen, J., Albertin, U., Brandsberg-Dahl, S.: System and method for 3D frequency domain waveform inversion based on 3D time-domain forward modeling. U. S. Patent, 11/756,384 (2007)Google Scholar
  23. 23.
    Sonneveld, P., van Gijzen, M.B.: IDR(s): a family of simple and fast algorithms for solving large nonsymmetric systems of linear equations. SIAM J. Sci. Comput. 31, 1035–1062 (2008)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Symes, W.W.: Migration velocity analysis and waveform inversion. Geophys. Prospect. 56(6), 765–790 (2008)CrossRefGoogle Scholar
  25. 25.
    Van Der Vorst, H.A.: BI-CGSTAB: a fast and smoothly converging variant of BI-CG for the solution of nonsymmetric linear systems. SIAM J. Sci. Stat. Comput. 13(2), 631–644 (1992)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Virieux, J.: Seismic wave modeling for seismic imaging. Lead. Edge 28, 538–544 (2009)CrossRefGoogle Scholar
  27. 27.
    Wang, S.V., de Hoop, M., Xia, J., Li, X.S.: Massively parallel structured multifrontal solver for time-harmonic elastic waves in 3-D anisotropic media. Geophys. J. Int. 191, 346–366 (2012)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Mikhail Belonosov
    • 1
  • Vladimir Cheverda
    • 2
  • Victor Kostin
    • 2
  • Dmitry Neklyudov
    • 2
  1. 1.Aramco Research Center - DelftAramco Overseas Company B.V.DelftThe Netherlands
  2. 2.Institute of Petroleum Geology and Geophysics SB RASNovosibirskRussia

Personalised recommendations