Computational Geosciences

, Volume 23, Issue 6, pp 1237–1258 | Cite as

Efficient use of sparsity by direct solvers applied to 3D controlled-source EM problems

  • Patrick R. Amestoy
  • Sébastien de la Kethulle de Ryhove
  • Jean-Yves L’ExcellentEmail author
  • Gilles Moreau
  • Daniil V. Shantsev
Original Paper


Controlled-source electromagnetic (CSEM) surveying becomes a widespread method for oil and gas exploration, which requires fast and efficient software for inverting large-scale EM datasets. In this context, one often needs to solve sparse systems of linear equations with a large number of sparse right-hand sides, each corresponding to a given transmitter position. Sparse direct solvers are very attractive for these problems, especially when combined with low-rank approximations which significantly reduce the complexity and the cost of the factorization. In the case of thousands of right-hand sides, the time spent in the sparse triangular solve tends to dominate the total simulation time, and here we propose several approaches to reduce it. A significant reduction is demonstrated for marine CSEM application by utilizing the sparsity of the right-hand sides (RHS) and of the solutions that results from the geometry of the problem. Large gains are achieved by restricting computations at the forward substitution stage to exploit the fact that the RHS matrix might have empty rows (vertical sparsity) and/or empty blocks of columns within a non-empty row (horizontal sparsity). We also adapt the parallel algorithms that were designed for the factorization to solve-oriented algorithms and describe performance optimizations particularly relevant for the very large numbers of right-hand sides of the CSEM application. We show that both the operation count and the elapsed time for the solution phase can be significantly reduced. The total time of CSEM simulation can be divided by approximately a factor of 3 on all the matrices from our set (from 3 to 30 million unknowns, and from 4 to 12 thousands RHSs).


Controlled-source electromagnetics (CSEM) Marine electromagnetics Numerical modeling Direct solver Multiple sparse right-hand sides 

Mathematics Subject Classification (2010)

15A06 15A23 65F05 65F50 65Y05 65Z05 68U20 68W10 78A25 86-04 86-08 86A20 86A22 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


Funding information

This work was partially supported by the MUMPS consortium and by LABEX MILYON (ANR-10-LABX-0070) of Université de Lyon, within the program “Investissements d’Avenir” (ANR-11-IDEX-0007) operated by the French National Research Agency (ANR).


  1. 1.
    Amestoy, P.R., Buttari, A., L’Excellent, J.Y., Mary, T.: On the complexity of the block low-rank multifrontal factorization. SIAM J. Sci. Comput. 39(4), A1710–A1740 (2017). CrossRefGoogle Scholar
  2. 2.
    Amestoy, P.R., Buttari, A., L’Excellent, J.Y., Mary, T.: Performance and scalability of the block low-rank multifrontal factorization on multicore architectures. ACM Trans. Math. Softw. 45(1), 2:1–2:26 (2019). CrossRefGoogle Scholar
  3. 3.
    Amestoy, P.R., Duff, I.S., Koster, J., L’Excellent, J.Y.: A fully asynchronous multifrontal solver using distributed dynamic scheduling. SIAM J Matrix Anal Appl 23(1), 15–41 (2001)CrossRefGoogle Scholar
  4. 4.
    Amestoy, P.R., Duff, I.S., L’Excellent, J.Y., Rouet, F.H.: Parallel computation of entries of A-1. SIAM J. Sci. Comput. 37(2), C268–C284 (2015)CrossRefGoogle Scholar
  5. 5.
    Amestoy, P.R., Guermouche, A., L’Excellent, J.Y., Pralet, S.: Hybrid scheduling for the parallel solution of linear systems. Parallel Comput. 32(2), 136–156 (2006)CrossRefGoogle Scholar
  6. 6.
    Amestoy, P.R., L’Excellent, J.Y., Moreau, G.: On exploiting sparsity of multiple right-hand sides in sparse direct solvers. SIAM J. Sci. Comput. 41, A269–A291 (2019)CrossRefGoogle Scholar
  7. 7.
    Amestoy, P.R., L’Excellent, J.Y., Rouet, F.H., Sid-Lakhdar, W.M.: Modeling 1D distributed-memory dense kernels for an asynchronous multifrontal sparse solver. In: High Performance Computing for Computational Science, VECPAR 2014 - 11th International Conference, Eugene, Oregon, USA, June 30 - July 3, 2014, Revised Selected Papers, pp. 156–169 (2014)Google Scholar
  8. 8.
    Avdeev, D.B.: Three-dimensional electromagnetic modelling and inversion from theory to application. Surv. Geophys. 26(6), 767–799 (2005). CrossRefGoogle Scholar
  9. 9.
    Blackford, L.S., Choi, J., Cleary, A., D’Azevedo, E., Demmel, J., Dhillon, I., Dongarra, J., Hammarling, S., Henry, G., Petitet, A., Stanley, K., Walker, D., Whaley, R.C.: ScaLAPACK users’ guide. SIAM Press (1997)Google Scholar
  10. 10.
    Börner, R.U.: Numerical modelling in geo-electromagnetics: advances and challenges. Surv. Geophys. 31(2), 225–245 (2010). CrossRefGoogle Scholar
  11. 11.
    Constable, S.: Ten years of marine CSEM for hydrocarbon exploration. Geophysics 75(5), 75A67–75A81 (2010). CrossRefGoogle Scholar
  12. 12.
    Dongarra, J.J., Du Croz, J., Duff, I.S., Hammarling, S.: Algorithm 679: a set of level 3 basic linear algebra subprograms. ACM Trans. Math. Softw. 16, 1–17 (1990)CrossRefGoogle Scholar
  13. 13.
    Duff, I.S., Erisman, A.M., Reid, J.K.: Direct Methods for Sparse Matrices, 2nd edn. Oxford University Press, London (2017)CrossRefGoogle Scholar
  14. 14.
    Duff, I.S., Reid, J.K.: The multifrontal solution of indefinite sparse symmetric linear systems. ACM Trans. Math. Softw. 9, 302–325 (1983)CrossRefGoogle Scholar
  15. 15.
    Ellingsrud, S., Eidesmo, T., Johansen, S., Sinha, M.C., MacGregor, L.M., Constable, S.: Remote sensing of hydrocarbon layers by seabed logging (SBL): results from a cruise offshore Angola. Lead. Edge 21(10), 972–982 (2002). CrossRefGoogle Scholar
  16. 16.
    George, J.A.: Nested dissection of a regular finite-element mesh. SIAM J. Numer. Anal. 10(2), 345–363 (1973)CrossRefGoogle Scholar
  17. 17.
    Gilbert, J.R.: Predicting structure in sparse matrix computations. SIAM J. Matrix Anal. Appl. 15, 62–79 (1994)CrossRefGoogle Scholar
  18. 18.
    Gilbert, J.R., Liu, J.W.H.: Elimination structures for unsymmetric sparse LU factors. SIAM J. Matrix Anal. Appl. 14, 334–352 (1993)CrossRefGoogle Scholar
  19. 19.
    Hanssen, P., Nguyen, A.K., Fogelin, L.T.T., Jensen, H.R., Skaro, M., Mittet, R., Rosenquist, M., Suilleabhain, L.O., van der Sman, P.: The next generation offshore CSEM acquisition system, pp. 1194–1198. Society of Exploration Geophysicists. (2017)
  20. 20.
    Hiner, M., Martinez, Y., Sun, S.: Delineating salt bodies with 3D CSEM technology. In: Salt Challenges in Hydrocarbon Exploration, SEG Annual Meeting Post-convention Workshop. New Orleans (2015)Google Scholar
  21. 21.
    Lötstedt, P., Nilsson, M.: A minimal residual interpolation method for linear equations with multiple right-hand sides. SIAM J. Sci. Comput. 25(6), 2126–2144 (2004)CrossRefGoogle Scholar
  22. 22.
    Mary, T.: Block low-rank multifrontal solvers: complexity, performance, and scalability. PhD thesis, Université de Toulouse (2017)Google Scholar
  23. 23.
    Nguyen, A.K., Nordskag, J.I., Wiik, T., Bjorke, A.K., Boman, L., Pedersen, O.M., Ribaudo, J., Mittet, R.: Comparing large-scale 3D Gauss-Newton and BFGS CSEM inversions, pp. 872–877. Society of Exploration Geophysicists (2016).
  24. 24.
    Plessix, R.E., Darnet, M., Mulder, W.A.: An approach for 3D multisource, multifrequency CSEM modeling. Geophysics 72(5), SM177–SM184 (2007)CrossRefGoogle Scholar
  25. 25.
    Pothen, A., Sun, C.: A mapping algorithm for parallel sparse Cholesky factorization. SIAM J. Sci. Comput. 14(5), 1253–1257 (1993)CrossRefGoogle Scholar
  26. 26.
    Rouet, F.H.: Memory and performance issues in parallel multifrontal factorizations and triangular solutions with sparse right-hand sides. PhD thesis, Institut National Polytechnique de Toulouse (2012)Google Scholar
  27. 27.
    Shantsev, D., Jaysaval, P., de la Kethulle de Ryhove, S., Amestoy, P.R., Buttari, A., L’Excellent, J.Y., Mary, T.: Large-scale 3-D EM modeling with a Block Low-Rank multifrontal direct solver. Geophys. J. Int. 209(3), 1558–1571 (2017)CrossRefGoogle Scholar
  28. 28.
    Slavova, Tz.: Parallel triangular solution in the out-of-core multifrontal approach for solving large sparse linear systems. Ph.D. dissertation, Institut National Polytechnique de Toulouse (2009). Available as CERFACS Report TH/PA/09/59Google Scholar
  29. 29.
    Streich, R.: Controlled-source electromagnetic approaches for hydrocarbon exploration and monitoring on land. Surv. Geophys. 37(1), 47–80 (2016). CrossRefGoogle Scholar
  30. 30.
    Yee, K.: Numerical solution of initial boundary value problems involving Maxwell’s equations in isotropic media. IEEE Trans. Antennas Propag. 14(3), 302–307 (1966)CrossRefGoogle Scholar
  31. 31.
    Zach, J., Bjorke, A., Storen, T., Maao, F.: 3D inversion of marine CSEM data using a fast finite-difference time-domain forward code and approximate Hessian-based optimization. In: SEG Technical Program Expanded Abstracts 2008, pp. 614–618 (2008).

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University Toulouse, INPT, IRIT UMR5505ToulouseFrance
  2. 2.Mumps TechnologiesLyonFrance
  3. 3.EMGSTrondheimNorway
  4. 4.Kongsberg Defence & AerospaceAskerNorway
  5. 5.University Lyon, CNRS, ENS Lyon, Inria, UCBL, LIP UMR5668LyonFrance
  6. 6.EMGSI&I Technology CenterOsloNorway

Personalised recommendations