A Review of High-Performance Computational Strategies for Modeling and Imaging of Electromagnetic Induction Data

Abstract

Many geoscientific applications exploit electrostatic and electromagnetic fields to interrogate and map subsurface electrical resistivity—an important geophysical attribute for characterizing mineral, energy, and water resources. In complex three-dimensional geologies, where many of these resources remain to be found, resistivity mapping requires large-scale modeling and imaging capabilities, as well as the ability to treat significant data volumes, which can easily overwhelm single-core and modest multicore computing hardware. To treat such problems requires large-scale parallel computational resources, necessary for reducing the time to solution to a time frame acceptable to the exploration process. The recognition that significant parallel computing processes must be brought to bear on these problems gives rise to choices that must be made in parallel computing hardware and software. In this review, some of these choices are presented, along with the resulting trade-offs. We also discuss future trends in high-performance computing and the anticipated impact on electromagnetic (EM) geophysics. Topics discussed in this review article include a survey of parallel computing platforms, graphics processing units to multicore CPUs with a fast interconnect, along with effective parallel solvers and associated solver libraries effective for inductive EM modeling and imaging.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2

Notes

  1. 1.

    I consider the beginning of the HPC era circa 1988, with the development of multiple instructions multiple data (MIMD) asynchronous computing architectures (cf. Fox 1988).

  2. 2.

    One petaflop is equivalent to 1015 operations per second.

References

  1. Alumbaugh DL, Newman GA (1997) 3-D massively parallel electromagnetic inversion—part II. Anal Crosswell Exp Geophys J Int 128:355–363

    Article  Google Scholar 

  2. Alumbaugh DL, Newman GA, Prevost L, Shadid JN (1996) Three dimensional, wideband electromagnetic modeling on massively parallel computers. Radio Sci 31:1–23

    Article  Google Scholar 

  3. Amestoy PR, Duff IS, Koster J, L’Excellent JY (2001) A fully asynchronous multifrontal solver using distributed dynamic scheduling. SIAM J Matrix Anal Appl 23(1):15–41

    Article  Google Scholar 

  4. Amestoy PR, Guermouche A, L’Excellent JY, Pralet S (2006) Hybrid scheduling for the parallel solution of linear systems. Parallel Comput 32(2):136–156

    Article  Google Scholar 

  5. Balay S, Brown J, Buschelman K, Eijkhout V, Gropp WD, Kaushik D, Knepley MG, McInnes LC, Smith BF, Zhang H (2010) PETSc users manual. Tech. Rep. Number ANL-95/11—revision 3.1, Argonne National Laboratory

  6. Börner R-U, Ernst OG, Spitzer K (2008) Fast 3-D simulation of transient electromagnetic fields by model reduction in the frequency domain using Krylov subspace projection. Geophys J Int 173:766–780. doi:10.1111/j.1365-246x.2008.03750.x

    Article  Google Scholar 

  7. Carazzone JJ, Burtz OM, Green KE, Pavlov DA, Xia C (2005) Three-dimensional imaging of marine CSEM data. In: 75th Annual international meeting, SEG, expanded abstracts, pp 575–578

  8. Carazzone JJ, Dickens TA, Green KE, Jing C, Wahrmund LA, Willen DE, Commer M, Newman GA (2008) Inversion study of a large marine CSEM survey. In: 78th Annual international meeting, SEG, expanded abstracts, pp 644–647

  9. Chen J, Dickens T (2009) Effects of uncertainty in rock-physics models on reservoir parameter estimation using seismic amplitude variation with angle and controlled-source electromagnetics data. Geophys Prospect 57:61–74

    Article  Google Scholar 

  10. Chen J, Hoversten GM, Vasco D, Rubin Y, Zhou Z (2007) A Bayesian model for gas saturation estimation using marine seismic AVA and CSEM data. Geophysics 72:WA85–WA95

    Article  Google Scholar 

  11. Chen J, Tompkins M, Zhang P, Wilt M, Mackie R (2012) Frequency-domain EM modeling of 3D anisotropic magnetic permeability and analytical analysis. In: 82nd Annual international meeting, SEG extended abstracts, pp 1–5. doi:10.1190/segam2012-0308.1

  12. Chen J, Hoversten GM, Key K, Nordquest G, Cumming W (2012b) Stochastic inversion of magnetotelluric data using a sharp boundary parameterization and application to a geothermal site. Geophysics 77(4):E265–E279. doi:10.1190/geo2011-0430.1

    Article  Google Scholar 

  13. Colombo D, Keho T, McNeice G (2012) Integrated seismic-electromagnetic workflow for sub-basalt exploration in northwest Saudi Arabia. Lead Edge 31:42–52

    Article  Google Scholar 

  14. Commer M, Newman GA (2008) New advances in three-dimensional controlled-source electromagnetic inversion. Geophys J Int 172:513–535

    Article  Google Scholar 

  15. Commer M, Newman GA (2009) Three-dimensional controlled-source electromagnetic and magnetotelluric joint inversion. Geophys J Int 178:1305–1316

    Article  Google Scholar 

  16. Commer M, Newman GA, Carazzone JJ, Dickens TA, Green KE, Wahrmund LA, Willen DE, Shiu J (2008) Massively parallel electrical-conductivity imaging of hydrocarbons using the IBM Blue Gene/L supercomputer. IBM J Res Dev 52(½):93–103

    Article  Google Scholar 

  17. Commer M, Maia FRN, Newman GA (2011) Iterative Krylov solution methods for geophysical electromagnetic simulations on throughput-oriented processing units. Int J High Perform Comput Appl 26(4):378–385. doi:10.1177/1094342011428145

    Article  Google Scholar 

  18. Constable S (2006) Marine electromagnetic methods—a new tool for offshore exploration. Lead Edge 25:438–444

    Article  Google Scholar 

  19. Druskin V, Knizhnerman L (1994) Spectral approach to solving three-dimensional Maxwell’s diffusion equations in the time and frequency domains. Radio Sci 29(4):937–953

    Article  Google Scholar 

  20. Druskin V, Knizhnerman LA, Ping L (1999) New spectral Lanczos decomposition method for induction modeling in arbitrary 3-D geometry. Geophysics 64(3):701–706

    Article  Google Scholar 

  21. Eidesmo T, Ellingsrud S, MacGregor LM, Constable S, Sinha MC, Johansen S, Kong S, Westerdahl FN (2002) Sea Bed Logging (SBL), a new method for remote and direct identification of hydrocarbon filled layers in deepwater areas. First Break 20(3):144–152

    Google Scholar 

  22. Ellingsrud S, Eidesmo T, Johansen S, Sinha MC, MacGregor LM, Constable S (2002) Remote sensing of hydrocarbon layers by seabed logging (SBL): results from a cruise offshore Angola. Lead Edge 21:972–982

    Article  Google Scholar 

  23. Farquharson CG, Oldenburg DW (1996) Approximate sensitivities for the electromagnetic inverse problem. Geophys J Int 126:235–252

    Article  Google Scholar 

  24. Fox GC (1988) Solving problems on concurrent processors. Prentice Hall, Old Tappan, NJ

    Google Scholar 

  25. Franke A, Börner R-U, Spitzer K (2007) Adaptive unstructured grid finite element simulation of two-dimensional magnetotelluric fields for arbitrary surface and seafloor topography. Geophys J Int 171:71–86

    Article  Google Scholar 

  26. Freund R (1992) Conjugate gradient type methods for linear systems with complex symmetric coefficient matrices. SIAM J Sci Stat Comput 13:425–448

    Article  Google Scholar 

  27. Freund R, Nachtigal N (1991) QMR: a quasi-minimal residual method for non-hermitian linear systems. Numer Math 60:315–339

    Article  Google Scholar 

  28. Grayver AV, Streich R, Ritter O (2013) Three-dimensional parallel distributed inversion of CSEM data using a direct forward solver. Geophys J Int 193(3):1432–1446. doi:10.1093/gji/ggt055

    Article  Google Scholar 

  29. Greenbaum A (1997) Iterative methods for solving linear systems. SIAM, Philadelphia, PA

    Google Scholar 

  30. Gribenko A, Zhdanov MS (2007) Rigorous 3D inversion of marine CSEM data based on the integral equation method. Geophysics 72:73–84

    Article  Google Scholar 

  31. Gropp W, Lusk E, Doss N, Skjellum A (1996) A high-performance, portable implementation of the MPI message passing interface standard. Parallel Comput 22:789–828

    Article  Google Scholar 

  32. Habashy TM, Groom RW, Spies BR (1993) Beyond the Born and Rytov approximations: a nonlinear approach to electromagnetic scattering. J Geophy Res 98:1759–1775. doi:10.1029/92JB02324

    Article  Google Scholar 

  33. Heroux MA, Willenbring JM, Heaphy R (2003) Trilinos developers guide part II: ASCI software quality engineering practices version 1.0. Tech. Rep. SAND2003-1899, Sandia National Laboratories

  34. Heroux MA, Salinger AG, Bartlett RA, Thornquist HK, Howle VE, Tuminaro RS, Hoekstra RJ, Willenbring JM, Hu JJ, Willina SA, Kolda T, Lehoucq RB, Long KR, Pawlowski RP, Philipps ET, Stanley KS (2005) An overview of the Trilinos project. ACM Trans Math Softw 31:397–423

    Article  Google Scholar 

  35. Hestenes MR, Stiefel E (1952) Methods of conjugate directions for solving linear systems. J Res Natl Bureau Stand 49:409–435

    Article  Google Scholar 

  36. Hohmann GW (1975) Three-dimensional induced polarization and electromagnetic modeling. Geophysics 40:309–324

    Article  Google Scholar 

  37. Hoversten G, Constable S, Morrison H (2000) Marine magnetotellurics for base-of-salt mapping: Gulf of Mexico field test at the Gemini structure. Geophysics 65:1476–1488

    Article  Google Scholar 

  38. Jegen MD, Hobbs R, Tarits P, Chave A (2009) Joint inversion of marine magnetotelluric and gravity data incorporating seismic constraints: preliminary results of sub-basalt imaging off the Faroe Shelf. Earth Planet Sci Lett 282:47–55

    Article  Google Scholar 

  39. Key K, Ovall J (2011) A parallel goal-oriented adaptive finite element method for 2.5-D electromagnetic modeling. Geophys J Int 186(1):137–154. doi:10.1111/j.1365-246X2011.05025.x

    Article  Google Scholar 

  40. Krylov A (1931) On the numerical solution of the equation by which in technical questions frequencies of small oscillations of material systems are determined. Izv. Akad. Nauk SSSR 7:491–539 (in Russian)

    Google Scholar 

  41. Lanczos C (1952) Solution of systems of linear equations by minimized iterations. J Res Natl Bureau Stand 49:33–53

    Article  Google Scholar 

  42. Lawlor OS (2009) Message passing for GPGPU clusters. In: CudaMPI: IEEE international conference on cluster computing and workshops, 2009. CLUSTER ‘09, pp 1–8

  43. Li XS, Demmel JW (2003) SuperLU_DIST: a scalable distributed-memory sparse direct solver for unsymmetric linear systems. ACM Trans Math Softw 29(2):110–140

    Article  Google Scholar 

  44. Liu B, Li SC, Nie LC, Wang J, Nie LC, Wang J, Zhang QS (2012) 3D resistivity inversion using an improved Genetic Algorithm based on control method of mutation direction. J Appl Geophys 87:1–8. doi:10.1016/j.jappgeo.2012.08.002

    Article  Google Scholar 

  45. Lu JJ, Wu XP, Spitzer K (2010) Algebraic multigrid methods for 3D DC resistivity modeling. Chin J Geophys. Special issue of the 19th international workshop on electromagnetic induction in the Earth, Beijing, Oct 23–29, 2008, vol 53, pp 700–707

  46. MacGregor L, Andeis D, Tomlinson T, Barker N (2006) Controlled-source electromagnetic imaging of the Nuggets-1 reservoir. Lead Edge 25:984–992

    Article  Google Scholar 

  47. Mackie RL, Madden TR (1993) Conjugate direction relaxation solutions for 3-D magnetotelluric modeling. Geophysics 58:1052–1057

    Article  Google Scholar 

  48. Maresh J, White RS (2005) Seeing through a glass, darkly: strategies for imaging through basalt. First Break 23:27–33

    Google Scholar 

  49. Moucha R, Bailey RC (2004) An accurate and robust multi-grid algorithm for 2D resistivity modeling. Geophys Prospect 52:197–212

    Article  Google Scholar 

  50. Mudge JC, Heinson GS, Thiel S (2011) Evolving inversion methods in geophysics with cloud computing—a case study of an eScience collaboration. In: Proceedings of IEEE eScience, pp 119–125

  51. Newman GA, Alumbaugh DL (1995) Frequency domain modeling of airborne electromagnetic responses using staggered finite differences. Geophys Prospect 43:1021–1042

    Article  Google Scholar 

  52. Newman GA, Alumbaugh DL (1997) 3-D massively parallel electromagnetic inversion—part I theory. Geophys J Int 128:345–354

    Article  Google Scholar 

  53. Newman GA, Alumbaugh DL (1999) 3-D electromagnetic modeling and inversion on massively parallel computers. In: Oristaglio MN, Spies BR (eds) Three-dimensional electromagnetics. Society of exploration geophysicists, Geophysical Developments No. 7, Tulsa OK, pp 299–321

  54. Newman GA, Alumbaugh DL (2000) Three-dimensional magnetotelluric inversion using non-linear conjugate gradients. Geophys J Int 140:410–424

    Article  Google Scholar 

  55. Newman GA, Commer M (2005) New advances in transient electromagnetic inversion. Geophys J Int 160:5–32

    Article  Google Scholar 

  56. Newman GA, Commer M (2009) Massively parallel electrical conductivity imaging of the subsurface. J Phys Conf Ser 180:012063

    Article  Google Scholar 

  57. Newman GA, Hohmann GW, Anderson WL (1986) Transient electromagnetic response of a three-dimensional body in a layered earth. Geophysics 51:1608–1627

    Article  Google Scholar 

  58. Newman GA, Commer M, Carazzone JJ (2010) Imaging CSEM data in the presence of electrical anisotropy. Geophysics 75:51–61

    Article  Google Scholar 

  59. Oldenburg DW, Haber E, Shekhtman R (2013) Three dimensional inversion of multisource time domain electromagnetic data. Geophysics 78:E47–E57

    Article  Google Scholar 

  60. Plessix RE, Mulder WA (2008) Resistivity imaging with controlled-source electromagnetic data: depth and data weighting. Inverse Prob 24:1–22

    Google Scholar 

  61. Plessix RE, van der Sman P (2007) 3D CSEM modeling and inversion in complex geological settings. In: 77th Annual international meeting, SEG, expanded abstracts, pp 589–593

  62. Plessix RE, van der Sman P (2008) Regularized and blocky 3D controlled source electromagnetic inversion. In: 24th Progress in electromagnetic research symposium, abstracts, pp 755–760

  63. Puzyrev V, Koldan J, de la Puente J, Houzeaux G, Vazquez M, Cele J (2013) A parallel finite-element method for 3D controlled-source electromagnetic forward modeling. Geophys J Int 193:678–693. doi:10.1093/gji/ggt027

    Article  Google Scholar 

  64. Schenk O, Gärtner K (2004) Solving unsymmetric sparse systems of linear equations with PARDISO. J Future Gen Comput Syst 20(3):475–487

    Article  Google Scholar 

  65. Schenk O, Gärtner K (2006) On fast factorization pivoting methods for symmetric indefinite systems. Elec Trans Numer Anal 23:158–179

    Google Scholar 

  66. Schwarzbach C, Haber E (2013) Finite-element based inversion for time-harmonic electromagnetic problems. Geophys J Int 193:615–634. doi:10.1093/gji/ggt006

    Article  Google Scholar 

  67. Schwarzbach C, Börner R-U, Spitzer K (2005) 2D inversion of direct current resistivity data using a parallel, multi-objective genetic algorithm. Geophys J Int 162:685–695

    Article  Google Scholar 

  68. Schwarzbach C, Börner R-U, Spitzer K (2011) Three-dimensional adaptive higher order finite element simulation for geo-electromagnetics—a marine CSEM example. Geophys J Int 187:63–74

    Article  Google Scholar 

  69. Smith JT (1992) Conservative modeling of 3-D electromagnetic fields. Paper presented at the 11th workshop on electromagnetic induction in the earth. International association of geomagnetism and aeronomy. Wellington, New Zealand, Aug 26–Sept 2

  70. Smith JT (1996a) Conservative modeling of 3-D electromagnetic fields, part I: properties and error analysis. Geophysics 61:1308–1318

    Article  Google Scholar 

  71. Smith JT (1996b) Conservative modeling of 3-D electromagnetic fields, part II: biconjugate gradient solution and an accelerator. Geophysics 61:1319–1324

    Article  Google Scholar 

  72. Smith JT, Booker JR (1991) Rapid inversion of two- and three dimensional magnetotelluric data. J Geophys Res 96:3905–3922

    Article  Google Scholar 

  73. Spitzer K (1995) A 3D finite difference algorithm for DC resistivity modeling using conjugate gradient methods. Geophys J Int 123:903–914

    Article  Google Scholar 

  74. Spitzer K, Wurmstich B (1999) Speed and accuracy in 3D resistivity modeling. In: Oristaglio ML, Spies BR (eds) Three-dimensional electromagnetics, SEG book series “Geophysical Developments”, No. 7, Society of exploration geophysicists, pp 161–176, Tulsa, OK

  75. Streich R (2009) 3D finite-difference frequency-domain modeling of controlled-source electromagnetic data: direct solution and optimization for high accuracy. Geophysics 74(5):F95–F105. doi:10.1190/1.3196241

    Article  Google Scholar 

  76. Stuart JA, Owens JD (2009) Message passing on data-parallel architectures. In: Proceedings of the 23rd IEEE international parallel and distributed processing symposium

  77. Torres-Verdin C, Habashy TM (1994) Rapid 2.5-dimensional forward modeling and inversion via a new nonlinear scattering approximation. Radio Sci 29:1051–1079

    Article  Google Scholar 

  78. Torres-Verdin C, Habashy TM (1995) A two step linear inversion of two dimensional electrical conductivity. IEEE Trans Antenna Propag 43:405–415

    Article  Google Scholar 

  79. Tuminaro RS, Heroux M, Hutchinson SA, Shadid JN (1999) Official Aztec user’s guide: version 2.1, sand report SAND99-8801J, Sandia National Laboratories

  80. Um E, Harris JM, Alumbaugh DL (2010) 3D time-domain simulation of electromagnetic diffusion phenomena: a finite-element electric-field approach. Geophysics 75:115–126

    Article  Google Scholar 

  81. Um E, Commer M, Newman GA (2013) Efficient pre-conditioned iterative solution strategies for the electromagnetic diffusion in the Earth: finite-element frequency-domain approach. Geophys J Int. doi: 10.1093/gji/ggt071

  82. van der Vorst H (1992) Bi-CGSTAB: a fast and smoothly converging variant of Bi-CG for the solution of non-symmetric linear systems. SIAM J Sci Statist Comput 13:631–644

    Article  Google Scholar 

  83. Vieira da Silva N, Morgan JV, Macgregor L, Warner M (2012) A finite element multifrontal method for 3D CSEM modeling in the frequency domain. Geophysics 77:101–115

    Article  Google Scholar 

  84. Wang T, Hohmann GW (1993) A finite-difference, time-domain solution for three-dimensional electromagnetic modeling. Geophysics 58:797–809

    Article  Google Scholar 

  85. Wang S, de Hoop MV, Xia J (1011) 2011, On 3D modeling of seismic wave propagation via a structured parallel multifrontal direct Helmholtz solver. Geophys Prospect 59:857–873. doi:11/j.1365-2478.2011.00982.x

    Google Scholar 

  86. Wang S, de Hoop MV, Xia J, Li XS (2012) Massively parallel structured multifrontal solver for time-harmonic elastic waves in 3-D anisotropic media. Geophys J Int 191(1):346–366. doi:10.1111/j.1365.246X.2012.05634.x

    Article  Google Scholar 

  87. Wannamaker PE, Hohmann GW, Ward SH (1984) Magnetotelluric responses of three-dimensional bodies in layered earths. Geophysics 49:1517–1533

    Article  Google Scholar 

  88. Weiss CJ, Schultz A (2011) An evaluation of parallelization strategies for low-frequency electromagnetic induction simulators using staggered grid discretizations. In: American geophysical union fall meeting conference proceedings, informatics session, San Francisco

  89. Yang C, Huang C, Lin C (2011) Hybrid CUDA, OpenMP, and MPI parallel programming on multicore GPU clusters. Comput Phys Commun 182(1):266–269

    Article  Google Scholar 

  90. Zach JJ, Bjørke AK, Støren T, Maaø F (2008) 3D inversion of marine CSEM data using a fast finite-difference time-domain forward code and approximate Hessian-based optimization. In: 78th Annual international meeting, SEG, expanded abstracts, pp 614–618

  91. Zhdanov MS, Fang S (1999) 3D electromagnetic inversion based on the quasi-linear approximation. In: Three-dimensional electromagnetics, SEG book series “Geophysical Developments”, No. 7, Society of exploration geophysicist, pp 233–255. Society of Exploration Geophysicists, Tulsa, OK

Download references

Acknowledgments

I wish to thank Yasuo Ogawa, Graham Heinson, and other members of the 21st EM Workshop Program Committee for the invitation and opportunity to write this review article. Input from the two referees, Klaus Spitzer and Chester Weiss, also improved the content of the review. Finally, I also wish to acknowledge my employer, Lawrence Berkeley Laboratory, and the U.S. Department of Energy Office of Science for funding, under contract number DE-AC02-05CH11231.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Gregory A. Newman.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Newman, G.A. A Review of High-Performance Computational Strategies for Modeling and Imaging of Electromagnetic Induction Data. Surv Geophys 35, 85–100 (2014). https://doi.org/10.1007/s10712-013-9260-0

Download citation

Keywords

  • Three-dimensional electromagnetic modeling and inversion
  • Magnetotelluric soundings
  • Electromagnetic induction
  • High-performance computing
  • Parallel solvers
  • Resistivity imaging for the Earth