Fast Algorithms for Bayesian Inversion

  • Sivaram Ambikasaran
  • Arvind K. Saibaba
  • Eric F. Darve
  • Peter K. Kitanidis
Conference paper
Part of the The IMA Volumes in Mathematics and its Applications book series (IMA, volume 156)


In this article, we review a few fast algorithms for solving large-scale stochastic inverse problems using Bayesian methods. After a brief introduction to the Bayesian stochastic inverse methodology, we review the following computational techniques, to solve large scale problems: the fast Fourier transform, the fast multipole method (classical and a black-box version), and finallym the hierarchical matrix approach. We emphasize that this is mainly a survey paper presenting a few fast algorithms applicable to large-scale Bayesian inversion techniques, applicable to applications arising from geostatistics. The article is presented at a level accessible to graduate students and computational engineers. Hence, we mainly present the algorithmic ideas and theoretical results.


Bayesian stochastic inverse modeling Large-scale problems Geostatistical estimation Numerical linear algebra Fast Fourier transform Ast multipole method Hierarchical matrices 

AMS(MOS) Subject Classifications

Primary 1234 5678 9101112 



The authors were supported by “NSF Award 0934596, Subsurface Imaging and Uncertainty Quantification,” “Army High Performance Computing Research Center” (AHPCRC, sponsored by the U.S. Army Research Laboratory under contract No. W911NF-07-2-0027) and “The Global Climate and Energy Project” (GCEP) at Stanford.


  1. 1.
    Ambikasaran, S., Li, J., Darve, E., Kitanidis, P.K.: Large-scale stochastic linear inversion using hierarchical matrices (2012). In reviewGoogle Scholar
  2. 2.
    Beatson, R., Greengard, L.: A short course on fast multipole methods. Wavelets, multilevel methods and elliptic PDEs pp. 1–37 (1997)Google Scholar
  3. 3.
    Beatson, R., Newsam, G.: Fast evaluation of radial basis functions: I. Computers & Mathematics with Applications 24(12), 7–19 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Bebendorf, M.: Approximation of boundary element matrices. Numerische Mathematik 86(4), 565–589 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Bebendorf, M.: Hierarchical Matrices: A Means to Efficiently Solve Elliptic Boundary Value Problems, Lecture Notes in Computational Science and Engineering (LNCSE), vol. 63. Springer-Verlag (2008). ISBN 978-3-540-77146-3Google Scholar
  6. 6.
    Bebendorf, M., Rjasanow, S.: Adaptive low-rank approximation of collocation matrices. Computing 70(1), 1–24 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Beylkin, G., Coifman, R., Rokhlin, V.: Fast wavelet transforms and numerical algorithms I. Communications on Pure and Applied Mathematics 44(2), 141–183 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Bjørnstad, O., Falck, W.: Nonparametric spatial covariance functions: estimation and testing. Environmental and Ecological Statistics 8(1), 53–70 (2001)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Börm, S., Grasedyck, L., Hackbusch, W.: Hierarchical matrices. Lecture notes 21 (2003)Google Scholar
  10. 10.
    Burrus, C., Gopinath, R., Guo, H.: Introduction to wavelets and wavelet transforms: A primer. Recherche 67, 02 (1998)Google Scholar
  11. 11.
    Chandrasekaran, S., Dewilde, P., Gu, M., Pals, T., Sun, X., van der Veen, A., White, D.: Some fast algorithms for sequentially semiseparable representations. SIAM Journal on Matrix Analysis and Applications 27(2), 341 (2006)CrossRefGoogle Scholar
  12. 12.
    Chandrasekaran, S., Gu, M., Pals, T.: A fast ULV decomposition solver for hierarchically semiseparable representations. SIAM Journal on Matrix Analysis and Applications 28(3), 603 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Cheng, H., Greengard, L., Rokhlin, V.: A fast adaptive multipole algorithm in three dimensions. Journal of Computational Physics 155(2), 468–498 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Coifman, R., Rokhlin, V., Wandzura, S.: The fast multipole method for the wave equation: A pedestrian prescription. Antennas and Propagation Magazine, IEEE 35(3), 7–12 (1993)CrossRefGoogle Scholar
  15. 15.
    Cooley, J., Tukey, J.: An algorithm for the machine calculation of complex fourier series. Math. Comput 19(90), 297–301 (1965)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Cornford, D., Csató, L., Opper, M.: Sequential, Bayesian geostatistics: a principled method for large data sets. Geographical Analysis 37(2), 183–199 (2005)Google Scholar
  17. 17.
    Darve, E.: The fast multipole method: numerical implementation. Journal of Computational Physics 160(1), 195–240 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Darve, E.: The fast multipole method - I: Error analysis and asymptotic complexity. SIAM Journal on Numerical Analysis pp. 98–128 (2001)Google Scholar
  19. 19.
    Darve, E., Havé, P.: Efficient fast multipole method for low-frequency scattering. Journal of Computational Physics 197(1), 341–363 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Darve, E., Havé, P.: A fast multipole method for Maxwell equations stable at all frequencies. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences 362(1816), 603–628 (2004)Google Scholar
  21. 21.
    Fong, W., Darve, E.: The black-box fast multipole method. Journal of Computational Physics 228(23), 8712–8725 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Fritz, J., Neuweiler, I., Nowak, W.: Application of FFT-based algorithms for large-scale universal kriging problems. Mathematical Geosciences 41(5), 509–533 (2009)CrossRefzbMATHGoogle Scholar
  23. 23.
    Golub, G., Van Loan, C.: Matrix computations, vol. 3. Johns Hopkins Univ Press (1996)Google Scholar
  24. 24.
    Goreinov, S., Tyrtyshnikov, E., Zamarashkin, N.: A theory of pseudoskeleton approximations. Linear Algebra and Its Applications 261(1–3), 1–21 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Grasedyck, L., Hackbusch, W.: Construction and arithmetics of \(\mathcal{H}\)-matrices. Computing 70, 2003 (2003)Google Scholar
  26. 26.
    Greengard, L., Rokhlin, V.: A fast algorithm for particle simulations. Journal of Computational Physics 73(2), 325–348 (1987)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Greengard, L., Rokhlin, V.: A new version of the fast multipole method for the Laplace equation in three dimensions. Acta Numerica 6(1), 229–269 (1997)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Greengard, L., Rokhlin, V., SCIENCE., Y.U.N.H.C.D.O.C.: On the efficient implementation of the fast multipole algorithm. Defense Technical Information Center (1988)Google Scholar
  29. 29.
    Hackbusch, W.: A sparse matrix arithmetic based on \(\mathcal{H}\)-matrices. Part I: Introduction to \(\mathcal{H}\)-matrices. Computing 62(2), 89–108 (1999)Google Scholar
  30. 30.
    Hackbusch, W., Börm, S.: Data-sparse approximation by adaptive \({\mathcal{H}}^{2}\)-matrices. Computing 69(1), 1–35 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Hackbusch, W., Grasedyck, L., Börm, S.: An introduction to hierarchical matrices. Max-Planck-Inst. für Mathematik in den Naturwiss. (2001)Google Scholar
  32. 32.
    Hackbusch, W., Khoromskij, B.: A sparse \(\mathcal{H}\)-matrix arithmetic. Computing 64(1), 21–47 (2000)MathSciNetzbMATHGoogle Scholar
  33. 33.
    Hackbusch, W., Nowak, Z.: On the fast matrix multiplication in the boundary element method by panel clustering. Numerische Mathematik 54(4), 463–491 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Hrycak, T., Rokhlin, V.: An improved fast multipole algorithm for potential fields. Tech. rep., DTIC Document (1995)Google Scholar
  35. 35.
    Kitanidis, P.K.: Statistical estimation of polynomial generalized covariance functions and hydrologic applications. Water Resources Research 19(4), 909–921 (1983)CrossRefGoogle Scholar
  36. 36.
    Kitanidis, P.K.: Generalized covariance functions in estimation. Mathematical Geology 25(5), 525–540 (1993)CrossRefGoogle Scholar
  37. 37.
    Kitanidis, P.K.: Quasi-linear geostatistical theory for inversing. Water Resources Research 31(10), 2411–2419 (1995)CrossRefGoogle Scholar
  38. 38.
    Kitanidis, P.K.: On the geostatistical approach to the inverse problem. Advances in Water Resources 19(6), 333–342 (1996)CrossRefGoogle Scholar
  39. 39.
    Kitanidis, P.K.: Introduction to geostatistics: applications to hydrogeology. Cambridge Univ Pr (1997)Google Scholar
  40. 40.
    Kitanidis, P.K.: Generalized covariance functions associated with the Laplace equation and their use in interpolation and inverse problems. Water Resources Research 35(5), 1361–1367 (1999)CrossRefGoogle Scholar
  41. 41.
    Kitanidis, P.K.: On stochastic inverse modeling. Geophysical Monograph-American Geophysical Union 171, 19 (2007)Google Scholar
  42. 42.
    Kitanidis, P.K., Vomvoris, E.G.: A geostatistical approach to the inverse problem in groundwater modeling (steady state) and one-dimensional simulations. Water Resources Research 19(3), 677–690 (1983)CrossRefGoogle Scholar
  43. 43.
    Liu, X., Illman, W., Craig, A., Zhu, J., Yeh, T.: Laboratory sandbox validation of transient hydraulic tomography. Water Resources Research 43(5), W05,404 (2007)CrossRefGoogle Scholar
  44. 44.
    Mallat, S.: A theory for multiresolution signal decomposition: The wavelet representation. Pattern Analysis and Machine Intelligence, IEEE Transactions on 11(7), 674–693 (1989)CrossRefzbMATHGoogle Scholar
  45. 45.
    Mallat, S.: A wavelet tour of signal processing. Academic Press (1999)Google Scholar
  46. 46.
    Nishimura, N.: Fast multipole accelerated boundary integral equation methods. Applied Mechanics Reviews 55, 299 (2002)CrossRefGoogle Scholar
  47. 47.
    Nowak, W., Cirpka, O.A.: Geostatistical inference of hydraulic conductivity and dispersivities from hydraulic heads and tracer data. Water Resources Research 42(8), 8416 (2006)CrossRefGoogle Scholar
  48. 48.
    Nowak, W., Tenkleve, S., Cirpka, O.: Efficient computation of linearized cross-covariance and auto-covariance matrices of interdependent quantities. Mathematical geology 35(1), 53–66 (2003)MathSciNetCrossRefGoogle Scholar
  49. 49.
    Pollock, D., Cirpka, O.: Fully coupled hydrogeophysical inversion of synthetic salt tracer experiments. Water Resources Research 46(7), W07,501 (2010)CrossRefGoogle Scholar
  50. 50.
    Rjasanow, S.: Adaptive cross approximation of dense matrices. IABEM 2002, International Association for Boundary Element Methods (2002)Google Scholar
  51. 51.
    Rjasanow, S., Steinbach, O.: The fast solution of boundary integral equations. Mathematical and Analytical Techniques with Applications to Engineering. Springer, New York (2007)zbMATHGoogle Scholar
  52. 52.
    Rokhlin, V.: Rapid solution of integral equations of classical potential theory. Journal of Computational Physics 60(2), 187–207 (1985)MathSciNetCrossRefzbMATHGoogle Scholar
  53. 53.
    Saibaba, A., Ambikasaran, S., Li, J., Darve, E., Kitanidis, P.: Application of hierarchical matrices to linear inverse problems in geostatistics. Oil & Gas Science and Technology–Revue d’IFP Energies nouvelles 67(5), 857–875 (2012)Google Scholar
  54. 54.
    Saibaba, A., Kitanidis, P.: Efficient methods for large-scale linear inversion using a geostatistical approach. Water Resources Research 48(5), W05,522 (2012)CrossRefGoogle Scholar
  55. 55.
    Starks, T., Fang, J.: On the estimation of the generalized covariance function. Mathematical Geology 14(1), 57–64 (1982)MathSciNetCrossRefGoogle Scholar
  56. 56.
    Vandebril, R., Barel, M., Golub, G., Mastronardi, N.: A bibliography on semiseparable matrices. Calcolo 42(3), 249–270 (2005)MathSciNetCrossRefGoogle Scholar
  57. 57.
    Xia, J., Chandrasekaran, S., Gu, M., Li, X.: Fast algorithms for hierarchically semiseparable matrices. Numerical Linear Algebra with Applications 17(6), 953–976 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  58. 58.
    Ying, L., Biros, G., Zorin, D.: A kernel-independent adaptive fast multipole algorithm in two and three dimensions. Journal of Computational Physics 196(2), 591–626 (2004)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Sivaram Ambikasaran
    • 1
  • Arvind K. Saibaba
    • 1
  • Eric F. Darve
    • 2
  • Peter K. Kitanidis
    • 3
  1. 1.Institute for Computational and Mathematical Engineering, Huang Engineering Center 053B, 475, Via OrtegaStanford UniversityStanfordUSA
  2. 2.Mechanical Engineering, Durand 209, 496, Lomita MallStanford UniversityStanfordUSA
  3. 3.Civil and Environmental EngineeringYang & Yamazaki Environment & Energy Building - MC 4020StanfordUSA

Personalised recommendations