Skip to main content
SpringerLink
Log in

Iterative numerical methods for sampling from high dimensional Gaussian distributions

  • Published:
Statistics and Computing Aims and scope Submit manuscript

Abstract

Many applications require efficient sampling from Gaussian distributions. The method of choice depends on the dimension of the problem as well as the structure of the covariance- (Σ) or precision matrix (Q). The most common black-box routine for computing a sample is based on Cholesky factorization. In high dimensions, computing the Cholesky factor of Σ or Q may be prohibitive due to accumulation of more non-zero entries in the factor than is possible to store in memory. We compare different methods for computing the samples iteratively adapting ideas from numerical linear algebra. These methods assume that matrix vector products, Qv, are fast to compute. We show that some of the methods are competitive and faster than Cholesky sampling and that a parallel version of one method on a Graphical Processing Unit (GPU) using CUDA can introduce a speed-up of up to 30x. Moreover, one method is used to sample from the posterior distribution of petroleum reservoir parameters in a North Sea field, given seismic reflection data on a large 3D grid.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Algorithm 2
Algorithm 3
Algorithm 4
Algorithm 5
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Akhiezer, N.I.: Elements of the Theory of Elliptic Functions. American Mathematical Society, Providence (1990)

    MATH  Google Scholar 

  • Allen, E.J., Baglama, J., Boyd, S.K.: Numerical approximation of the product of the square root of a matrix with a vector. Linear Algebra Appl. 310, 167–181 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  • Aune, E., Simpson, D.P.: Parameter estimation in high dimensional Gaussian distributions. arXiv:1105.5256v1 [stat.CO] (2011)

  • Banerjee, S., Gelfand, A.E., Finley, A., Sang, H.: Gaussian predictive process models for large spatial data sets. J. R. Stat. Soc. B 70, 209–226 (2008)

    Article  MathSciNet  Google Scholar 

  • Belabbas, M., Wolfe, P.: Spectral methods in machine learning and new strategies for very large datasets. Proc. Natl. Acad. Sci. 106(2), 369 (2009)

    Article  Google Scholar 

  • Bell, N., Garland, M.: Implementing sparse matrix-vector multiplication on throughput-oriented processors. In: SC’09: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, pp. 1–11. ACM, New York (2009)

    Chapter  Google Scholar 

  • Benzi, M., Bertaccini, D.: Approximate inverse preconditioning for shifted linear systems. BIT Numer. Math. 43(2), 231–244 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  • Besag, J., York, J., Mollie, A.: Bayesian image restoration, with two applications in spatial statistics. Ann. Inst. Stat. Math. 43, 1–59 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  • Björck, Å.: Numerical Methods for Least Squares Problems, vol. 51. Society for Industrial Mathematics, Philadelphia (1996)

    Book  MATH  Google Scholar 

  • Buland, A., Omre, H.: Bayesian linearized AVO inversion. Geophysics 68, 185–198 (2003)

    Article  Google Scholar 

  • Buland, A., Kolbjørnsen, O., Omre, H.: Rapid spatially coupled AVO inversion in the Fourier domain. Geophysics 68, 824–836 (2003)

    Article  Google Scholar 

  • Cressie, N., Johannesson, G.: Fixed rank kriging for large spatial datasets. J. R. Stat. Soc. B 70, 209–226 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  • Davies, P.I., Higham, N.J.: Computing f(A)b for matrix functions f. In: QCD and Numerical Analysis III, pp. 15–24. Springer, Berlin (2005)

    Chapter  Google Scholar 

  • Frommer, A., Simoncini, V.: Matrix functions. In: Model Order Reduction: Theory, Research Aspects and Applications, pp. 275–304. Springer, Berlin (2008)

    Chapter  Google Scholar 

  • Golub, G.H., van Loan, C.F.: Matrix Computations, 3rd edn. John Hopkins University Press, Baltimore (1996)

    MATH  Google Scholar 

  • Gray, R.: Toeplitz and Circulant Matrices: A Review. E-book (2006)

  • Hale, N., Higham, N.J., Trefethen, L.N.: Computing A α, log(A) and related matrix functions by contour integrals. SIAM J. Numer. Anal. 46, 2505–2523 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  • Hestenes, M., Stiefel, E.: Methods of conjugate gradients for solving linear systems. J. Res. Natl. Bur. Stand. 49, 409–436 (1952)

    Article  MathSciNet  MATH  Google Scholar 

  • Higham, N.J.: Functions of Matrices: Theory and Computation. Society for Industrial and Applied Mathematics, Philadelphia (2008)

    Book  Google Scholar 

  • Ilic, M., Turner, I.W., Anh, V.: A numerical solution using an adaptively preconditioned lanczos method for a class of linear systems related with the fractional Poisson equation. J. Appl. Math. Stoch. Anal. 2008, 104525 (2008)

    Article  MathSciNet  Google Scholar 

  • Ilic, M., Turner, I.W., Simpson, D.P.: A restarted Lanczos approximation to functions of a symmetric matrix. IMA J. Numer. Anal. 30, 1044–1061 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  • Jegerlehner, B.: Krylov space solvers for shifted linear systems. arXiv:hep-lat/9612014v1 (1996)

  • Lee, A., Yau, C., Giles, M.B., Doucet, A., Holmes, C.C.: On the utility of graphics cards to perform massively parallel simulation of advanced Monte Carlo methods. J. Comput. Graph. Stat. 19, 769–789 (2010)

    Article  Google Scholar 

  • Lindgren, F., Lindstrøm, J., Rue, H.: An explicit link between Gaussian fields and Gaussian Markov random fields: the SPDE approach. J. R. Stat. Soc. B 73, 423–498 (2011)

    Article  MATH  Google Scholar 

  • MATLAB: Version 7.11.0 (R2010b). The MathWorks Inc., Natick (2010)

    Google Scholar 

  • Meurant, G., Strakos, Z.: The Lanczos and conjugate gradient algorithms in finite precision arithmetic. Acta Numer. 15, 471–542 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  • Parker, A., Fox, C.: Sampling Gaussian distributions in Krylov spaces with conjugate gradients. SIAM J. Sci. Comput. (2011, submitted)

  • Paul, M., Held, L., Toschke, A.M.: Multivariate modelling of infectious disease surveillance data. Stat. Med. 27, 6250–6267 (2008)

    Article  MathSciNet  Google Scholar 

  • Rabben, T.E., Ursin, B., Tjelmeland, H.: Non-linear Bayesian joint inversion of seismic reflection coefficients. Geophys. J. Int. 173, 265–280 (2008)

    Article  Google Scholar 

  • Rasmussen, C., Wiliams, C.: Gaussian Processes for Machine Learning. MIT Press, Cambridge (2006)

    MATH  Google Scholar 

  • Riley, J.: Solving systems of linear equations with a positive definite, symmetric, but possibly ill-conditioned matrix. Math. Tables Other Aids Comput. 9(51), 96–101 (1955)

    Article  MathSciNet  MATH  Google Scholar 

  • Roberts, G., Sahu, S.: Updating schemes, correlation structure, blocking and parameterization for the Gibbs sampler. J. R. Stat. Soc., Ser. B, Stat. Methodol. 59(2), 291–317 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  • Rue, H.: Fast sampling of Gaussian Markov random fields. J. R. Stat. Soc. B 63, 325–338 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  • Rue, H., Held, L.: Gaussian Markov Random Fields. Chapman & Hall, London (2005)

    Book  MATH  Google Scholar 

  • Rue, H., Tjelmeland, H.: Fitting Gaussian Markov random fields to Gaussian fields. Scand. J. Stat. 29, 31–49 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  • Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd edn. SIAM, Philadelphia (2003)

    Book  MATH  Google Scholar 

  • Saad, Y., Yeung, M., Erhel, J., Guyomarc’h, F.: A deflated version of the conjugate gradient algorithm. SIAM J. Sci. Comput. 21, 1909–1926 (1999)

    Article  MathSciNet  Google Scholar 

  • Schneider, M., Willsky, A.: A Krylov subspace method for covariance approximation and simulation of random processes and fields. Multidimens. Syst. Signal Process. 14, 295–318 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  • Shewchuk, J.: An introduction to the conjugate gradient method without the agonizing pain. http://www.cs.colorado.edu/~jessup/csci5646/READINGS/painless-conjugate-gradient.pdf (1994)

  • Simon, H.: Analysis of the symmetric Lanczos algorithm with reorthogonalization methods. Linear Algebra Appl. 61, 101–131 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  • Simpson, D.: Krylov subspace methods for approximating functions of symmetric positive definite matrices with applications to applied statistics and anomalous diffusion. PhD thesis, School of Mathematical Sciences, Queensland Univ of Tech (2008)

  • Simpson, D., Turner, I., Pettitt, A.: Fast sampling from a Gaussian Markov random field using Krylov subspace approaches. Technical report, School of Mathematical Sciences, Queensland Univ. of Tech. (2008)

  • Stein, E.M., Shakarchi, R.: Complex Analysis. Princeton University Press, Princeton (2003)

    MATH  Google Scholar 

  • Stovas, A., Ursin, B.: Reflection and transmission responses of layered transversely isotropic viscoelastic media. Geophys. Prospect. 51, 447–477 (2003)

    Article  Google Scholar 

  • Trefethen, L., Bau, D.: Numerical Linear Algebra. SIAM, Philadelphia (1997)

    Book  MATH  Google Scholar 

  • Tyrtyshnikov, E.E.: Optimal and superoptimal circulant preconditioners. SIAM J. Matrix Anal. Appl. 13, 459–473 (1990)

    Article  MathSciNet  Google Scholar 

  • van den Eshof, J., Sleijpen, G.: Accurate conjugate gradients methods for families of shifted systems. Appl. Numer. Math. 49, 17–37 (2003)

    Article  Google Scholar 

  • Zolotarev, E.I.: Applications of elliptic functions to questions of functions deviating least and most from zero. Zap. Imp. Nauk St. Petersb. 30 (1877)

Download references

Acknowledgements

We thank Statoil for permission to use the Norne dataset, François Alouges for helpful discussion of the deformation method and Daniel P. Simpson for insightful discussions on the use of Krylov methods for computing matrix functions.

Author information

Authors and Affiliations

  1. Norwegian University of Science and Technology, Trondheim, Norway

    Erlend Aune & Jo Eidsvik

  2. University College, London, London, UK

    Yvo Pokern

Authors
  1. Erlend Aune
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jo Eidsvik
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yvo Pokern
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Erlend Aune.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Aune, E., Eidsvik, J. & Pokern, Y. Iterative numerical methods for sampling from high dimensional Gaussian distributions. Stat Comput 23, 501–521 (2013). https://doi.org/10.1007/s11222-012-9326-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-012-9326-8

Keywords

Search

Navigation