Skip to main content

Hierarchical-block conditioning approximations for high-dimensional multivariate normal probabilities

Abstract

This paper presents a new method to estimate large-scale multivariate normal probabilities. The approach combines a hierarchical representation with processing of the covariance matrix that decomposes the n-dimensional problem into a sequence of smaller m-dimensional ones. It also includes a d-dimensional conditioning method that further decomposes the m-dimensional problems into smaller d-dimensional problems. The resulting two-level hierarchical-block conditioning method requires Monte Carlo simulations to be performed only in d dimensions, with \(d \ll n\), and allows the complexity of the algorithm’s major cost to be \(O(n \log n)\). The run-time cost of the method depends on two parameters, m and d, where m represents the diagonal block size and controls the sizes of the blocks of the covariance matrix that are replaced by low-rank approximations, and d allows a trade-off of accuracy for expensive computations in the evaluation of the probabilities of m-dimensional blocks. We also introduce an inexpensive block reordering strategy to provide improved accuracy in the overall probability computation. The downside of this method, as with other such conditioning approximations, is the absence of an internal estimate of its error to use in tuning the approximation. Numerical simulations on problems from 2D spatial statistics with dimensions up to 16,384 indicate that the algorithm achieves a \(1\%\) error level and improves the run time over a one-level hierarchical Quasi-Monte Carlo method by a factor between 10 and 15.

This is a preview of subscription content, access via your institution.

Fig. 1

References

  • Anderson, E., Bai, Z., Bischof, C., Blackford, S., Demmel, J., Dongarra, J., Du Croz, J., Greenbaum, A., Hammerling, S., McKenney, A., et al.: LAPACK Users’ Guide, 3rd edn. SIAM Publications, Philadelphia (1999)

    Book  MATH  Google Scholar 

  • Azzalini, A., Capitanio, A.: The Skew-Normal and Related Families. Cambridge University Press, Cambridge (2014)

    MATH  Google Scholar 

  • Brown, B.M., Resnick, S.I.: Extreme values of independent stochastic processes. J. Appl. Probab. 14, 732–739 (1977)

    MathSciNet  Article  MATH  Google Scholar 

  • Caflisch, R.E.: Monte Carlo and quasi-Monte Carlo methods. Acta Numer. 7, 1–49 (1998)

    MathSciNet  Article  MATH  Google Scholar 

  • Connors, R.D., Hess, S., Daly, A.: Analytic approximations for computing probit choice probabilities. Transp. A Transp. Sci. 10, 119–139 (2014)

    Google Scholar 

  • Genton, M.G.: Skew-Elliptical Distributions and Their Applications: A Journey Beyond Normality. CRC Press, Boca Raton (2004)

    Book  MATH  Google Scholar 

  • Genton, M.G., Keyes, D.E., Turkiyyah, G.M.: Hierarchical decompositions for the computation of high-dimensional multivariate normal probabilities. J. Comput. Graph. Stat. 27, 268–277 (2018)

    MathSciNet  Article  Google Scholar 

  • Genz, A.: Numerical computation of multivariate normal probabilities. J. Comput. Graph. Stat. 1, 141–149 (1992)

    Google Scholar 

  • Genz, A., Bretz, F.: Computation of Multivariate Normal and t Probabilities, vol. 195. Springer, Berlin (2009)

    MATH  Google Scholar 

  • Gupta, R.C., Brown, N.: Reliability studies of the skew-normal distribution and its application to a strength–stress model. Commun. Stat. Theory Methods 30, 2427–2445 (2001)

    MathSciNet  Article  MATH  Google Scholar 

  • Hackbusch, W.: Hierarchical Matrices: Algorithms and Analysis, vol. 49. Springer, Berlin (2015)

    Book  MATH  Google Scholar 

  • Kamakura, W.A.: The estimation of multinomial probit models: a new calibration algorithm. Transp. Sci. 23, 253–265 (1989)

    Article  MATH  Google Scholar 

  • Kan, R., Robotti, C.: On moments of folded and truncated multivariate normal distributions. J. Comput. Graph. Stat. 26(4), 930–934 (2017)

    MathSciNet  Article  Google Scholar 

  • Kotz, S., Nadarajah, S.: Multivariate t-Distributions and Their Applications. Cambridge University Press, Cambridge (2004)

    Book  MATH  Google Scholar 

  • Mendell, N.R., Elston, R.: Multifactorial qualitative traits: genetic analysis and prediction of recurrence risks. Biometrics 30, 41–57 (1974)

    MathSciNet  Article  Google Scholar 

  • Morton, G.M.: A Computer Oriented Geodetic Data Base and a New Technique in File Sequencing. International Business Machines Company, New York (1966)

    Google Scholar 

  • Muthen, B.: Moments of the censored and truncated bivariate normal distribution. Br. J. Math. Stat. Psychol. 43, 131–143 (1990)

    MathSciNet  Article  MATH  Google Scholar 

  • Niederreiter, H.: New methods for pseudorandom numbers and pseudorandom vector generation. In: Proceedings of the 24th Conference on Winter Simulation, New York, NY, USA, WSC ’92, pp. 264–269. ACM (1992)

  • Owen, A., Zhou, Y.: Safe and effective importance sampling. J. Am. Stat. Assoc. 95, 135–143 (2000)

    MathSciNet  Article  MATH  Google Scholar 

  • R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2016)

    Google Scholar 

  • Schlather, M.: Models for stationary max-stable random fields. Extremes 5, 33–44 (2002)

    MathSciNet  Article  MATH  Google Scholar 

  • Smith, R., Tawn, J., Yuen, H.: Statistics of multivariate extremes. Int. Stat. Rev. 58, 47–58 (1990)

    Article  MATH  Google Scholar 

  • Stewart, G.W.: The efficient generation of random orthogonal matrices with an application to condition estimators. SIAM J. Numer. Anal. 17, 403–409 (1980)

    MathSciNet  Article  MATH  Google Scholar 

  • Trinh, G., Genz, A.: Bivariate conditioning approximations for multivariate normal probabilities. Stat. Comput. 25, 989–996 (2015)

    MathSciNet  Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jian Cao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research was supported by King Abdullah University of Science and Technology (KAUST).

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Cao, J., Genton, M.G., Keyes, D.E. et al. Hierarchical-block conditioning approximations for high-dimensional multivariate normal probabilities. Stat Comput 29, 585–598 (2019). https://doi.org/10.1007/s11222-018-9825-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-018-9825-3

Keywords

  • Block reordering
  • d-Dimensional conditioning
  • Hierarchical representation
  • Spatial covariance functions
  • Univariate reordering