Laplace Approximation in High-Dimensional Bayesian Regression

  • Rina Foygel Barber
  • Mathias Drton
  • Kean Ming Tan
Conference paper
Part of the Abel Symposia book series (ABEL, volume 11)


We consider Bayesian variable selection in sparse high-dimensional regression, where the number of covariates p may be large relative to the sample size n, but at most a moderate number q of covariates are active. Specifically, we treat generalized linear models. For a single fixed sparse model with well-behaved prior distribution, classical theory proves that the Laplace approximation to the marginal likelihood of the model is accurate for sufficiently large sample size n. We extend this theory by giving results on uniform accuracy of the Laplace approximation across all models in a high-dimensional scenario in which p and q, and thus also the number of considered models, may increase with n. Moreover, we show how this connection between marginal likelihood and Laplace approximation can be used to obtain consistency results for Bayesian approaches to variable selection in high-dimensional regression.


  1. 1.
    Anderson, T.W.: An Introduction to Multivariate Statistical Analysis. Wiley Series in Probability and Statistics, 3rd edn. Wiley-Interscience, Hoboken, NJ (2003)Google Scholar
  2. 2.
    Barber, R.F., Drton, M.: High-dimensional Ising model selection with Bayesian information criteria. Electron. J. Stat. 9, 567–607 (2015)CrossRefMathSciNetzbMATHGoogle Scholar
  3. 3.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, New York (2006)zbMATHGoogle Scholar
  4. 4.
    Bogdan, M., Ghosh, J.K., Doerge, R.: Modifying the Schwarz Bayesian information criterion to locate multiple interacting quantitative trait loci. Genetics 167(2), 989–999 (2004)CrossRefGoogle Scholar
  5. 5.
    Chen, J., Chen, Z.: Extended Bayesian information criteria for model selection with large model spaces. Biometrika 95(3), 759–771 (2008)CrossRefMathSciNetzbMATHGoogle Scholar
  6. 6.
    Chen, J., Chen, Z.: Extended BIC for small-n-large-P sparse GLM. Stat. Sinica 22(2), 555–574 (2012)zbMATHGoogle Scholar
  7. 7.
    Friel, N., Wyse, J.: Estimating the evidence—a review. Statistica Neerlandica 66(3), 288–308 (2012)CrossRefMathSciNetGoogle Scholar
  8. 8.
    Frommlet, F., Ruhaltinger, F., Twaróg, P., Bogdan, M.: Modified versions of Bayesian information criterion for genome-wide association studies. Comput. Stat. Data Anal. 56(5), 1038–1051 (2012)CrossRefGoogle Scholar
  9. 9.
    Haughton, D.M.A.: On the choice of a model to fit data from an exponential family. Ann. Stat. 16(1), 342–355 (1988)CrossRefMathSciNetzbMATHGoogle Scholar
  10. 10.
    Laurent, B., Massart, P.: Adaptive estimation of a quadratic functional by model selection. Ann. Stat. 28(5), 1302–1338 (2000)CrossRefMathSciNetzbMATHGoogle Scholar
  11. 11.
    Luo, S., Chen, Z.: Selection consistency of EBIC for GLIM with non-canonical links and diverging number of parameters. Stat. Interface 6(2), 275–284 (2013)CrossRefMathSciNetGoogle Scholar
  12. 12.
    Luo, S., Xu, J., Chen, Z.: Extended Bayesian information criterion in the Cox model with a high-dimensional feature space. Ann. Inst. Stat. Math. 67(2), 287–311 (2015)CrossRefMathSciNetGoogle Scholar
  13. 13.
    McCullagh, P., Nelder, J.A.: Generalized Linear Models. Monographs on Statistics and Applied Probability, 2nd edn. Chapman & Hall, London (1989)Google Scholar
  14. 14.
    Natalini, P., Palumbo, B.: Inequalities for the incomplete gamma function. Math. Inequal. Appl. 3(1), 69–77 (2000)MathSciNetGoogle Scholar
  15. 15.
    Schwarz, G.: Estimating the dimension of a model. Ann. Stat. 6(2), 461–464 (1978)CrossRefzbMATHGoogle Scholar
  16. 16.
    Scott, J.G., Berger, J.O.: Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem. Ann. Stat. 38(5), 2587–2619 (2010)CrossRefMathSciNetzbMATHGoogle Scholar
  17. 17.
    Żak-Szatkowska, M., Bogdan, M.: Modified versions of the Bayesian information criterion for sparse generalized linear models. Comput. Stat. Data Anal. 55(11), 2908–2924 (2011)CrossRefzbMATHGoogle Scholar
  18. 18.
    Zhang, F. (ed.): The Schur Complement and Its Applications. Numerical Methods and Algorithms, vol. 4. Springer, New York (2005)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Rina Foygel Barber
    • 1
  • Mathias Drton
    • 2
  • Kean Ming Tan
    • 3
  1. 1.Department of StatisticsThe University of ChicagoChicagoUSA
  2. 2.Department of StatisticsUniversity of WashingtonSeattleUSA
  3. 3.Department of BiostatisticsUniversity of WashingtonSeattleUSA

Personalised recommendations