Abstract
We consider Bayesian variable selection in sparse high-dimensional regression, where the number of covariates p may be large relative to the sample size n, but at most a moderate number q of covariates are active. Specifically, we treat generalized linear models. For a single fixed sparse model with well-behaved prior distribution, classical theory proves that the Laplace approximation to the marginal likelihood of the model is accurate for sufficiently large sample size n. We extend this theory by giving results on uniform accuracy of the Laplace approximation across all models in a high-dimensional scenario in which p and q, and thus also the number of considered models, may increase with n. Moreover, we show how this connection between marginal likelihood and Laplace approximation can be used to obtain consistency results for Bayesian approaches to variable selection in high-dimensional regression.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
In the proof of this theorem, we cite several results from Sect. B.2 and Lemma B.1 in [2]. Although that paper treats the specific case of logistic regression, by examining the proofs of their results that we cite here, we can see that they hold more broadly for the general GLM case as long as we assume that the Hessian conditions hold, i.e., Conditions (A1)–(A3), and therefore we may use these results for the setting considered here.
References
Anderson, T.W.: An Introduction to Multivariate Statistical Analysis. Wiley Series in Probability and Statistics, 3rd edn. Wiley-Interscience, Hoboken, NJ (2003)
Barber, R.F., Drton, M.: High-dimensional Ising model selection with Bayesian information criteria. Electron. J. Stat. 9, 567–607 (2015)
Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, New York (2006)
Bogdan, M., Ghosh, J.K., Doerge, R.: Modifying the Schwarz Bayesian information criterion to locate multiple interacting quantitative trait loci. Genetics 167(2), 989–999 (2004)
Chen, J., Chen, Z.: Extended Bayesian information criteria for model selection with large model spaces. Biometrika 95(3), 759–771 (2008)
Chen, J., Chen, Z.: Extended BIC for small-n-large-P sparse GLM. Stat. Sinica 22(2), 555–574 (2012)
Friel, N., Wyse, J.: Estimating the evidence—a review. Statistica Neerlandica 66(3), 288–308 (2012)
Frommlet, F., Ruhaltinger, F., Twaróg, P., Bogdan, M.: Modified versions of Bayesian information criterion for genome-wide association studies. Comput. Stat. Data Anal. 56(5), 1038–1051 (2012)
Haughton, D.M.A.: On the choice of a model to fit data from an exponential family. Ann. Stat. 16(1), 342–355 (1988)
Laurent, B., Massart, P.: Adaptive estimation of a quadratic functional by model selection. Ann. Stat. 28(5), 1302–1338 (2000)
Luo, S., Chen, Z.: Selection consistency of EBIC for GLIM with non-canonical links and diverging number of parameters. Stat. Interface 6(2), 275–284 (2013)
Luo, S., Xu, J., Chen, Z.: Extended Bayesian information criterion in the Cox model with a high-dimensional feature space. Ann. Inst. Stat. Math. 67(2), 287–311 (2015)
McCullagh, P., Nelder, J.A.: Generalized Linear Models. Monographs on Statistics and Applied Probability, 2nd edn. Chapman & Hall, London (1989)
Natalini, P., Palumbo, B.: Inequalities for the incomplete gamma function. Math. Inequal. Appl. 3(1), 69–77 (2000)
Schwarz, G.: Estimating the dimension of a model. Ann. Stat. 6(2), 461–464 (1978)
Scott, J.G., Berger, J.O.: Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem. Ann. Stat. 38(5), 2587–2619 (2010)
Żak-Szatkowska, M., Bogdan, M.: Modified versions of the Bayesian information criterion for sparse generalized linear models. Comput. Stat. Data Anal. 55(11), 2908–2924 (2011)
Zhang, F. (ed.): The Schur Complement and Its Applications. Numerical Methods and Algorithms, vol. 4. Springer, New York (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Barber, R.F., Drton, M., Tan, K.M. (2016). Laplace Approximation in High-Dimensional Bayesian Regression. In: Frigessi, A., Bühlmann, P., Glad, I., Langaas, M., Richardson, S., Vannucci, M. (eds) Statistical Analysis for High-Dimensional Data. Abel Symposia, vol 11. Springer, Cham. https://doi.org/10.1007/978-3-319-27099-9_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-27099-9_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-27097-5
Online ISBN: 978-3-319-27099-9
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)