Skip to main content
SpringerLink
Log in

Bayesian Variable Selection

  • Chapter
  • First Online:
Case Studies in Applied Bayesian Data Science

Part of the book series: Lecture Notes in Mathematics ((LNM,volume 2259))

Abstract

In this chapter we survey Bayesian approaches for variable selection and model choice in regression models. We explore the methodological developments and computational approaches for these methods. In conclusion we note the available software for their implementation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. T.J. Mitchell, J.J. Beauchamp, Bayesian variable selection in linear regression. J. Am. Stat. Assoc. 83(404), 1023–1032 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  2. M. Barbieri, J.O. Berger, E.I. George, V. Rockova, The median probability model and correlated variables. arXiv:1807.08336 (2020)

    Google Scholar 

  3. M.M. Barbieri, J.O. Berger, Optimal predictive model selection. Ann. Stat. 32(3), 870–897 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  4. F. Liang, Q. Song, K. Yu, Bayesian subset modeling for high-dimensional generalized linear models. J. Am. Stat. Assoc. 108(502), 589–606 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  5. W. Jiang, Bayesian variable selection for high dimensional generalized linear models: convergence rates of the fitted densities. Ann. Stat. 35(4), 1487–1511 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  6. E.I. George, R.E. McCulloch, Variable selection via Gibbs sampling. J. Am. Stat. Assoc. 88(423), 881–889 (1993)

    Article  Google Scholar 

  7. M. Smith, R. Kohn, A Bayesian approach to nonparametric bivariate regression. J. Am. Stat. Assoc. 92(440), 1522–1535 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  8. I. Castillo, J. Schmidt-Hieber, A. van der Vaart, Bayesian linear regression with sparse priors. Ann. Stat. 43(5), 1986–2018 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  9. J.G. Scott, J.O. Berger, Bayes and empirical-bayes multiplicity adjustment in the variable-selection problem. Ann. Stat. 38(5), 2587–2619 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  10. F. Li, N.R. Zhang, Bayesian variable selection in structured high-dimensional covariate spaces with applications in genomics. J. Am. Stat. Assoc. 105(491), 1202–1214 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  11. M.E.J. Newman, G.T. Barkema, Monte Carlo Methods in Statistical Physics (Clarendon Press, Oxford, 1999)

    MATH  Google Scholar 

  12. E.E. Leamer, Specification Searches: Ad hoc Inference with Nonexperimental Data, vol. 53 (Wiley, Hoboken, 1978)

    MATH  Google Scholar 

  13. I. Castillo, R. Mismer, Empirical bayes analysis of spike and slab posterior distributions. Electron. J. Stat. 12, 3953–4001 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  14. R.B. O’Hara, M.J. Sillanpää, A review of Bayesian variable selection methods: what, how and which. Bayesian Anal. 4(1), 85–117 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  15. L. Kuo, B. Mallick, Variable selection for regression models. Sankhyā Indian J. Stat. Ser. B (1960–2002) 60(1), 65–81 (1998)

    Google Scholar 

  16. H. Chipman, E.I. George, R.E. McCulloch, The Practical Implementation of Bayesian Model Selection. Lecture Notes–Monograph Series, vol. 38 (Institute of Mathematical Statistics, Beachwood, 2001), pp. 65–116. https://doi.org/10.1214/lnms/1215540964

    Google Scholar 

  17. P. Dellaportas, J.J. Forster, I. Ntzoufras, Bayesian variable selection using the Gibbs sampler. BIOSTATISTICS-BASEL- 5, 273–286 (2000)

    MathSciNet  MATH  Google Scholar 

  18. B.P. Carlin, S. Chib, Bayesian model choice via Markov chain Monte Carlo methods. J. R. Stat. Soc. Ser. B Stat. Methodol. 57(3), 473–484 (1995)

    MATH  Google Scholar 

  19. N.N. Narisetty, X. He, Bayesian variable selection with shrinking and diffusing priors. Ann. Stat. 42(2), 789–817 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  20. H. Ishwaran, J.S. Rao, Detecting differentially expressed genes in microarrays using Bayesian model selection. J. Am. Stat. Assoc. 98(462), 438–455 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  21. L. Fahrmeir, T. Kneib, S. Konrath, Bayesian regularisation in structured additive regression: a unifying perspective on shrinkage, smoothing and predictor selection. Stat. Comput. 20(2), 203–219 (2010)

    Article  MathSciNet  Google Scholar 

  22. V. Ročková, E.I. George, The spike-and-slab lasso. J. Am. Stat. Assoc. 113(521), 431–444 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  23. V.E. Johnson, D. Rossell, Bayesian model selection in high-dimensional settings. J. Am. Stat. Assoc. 107(498), 649–660 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  24. D. Rossell, D. Telesca, Non-local priors for high-dimensional estimation. J. Am. Stat. Assoc. 112(517), 254–265 (2017)

    Article  Google Scholar 

  25. A. Nikooienejad, W. Wang, V.E. Johnson, Bayesian variable selection for binary outcomes in high-dimensional genomic studies using non-local priors. Bioinformatics 32(9), 1338–1345 (2016)

    Article  Google Scholar 

  26. R. Tibshirani, Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58(1), 267–288 (1996)

    MathSciNet  MATH  Google Scholar 

  27. J. Fan, J. Lv, A selective overview of variable selection in high dimensional feature space. Stat. Sin. 20(1), 101–148 (2010)

    MathSciNet  MATH  Google Scholar 

  28. N.G. Polson, J.G. Scott, Local shrinkage rules, lévy processes and regularized regression. J. R. Stat. Soc. Ser. B Stat. Methodol. 74(2), 287–311 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  29. J.E. Griffin, P.J. Brown, Inference with normal-gamma prior distributions in regression problems. Bayesian Anal. 5(1), 171–188 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  30. C.M. Carvalho, N.G. Polson, J.G. Scott, The horseshoe estimator for sparse signals. Biometrika 97(2), 465–480 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  31. A. Armagan, D.B. Dunson, J. Lee, Generalized double pareto shrinkage. Stat. Sin. 23(1), 119–143 (2013)

    MathSciNet  MATH  Google Scholar 

  32. A. Bhattacharya, D. Pati, N.S. Pillai, D.B. Dunson, Dirichlet–laplace priors for optimal shrinkage. J. Am. Stat. Assoc. 110(512), 1479–1490 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  33. A. Bhadra, J. Datta, N.G. Polson, B. Willard, The horseshoe+ estimator of ultra-sparse signals. Bayesian Anal. 12(4), 1105–1131 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  34. P. Ghosh, X. Tang, M. Ghosh, A. Chakrabarti, Asymptotic properties of bayes risk of a general class of shrinkage priors in multiple hypothesis testing under sparsity. Bayesian Anal. 11(3), 753–796 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  35. R. Bai, M. Ghosh, High-dimensional multivariate posterior consistency under global–local shrinkage priors. J. Multivar. Anal. 167, 157–170 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  36. S. van der Pas, B. Szabó, A. van der Vaart, Uncertainty quantification for the horseshoe (with discussion). Bayesian Anal. 12(4), 1221–1274 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  37. P.R. Hahn, C.M. Carvalho, Decoupling shrinkage and selection in Bayesian linear models: a posterior summary perspective. J. Am. Stat. Assoc. 110(509), 435–448 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  38. D. Madigan, J. York, D. Allard, Bayesian graphical models for discrete data. Int. Stat. Rev./Rev. Int. de Stat. 63(2), 215–232 (1995)

    MATH  Google Scholar 

  39. C. Hans, A. Dobra, M. West, Shotgun stochastic search for “large p” regression. J. Am. Stat. Assoc. 102(478), 507–516 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  40. F. Liang, W.H. Wong, Evolutionary monte carlo: applications to C p model sampling and change point problem. Stat. Sin. 10(2), 317–342 (2000)

    MATH  Google Scholar 

  41. L. Bottolo, S. Richardson, Evolutionary stochastic search for Bayesian model exploration. Bayesian Anal. 5(3), 583–618 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  42. G. Zanella, G. Roberts, Scalable importance tempering and Bayesian variable selection. J. R. Statist. Soc. B 81, 489–517 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  43. R Core Team, R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna, 2013)

    Google Scholar 

  44. I. Ntzoufras, Gibbs variable selection usingbugs. J. Stat. Softw. 7(7), 1–19 (2002)

    Article  Google Scholar 

  45. D.J. Lunn, A. Thomas, N. Best, D. Spiegelhalter, Winbugs-a Bayesian modelling framework: concepts, structure, and extensibility. Stat. Comput. 10(4), 325–337 (2000)

    Article  Google Scholar 

  46. M. Plummer, et al., JAGS: A program for analysis of Bayesian graphical models using gibbs sampling, in Proceedings of the 3rd International Workshop on Distributed Statistical Computing, vol. 124 (2003)

    Google Scholar 

  47. B. Carpenter, A. Gelman, M.D. Hoffman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, A. Riddell, Stan: A probabilistic programming language. J. Stat. Softw. 76(1), 1–32 (2017)

    Article  Google Scholar 

  48. J. Piironen, A. Vehtari, Projection predictive model selection for gaussian processes, in 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP) (2016), pp. 1–6

    Google Scholar 

  49. A. Forte, G. Garcia-Donato, M. Steel, Methods and tools for Bayesian variable selection and model averaging in normal linear regression. Int. Stat. Rev./Rev. Int. de Stat. 86(2), 237–258 (2018)

    Article  MathSciNet  Google Scholar 

  50. G. Csárdi, pkgsearch: Search CRAN R Packages. R package version 2.0.1. (2018). https://CRAN.R-project.org/package=pkgsearch

  51. H. Ishwaran, U.B. Kogalur, J.S. Rao, spikeslab: prediction and variable selection using spike and slab regression. R J. 2, 68–73 (2010)

    Article  Google Scholar 

  52. P. Carbonetto, M. Stephens, Scalable variational inference for Bayesian variable selection in regression, and its accuracy in genetic association studies. Bayesian Anal. 7, 73–108 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  53. D. Rossell, J.D. Cook, D. Telesca, P. Roebuck, mombf: moment and inverse moment bayes factors. R Package Version 1. 0, vol. 3 (2008)

    Google Scholar 

Download references

Acknowledgement

The author would like to acknowledge the Australian Research Council Centre of Excellence in Mathematical and Statistical Frontiers for funding.

Author information

Authors and Affiliations

  1. Queensland University of Technology, Brisbane, QLD, Australia

    Matthew Sutton

Authors
  1. Matthew Sutton
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matthew Sutton .

Editor information

Editors and Affiliations

  1. Mathematical Sciences, Queensland University of Technology, Brisbane, QLD, Australia

    Kerrie L. Mengersen

  2. I2M, CNRS, Centrale Marseille, Aix-Marseille University, Marseille, France

    Pierre Pudlo

  3. CEREMADE, Université Paris Dauphine, Paris, France

    Christian P. Robert

Rights and permissions

Reprints and permissions

Copyright information

© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Sutton, M. (2020). Bayesian Variable Selection. In: Mengersen, K., Pudlo, P., Robert, C. (eds) Case Studies in Applied Bayesian Data Science. Lecture Notes in Mathematics, vol 2259. Springer, Cham. https://doi.org/10.1007/978-3-030-42553-1_5

Download citation

Publish with us

Policies and ethics

Search

Navigation