Abstract
This article proposes a stochastic version of the matching pursuit algorithm for Bayesian variable selection in linear regression. In the Bayesian formulation, the prior distribution of each regression coefficient is assumed to be a mixture of a point mass at 0 and a normal distribution with zero mean and a large variance. The proposed stochastic matching pursuit algorithm is designed for sampling from the posterior distribution of the coefficients for the purpose of variable selection. The proposed algorithm can be considered a modification of the componentwise Gibbs sampler. In the componentwise Gibbs sampler, the variables are visited by a random or a systematic scan. In the stochastic matching pursuit algorithm, the variables that better align with the current residual vector are given higher probabilities of being visited. The proposed algorithm combines the efficiency of the matching pursuit algorithm and the Bayesian formulation with well defined prior distributions on coefficients. Several simulated examples of small n and large p are used to illustrate the algorithm. These examples show that the algorithm is efficient for screening and selecting variables.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Barbieri, M., Berger, J.O.: Optimal predictive model selection. Ann. Stat. 32, 870–897 (2004)
Beattie, S.D., Fong, D.K.H., Lin, D.K.J.: A two-stage Bayesian model selection strategy for supersaturated designs. Technometrics 44, 55–63 (2002)
Chipman, H.: Bayesian variable selection with related predictors. Can. J. Stat. 24, 17–36 (1996)
Chipman, H., Hamada, M., Wu, C.F.J.: A Bayesian variable selection approach for analyzing designed experiments with complex aliasing. Technometrics 39, 372–381 (1997)
Févotte, C., Godsill, S.J.: Sparse linear regression in unions of bases via Bayesian variable selection. IEEE Signal Process. Lett. 13, 441–444 (2006)
George, E.I., McCulloch, R.E.: Variable selection via Gibbs sampling. J. Am. Stat. Assoc. 88, 881–889 (1993)
George, E.I., McCulloch, R.E.: Approaches for Bayesian variable selection. Stat. Sin. 7, 339–374 (1997)
Geweke, J.: Variable selection and model comparison in regression. In: Bernardo, J.M., Berger, J.O., Dawid, A.P., Smith, A.F.M. (eds.) Bayesian Statistics, vol. 5, pp. 609–620. Oxford Press, Oxford (1996)
Lai, T.-W.: Variable selection via MCMC matching pursuit. M.S. Thesis, Institute of Statistics, National University of Kaohsiung, Kaohsiung, Taiwan (2007)
Lee, K.E., Sha, N., Dougherty, E.R., Vannucci, M., Mallick, B.: Gene selection: a Bayesian variable selection approach. Bioinformatics 19, 90–97 (2003)
Mallat, S.G., Zhang, Z.: Matching pursuit with time-frequency dictionaries. IEEE Trans. Signal Process. 41, 3397–3415 (1993)
Shao, J., Chow, S.-C.: Variable screening in predicting clinical outcome with high-dimensional microarrays. J. Multivar. Anal. 98, 1529–1538 (2007)
Smith, M., Kohn, R.: Nonparametric regression using Bayesian variable selection. J. Econom. 75, 317–343 (1996)
Tibshirani, R.: Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. B 58, 267–288 (1996)
Wolfe, P.J., Godsill, S.J., Ng, W.J.: Bayesian variable selection and regularization for time-frequency surface estimation. J. R. Stat. Soc. B 66, 575–589 (2004)
Wu, Y.N., Zhu, S.C., Guo, C.: Statistical modeling of texture sketch. In: Proceedings of European Conference of Computer Vision, pp. 240–254 (2002)
Yi, N., George, V., Allison, D.B.: Stochastic search variable selection for identifying multiple quantitative trait loci. Genetics 164, 1129–1138 (2003)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Chen, RB., Chu, CH., Lai, TY. et al. Stochastic matching pursuit for Bayesian variable selection. Stat Comput 21, 247–259 (2011). https://doi.org/10.1007/s11222-009-9165-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11222-009-9165-4