Acceleration of the stochastic search variable selection via componentwise Gibbs sampling

Abstract

The stochastic search variable selection proposed by George and McCulloch (J Am Stat Assoc 88:881–889, 1993) is one of the most popular variable selection methods for linear regression models. Many efforts have been proposed in the literature to improve its computational efficiency. However, most of these efforts change its original Bayesian formulation, thus the comparisons are not fair. This work focuses on how to improve the computational efficiency of the stochastic search variable selection, but remains its original Bayesian formulation unchanged. The improvement is achieved by developing a new Gibbs sampling scheme different from that of George and McCulloch (J Am Stat Assoc 88:881–889, 1993). A remarkable feature of the proposed Gibbs sampling scheme is that, it samples the regression coefficients from their posterior distributions in a componentwise manner, so that the expensive computation of the inverse of the information matrix, which is involved in the algorithm of George and McCulloch (J Am Stat Assoc 88:881–889, 1993), can be avoided. Moreover, since the original Bayesian formulation remains unchanged, the stochastic search variable selection using the proposed Gibbs sampling scheme shall be as efficient as that of George and McCulloch (J Am Stat Assoc 88:881–889, 1993) in terms of assigning large probabilities to those promising models. Some numerical results support these findings.

Appendix

Proof of Theorem 1

The joint probability density function (pdf) of all the variables can be expressed as

$$\begin{aligned}{}[\varvec{\beta }, \sigma ^2,\varvec{\gamma }, {\mathbf {Y}}]= & {} [{\mathbf {Y}}|\varvec{\beta }, \sigma ^2,\varvec{\gamma }][\varvec{\beta }|\sigma ^2,\varvec{\gamma }][\sigma ^2,\varvec{\gamma }]\nonumber \\= & {} [{\mathbf {Y}}|\varvec{\beta }, \sigma ^2][\varvec{\beta }| \varvec{\gamma }][\varvec{\gamma }][\sigma ^2], \end{aligned}$$

(9)

where the last equality follows from the assumptions (1)–(4).

For $i=1,\ldots ,p$, the full conditional pdf of the pair $(\beta _i,\gamma _i)$ can be expressed as

$$\begin{aligned}{}[\beta _i,\gamma _i|\varvec{\beta }_{(-i)},\varvec{\gamma }_{(-i)},\sigma ^2,{\mathbf {Y}}]= [\beta _i|\gamma _i,\varvec{\beta }_{(-i)},\varvec{\gamma }_{(-i)},\sigma ^2,{\mathbf {Y}}][\gamma _i|\varvec{\beta }_{(-i)},\varvec{\gamma }_{(-i)},\sigma ^2,{\mathbf {Y}}],\nonumber \\ \end{aligned}$$

(10)

where the notation $(-i)$ means all the components except the i-th one.

Next, we derive the closed forms for the two conditional pdf’s on the right-hand side of (10). Notice that the prior distribution of $\gamma _i$ follows a Bernoulli distribution, by Tanner and Wong (1987), given other variables the conditional distribution of $\gamma _i$ follows a Bernoulli distribution as well. Thus

$$\begin{aligned} P(\gamma _i=1|\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2, {\mathbf {Y}})= & {} \frac{[{\mathbf {Y}}|\gamma _i=1,\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2][\gamma _i=1,\varvec{\beta }_{(-i)},\varvec{\gamma }_{(-i)}, \sigma ^2]}{[{\mathbf {Y}}|\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2]} \\= & {} \frac{[{\mathbf {Y}}|\gamma _i=1,\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2][\gamma _i=1]}{[{\mathbf {Y}}|\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2]}. \end{aligned}$$

Similarly,

$$\begin{aligned} P(\gamma _i=0|\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2, {\mathbf {Y}}) = \frac{[{\mathbf {Y}}|\gamma _i=0,\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2][\gamma _i=0]}{[{\mathbf {Y}}|\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2]}. \end{aligned}$$

Since $P(\gamma _i=1|\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2, {\mathbf {Y}})+ P(\gamma _i=0|\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2, {\mathbf {Y}})=1$, after some basic algebra we obtain that

$$\begin{aligned} P(\gamma _i=1|\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2, {\mathbf {Y}}) =\frac{z_i\pi _i}{(1-\pi _i)+z_i\pi _i}\quad \text{ for }\,\, i=1,\ldots ,p, \end{aligned}$$

where

$$\begin{aligned} z_i=\frac{\left[ {\mathbf {Y}}|\gamma _i=1,\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2\right] }{\left[ {\mathbf {Y}}|\gamma _i=0,\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2\right] }. \end{aligned}$$

(11)

Now we derive the closed form for $z_i$. Notice that

$$\begin{aligned} \left[ {\mathbf {Y}}|\gamma _i=1,\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2\right]= & {} \int [{\mathbf {Y}},\beta _i|\gamma _i=1,\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2]d\beta _i \\= & {} \int \left[ {\mathbf {Y}}|\beta _i,\gamma _i=1,\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2\right] \left[ \beta _i|\gamma _i \right. \\= & {} \left. 1,\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2\right] d\beta _i \\= & {} \int \left[ {\mathbf {Y}}|\varvec{\beta },\sigma ^2\right] \left[ \beta _i|\gamma _i=1\right] d\beta _i. \end{aligned}$$

The closed form of the integrand in the above equation can be calculated from (1) and (2). After some calculations, the integration yields

$$\begin{aligned}{}[{\mathbf {Y}}|\gamma _i= & {} 1,\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2]\\= & {} C \exp \{-\frac{R_i^TR_i}{2\sigma ^2}\}\exp \left\{ \frac{(\omega ^{-2}_i+c_i^{-2}\tau _i^{-2})^{-1}\omega ^{-4}_ib^2}{2}\right\} \sqrt{\frac{1}{\omega ^{-2}_ic_i^2\tau _i^2+1}}, \end{aligned}$$

where C is a normalization constant, $\omega _i^2=\sigma ^2/({\mathbf {X}}_i^T{\mathbf {X}}_i)$, and $b_i={\mathbf {X}}_i^T{\mathbf {R}}_i/({\mathbf {X}}_i^T{\mathbf {X}}_i)$ with ${\mathbf {R}}_i={\mathbf {Y}}-\sum _{j\ne i}\beta _j{\mathbf {X}}_j$. The closed form of $[{\mathbf {Y}}|\gamma _i=0,\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)}, \sigma ^2]$ can be obtained from the right-hand side of the above equation with $c_i$ being replaced by 1. After some calculations on (11), we have

$$\begin{aligned} z_i=\exp \left\{ \frac{\omega ^{-4}_ib_i^2\tau _i^{-2}(1-c_i^{-2})(\sigma _1^i)^2(\sigma _2^i)^2}{2} \right\} \sqrt{\frac{\omega ^{-2}_i+\tau _i^{-2}}{c_i^2\omega ^{-2}_i+\tau _i^{-2}}}, \end{aligned}$$

where $(\sigma _1^i)^2=(\omega ^{-2}_i+c_i^{-2}\tau _i^{-2})^{-1}$ and $(\sigma _2^i)^2=(\omega ^{-2}_i+\tau _i^{-2})^{-1}$. From (9), we know that $[\beta _i|\gamma _i,\varvec{\beta }_{(-i)}, \varvec{\gamma }_{(-i)},\sigma ^2,{\mathbf {Y}}] \propto [{\mathbf {Y}}|\beta _i, \varvec{\beta }_{(-i)}, \sigma ^2][\beta _i|\gamma _i]$, which can be calculated from (1) and (2). In particular,

$$\begin{aligned}{}[\beta _i|\gamma _i,\varvec{\beta }_{(-i)},\varvec{\gamma }_{(-i)},\sigma ^2,{\mathbf {Y}}] \sim \left\{ \begin{array}{ll} N\left( (\sigma _1^i)^2\omega _i^{-2}b_i, (\sigma _1^i)^2\right) &{} \text{ when } \gamma _i=1;\\ N\left( (\sigma _2^i)^2\omega _i^{-2}b_i, (\sigma _2^i)^2\right) &{} \text{ when } \gamma _i=0. \end{array} \right. \end{aligned}$$

Finally, by the same argument in George and McCulloch (1993), we have

$$\begin{aligned}{}[\sigma ^2|\varvec{\beta }, \varvec{\gamma },{\mathbf {Y}}]\sim \mathrm {IG}\left( \frac{n+v}{2}, \frac{({\mathbf {Y}}-{\mathbf {X}}\varvec{\beta })^T({\mathbf {Y}}-{\mathbf {X}}\varvec{\beta })+v\lambda }{2}\right) , \end{aligned}$$

where ${\mathrm {IG}}$ denotes an inverted gamma distribution. Then from the well-known relation between the inverted gamma distribution and the chi-square distribution (cf., Wu and Hamada 2009), we conclude that

$$\begin{aligned} \left[ (v\lambda +({\mathbf {Y}}-{\mathbf {X}}\varvec{\beta })^T({\mathbf {Y}}-{\mathbf {X}}\varvec{\beta }))/\sigma ^2|\varvec{\beta }, \varvec{\gamma }, {\mathbf {Y}}\right] \sim \chi ^2(v+n). \end{aligned}$$

$\square$