Sparse estimation of linear model via Bayesian method$^*$

Abstract

This paper considers the sparse estimation problem of regression coefficients in the linear model. Note that the global–local shrinkage priors do not allow the regression coefficients to be truly estimated as zero, we propose three threshold rules and compare their contraction properties, and also tandem those rules with the popular horseshoe prior and the horseshoe+ prior that are normally regarded as global–local shrinkage priors. The hierarchical prior expressions for the horseshoe prior and the horseshoe+ prior are obtained, and the full conditional posterior distributions for all parameters for algorithm implementation are also given. Simulation studies indicate that the horseshoe/horseshoe+ prior with the threshold rules are both superior to the spike-slab models. Finally, a real data analysis demonstrates the effectiveness of variable selection of the proposed method.

Appendix

The more details about the Gibbs sampling of the horseshore+ prior model as follows:

From the joint posterior (2.7), it is easy to obtain

$$\begin{aligned} \varvec{\beta }\big | \varvec{\lambda }, \varvec{\eta }, \tau , \sigma ^2 \sim MVN(\mu _n,\sigma ^2{\mathrm {\Lambda }_n}^{-1}), \end{aligned}$$

where $\mu _n={\mathrm {\Lambda }_n}^{-1}X^\prime \varvec{y}$, $\ \mathrm {\Lambda }_n=X^\prime X+\mathrm {\Lambda }_0$ and $\mathrm {\Lambda }_0^{-1}=\tau ^2$ diag$\{{\lambda _i}^2{\eta _i}^2\}$ with diag$\{.\}$ being the diagonal matrix whose elements are ${\lambda _i}^2{\eta _i}^2(i=1,...,p)$.

And, for the parameter $\tau$ we have

$$\begin{aligned} p_8\left( \tau |\varvec{\lambda },\varvec{\eta },\sigma ^2,\varvec{\beta }\right)&\propto p_5\left( \varvec{\beta }|\varvec{\lambda },\varvec{\eta },\tau ,\sigma ^2\right) p_2\left( \tau \right) \\&=\frac{1}{\sqrt{\left( 2\pi \right) ^p\left| \mathrm {\Sigma }_{\varvec{\beta }}\right| }} \exp \left\{ -\frac{1}{2}\varvec{\beta }^\prime {\mathrm {\Sigma }_{\varvec{\beta }}^{-1} \varvec{\beta }}\right\} \frac{2}{\pi (1+\tau ^2)}\\&\propto \ \frac{1}{\tau ^p}\exp \left\{ -\frac{1}{2} \varvec{\beta }^\prime {\mathrm {\Sigma }_{\varvec{\beta }}}^{-1}\varvec{\beta }\right\} \frac{1}{1+\tau ^2}\\&=\frac{1}{\tau ^p}\exp \left\{ -\frac{1}{2\sigma ^2\tau ^2}\sum _{i=1}^{p} \frac{{\beta _i}^2}{{\eta _i}^2{\lambda _i}^2}\right\} \frac{1}{1+\tau ^2}, \end{aligned}$$

where $\mathrm {\Sigma }_{\varvec{\beta }}=\sigma ^2{\mathrm {\Lambda }_0}^{-1}$.

Let $\gamma =\frac{1}{\tau ^2}$. Together with the above formula we obtain

$$\begin{aligned} p_9(\gamma | \varvec{\lambda }, \varvec{\eta }, \sigma ^2, \varvec{\beta } ) \propto \exp \left\{ -\frac{1}{2\sigma ^2}\left( \sum _{i=1}^{p}\frac{{\beta _i}^2}{{\eta _i}^2{\lambda _i}^2}\right) \gamma \right\} \frac{\gamma ^\frac{p-1}{2}}{\gamma +1}. \end{aligned}$$

Let ${\widetilde{\mu }}^2=\sum _{i=1}^{p}\left( \frac{\beta _i}{\lambda _i\eta _i\sigma }\right) ^2$. Using the uniform distribution, we can employ the following sampling steps to generate $\gamma$, i.e.,

$$\begin{aligned} u_{11} \sim U\left( 0,~\left( 1+\gamma \right) ^{-1}\right) \end{aligned}$$

and

$$\begin{aligned} \gamma |\varvec{\lambda },\varvec{\eta },\sigma ^2,\varvec{\beta }~ \sim Ga\left( \frac{1}{2}\left( p+1\right) ,\frac{1}{2}{\widetilde{\mu }}^2\right) I\left( \gamma <\frac{1-u_{11}}{u_{11}}\right) , \end{aligned}$$

where $I(\cdot )$ is the indicator function.

For $\eta _i$, we have

$$\begin{aligned} p_{10}\left( \eta _i |\lambda _i, \tau , \sigma ^2,\beta _i\right)&\propto p_{11}\left( \beta _i|\lambda _i,\eta _i,\tau ,\sigma ^2\right) p_4\left( \eta _i\right) \nonumber \\&\propto \frac{1}{\eta _i}\exp \left\{ \frac{-{\beta _i}^2}{2\sigma ^2\tau ^2{\eta _i}^2{\lambda _i}^2} \right\} \frac{1}{1+{\eta _i}^2}. \end{aligned}$$

Let  $\vartheta _i=\frac{1}{{\eta _i}^2}$, then $\eta _i={\vartheta _i}^{-\frac{1}{2}}$. Substituting $\eta _i={\vartheta _i}^{-\frac{1}{2}}$ into the above formula, we have

$$\begin{aligned} p_{12}(\vartheta _i|\lambda _i, \tau , \sigma ^2, \beta _i) \propto \exp \left\{ \frac{-{\beta _i}^2}{2\sigma ^2\tau ^2{\lambda _i}^2}\vartheta _i\right\} \frac{1}{\vartheta _i+1}. \end{aligned}$$

Using the uniform distribution again, we obtain

$$\begin{aligned} u_{12}\sim U\left( 0,\left( 1+\vartheta _i\right) ^{-1}\right) \end{aligned}$$

and

$$\begin{aligned} \vartheta _i|\lambda _i, \tau , \sigma ^2, \beta _i&\sim Exp\left( \frac{1}{2}\frac{{\beta _i}^2}{\sigma ^2\tau ^2{\lambda _i}^2}\right) I\left( \vartheta _i<\frac{1-u_{12}}{u_{12}}\right) . \end{aligned}$$

A similar sampling method for  $\lambda _i$  is given as follows:

$$\begin{aligned} u_{13}&\sim U\left( 0,\left( 1+\frac{1}{{\lambda _i}^2}\right) ^{-1}\right) \end{aligned}$$

and

$$\begin{aligned} \frac{1}{{\lambda _i}^2}|\eta _i, \tau , \sigma ^2, \beta _i&\sim Exp\left( \frac{1}{2}\frac{{\beta _i}^2}{\sigma ^2\tau ^2{\eta _i}^2}\right) I\left( \frac{1}{{\lambda _i}^2}<\frac{1-u_{13}}{u_{13}}\right) . \end{aligned}$$

Note that

$$\begin{aligned} p_{13}\left( \sigma ^2|\varvec{\lambda },\varvec{\eta },\tau ,\varvec{\beta }\right)&\propto p_6\left( y|\varvec{\beta },\sigma ^2\right) p_5\left( \varvec{\beta }| \varvec{\lambda },\varvec{\eta },\tau ,\sigma ^2\right) p_1\left( \sigma ^2\right) \\&\propto \sigma ^{2({-a}_1-\frac{p+n}{2}-1)}\exp {\left\{ {-\frac{1}{2\sigma ^2} \left[ (\varvec{\beta }-\mu _{n})^{\prime }\varLambda _{n}(\varvec{\beta }-\mu _{n})\right] } -\frac{b_1}{\sigma ^2}\right\} }. \end{aligned}$$

Thus, its full conditional distribution is also an inverse gamma distribution, i.e.,

$$\begin{aligned} \sigma ^2|\varvec{\lambda },\varvec{\eta },\tau ,\varvec{\beta }&\sim IG\left( a_1+\frac{n+p}{2},b_1+\frac{1}{2}\left[ (\varvec{\beta }-\mu _{n})^{\prime } \varLambda _{n}(\varvec{\beta }-\mu _{n})\right] \right) . \end{aligned}$$