Bayesian structured variable selection in linear regression models

Abstract

In this paper we consider the Bayesian approach to the problem of variable selection in normal linear regression models with related predictors. We adopt a generalized singular $g$-prior distribution for the unknown model parameters and the beta-prime prior for the scaling factor $g$, which results in a closed-form expression of the marginal posterior distribution without integral representation. A special prior on the model space is then advocated to reflect and maintain the hierarchical or structural relationships among predictors. It is shown that under some nominal assumptions, the proposed approach is consistent in terms of model selection and prediction. Simulation studies show that our proposed approach has a good performance for structured variable selection in linear regression models. Finally, a real-data example is analyzed for illustrative purposes.

Appendix

Proof of Theorem 2

The well-known Stirling’s formula is the asymptotic relation given by

$$\begin{aligned} \Gamma {(c_{1}x +c_{2})} \approx \sqrt{2\pi }e^{-c_{1}x}(c_{1}x)^{c_{1}x + c_{2} - 1/2}, \end{aligned}$$

(28)

when $x$ is sufficiently large. Here, ‘$f \approx g$’ means that the ratio of the two sides approaches 1 as $x$ goes to infinity. For our problem, when $n$ approaches infinity, it may be verified that

$$\begin{aligned} \frac{\Gamma {\big ((n+a_{0} - m_{\gamma '})/2\big )}}{\Gamma {\big ((n+a_{0} - m_{\gamma })/2\big )}} \approx \biggl (\frac{n}{2}\biggr )^{(m_{\gamma }-m_{\gamma '})/2}. \end{aligned}$$

(29)

For comparing an arbitrary model $M_{\gamma '}$ and the true model $M_\gamma$, we assume the design matrix of the linear models satisfies the assumption in (22). Then the posterior consistency for model selection means that when sampling from $M_\gamma$, it follows that

$$\begin{aligned} \mathop {\hbox {plim}}\limits _{n\rightarrow \infty }{\frac{p(M_{\gamma '} \mid \mathbf{Y})}{p(M_{\gamma } \mid \mathbf{Y})}} = \mathop {\hbox {plim}}\limits _{n\rightarrow \infty }\frac{f(\gamma '\mid \mathbf{Y})}{f(\gamma \mid \mathbf{Y})}= 0, \end{aligned}$$

(30)

where the model $M_{\gamma '} \ne M_{\gamma }$ and $f(\gamma \mid \mathbf{Y})$ is given by Eq. (16).

In what follows, for simplicity of notation, let $c_i$ be some constants independent of the sample size $n$ for $i =1, \ldots , 4$. It can be seen from Lemma A. 1 of Fernández et al. (2001) that when sampling from the model $M_\gamma$ which is nested within or equal to model $M_{\gamma '}$, we have

$$\begin{aligned} \mathop {\hbox {plim}}\limits _{n\rightarrow \infty } \frac{\mathbf{Y}'(\mathbf{I}_n - \mathbf{H}_{\gamma '})\mathbf{Y}}{n} = \sigma ^2, \end{aligned}$$

and that when sampling from the model $M_{\gamma '}$ which does not nest $M_{\gamma }$, we have

$$\begin{aligned} \mathop {\hbox {plim}}\limits _{n\rightarrow \infty } \frac{\mathbf{Y}'(\mathbf{I}_n - \mathbf{H}_{\gamma '})\mathbf{Y}}{n} = \sigma ^2 + c_{\gamma '}, \end{aligned}$$

where $c_{\gamma '}$ is given by Eq. (22). To show the consistency of model selection, we now consider the following two situations.

• (a) If $M_\gamma \not \subseteq M_{\gamma '}$, then by using the relationship in (29), as $n$ approaches infinity, Eq. (30) can be written as

  $$\begin{aligned} \mathop {\hbox {plim}}\limits _{n\rightarrow \infty }\frac{f(\gamma '\mid \mathbf{Y})}{f(\gamma \mid \mathbf{Y})}&\,{=}\, \frac{\Gamma {\Bigl (\frac{m_{\gamma '} {+} 2a {+} 2}{2}\Bigr )}\Gamma {\Bigl (\frac{n + a_0 - m_{\gamma '}}{2}\Bigr )}\bigl (1 - \widetilde{R}^{2}_{\gamma '}\bigr )^{-(n+ a_ 0 - m_{\gamma '})/2 + a + 1}\pi ({\gamma '})}{\Gamma {\Bigl (\frac{m_\gamma + 2a + 2}{2}\Bigr )}\Gamma {\Bigl (\frac{n + a_0 - m_\gamma }{2}\Bigr )}\bigl (1 - \widetilde{R}^{2}_\gamma \bigr )^{-(n+ a_ 0 -m_\gamma )/2 + a +1}\pi (\gamma )}\\&= c_1 \mathop {\hbox {plim}}\limits _{n\rightarrow \infty } n^{(m_{\gamma }-m_{\gamma '})/2} \biggl (\frac{1 - \tilde{R}^2_{\gamma '}}{1 - \tilde{R}^2_{\gamma }}\biggr )^{-n/2}\\&= c_2 \mathop {\hbox {plim}}\limits _{n\rightarrow \infty } n^{(m_{\gamma }-m_{\gamma '})/2} \biggl (\frac{b_0 + \mathbf{Y}'(\mathbf{I}_n - \mathbf{H}_{\gamma '})\mathbf{Y}}{b_0 + \mathbf{Y}'(\mathbf{I}_n - \mathbf{H}_\gamma )\mathbf{Y}}\biggr )^{-n/2}\\&= c_2 \mathop {\hbox {plim}}\limits _{n\rightarrow \infty } n^{(m_{\gamma }-m_{\gamma '})/2} \biggl (\frac{b_0/n + \mathbf{Y}'(\mathbf{I}_n - \mathbf{H}_{\gamma '})\mathbf{Y}/n}{b_0/n + \mathbf{Y}'(\mathbf{I}_n - \mathbf{H}_\gamma )\mathbf{Y}/n}\biggr )^{-n/2}\\&= c_3 \mathop {\hbox {plim}}\limits _{n\rightarrow \infty } n^{(m_{\gamma }-m_{\gamma '})/2}\biggl (\frac{\sigma ^2}{\sigma ^2 + c_{\gamma '}}\biggr )^{n/2} \\&= 0, \end{aligned}$$

  because $c_{\gamma '} >0$ and the term $\big (\sigma ^2/(\sigma ^2 + c_{\gamma '})\big )^{n/2}$ converges to 0 in probability exponentially fast as $n$ approaches infinity, no matter what value of $m_{\gamma }-m_{\gamma '}$. Thus, the limit of Eq. (30) converges to 0 with respect to $n$.

• (b) If $M_{\gamma } \subseteq M_{\gamma '}$, it is immediate from the result of Fernández et al. (2001) that we have

  $$\begin{aligned} \mathop {\hbox {plim}}\limits _{n\rightarrow \infty } {\biggl (\frac{\mathbf{Y}'(\mathbf{I}_n - \mathbf{H}_{\gamma })\mathbf{Y}/n}{\mathbf{Y}'(\mathbf{I}_n - \mathbf{H}_{\gamma '})\mathbf{Y}/n}\biggr )^{n/2}} \mathop {\longrightarrow }\limits ^{D} \exp {\biggl (\frac{\chi ^{2}_{m_{\gamma '} - m_{\gamma }}}{2}\biggr )}, \end{aligned}$$

  where $\mathop {\longrightarrow }\limits ^{D}$ means convergence in distribution. As $n$ approaches infinity, the limit of Eq. (30) becomes

  $$\begin{aligned} \mathop {\hbox {plim}}\limits _{n\rightarrow \infty }\frac{f(\gamma '\mid \mathbf{Y})}{f(\gamma \mid \mathbf{Y})}&= c_2 \mathop {\hbox {plim}}\limits _{n\rightarrow \infty } n^{(m_{\gamma }-m_{\gamma '})/2} \biggl (\frac{b_0/n + \mathbf{Y}'(\mathbf{I}_n - \mathbf{H}_{\gamma '})\mathbf{Y}/n}{b_0/n + \mathbf{Y}'(\mathbf{I}_n - \mathbf{H}_\gamma )\mathbf{Y}/n}\biggr )^{-n/2}\\&= c_4\mathop {\hbox {plim}}\limits _{n\rightarrow \infty }{n^{(m_\gamma -m_{\gamma '})/2}}\exp {\biggl (\frac{\chi ^{2}_{m_{\gamma '} - m_\gamma }}{2}\biggr )} \\&= 0, \end{aligned}$$

  because $m_\gamma - m_{\gamma '} < 0$ for $M_\gamma \subseteq M_{\gamma '}$. This completed the proof.

$\square$

Proof of Theorem 3:

We consider the following two situations.

• (a) When $\mathbf{M}_\gamma = \mathbf{M}_N$, it follows directly from the consistency of least squares estimators that $\Vert \hat{\varvec{\beta }}_\gamma \Vert \rightarrow 0$, and therefore, the consistency of the BMA estimates follows.

• (b) When $\mathbf{M}_\gamma \ne \mathbf{M}_N$, if follows from Theorem 2 that $\mathop {\hbox {plim}}\limits \nolimits _{n\rightarrow \infty } P(M_\gamma \mid \mathbf{Y}) = 1$ when $M_\gamma$ is the true model. In addition, following the result of Liang et al. (2008), we have

  $$\begin{aligned} \mathop {\hbox {plim}}\limits _{n\rightarrow \infty }\int _0^\infty \frac{g}{1+g}\pi (g \mid M_\gamma , Y) = \frac{\hat{g}}{1 + \hat{g}}\biggl (1 + O\Bigl (\frac{1}{n}\Bigr )\biggr ), \end{aligned}$$

  where $\hat{g}$ can be obtained by maximizing the function of the form

  $$\begin{aligned} L(g) = (1+g)^{(n-m_\gamma +a_0)/2} \bigl (1 + g(1-\tilde{R}^2_\gamma )\bigr )^{-(n+a_0)/2}. \end{aligned}$$

  Taking the first partial derivative of $\log (L(g))$ with respect to $g$ and putting it equal to 0, it provides

  $$\begin{aligned} \hat{g} = \max \biggl \{\frac{\tilde{R}^2_\gamma /m_\gamma }{(1-\tilde{R}^2_\gamma )/(n-m_\gamma +a_0)}-1, 0 \biggr \}. \end{aligned}$$

  Note that $\hat{g}$ approaches infinity under the true model $M_\gamma$ as $n$ tends to infinity, and therefore, we obtain

  $$\begin{aligned} \mathop {\hbox {plim}}\limits _{n\rightarrow \infty }\int _0^\infty \frac{g}{1+g}\pi (g \mid M_\gamma , Y) = 1. \end{aligned}$$

  Using the consistency property of the least squares estimators, it is easy to show that

  $$\begin{aligned} \mathop {\hbox {plim}}\limits _{n\rightarrow \infty }\hat{Y}_f = \mathbb {E}[Y_f] = \alpha {\mathbf{1}_m} + \mathbf{X}_f \varvec{\beta }_\gamma . \end{aligned}$$

  This completes the proof.$\square$