A sparse estimate based on variational approximations for semiparametric generalized additive models

Abstract

In semiparametric regression, traditional methods such as mixed generalized additive models (GAM), computed via Laplace approximation or variational approximation using penalized marginal likelihood estimation, may not achieve sparsity and unbiasedness simultaneously, and may sometimes suffer from convergence problems. To address these issues, we propose an estimator for semiparametric generalized additive models based on the marginal likelihood. Our approach provides sparsity estimates and allows for statistical inference. To estimate and select variables, we use the smoothly clipped absolute deviation penalty (SCAD) within the framework of variational approximation. We also propose efficient iterative algorithms to obtain estimations. Simulation results support our theoretical characteristics, and we demonstrate that our method is more effective than the original variational approximations framework and many other penalized methods under certain conditions. Moreover, applications with actual data further demonstrate the superior performance of the proposed method.

Appendix

Proof of Theorem 1

Set $\delta =d^{1/2}n^{-1/2}$, where $d = o(n^{1/4})$. The goal is to prove $\Vert {\hat{\theta }}-\theta ^0\Vert _2=O_p(\delta )$. Let $\theta = \theta ^0+\delta u$, then it is equivalent to prove that there exists a large constant C such that $\forall \varepsilon >0$,

$$\begin{aligned} P\{\sup \limits _{\Vert u\Vert _2=C}{l_{\lambda }\left( \theta ,\text {vech}(A)\right) }\}>l_{\lambda } \theta ^0,\text {vech}(A)\} \ge 1-\varepsilon , \end{aligned}$$

which implies that with probability at least $1-\varepsilon$ there is a local solution in the ball $\{\theta : \Vert \theta -\theta ^0\Vert _2=\delta u,\Vert u\Vert _2 \le C\}$ for ${\arg \max }_{\theta }\ l_{\lambda }\left( \theta ,\text {vech}(A)\right)$. According to Lemma 1, when ${\hat{\theta }} \in \{\theta : \Vert \theta -\theta ^0\Vert _2=\delta u,\Vert u\Vert _2 \le C\}$, the covariance ${\hat{A}}$ estimated by VA approximation satisfies ${\hat{A}} = O_p(1/n)$. For any $\theta \in \{\theta : \Vert \theta -\theta ^0\Vert _2=\delta u,\Vert u\Vert _2 \le C\}$, by Taylor expansion, we have

$$\begin{aligned} \beta _n(\theta )&\equiv l_{\lambda }\left( \theta ,\text {vech}(A)\right) -l_{\lambda }\left( \theta ^0,\text {vech}(A)\right) \\&=\left( \theta -\theta ^{0}\right) ^\mathrm {\scriptscriptstyle T} \nabla _{\theta } \underline{\ell }\left\{ \theta ^{0}, {\text {vech}}(A)\right\} -\frac{1}{2}\left( \theta -\theta ^{0}\right) ^\mathrm {\scriptscriptstyle T} \left[ -\nabla _{\theta }^{2} \underline{\ell }\left\{ \theta ^{0}, {\text {vech}}(A)\right\} \right] \left( \theta -\theta ^{0}\right) \\&~~~+\frac{1}{6} \sum _{r, s, t=1}^{{\text {dim}}(\theta )} \frac{\partial ^{3} \underline{\ell }\{\overline{\theta }, {\text {vech}}(A)\}}{\partial \theta _{r} \partial \theta _{s} \partial \theta _{t}}\left( \theta -\theta ^{0}\right) _{r}\left( \theta -\theta ^{0}\right) _{s}\left( \theta -\theta ^{0}\right) _{t}\\&~~~+n\sum _{l=1}^{p}\Big \{p_{\lambda _0}\big (\sqrt{\beta _l^{\mathrm {\scriptscriptstyle T} }H_l\beta _l}\big ) -p_{\lambda _0}\big (\sqrt{\beta _{0l}^{\mathrm {\scriptscriptstyle T} }H_l\beta _{0l}}\big )\Big \} \\&\triangleq T_{1}+T_{2}+T_{3}+T_{\lambda }. \end{aligned}$$

We already have $T_1=C\cdot O_p(d)$, $T_2\le -K^2\tau _{\min }d$, $T_3=o_p(d)$. In addition, by the inequality $p_\lambda (|x|)-p_\lambda (|y|)\le \lambda |x-y|$, the conditions $\lambda _0=o(1)$ and $\lambda _0=O\left( d^{1/2}n^{-1/2}\right)$, we have

$$\begin{aligned} T_\lambda&=n\sum _{l=1}^{p}\Big \{p_{\lambda _0}\Big (\sqrt{\beta _l^{\mathrm {\scriptscriptstyle T} }H_l\beta _l} \Big ) -p_{\lambda _0}\Big (\sqrt{\beta _{0l}^{\mathrm {\scriptscriptstyle T} }H_l\beta _{0l}}\Big )\Big \} \\&\le n\lambda _0 \sum _{l=1}^{p} \Big | \sqrt{\beta _l^{\mathrm {\scriptscriptstyle T} }H_l\beta _l} -\sqrt{\beta _{0l}^{\mathrm {\scriptscriptstyle T} }H_l\beta _{0l}} \Big | \\&= n\lambda _0\sum _{l=1}^{p}\frac{\Big | \beta _l^{\mathrm {\scriptscriptstyle T} }H_l\beta _l -\beta _0l^{\mathrm {\scriptscriptstyle T} }H_l\beta _0l \Big |}{\Big | \sqrt{\beta _l^{\mathrm {\scriptscriptstyle T} }H_l\beta _l} +\sqrt{\beta _{0l}^{\mathrm {\scriptscriptstyle T} }H_l\beta _{0l}} \Big |}. \end{aligned}$$

By Condition C6, we can obtain that there exists a constant $C_3$ such that $\big | \sqrt{\beta _l^{\mathrm {\scriptscriptstyle T} }H_l\beta _l}+\sqrt{\beta _{0\,l}^{\mathrm {\scriptscriptstyle T} }H_l\beta _{0\,l}} \big |>C_3$, then

$$\begin{aligned} T_\lambda&\le nC\lambda _0\sum _{l=1}^{p}\frac{\Vert \beta _l-\beta _{0l}\Vert _2}{\sqrt{C_3}} =O_p\big (n\lambda _0d^{1/2}n^{-1/2})\\&= O_p(\lambda _0d^{1/2}n^{1/2})=O_p(d). \end{aligned}$$

In summary, $T_1=C \cdot O_p(d)$, $T_2\le -K^2\tau _{\min }d$, $T_3=o_p(d)$, and $T_\lambda =O_p(d)$. If we choose a large constant K, $T_2$ will dominate $T_1$, $T_2$ and $T_\lambda$. We thus conclude that there exists a local solve in the $\{\theta : \Vert \theta -\theta ^0\Vert _2=\delta u,\Vert u\Vert _2 \le C \}$ for ${\arg \max }_{\theta } l_{\lambda }\left( \theta ,\text {vech}(A)\right)$. Then $\Vert \theta -\theta ^0\Vert _2=O_p(\delta )=O_p(d^{1/2}n^{-1/2})$ is able to proved, completing the proof. $\square$

Proof of Theorem 2

Based on Theorem 1, we can obtain that ${\hat{\theta }}$ is the consistent estimator of $\theta$, thus there exists a local maximizer ${\hat{\theta }}$ with the convergence rate $d^{1/2}n^{-1/2}$ such that ${l_{\lambda }\left( \theta ,\text {vech}(A)\right) }$ is maximum. That is

$$\begin{aligned} \left. \frac{\partial l_{\lambda }\left( \theta ,\text {vech}(A)\right) }{\partial \theta _{j}}\right| _{\theta ^\mathrm {\scriptscriptstyle T} = \big ({\hat{\alpha }}^\mathrm {\scriptscriptstyle T} , {\hat{\beta }}^\mathrm {\scriptscriptstyle T} \big )} =0 \quad \text {for} j=1, \dots , p+q. \end{aligned}$$

Note that ${\hat{\theta }}$ converges to $\theta$ in probability and by Taylor expansion, we have when $j=1,\dots ,p$, the score function of the likelihood $\partial l_{\lambda }\left( \theta ,\text {vech}(A)\right) /\partial \theta _{j}=\partial l_{}\left( \theta ,\text {vech}(A)\right) /\partial \theta _{j}$ because the penalty item does not contain the parameter component $\psi =(\alpha ,\phi )$, and when $j=p+1,\dots ,q$, the following holds

$$\begin{aligned} \partial l_{\lambda }\left( \theta ,\text {vech}(A)\right) /\partial \theta _{j}=\dfrac{\partial l_{\lambda }\left( \theta ,\text {vech}(A)\right) }{\partial \theta _{j}} \Big |_{\theta =\left( {\hat{\alpha }},{\hat{\beta }} \right) }-n p_{\lambda _{}}^{\prime }\left( \sqrt{{\hat{\beta }}_j^\mathrm {\scriptscriptstyle T} H_j{\hat{\beta }}_j}\right) H_j{\hat{\beta }}_j/(\sqrt{{\hat{\beta }}_j^\mathrm {\scriptscriptstyle T} H_j{\hat{\beta }}_j}). \end{aligned}$$

Therefore, we only need to prove that the linear part of VA estimator is asymptotically normal solution the linear part $\alpha$ to $\partial l_{}\left( \theta ,\text {vech}(A)\right) /\partial \theta _{j}=0$ is one and the same as $\partial l_{\lambda }\left( \theta ,\text {vech}(A)\right) /\partial \theta _{j}$. According to Hui et al. (2019), the VA estimators of normal response, poisson response and bernoulli response satisfy:

$$\begin{aligned} n^{1 / 2} G\left( \hat{\theta }-\theta ^{0}\right) =\frac{1}{n^{1 / 2}} G \mathcalligra{I}^{-1}\left( \theta ^{0}\right) \nabla _{\theta } \ell \left( \theta ^{0}\right) +G \mathcalligra {I}^{-1}\left( \theta ^{0}\right) \delta , \end{aligned}$$

and the last term on the right-hand side is asymptotically negligible. We also have $\sum _{i=1}^{n} L_{i}=n^{-1 / 2} G \mathcalligra {I}^{-1}\left( \theta ^{0}\right) \nabla _{\theta } \ell \left( \theta ^{0}\right)$ such that $\sum _{i=1}^{n} L_{i}=n^{-1 / 2} G \mathcalligra {I}^{-1}\left( \theta ^{0}\right) \nabla _{\theta } \ell \left( \theta ^{0}\right)$. Thus, we conclude $E(l_i)=0$ and ${\text {Var}}\left( L_{i}\right) =n^{-1} G \mathcalligra {I}^{-1}\left( \theta ^{0}\right) G^\mathrm {\scriptscriptstyle T}$. By multivariate Lindeberg-Feller central limit theorem, we have $\jmath ^{-1} \ell (\theta ^0)\nabla _0 (\theta ^0){\mathop {\longrightarrow }\limits ^{d}}N(0,G \jmath ^{-1} ( \theta ^0)G^\top )$, completing the proof. $\square$