Two stage smoothing in additive models with missing covariates

Abstract

This paper considers sparse additive models with missing covariates. The missing mechanism is assumed to be missing at random. The additive components are estimated via a two stage method. First, the penalized weighted least squares method is used. The weight is the inverse of the selection probability, which is the probability of observing covariates. As the penalty, we utilize the adaptive group lasso to distinguish between the zero and the nonzero components. Thus, the penalty is used to investigate the sparse structure and the weight reflects the missing structure. The estimator obtained from the penalized weighted least squares method is denoted by the first stage estimator (FSE). We show the sparsity and consistency properties of the FSE. However, the asymptotic distribution of the FSE of the nonzero components is not derived as it is difficult. Therefore for each nonzero component, we apply the penalized spline methods for univariate regression with the residual of the FSE of other component. The asymptotic normality of the second stage estimator is shown. To confirm the performance of the proposed estimator, simulation studies and a real data application are implemented.

Appendix

We describe the proofs of Theorems 1–2. To prove the main theorem of this paper, we first show the consistency of the centered B-spline model.

Lemma 1

For $j=1,\ldots ,p$, under Assumptions A and B, there exists $\mathbf {b}_{0j}\in \mathbb {R}^{K+r}$ such that

$$\begin{aligned} f_j(x_j)-\mathbf {B}(x_j)^T\mathbf {b}_{0j}=O_P(K^{-d}). \end{aligned}$$

proof of Lemma 1

For $j=1,\ldots ,p$, from Barrow and Smith (1978), there exists $\mathbf {b}_{0j}\in \mathbb {R}^{K+r}$ such that $\sup _{x_j\in [a,b]}|f_j(x)-\mathbf {\psi }(x_j)^T\mathbf {b}_{0j}|=O(K^{-d})$. For $x_{ij}=x_{ij,2}$, we have

$$\begin{aligned} \mathbf {B}(x_j)^T\mathbf {b}_j^*=\mathbf {\psi }(x_j)^T\mathbf {b}_{0j}-\frac{1}{n}\sum _{i=1}^n\mathbf {\psi }(x_{ij})^T\mathbf {b}_{0j} \end{aligned}$$

and

$$\begin{aligned} f_j(x_j)-\mathbf {B}(x_j)^T\mathbf {b}_j^*= & {} O(K^{-d})+\frac{1}{n}\sum _{i=1}^n\mathbf {\psi }(x_{ij})^T\mathbf {b}_{0j}\\= & {} O(K^{-d})+\frac{1}{n}\sum _{i=1}^n\{f_j(x_{ij})+O_P(K^{-d})\}. \end{aligned}$$

From Assumption A 2, we obtain

$$\begin{aligned} E\left[ \frac{1}{n}\sum _{i=1}^n\{f_j(X_{ij})+O_P(K^{-d})\}\right]= & {} O(n^{-d}),\\ V\left[ \frac{1}{n}\sum _{i=1}^n\{f_j(X_{ij})+O_P(K^{-d})\}\right]= & {} O(n^{-1}). \end{aligned}$$

Therefore, Assumption A 5 yields that $f_j(x_j)-\mathbf {B}(x_j)^T\mathbf {b}_j^*=O(K^{-d})$. For $x_{ij}=x_{ij,1}$, since

$$\begin{aligned} \mathbf {B}(x_j)=\mathbf {\psi }(x_j)-\frac{1}{n}\sum _{i=1}^n \frac{\delta _i}{\hat{\pi }_i}\mathbf {\psi }(x_{ij}), \end{aligned}$$

we have

$$\begin{aligned} f_j(x_j)-\mathbf {B}(x_j)^T\mathbf {b}_j^*= & {} O(K^{-d})+\frac{1}{n}\sum _{i=1}^n\frac{\delta _i}{\hat{\pi }_i}\mathbf {\psi }(x_{ij})^T\mathbf {b}_{0j}\\= & {} O(K^{-d})+\frac{1}{n}\sum _{i=1}^n\frac{\delta _i}{\hat{\pi }_i}\big \{f_j(x_{ij})+O_P(K^{-d})\big \} \end{aligned}$$

From Assumptions A 2 and B, we have

$$\begin{aligned} E\left[ \frac{1}{n}\sum _{i=1}^n\frac{\delta _i}{\hat{\pi }_i}\{f_j(X_{ij})+O(K^{-d})\}\right]= & {} O(K^{-d})+O(n^{-\alpha }K^{-d})=O(K^{-d}),\\ V\left[ \frac{1}{n}\sum _{i=1}^n\frac{\delta _i}{\hat{\pi }_i}f_j(X_{ij})\right]= & {} V\left[ \frac{1}{n}\sum _{i=1}^n\frac{\delta _i}{\pi _i}f_j(X_{ij})\right] =O(n^{-1}). \end{aligned}$$

Thus, we obtain $f_j(x_j)-\mathbf {B}(x_j)^T\mathbf {b}_j^*=O(K^{-d})$ and this completes the proof.

Define $Z_A=(Z_j, j\in A)$ and $\hat{\mathbf {b}}_A=(\hat{\mathbf {b}}_j, j\in A)$. From Lemma 1, there exists $\mathbf {b}_{0,j}\in \mathbb {R}^{K+r}$ such that $f_j(x_j)-\mathbf {B}(x_j)^T\mathbf {b}_j^*=O_P(K^{-d})$ for $j\in A$. From Assumption A 3, for $j\in A$, there exist $c_j>0$ such that $||\mathbf {b}_{0,j}||>c_j$ and for $j\not \in A$, $||\mathbf {b}_j||=0$. Define $\mathbf {b}_A=(\mathbf {b}_{0,j},j\in A)$.

Proof of Theorem 1

From KKT conditions, a necessary and sufficient condition for $\hat{\mathbf {b}}$ is

$$\begin{aligned} \left\{ \begin{array}{ll} 2||Z_j^T \hat{\Pi }(\mathbf {Y}-Z\hat{\mathbf {b}})||\le \lambda w_j&{}\quad ||\hat{\mathbf {b}}_{j}||\not =0.\\ 2Z_j^T \hat{\Pi }(\mathbf {Y}-Z\hat{\mathbf {b}})=\lambda w_j \frac{\hat{\mathbf {b}}_{j}}{||\hat{\mathbf {b}}_j||}, &{}\quad ||\hat{\mathbf {b}}_{j}||=0.\\ \end{array} \right. \end{aligned}$$

We first show

$$\begin{aligned}&P(||\hat{\mathbf {b}}_j||\not =0),\quad \ j\in A,\\&P(||\hat{\mathbf {b}}_j||=0),\quad \ j\not \in A. \end{aligned}$$

For this, it is sufficient to prove that as $n\rightarrow \infty$,

$$\begin{aligned}&P(||\mathbf {b}_{0,j}-\hat{\mathbf {b}}_j||\ge ||\mathbf {b}_{0,j}||)\mathop {\rightarrow }\limits ^{P}0,\quad \ j\in A,\nonumber \\&P(2||Z_j^T \hat{\Pi }(\mathbf {Y}-Z\hat{\mathbf {b}})||> \lambda w_)\mathop {\rightarrow }\limits ^{P}0,\quad \ j\not \in A. \end{aligned}$$

(4)

Since for $j\in A$, $||\hat{\mathbf {b}}_j||\not =0$ is necessary and sufficient for $||\hat{\mathbf {b}}_j-\mathbf {b}_{0.j}||<||\mathbf {b}_{0,j}||$, we obtain for $j\in A$ that

$$\begin{aligned} 2Z_j^T \hat{\Pi }(\mathbf {Y}-Z\hat{\mathbf {b}})=\lambda w_j \frac{\hat{\mathbf {b}}_{j}}{||\hat{\mathbf {b}}_j||} \end{aligned}$$

(5)

if and only if $||\hat{\mathbf {b}}_j-\mathbf {b}_{0.j}||<||\mathbf {b}_{0,j}||$. Thus under (5), we show

$$\begin{aligned} P(||\mathbf {b}_{0,j}-\hat{\mathbf {b}}_j||\ge ||\mathbf {b}_{0,j}||,\quad j\in A)\mathop {\rightarrow }\limits ^{P}0. \end{aligned}$$

The estimator $\hat{\mathbf {b}}_A$ can be written as

$$\begin{aligned} \hat{\mathbf {b}}_A=(Z_A^T\hat{\Pi }Z_A)^{-1}Z_A^T\hat{\Pi }(\mathbf {Y}-\lambda \mathbf {\nu }_A), \end{aligned}$$

where

$$\begin{aligned} \mathbf {\nu }_A=\left( w_j \frac{\hat{\mathbf {b}}_{j}}{||\hat{\mathbf {b}}_j||}, j\in A \right) ^T. \end{aligned}$$

Let $\mathbf {f}_j=(f_j(x_{1j}),\ldots ,f_j(x_{nj}))^T$ and $\mathbf {f}_A=\sum _{j\in A} \mathbf {f}_j$. Then from Lemma 1, we have $\mathbf {\delta }_A=\mathbf {f}_A-Z_A\mathbf {b}_A=O_P(K^{-d})$. Since $\mathbf {Y}=\mathbf {f}_A+\mathbf {\varepsilon }=Z_A\mathbf {b}_A+\mathbf {\delta }_A+\mathbf {\varepsilon }$, we obtain

$$\begin{aligned} \hat{\mathbf {b}}_A-\mathbf {b}_A=(Z_A^T\hat{\Pi }Z_A)^{-1}Z_A^T\hat{\Pi }(\mathbf {\delta }+\mathbf {\varepsilon }-\lambda \mathbf {\nu }_A), \end{aligned}$$

From Assumption B, $\hat{\Pi }=\Pi +O_P(n^{-\alpha })$ and hence $\hat{\Pi }=O_P(1)$. Therefore, similar to proof of Lemma 5 of Huang et al. (2010),

$$\begin{aligned} ||(Z_A^T\hat{\Pi }Z_A)^{-1}Z_A^T\hat{\Pi }\mathbf {\delta }||\le O_P(K^{-d+1/2}),\ \ ||(Z_A^T\hat{\Pi }Z_A)^{-1}Z_A^T\hat{\Pi }\mathbf {\varepsilon }||\le O_P(n^{-1/2}K) \end{aligned}$$

and

$$\begin{aligned} \lambda ||(Z_A^T\hat{\Pi }Z_A)^{-1}Z_A^T\hat{\Pi }\mathbf {\nu }_A||\le O_P(\lambda K n^{-1}). \end{aligned}$$

Thus, $||\hat{\mathbf {b}}_A-\mathbf {b}_A||=o_P(1)$. Together with $||\mathbf {b}_{0,j}||=O(1) (j\in A)$, the first assertion of (4) holds. The analogy of the proof of second assertion of (4) is the same manner as the proof of Lemma 6 of Huang et al. (2010) and hence it is omitted. Thus, the first assertion of Theorem 1 can be derived.

We next show the second assertion of Theorem 1. From the property of the B-spline function, for $j\in A$, there exist $\eta _1, \eta _2>0$ such that

$$\begin{aligned} \eta _1K^{-1}||\hat{\mathbf {b}}_j-\mathbf {b}_{j}||^2\le |\mathbf {B}(x_j)^T\hat{\mathbf {b}}_j-\mathbf {B}(x_j)^T\mathbf {b}_{0,j}|^2\le \eta _2 K^{-1}||\hat{\mathbf {b}}_j-\mathbf {b}_{j}||^2. \end{aligned}$$

Since

$$\begin{aligned} ||\hat{\mathbf {b}}_A-\mathbf {b}_A||^2=O_P(K^{-2d+1})+O_P(K^2/n)+O_P(\lambda ^2 K^2/n^2), \end{aligned}$$

we obtain

$$\begin{aligned} |\hat{f}(x_j)-f(x_j)|^2=O_P(K^{-2d})+O_P(K/n)+O_P(\lambda ^2 K/n^2). \end{aligned}$$

The proof was completed.

Proof of Theorem 2

For simplicity, we rewrite $Z_j\equiv Z_{j,S}, j\in \hat{A}$. For $j\in A\cap \hat{A}$, the second stage estimator can be written as

$$\begin{aligned} \hat{f}_{j,S}(x_j)=\mathbf {B}_S(x_j)^T(Z_j^T\hat{\Pi }Z_j+\lambda _j D_m^TRD_m)^{-1}Z_j^T\hat{\Pi }(\mathbf {Y}-Z_{(-j),\hat{A}}\hat{\mathbf {b}}_{(-j,\hat{A})}), \end{aligned}$$

where $Z_{(-j),\hat{A}}=(Z_k, k\in \hat{A}-\{j\})^T$ and $\hat{\mathbf {b}}_{(-j,\hat{A})}=(\hat{\mathbf {b}}_k, k\in \hat{A}-\{j\})$. Let $\mathbf {\delta }_{(-j),\hat{A}}=\mathbf {f}_{(-j),\hat{A}}-Z_{(-j),\hat{A}}\mathbf {b}_{(-j),\hat{A}}=O_P(K^{-d})$. Then we have

$$\begin{aligned} \mathbf {Y}-Z_{(-j),\hat{A}}\hat{\mathbf {b}}_{(-j,\hat{A})}=\mathbf {f}_j+\mathbf {\delta }_{(-j),\hat{A}}+Z_{(-j),\hat{A}}(\mathbf {b}_{(-j),\hat{A}}-\hat{\mathbf {b}}_{(-j),\hat{A}})+\mathbf {\varepsilon } \end{aligned}$$

and so

$$\begin{aligned} \hat{f}_{j,S}(x_j)= & {} \mathbf {B}_S(x_j)^T(Z_j^T\hat{\Pi }Z_j+\lambda _j D_m^TRD_m)^{-1}Z_j^T\hat{\Pi }\mathbf {f}_j\\&+\;\mathbf {B}_S(x_j)^T(Z_j^T\hat{\Pi }Z_j+\lambda _j D_m^TRD_m)^{-1}Z_j^T\hat{\Pi }\mathbf {\delta }_{(-j),\hat{A}}\\&+ \;\mathbf {B}_S(x_j)^T(Z_j^T\hat{\Pi }Z_j+\lambda _j D_m^TRD_m)^{-1}Z_j^T\hat{\Pi }Z_{(-j),\hat{A}}(\mathbf {b}_{(-j),\hat{A}}-\hat{\mathbf {b}}_{(-j),\hat{A}})\\&+\;\mathbf {B}_S(x_j)^T(Z_j^T\hat{\Pi }Z_j+\lambda _j D_m^TRD_m)^{-1}Z_j^T\hat{\Pi }\mathbf {\varepsilon }\\\equiv & {} D_1+D_2+D_3+D_4. \end{aligned}$$

Similar to Theorem 2(b) of Claeskens et al. (2009), we have

$$\begin{aligned} D_1=f_j(x_j)+b_{j,\lambda }(x_j)+b_{j,a}(x_j)+o_P((\lambda /n)^{1/2})+o_P(K_S^{-d}). \end{aligned}$$

From Lemma A3 of Claeskens et al. (2009) and the fundamental property of B-splines, $D_2=O_P(K_SK_{j,m}^{-1}K^{-d})$. The result of Theorem 1 and Lemma A3 of Claeskens et al. (2009) yields that

$$\begin{aligned} D_3=O_P(K_SK_{j,m}^{-1}\{K^{-d}+K^{1/2}/n^{1/2}+\lambda K^{1/2}/n\}). \end{aligned}$$

We note that $K_{j,m}$ is the maximum eigenvalue of $(Z_j^TZ_j)^{-1/2}D_m^TRD_m(Z_j^TZ_j)^{-1/2}$ and this has an asymptotic order $O(K_S^{2m}(\lambda /n))$ from the proof of Lemma A1 of Claeskens et al. (2009). If $K_S=O(n^{\eta }), \eta >1/(2m+1)$ and $\lambda _j=O(n^{m/(2m+1)})$, $K_{j,m}\rightarrow \infty$ as $n\rightarrow \infty$. Thus under Assumption C, we have $E[D_2]=o((\lambda _j/n)^{1/2})$ and $E[D_3]=o((\lambda _j/n)^{1/2})$. From

$$\begin{aligned}&(Z_j^T\hat{\Pi }Z_j+\lambda _j D_m^TRD_m)^{-1} =(Z_j^T\Pi Z_j)^{-1}\\&\quad +\;((Z_j^T\hat{\Pi }Z_j+\lambda _j D_m^TRD_m)^{-1})(I+Z_j^T(\hat{\Pi }-\Pi )Z_j(Z_j^T\Pi Z_j)^{-1}), \end{aligned}$$

we obtain $E[D_4]=o((\lambda _j/n)^{1/2})$. Thus,

$$\begin{aligned} E[\hat{f}_{j,S}(x_j)]=f_j(x_j)+b_{j,\lambda }(x_j)+b_{j,a}(x_j)+o_P((\lambda _j/n)^{1/2})+o_P(K_S^{-d}). \end{aligned}$$

Similarly, the variance of $\hat{f}_{j,S}(x_j)$ is dominated by

$$\begin{aligned} V[D_4]=V\left[ \mathbf {B}_S(x_j)^T(Z_j^T\Pi Z_j+\lambda _j D_m^TRD_m)^{-1}Z_j^T\hat{\Pi }\mathbf {\varepsilon }\right] (1+o(1)), \end{aligned}$$

which can be calculated by similar manner as Theorem 2(b) of Claeskens et al. (2009). As the result, $V[\hat{f}_{j,S}(x_j)]=v_j(x_j)(1+o(1))$, where $v_j(x_j)=O(n^{-1}(\lambda _j/n)^{-1/(2m)})$. This completes the proof.