Robust variable selection with exponential squared loss for partially linear spatial autoregressive models

Abstract

In this paper, we consider variable selection for a class of semiparametric spatial autoregressive models based on exponential squared loss (ESL). Using the orthogonal projection technique, we propose a novel orthogonality-based variable selection procedure that enables simultaneous model selection and parameter estimation, and identifies the significance of spatial effects. Under appropriate conditions, we show that the proposed procedure is consistent and the resulting estimator has oracle properties. Furthermore, some simulation studies and an analysis of the Boston housing price data are also carried out to examine the finite-sample performance of the proposed method.

Appendix

Proof of Theorem 1

Let $\xi = n^{-1/2} + a_n$. Similar to Fan and Li (2001), we first prove that for any given $\epsilon > 0$, there exists a constant C such that

$$\begin{aligned} P\left\{ \sup \limits _{\Vert {{\textbf {u}}}\Vert = C} \ell (\theta _0 + \xi {{\textbf {u}}}) < \ell (\theta _0) \right\} \ge 1 - \epsilon , \end{aligned}$$

(18)

where ${{\textbf {u}}}$ is a $(p+1)$-dimensional vector such that $\Vert {{\textbf {u}}}\Vert = C$, and C is a large enough constant. This means that the probability that there exists a local maximum in the sphere $\{\theta _0 + \xi {{\textbf {u}}} : \Vert {{\textbf {u}}}\Vert \le C \}$ is at least $1 - \epsilon$. Hence, we prove that there exists a local maximizer ${\hat{\theta }}_n$ such that $\Vert {\hat{\theta }}_n - \theta _0\Vert = O_p(\xi )$. Let

$$\begin{aligned} D(\theta , \gamma ) = \sum _{i=L+1}^n \exp \left\{ -({\tilde{Y}}_i - {\tilde{Z}}_i^T \theta )^2 / \gamma \right\} \frac{2({\tilde{Y}}_i - {\tilde{Z}}_i^T \theta )}{\gamma } {\tilde{Z}}_i . \end{aligned}$$

Since $p_{\lambda _j}(0)=0$ for $j = 1,\ldots , p+1$ and $\gamma _n - \gamma _0 =o_p(1)$ , by Taylor’s expansion we have

$$\begin{aligned}&\ell (\theta _0 + \xi {{\textbf {u}}}) - \ell (\theta _0)\nonumber \\&\quad =\sum _{i=L+1}^n \exp \left\{ -\frac{({\tilde{Y}}_i - {\tilde{Z}}_i^T(\theta _0 + \xi {{\textbf {u}}}))^2}{\gamma _n} \right\} - \sum _{i=L+1}^n \exp \left\{ -\frac{({\tilde{Y}}_i - {\tilde{Z}}_i^T\theta _0)^2}{\gamma _n} \right\} \nonumber \\&\qquad - (n-L) \sum _{j=1}^{p+1} \{ p_{\lambda _j}(|\theta _{0j} + \xi _j|) - p_{\lambda _j}(|\theta _{0j}|)\} \nonumber \\&\quad \le \sum _{i=L+1}^n \exp \left\{ -\frac{({\tilde{Y}}_i - {\tilde{Z}}_i^T(\theta _0 + \xi {{\textbf {u}}}))^2}{\gamma _n} \right\} - \sum _{i=L+1}^n \exp \left\{ -\frac{({\tilde{Y}}_i - {\tilde{Z}}_i^T\theta _0)^2}{\gamma _n} \right\} \nonumber \\&\qquad - (n-L) \sum _{j=1}^{s} \{ p_{\lambda _j}(|\theta _{0j} + \xi _j|) - p_{\lambda _j}(|\theta _{0j}|)\} \nonumber \\&\quad = \xi D(\theta _0, \gamma _n)^T {{\textbf {u}}} -\frac{1}{2} {{\textbf {u}}}^T[- I(\theta _0, \gamma _n) ] {{\textbf {u}}} (n-L) \xi ^2\{1+o(1) \} \nonumber \\&\qquad - \sum _{j=1}^s[(n-L) \xi p_{\lambda _j}'(|\theta _{0j}|) \textrm{sign}(\theta _{0j}) u_j + (n-L) \xi ^2 p_{\lambda _j}''(|\theta _{0j}|) u_j^2\{1+o(1) \} ] \nonumber \\&\quad = \xi \{D(\theta _0, \gamma _0) + o_p( \sqrt{n} )\}^T {{\textbf {u}}} -\frac{1}{2} {{\textbf {u}}}^T[- I(\theta _0, \gamma _0) +o(1) ] {{\textbf {u}}} (n-L) \xi ^2\{1+o(1) \} \nonumber \\&\qquad - \sum _{j=1}^s[(n-L) \xi p_{\lambda _j}'(|\theta _{0j}|) \textrm{sign}(\theta _{0j}) u_j + (n-L) \xi ^2 p_{\lambda _j}''(|\theta _{0j}|) u_j^2\{1+o(1) \} ] \nonumber \\&\quad \le \xi \{D(\theta _0, \gamma _0) + o_p( \sqrt{n}) )\}^T {{\textbf {u}}} -\frac{1}{2} {{\textbf {u}}}^T[- I(\theta _0, \gamma _0) +o(1) ] {{\textbf {u}}} (n-L) \xi ^2\{1+o(1) \} \nonumber \\&\qquad -[ \sqrt{s} (n-L) \xi a_n \Vert {{\textbf {u}}}\Vert + (n-L)\xi ^2b_n\Vert {{\textbf {u}}}\Vert ^2 ] . \end{aligned}$$

(19)

Note that $n^{-1/2} D(\theta _0, \gamma _0)=O_p(1)$. Therefore, the order of the first term on the right side of Eq. (18) is equal to $O_p(n^{1/2} \xi ) = O_p(n \xi ^2)$. By choosing a sufficiently large C, the second term dominates the first term uniformly in $\Vert {{\textbf {u}}}\Vert = C$. Since $b_n=o_p(1)$, the third term is also dominated by the second term of (18). Therefore, (17) holds by choosing a sufficiently large C. The proof of Theorem 1 is completed. $\square$

Proof of Theorem 2(a)

We now prove the sparsity. We will prove that with probability 1, for any $\theta _1$ satisfying $\theta _1-\theta _{01} = O_p(n^{-1/2})$, and for some small $\epsilon _n = Cn^{-1/2}$ and $j=s+1,\ldots , p+1$, we have $\partial \ell (\theta )/ \partial \theta _j >0$, for $0<\theta _j<\epsilon _n$, and $\partial \ell (\theta )/ \partial \theta _j <0$, for $-\epsilon _n<\theta _j<0$. Let

$$\begin{aligned} Q_n(\theta ,\gamma )= \sum _{i=L+1}^n \exp \left\{ -({\tilde{Y}}_i - {\tilde{Z}}_i^T \theta )^2 / \gamma \right\} . \end{aligned}$$

(20)

By Talylor’s expansion, we have

$$\begin{aligned}&\frac{ \partial \ell (\theta )}{\partial \theta _j} = \frac{ \partial Q_n(\theta ,\gamma _n)}{\partial \theta _j} - (n-L)p_{\lambda _{j}}'(|\theta _{j}|)\textrm{sign}(\theta _{j}) \\&\quad = \frac{ \partial Q_n(\theta _0,\gamma _n)}{\partial \theta _j} + \sum _{l=1}^{p+1} \frac{ \partial ^2 Q_n(\theta _0,\gamma _n)}{\partial \theta _j \partial \theta _l }(\theta _l - \theta _{0l}) \\&\qquad + \sum _{l=1}^{p+1} \sum _{k=1}^{p+1} \frac{ \partial ^3 Q_n(\theta ^*,\gamma _n)}{\partial \theta _j \partial \theta _l \partial \theta _k}(\theta _l - \theta _{0l})(\theta _k - \theta _{0k}) - (n-L)p_{\lambda _{j}}'(|\theta _{j}|)\textrm{sign}(\theta _{j}), \end{aligned}$$

where $\theta ^*$ lies between $\theta$ and $\theta _0$. Here we assume $\left| (n-L)^{-1}\frac{ \partial ^3 Q_n(\theta ,\gamma _n)}{\partial \theta _j \partial \theta _l \partial \theta _k} \right| \le M_{jlk}$, where $E(M_{jlk}) < \infty$. Note that

$$\begin{aligned}&(n-L)^{-1/2} \frac{ \partial Q_n(\theta _0,\gamma _0)}{\partial \theta _j} = O_p(1), \\&(n-L)^{-1} \frac{ \partial ^2 Q_n(\theta _0,\gamma _0)}{\partial \theta _j \partial \theta _l } =E\left\{ \frac{ \partial ^2 Q_n(\theta _0)}{\partial \theta _j \partial \theta _l } \right\} +o_p(1) , \end{aligned}$$

and

$$\begin{aligned} (n-L)^{-1}\frac{ \partial ^3 Q_n(\theta ^*,\gamma _n)}{\partial \theta _j \partial \theta _l \partial \theta _k} = O_p(1). \end{aligned}$$

Since $b_n = o_p(1)$ and $\sqrt{n}a_n = o_p(1)$, we obtain $\theta - \theta _0 = O_p(n^{-1/2})$. By $\sqrt{n}(\gamma _n-\gamma _0) = o_p(1)$, we have

$$\begin{aligned} \frac{ \partial \ell (\theta )}{\partial \theta _j} = (n-L) \lambda _j \left\{ -\lambda _j^{-1} p_{\lambda _{j}}'(|\theta _{j}|)\textrm{sign}(\theta _{j})+O_p((n-L)^{-1/2}/\lambda _j) \right\} . \end{aligned}$$

Since $\frac{1}{\min _{s+1 \le j \le p+1} \sqrt{n}\lambda _j } =o_p(1)$ and $\lim \inf _{n\rightarrow \infty } \lim \inf _{t\rightarrow 0^+} \lambda ^{-1} p_{\lambda }'(|t|) >0$ with probability 1, the sign of the derivative is completely determined by that of $\theta _j$. This completes the proof of Theorem 2(a).

Proof of Theorem 2(b)

It can be shown easily that there exists a ${\hat{\theta }}_{n1}$ in Theorem 1 that is a $\sqrt{n}$-consistent local maximizer of $\ell \{(\theta _1,0)\}$, satisfying that

$$\begin{aligned} \frac{ \partial \ell \{({\hat{\theta }}_{n1},0)\}}{\partial \theta _j} =0, \quad \textrm{for} \, j=1,\ldots ,s. \end{aligned}$$

(21)

Note that ${\hat{\theta }}_{n1}$ is a consistent estimator,

$$\begin{aligned}&\frac{ \partial Q_n\{({\hat{\theta }}_{n1},0), \gamma _n\}}{\partial \theta _j} - (n-L) p_{\lambda _{j}}'(|\theta _{j}|)\textrm{sign}(\theta _{j}) \\&\quad = \frac{ \partial Q_n(\theta _0,\gamma _n)}{\partial \theta _j} + \sum _{l=1}^{s} \left\{ \frac{ \partial ^2 Q_n(\theta _0,\gamma _n)}{\partial \theta _j \partial \theta _l } + o_p(1) \right\} ({\hat{\theta }}_l - \theta _{0l}) \\&\qquad - (n-L) \left[ p_{\lambda _{j}}'(|\theta _{0j}|)\textrm{sign}(\theta _{0j}) + \left\{ p_{\lambda _{j}}''(|\theta _{0j}|) +o_p(1) \right\} ({\hat{\theta }}_j - \theta _{0j}) \right] =0 . \end{aligned}$$

The above equation can be rewritten as follows

$$\begin{aligned} \frac{ \partial Q_n(\theta _0,\gamma _n)}{\partial \theta _j}&= \sum _{l=1}^{s} \left\{ E \left\{ - \frac{ \partial ^2 Q_n(\theta _0,\gamma _n)}{\partial \theta _j \partial \theta _l }\right\} + o_p(1) \right\} (n-L)({\hat{\theta }}_l - \theta _{0l}) \\&\qquad +(n-L)\varDelta +(n-L)(\varSigma _1 + O_p(1) )({\hat{\theta }}_{n1} - \theta _{01}), \end{aligned}$$

and

$$\begin{aligned}&(n-L) I_1(\theta _{01},\gamma _0)({\hat{\theta }}_{n1} - \theta _{01}) +(n-L)\varDelta +(n-L)(\varSigma _1 + O_p(1) )({\hat{\theta }}_{n1} - \theta _{01}) \\&\quad =(n-L) (I_1(\theta _{01},\gamma _0)+ \varSigma _1 )({\hat{\theta }}_{n1} - \theta _{01}) +(n-L)\varDelta \\&\quad =(n-L) (I_1(\theta _{01},\gamma _0)+ \varSigma _1 ) \{({\hat{\theta }}_{n1} - \theta _{01}) + (I_1(\theta _{01},\gamma _0)+ \varSigma _1 )^{-1}\varDelta \} \\&\quad =\frac{ \partial Q_n(\theta _0,\gamma _n)}{\partial \theta _j} + o_p(1). \end{aligned}$$

Since $\sqrt{n}(\gamma _n- \gamma _0)=o_p(1)$, invoking the Slutsky’s lemma and the Lindeberg-Feller central limit theorem, we have

$$\begin{aligned} \sqrt{n-L} (I_1(\theta _{01},\gamma _0)+ \varSigma _1 ) \{({\hat{\theta }}_{n1} - \theta _{01}) + (I_1(\theta _{01},\gamma _0)+ \varSigma _1 )^{-1}\varDelta \} \rightarrow N({{\textbf {0}}},\varSigma _2), \end{aligned}$$

where $\varSigma _1 = {\text {diag}}\{p_{\lambda _{j}}''(|\theta _{01}|) ,\ldots , p_{\lambda _{j}}''(|\theta _{0s}|) \}$, $\varSigma _2 = \textrm{Cov}(\exp (-r^2/\gamma _0)\frac{2r}{\gamma _0} {\tilde{Z}}_{i1})$, $\varDelta = (p_{\lambda _{j}}'(|\theta _{01}|)\textrm{sign}(\theta _{01}),\ldots , p_{\lambda _{j}}'(|\theta _{0s}|)\textrm{sign}(\theta _{0s}) )^T$, and $I_1(\theta _{01},\gamma _0) = \frac{2}{\gamma _0} E[\exp (-r^2/\gamma _0)( \frac{2r^2}{\gamma _0}$ $-1) ] \times (E{\tilde{Z}}_{i1}{\tilde{Z}}_{i1}^T)$. Then the proof of Theorem 2(b) is completed. $\square$

Proof of Theorem 3

Let that $R(U_i) = g(U_i)-B(U_i)^T \eta$ and $R(U) = (R(U_1), \ldots ,$ $R(U_n))^T$. To facilitate expression, we set $Z = (WY, X), g(U) =(g(U_1), \ldots , g(U_n))^T$. Similar to Zhao et al. (2021), a simple calculation gives that

$$\begin{aligned}&{\hat{\eta }} - \eta =(S^T S)^{-1} S^T(Y - Z{\hat{\theta }}_n ) - \eta \nonumber \\&\quad =(S^T S)^{-1} S^T(Z\theta _0 + g(U) + \epsilon - Z{\hat{\theta }}_n ) - \eta \nonumber \\&\quad =(S^T S)^{-1} S^T(Z\theta _0 +g(U) - S\eta +S\eta +\epsilon - Z{\hat{\theta }}_n) - \eta \nonumber \\&\quad =(S^T S)^{-1} S^T(Z\theta _0 + R(U) + S\eta + \epsilon - Z{\hat{\theta }}_n) - \eta \nonumber \\&\quad =(S^T S)^{-1} S^T(Z(\theta _0 - {\hat{\theta }}_n) + R(U) + \epsilon +S\eta ) -\eta \nonumber \\&\quad =(S^T S)^{-1} S^TR(U) + (S^T S)^{-1} S^T\epsilon +(S^T S)^{-1} S^TZ(\theta _0 - {\hat{\theta }}_n) \nonumber \\&\quad =R_1 + R_2 + R_3 . \end{aligned}$$

(22)

By $\Vert R(u)\Vert =O(n^{-v/(2v+1)})$, we have

$$\begin{aligned} \Vert R_1\Vert \le \left( \frac{1}{n}\sum _{i=1}^n \Vert B(U_i) B(U_i)^T \Vert \right) ^{-1} \frac{1}{n}\sum _{i=1}^n \Vert B(U_i) R(U_i)\Vert = O_p(n^{-v/(2v+1)}). \end{aligned}$$

Note that $E\{B(U_i)\epsilon |X_i,U_i\}=0$, then by the central limit theorem we have $n^{-1/2}\sum _{i=1}^n$ $B(U_i)\epsilon _i = O_p(1)$. Therefore, we have

$$\begin{aligned} \Vert R_2\Vert \le \frac{1}{\sqrt{n}}\left( \frac{1}{n}\sum _{i=1}^n\Vert B(U_i)B(U_i)^T\Vert \right) ^{-1} \left\| \frac{1}{\sqrt{n}}\sum _{i=1}^n B(U_i)\epsilon _i\right\| =O_p(n^{-1/2}). \end{aligned}$$

By Theorem 1, we have $\sqrt{n}(\theta -{\hat{\theta }}_n) = O_p(1)$. Similar to the above proof, we can obtain $\Vert R_3\Vert = O_p(n^{-1/2})$. Hence, we have

$$\begin{aligned} \Vert {\hat{\eta }}-\eta \Vert = O_p(n^{-v/(2v+1)} + n^{-1/2} ) = O_p(n^{-v/(2v+1)}). \end{aligned}$$

(23)

Therefore,

$$\begin{aligned}&\Vert {\hat{g}}(u) - g(u) \Vert ^2 = \int _0^1 \{{\hat{g}}(u)-g(u) \}^2du \nonumber \\&\quad =\int _0^1\{B^T(u) {\hat{\eta }} - B^T(u)\eta +R(u) \}^2du \nonumber \\&\quad \le 2 \int _0^1\{B^T(u) {\hat{\eta }} - B^T(u)\eta \}^2du + 2\int _0^1R(u)^2du \nonumber \\&\quad = 2({\hat{\eta }}-\eta )^T \int _0^1B(u)B(u)^Tdu({\hat{\eta }}-\eta ) + 2\int _0^1R(u)^2du , \end{aligned}$$

(24)

Note that $\Vert \int _0^1B(u)B(u)^Tdu\Vert = O(1)$, and thus invoking (22) gives

$$\begin{aligned} ({\hat{\eta }}-\eta )^T \int _0^1B(u)B(u)^Tdu({\hat{\eta }}-\eta ) = O_p(n^{-2v/(2v+1)}). \end{aligned}$$

From $\Vert R(u)\Vert =O(n^{-v/(2v+1)})$, we have

$$\begin{aligned} \int _0^1R(u)^2du = O_p(n^{-2v/(2v+1)}). \end{aligned}$$

As a result, $\Vert {\hat{g}}(u) - g(u) \Vert ^2 = O_p(n^{-2v/(2v+1)})$. This completes the proof of Theorem 3. $\square$