Estimation and clustering for partially heterogeneous single index model

Abstract

In this paper, our goal is to estimate the homogeneous parameter and cluster the heterogeneous parameters in a partially heterogeneous single index model (PHSIM). To achieve the goal, the minimization criterion for such a single index model is first transformed into a least-squares optimization problem in the population form. Based on the least-squares objective function, we introduce an empirical version for the PHSIM. By minimizing such an empirical version, we estimate the homogeneous parameter and the subgroup-averages of the heterogeneous index directions, and then use a fusion penalized method to identify the subgroup structure of the PHSIM. By the proposed methodologies, the homogeneous parameter and the heterogeneous index directions can be consistently estimated, and the heterogeneous parameters can be consistently clustered. Moreover, the new clustering procedure is simple and robust. Simulation studies are carried out to examine the performance of the proposed methodologies.

Appendix

Proof of Proposition 1

According to the Proposition 2 in Wang et al. (2015), under conditions (B0),(B1)and (B2), there exist constants $\phi _1\ne 0$ and $\phi _2\ne 0$ such that $\varvec{\rho }_{LS}=(\phi _1\varvec{\beta }^T,\phi _2\varvec{\theta }^T)^T$. In the spirit of the justification of Theorem 2 of Wang et al. (2015), we only give a sketch to prove $\phi _1=1$.

(I) Note that

$$\begin{aligned} \begin{aligned} \varvec{\rho }_{LS}&={\begin{pmatrix}\Sigma _{X,X}&{}\Sigma _{X,Z}\\ \Sigma _{Z,X}&{}\Sigma _{Z,Z}\\ \end{pmatrix}}^{-1} \begin{pmatrix}\text{ Cov }(X,Y)\\ \text{ Cov }(Z,Y) \end{pmatrix}. \end{aligned} \end{aligned}$$

(6.1)

When the condition (A1) of the Proposition 1 holds,

$$\begin{aligned} \begin{aligned} \varvec{\rho }_{LS}&={\begin{pmatrix}\Sigma _{X,X}&{}0\\ 0&{}\Sigma _{Z,Z}\\ \end{pmatrix}}^{-1} \begin{pmatrix}\Sigma _{X,X}^T\varvec{\beta }\\ E[Zg_0(Z^T\varvec{\theta })] \end{pmatrix}\\&=\begin{pmatrix}\varvec{\beta }\\ \Sigma _{Z,Z}^{-1}E[Zg_0(Z^T\varvec{\theta })] \end{pmatrix}. \end{aligned} \end{aligned}$$

(6.2)

Clearly, we have $\phi _1=1$.

(II) Under the condition (A2) of the Proposition 1, define

$$\begin{aligned} \begin{aligned} S&={\begin{pmatrix}\varvec{\beta }&{}0_{p\times 1}\\ 0_{q\times 1}&{}\varvec{\theta } \end{pmatrix}}, \end{aligned} \end{aligned}$$

(6.3)

and

$$\begin{aligned} \begin{aligned} \phi =(S^T\Sigma S)^{-1}S^T{\begin{pmatrix}E(XY)\\ E(ZY) \end{pmatrix}}, \end{aligned} \end{aligned}$$

(6.4)

where $\Sigma ={\begin{pmatrix}\Sigma _{X,X}&{}\Sigma _{X,Z}\\ \Sigma _{Z,X}&{}\Sigma _{Z,Z}\\ \end{pmatrix}}$. Because the link function $g_0(\cdot )$ is analytic, it suffices to consider the case where $g_0(t)=\Sigma _{k=1}^{\infty }a_kt^k$. Under the condition (A2) of the Proposition 1, the bivariate random vector $(\tilde{X},\tilde{Z})=(X^T\varvec{\beta }, Z^T\varvec{\theta })$ has an elliptically symmetric distribution with mean vector zero. It follows that $E(\tilde{Z}^{k+1})=0$ if k is even; $E(\tilde{Z}^{k+1})=c_r\{E(\tilde{Z}^2)\}^r$ and $E(\tilde{X}\tilde{Z}^k)=c_rE(\tilde{X}\tilde{Z})\{E(\tilde{Z}^2)\}^{r-1}$ for some constant $c_r$ if $k+1=2r$. As a result, we have

$$\begin{aligned} \sum _{k+1=2r}a_kc_rE(\tilde{X}\tilde{Z})\{E(\tilde{Z}^2)\}^{r-1}=\phi _2E(\tilde{X}\tilde{Z}), \sum _{k+1=2r}a_kc_r\{E(\tilde{Z}^2)\}^r=\phi _2E(\tilde{Z}^2), \end{aligned}$$

(6.5)

where the summation $\sum _{k+1=2r}$ is taken over the set $\{k:k\ge 1, k+1=2r\}.$ Therefore, we can take $\phi _2=\sum _{k+1=2r}a_kc_r\{E(\tilde{Z}^2)\}^{r-1}.$ We have completed the proof of the Proposition 1. $\square$

Proof of Theorem 1

Note that $\varvec{\theta }$ is heterogeneous. By objective function (2.4), we have the following empirical versions:

$$\begin{aligned} \overline{\mathbf {X} \mathbf {y}}= & {} \overline{\mathbf {X} \mathbf {X}^T}\varvec{\beta }+\frac{1}{n} \sum _{i=1}^n \mathbf {x}_i\mathbf {z}_i^T\varvec{\theta }_i+\frac{1}{n} \sum _{i=1}^n \mathbf {x}_i\varepsilon _i,\\ \overline{\mathbf {Z} \mathbf {y}}= & {} \overline{\mathbf {Z} \mathbf {X}^T}\varvec{\beta }+\frac{1}{n} \sum _{i=1}^n \mathbf {z}_i\mathbf {z}_i^T\varvec{\theta }_i+\frac{1}{n} \sum _{i=1}^n \mathbf {z}_i\varepsilon _i. \end{aligned}$$

Then

$$\begin{aligned}&\overline{\mathbf {X} \mathbf {y}}-\overline{\mathbf {X} \mathbf {Z}^T}(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\overline{\mathbf {Z} \mathbf {y}}\\&\quad =\Big (\overline{\mathbf {X} \mathbf {X}^T}-\overline{\mathbf {X} \mathbf {Z}^T}(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\overline{\mathbf {Z} \mathbf {X}^T}\Big )\varvec{\beta } +\frac{1}{n} \sum _{i=1}^n \mathbf {x}_i\mathbf {z}_i^T\Big (\varvec{\theta }_i-(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\frac{1}{n} \sum _{i=1}^n \mathbf {z}_i\mathbf {z}_i^T\varvec{\theta }_i\Big )\\&\qquad +\frac{1}{n} \sum _{i=1}^n\Big (\mathbf {x}_i-\overline{\mathbf {X} \mathbf {Z}^T}(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\mathbf {z}_i\Big )\varepsilon _i. \end{aligned}$$

By the result above and estimator (2.7), we get

$$\begin{aligned} \widehat{\varvec{\beta }}-\varvec{\beta }= & {} \Big (\overline{\mathbf {X} \mathbf {X}^T}-\overline{\mathbf {X} \mathbf {Z}^T}(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\overline{\mathbf {Z} \mathbf {X}^T}\Big )^{-1}\frac{1}{n} \sum _{i=1}^n \mathbf {x}_i\mathbf {z}_i^T\Big (\varvec{\theta }_i-(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\frac{1}{n} \sum _{i=1}^n \mathbf {z}_i\mathbf {z}_i^T\varvec{\theta }_i\Big )\\&+\Big (\overline{\mathbf {X} \mathbf {X}^T}-\overline{\mathbf {X} \mathbf {Z}^T}(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\overline{\mathbf {Z} \mathbf {X}^T}\Big )^{-1}\frac{1}{n} \sum _{i=1}^n\Big (\mathbf {x}_i-\overline{\mathbf {X} \mathbf {Z}^T}(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\mathbf {z}_i\Big )\varepsilon _i. \end{aligned}$$

Because

$$\begin{aligned} \overline{\mathbf {X} \mathbf {X}^T}-\overline{\mathbf {X} \mathbf {Z}^T}(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\overline{\mathbf {Z} \mathbf {X}^T}\xrightarrow {P}\Phi =E[XX^T]-E[XZ^T]\big (E[ZZ^T]\big )^{-1}E[ZX^T], \end{aligned}$$

we can obtain that $\sqrt{n}\Phi (\widehat{\varvec{\beta }}-\varvec{\beta })$ is asymptotically identically distributed as

$$\begin{aligned} \frac{1}{\sqrt{n}}\sum _{i=1}^n\mathbf {x}_i\mathbf {z}_i^T\Omega _i+\frac{1}{\sqrt{n}}\sum _{i=1}^n\Big (\mathbf {x}_i-\overline{\mathbf {X} \mathbf {Z}^T}(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\mathbf {z}_i\Big )\varepsilon _i, \end{aligned}$$

(6.6)

where the weight $\Omega _i=\Big (\varvec{\theta }_i-(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\frac{1}{n} \sum _{i=1}^n \mathbf {z}_i\mathbf {z}_i^T\varvec{\theta }_i\Big )$. Note that $(\mathbf {x}_i^T, \mathbf {z}_i^T)^T, i=1, \cdots , n$, are independent and identically distributed as $(X^T,Z^T)^T$. We get

$$\begin{aligned}&E\bigg (\frac{1}{n} \sum _{i=1}^n \mathbf {z}_i\mathbf {z}_i^T\varvec{\theta }_i\bigg )=\frac{1}{n} \sum _{i=1}^n E\big (\mathbf {z}_i\mathbf {z}_i^T\big )\varvec{\theta }_i=E\big (ZZ^T\big )\frac{1}{n} \sum _{i=1}^n\varvec{\theta }_i,\\&\quad Cov\bigg (\frac{1}{n} \sum _{i=1}^n \mathbf {z}_i\mathbf {z}_i^T\varvec{\theta }_i\bigg )=\frac{1}{n^2}\sum _{i=1}^nCov\big (\mathbf {z}_i\mathbf {z}_i^T\varvec{\theta }_i\big )=\frac{1}{n^2}\sum _{i=1}^n\varvec{\theta }_i^T\Delta \varvec{\theta }_i=O(1\sqrt{n}), \end{aligned}$$

where $\Delta =E\bigg (\big (ZZ^T-E(ZZ^T)\big )\big (ZZ^T-E(ZZ^T)\big )\bigg )$. The above indicates that

$$\begin{aligned} \frac{1}{n} \sum _{i=1}^n \mathbf {z}_i\mathbf {z}_i^T\varvec{\theta }_i=E\big (ZZ^T\big )\frac{1}{n} \sum _{i=1}^n\varvec{\theta }_i+O_p(1/\sqrt{n}) \end{aligned}$$

and then $\Omega _i\xrightarrow {P}(\varvec{\theta }_i-\overline{\varvec{\theta }})$. Thus, the weighted sum $\frac{1}{\sqrt{n}}\sum _{i=1}^n\mathbf {x}_i\mathbf {z}_i^T\Omega _i$ can be expressed as

$$\begin{aligned} \frac{1}{\sqrt{n}}\sum _{i=1}^n\mathbf {x}_i\mathbf {z}_i^T(\varvec{\theta }_i-\overline{\varvec{\theta }})+o_p(1)\frac{1}{\sqrt{n}}\sum _{i=1}^n\mathbf {x}_i\mathbf {z}_i^T(\varvec{\theta }_i-\overline{\varvec{\theta }}) \end{aligned}$$

and then is asymptotically identically distributed as $\frac{1}{\sqrt{n}}\sum _{i=1}^n\mathbf {x}_i\mathbf {z}_i^T(\varvec{\theta }_i-\overline{\varvec{\theta }})$. It can be seen that

$$\begin{aligned} E\bigg (\frac{1}{\sqrt{n}}\sum _{i=1}^n\mathbf {x}_i\mathbf {z}_i^T(\varvec{\theta }_i-\overline{\varvec{\theta }})\bigg )=0 \end{aligned}$$

and

$$\begin{aligned}&Var\bigg (\frac{1}{\sqrt{n}}\sum _{i=1}^n\mathbf {x}_i\mathbf {z}_i^T(\varvec{\theta }_i-\overline{\varvec{\theta }})\bigg )\\&\quad =\frac{1}{n}\sum _{i=1}^n(\varvec{\theta }_i-\overline{\varvec{\theta }})^TE\bigg (\big (ZX^T-E(ZX^T)\big )\big (ZX^T-E(ZX^T)\big )^T\bigg )(\varvec{\theta }_i-\overline{\varvec{\theta }}). \end{aligned}$$

Therefore,

$$\begin{aligned} \frac{1}{\sqrt{n}}\sum _{i=1}^n\mathbf {x}_i\mathbf {z}_i^T(\varvec{\theta }_i-\overline{\varvec{\theta }})\xrightarrow {D}N(0,\Psi ). \end{aligned}$$

(6.7)

In addition, we can also get that

$$\begin{aligned} \frac{1}{\sqrt{n}} \sum _{i=1}^n\Big (\mathbf {x}_i-\overline{\mathbf {X} \mathbf {Z}^T}(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\mathbf {z}_i\Big )\varepsilon _i\xrightarrow {D}N(0,\sigma ^2\Lambda ), \end{aligned}$$

(6.8)

where $\Lambda =Var\bigg (X-E\big (XZ^T\big )\big (E(ZZ^T)\big )^{-1}Z\bigg )$. By (6.6), (6.7), (6.8) and the uncorrelation between (X, Z) and $\varepsilon$, we get that

$$\begin{aligned} \sqrt{n}(\widehat{\varvec{\beta }}-\varvec{\beta })\xrightarrow {D}N(0, \Phi ^{-1}(\Psi +\sigma ^2\Lambda )\Phi ^{-1}). \end{aligned}$$

(6.9)

Thus, the theorem is proved. $\square$

Proof of Theorem 2

We only prove the first result. The proof of the second result is similar. Based on the objective function (2.4), the estimator (2.8) can be denoted as the following empirical form:

$$\begin{aligned} \begin{aligned} \widehat{\overline{\varvec{\theta }}}&=\big (\overline{\mathbf {Z} \mathbf {Z}^T}\big )^{-1}\big (\overline{\mathbf {Z} \mathbf {y} }-\overline{\mathbf {Z} \mathbf {X}^T}\widehat{\varvec{\beta }}\big )\\&=\big (\overline{\mathbf {Z} \mathbf {Z}^T}\big )^{-1}\bigg (\overline{\mathbf {Z} \mathbf {X}^T}\varvec{\beta }+\frac{1}{n} \sum _{i=1}^n \mathbf {z}_i\mathbf {z}_i^T\varvec{\theta }_i+\frac{1}{n} \sum _{i=1}^n \mathbf {z}_i\varepsilon _i-\overline{\mathbf {Z} \mathbf {X}^T}\widehat{\varvec{\beta }}\bigg )\\&=\overline{\varvec{\theta }}+\big (\overline{\mathbf {Z} \mathbf {Z}^T}\big )^{-1}\bigg (\frac{1}{n} \sum _{i=1}^n \mathbf {z}_i\mathbf {z}_i^T(\varvec{\theta }_i-\overline{\varvec{\theta }})+\frac{1}{n} \sum _{i=1}^n \mathbf {z}_i\varepsilon _i-\overline{\mathbf {Z} \mathbf {X}^T}(\widehat{\varvec{\beta }}-\varvec{\beta })\bigg ). \end{aligned} \end{aligned}$$

(6.10)

From the proof of the Theorem 1, $\sqrt{n}\Phi (\widehat{\varvec{\beta }}-\varvec{\beta })$ is asymptotically identically distributed as

$$\begin{aligned} \frac{1}{\sqrt{n}}\sum _{i=1}^n \mathbf {x}_i\mathbf {z}_i^T(\varvec{\theta }_i-\overline{\varvec{\theta }})+\frac{1}{\sqrt{n}}\sum _{i=1}^n\big (\mathbf {x}_i-\overline{\mathbf {X} \mathbf {Z}^T}(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\mathbf {z}_i\big )\varepsilon _i. \end{aligned}$$

By the above results, we can easily see that $\sqrt{n}(\widehat{\overline{\varvec{\theta }}}-\overline{\varvec{\theta }})$ is asymptotically identically distributed as

$$\begin{aligned}&(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\frac{1}{\sqrt{n}}\sum _{i=1}^n\Big (\mathbf {z}_i\mathbf {z}_i^T-\overline{\mathbf {Z} \mathbf {X}^T}\Phi ^{-1}\mathbf {x}_i\mathbf {z}_i^T\Big )(\varvec{\theta }_i-\overline{\varvec{\theta }})\\&\quad +(\overline{\mathbf {Z} \mathbf {Z}^T})^{-1}\Bigg (\frac{1}{\sqrt{n}}\sum _{i=1}^n\bigg (\mathbf {z}_i-\overline{\mathbf {Z} \mathbf {X}^T}\Phi ^{-1}\Big (\mathbf {x}_i-\overline{\mathbf {X} \mathbf {Z}^T}\big (\overline{\mathbf {Z} \mathbf {Z}^T}\big )^{-1}\mathbf {z}_i\Big )\bigg )\varepsilon _i\Bigg ). \end{aligned}$$

Thus,

$$\begin{aligned} \sqrt{n}(\widehat{\overline{\varvec{\theta }}}-\overline{\varvec{\theta }})\xrightarrow {D}N\Big (0, \big (E(ZZ^T)\big )^{-1}\big (\Gamma +\sigma ^2\Upsilon \big )\big (E(ZZ^T)\big )^{-1}\Big ). \end{aligned}$$

(6.11)

$\square$

Proof of Theorem 3

For given $\beta ^0$, based on the sample $W_i(\beta ^0)=\mathbf {z}_i(y_i-\mathbf {x}_i^T\beta ^0), i=1,\cdots ,n$ of $W(\beta ^0)$, the corresponding estimators of $L, R, G_{L,R}, \cdots$, are denoted by $\tilde{L}(\beta ^0), \tilde{R}(\beta ^0), \tilde{G}_{L,R}(\beta ^0), \cdots ,$ respectively. Using the sample $\widehat{W}_i=\mathbf {z}_i(y_i-\mathbf {x}_i\widehat{\beta }), i=1,\cdots ,n$, the corresponding estimators are denoted by $\widehat{L}, \widehat{R}, \widehat{G}_{L,R}, \cdots$, respectively. By the asymptotic property of the estimator of $\widehat{\beta }$ given in Theorem 1, and the continuous-mapping theorem for strong consistency (see, e.g., Van der Vaart (2000)), we have $\widehat{L}-\tilde{L}(\beta ^0)\xrightarrow {a.s.}0, \widehat{R}-\tilde{R}(\beta ^0)\xrightarrow {a.s.}0$ and $\widehat{G}_{L,R}-\tilde{G}_{L,R}(\beta ^0)\xrightarrow {a.s.}0, \cdots$. Then, for any $\varepsilon >0$ and for all sufficiently large n, the following inequalities hold almost everywhere:

$$\begin{aligned} |\widehat{L}-\tilde{L}(\beta ^0)|<\varepsilon ,| \widehat{R}-\tilde{R}(\beta ^0)|<\varepsilon ,|\widehat{G}_{L,R}-\tilde{G}_{L,R}(\beta ^0)|<\varepsilon ,\cdots . \end{aligned}$$

This implies that the estimator $\widehat{L}, \widehat{R}, \widehat{G}_{L,R}, \cdots$, can be replaced by $\tilde{L}(\beta ^0), \tilde{L}(\beta ^0), \tilde{R}(\beta ^0), \tilde{G}_{L,R}(\beta ^0),\cdots$, respectively, in the proof procedure (see Lemma 1-Lemma 3 in supplementary Material of Radchenko and Mukherjee 2017). Then, based on the population $W(\beta ^0)$, its expectation $\alpha$ and the sample from $W(\beta ^0)$, by the proof of Theorem 1 of Radchenko and Mukherjee (2017), we can prove Theorem 3. $\square$