Robust dimension reduction using sliced inverse median regression

Abstract

Dimension reduction is a useful technique when working with high-dimensional predictors, as meaningful data visualizations and graphical analyses using fewer predictors can be achieved. We propose a new non-iterative and robust against extreme values estimation of the effective dimension reduction (e.d.r) subspace, which is based on the estimation of the conditional median function of the predictors given the response. The existing literature on robust estimation of the e.d.r subspace relies on iterative algorithms, such as the composite quantile minimum average variance estimation and the sliced regression. Compared with these existing robust dimension reduction methods, the new method avoids iterations by directly estimating the e.d.r subspace and has better finite sample performance. It is shown that the inverse Tukey and Oja median regression curve falls into the e.d.r subspace, and that its directions can be estimated $\sqrt{n}$-consistently.

Appendix

Proof of Theorem 2.4: Without loss of generality, assume that $med(\mathbf {X})=\mathbf {0}$. We will first prove the result for $\varvec{\Sigma }_{\mathbf {xx}}=\mathbf {I}_{p}$ (spherical distribution), and then we will extend it for a general $\varvec{\Sigma }_{\mathbf {xx}}$ (elliptical distribution).

Let $\varOmega _{Y}$ be the sample space of Y, J be a subset of $\varOmega _{Y}$, and define $S_{1}(\mathbf {m})=dep_{\mathbf {X}| Y \in J}(\mathbf {m})$ and $S_{2}(\mathbf {m})=E \{ \mathcal {V}(\mathbf {X}_{1}, \ldots , \mathbf {X}_{p}, \mathbf {m}) | Y \in J\}$. Let $\mathbf {A}$ be a $(p-d_{0}) \times (p-d_{0})$ orthogonal matrix that is not $\mathbf {I}_{p-d_{0}}$, $\mathbf {u}_{1}, \ldots , \mathbf {u}_{d_{0}}$ be an orthonormal basis for $\mathcal {S}_{y|\mathbf {x}}$, and $\mathbf {w}_{1}, \ldots , \mathbf {w}_{p-d_{0}}$ be an orthonormal basis for $\mathcal {S}^{\perp }_{y|\mathbf {x}}$. Define $\mathbf {R}=\mathbf {U}\mathbf {U}^{\top }+\mathbf {W}\mathbf {A}\mathbf {W}^{\top }$, where $\mathbf {U}=(\mathbf {u}_{1}, \ldots , \mathbf {u}_{d_{0}})$, and $\mathbf {W}=(\mathbf {w}_{1}, \ldots , \mathbf {w}_{p-d_{0}})$. Note that (a) if $\mathbf {v} \in \mathcal {S}_{y|\mathbf {x}}^{\perp }$, then $\mathbf {R}$ rotates $\mathbf {v}$, and (b) if $\mathbf {v} \in \mathcal {S}_{y|\mathbf {x}}$, then $\mathbf {R}$ leaves $\mathbf {v}$ unchanged.

Since $\mathbf {X}$ has a spherical distribution, $\mathbf {X}$ and $\mathbf {R}\mathbf {X}$ have the same distribution. Let $\mathbf {P}$ be the projection onto $\mathcal {S}_{y|\mathbf {x}}$, and $\mathbf {Q}=\mathbf {I}_{p}-\mathbf {P}$ be the projection onto $\mathcal {S}_{y|\mathbf {x}}^{\perp }$. Moreover, since , then $Y | \mathbf {X}$ and $Y | \mathbf {R} \mathbf {X}$ have the same conditional distribution. Therefore, $(Y,\mathbf {X})$ and $(Y,\mathbf {R}\mathbf {X})$ have the same distribution.

Moreover, the depth and the volume are invariant under rotation. Therefore, $dep_{\mathbf {X}}(\mathbf {m})=dep_{\mathbf {RX}}(\varvec{\mathbf {R}m})$ and $\mathcal {V}(\mathbf {X}_{1} ,\ldots , \mathbf {X}_{p}, \mathbf {m})=\mathcal {V}(\mathbf {RX}_{1}, \ldots , \mathbf {RX}_{p},$$\mathbf {Rm})$. Hence,

$$\begin{aligned} S_{1}(\mathbf {m})=dep_{\mathbf {RX}|Y \in J}(\mathbf {Rm})=dep_{\mathbf {X}|Y \in J}(\mathbf {Rm})=S_{1}(\mathbf {Rm}). \end{aligned}$$

Similarly,

$$\begin{aligned} S_{2}(\mathbf {m})= & {} E \{\mathcal {V}(\mathbf {R}\mathbf {X}_{1}, \ldots , \mathbf {R}\mathbf {X}_{p}, \mathbf {Rm}) | Y \in J\}\\= & {} E \{\mathcal {V}(\mathbf {X}_{1}, \ldots , \mathbf {X}_{p}, \mathbf {Rm}) | Y \in J\}=S_{2}(\mathbf {Rm}). \end{aligned}$$

Let $\varvec{\alpha }_{1}^{*}=tmed(\mathbf {X}|Y\in J)$ and suppose that $\varvec{\alpha }_{1}^{*} \notin \mathcal {S}_{y|\mathbf {x}}$. Then, $\varvec{\alpha }_{1}^{*}=\mathbf {P}\varvec{\alpha }_{1}^{*}+\mathbf {Q}\varvec{\alpha }_{1}^{*}$, where $\mathbf {Q}\varvec{\alpha }_{1}^{*} \ne \mathbf {0}$. By the definition of $\mathbf {R}$, $\mathbf {R}\varvec{\alpha }_{1}^{*}=\mathbf {P}\varvec{\alpha }_{1}^{*}+\mathbf {WAW}^{\top }\mathbf {Q}\varvec{\alpha }_{1}^{*}$. Then $\mathbf {WAW}^{\top }\mathbf {Q}\varvec{\alpha }_{1}^{*} \ne \mathbf {Q}\varvec{\alpha }_{1}^{*}$. But, $S_{1}(\varvec{\alpha }_{1}^{*})=S_{1}(\mathbf {R}\varvec{\alpha }_{1}^{*})=\mathbf {R}S_{1}(\varvec{\alpha }_{1}^{*})$ (affine equivariance property) and $\varvec{\alpha }_{1}^{*}$ cannot be $tmed(\mathbf {X}|Y \in J)$. Contradiction. Similar steps give the result for the Oja median.

Now, if $\varvec{\Sigma }_{\mathbf {xx}} \ne \mathbf {I}_{p}$ (elliptical distribution), then $\varvec{\Sigma }_{xx}^{-1}\mathbf {X}$ has a covariance matrix of $\mathbf {I}_{p}$ (spherical distribution), and $\varvec{\Sigma }_{\mathbf {xx}}^{-1} med(\mathbf {X}|Y)= med(\varvec{\Sigma }_{\mathbf {xx}}^{-1} \mathbf {X}|Y) \in span\{\varvec{\beta }_{01}, \ldots , \varvec{\beta }_{0d_{0}}\}$, where the first equality follows from the affine equivariant property.

Proof of Corollary 3.1: We will first show that $\widehat{\mathbf {V}}$, defined in (2), converges to

$$\begin{aligned} \mathbf {V}= \sum _{h=1}^{H} p_{h} med(\mathbf {Z}|Y \in I_{h})med(\mathbf {Z}^{\top }|Y \in I_{h}). \end{aligned}$$

Letting $\left\| \cdot \right\|$ to be the Frobenius norm, consider the difference

$$\begin{aligned} \left\| \widehat{\mathbf {V}}-\mathbf {V} \right\|= & {} \left\| \sum _{h=1}^{H} \left( \widehat{p}_{h}med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h})med_{n}(\widetilde{\mathbf {Z}}^{\top }|Y \in I_{h}) \right. \right. \\&\left. \left. - p_{h} med(\mathbf {Z}|Y \in I_{h})med(\mathbf {Z}^{\top }|Y \in I_{h}) \right) \right\| \\\le & {} \sum _{h=1}^{H} \left\| \widehat{p}_{h}med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h})med_{n}(\widetilde{\mathbf {Z}}^{\top }|Y \in I_{h}) \right. \\&\left. - p_{h} med(\mathbf {Z}|Y \in I_{h})med(\mathbf {Z}^{\top }|Y \in I_{h}) \right\| \\\le & {} H \max _{h=1,...,H} \left\| \widehat{p}_{h}med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h})med_{n}(\widetilde{\mathbf {Z}}^{\top }|Y \in I_{h}) \right. \\&\left. - p_{h} med(\mathbf {Z}|Y \in I_{h})med(\mathbf {Z}^{\top }|Y \in I_{h}) \right\| , \end{aligned}$$

where the first inequality follows from the triangular inequality. Therefore, since $H<\infty$, it is enough to show that $\widehat{p}_{h}med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h})med_{n}(\widetilde{\mathbf {Z}}^{\top }|Y \in I_{h})$ converge to $p_{h} med(\mathbf {Z}|Y \in I_{h})med(\mathbf {Z}^{\top }|Y \in I_{h})$, for every $h=1, \ldots , H$. Note that

$$\begin{aligned}&\left\| \widehat{p}_{h}med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h})med_{n}(\widetilde{\mathbf {Z}}^{\top }|Y \in I_{h}) - p_{h} med(\mathbf {Z}|Y \in I_{h})med(\mathbf {Z}^{\top }|Y \in I_{h}) \right\| \\&\quad =\left\| \widehat{p}_{h}med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h})med_{n}(\widetilde{\mathbf {Z}}^{\top }|Y \in I_{h}) - p_{h} med(\mathbf {Z}|Y \in I_{h})med(\mathbf {Z}^{\top }|Y \in I_{h}) \right. \\&\qquad \left. \pm \, \widehat{p}_{h}med(\mathbf {Z}|Y \in I_{h})med(\mathbf {Z}^{\top }|Y \in I_{h}) \right\| \\&\quad \le \left\| \widehat{p}_{h} \left( med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h})med_{n}(\widetilde{\mathbf {Z}}^{\top }|Y \in I_{h}) - med(\mathbf {Z}|Y \in I_{h})med(\mathbf {Z}^{\top }|Y \in I_{h}) \right) \right\| \\&\qquad +\, \left\| (\widehat{p}_{h}-p_{h}) med(\mathbf {Z}|Y \in I_{h})med(\mathbf {Z}^{\top }|Y \in I_{h}) \right\| \\&\quad = \left\| \widehat{p}_{h} \left( med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h})med_{n}(\widetilde{\mathbf {Z}}^{\top }|Y \in I_{h}) - med(\mathbf {Z}|Y \in I_{h})med(\mathbf {Z}^{\top }|Y \in I_{h}) \right. \right. \\&\qquad \left. \left. \pm \, med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h}) med(\mathbf {Z}^{\top }|Y \in I_{h}) \right) \right\| \\&\qquad +\, \left\| (\widehat{p}_{h}-p_{h}) med(\mathbf {Z}|Y \in I_{h})med(\mathbf {Z}^{\top }|Y \in I_{h}) \right\| \\&\quad \le \widehat{p}_{h} \left\| med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h}) \left( med_{n}(\widetilde{\mathbf {Z}}^{\top }|Y \in I_{h}) - med(\mathbf {Z}^{\top }|Y \in I_{h}) \right) \right\| \\&\qquad +\, \widehat{p}_{h} \left\| \left( med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h}) - med(\mathbf {Z}|Y \in I_{h}) \right) med(\mathbf {Z}^{\top }|Y \in I_{h}) \right\| \\&\qquad +\, |\widehat{p}_{h}-p_{h}| \left\| med(\mathbf {Z}|Y \in I_{h}) med(\mathbf {Z}^{\top }|Y \in I_{h}) \right\| \end{aligned}$$

Note that

• 1. elementary probability theory shows that $\widehat{p}_{h}$ converges to $p_{h}$,

• 2. $med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h})$ converges to $med(\mathbf {Z} | Y \in I_{h})$ for both the Tukey and Oja medians. The convergence of the Tukey and Oja medians follows from their asymptotic normality distribution. See for example Bai and He (1999), Nolan (1999), and Massé (2002) for the regularity conditions and the proof of the asymptotic normality for the Tukey median, and Arcones et al. (1994), Hettmansperger et al. (1997), and Shen (2008) for the corresponding conditions and proof for the Oja median.

• 3. $\widehat{p}_{h}$ is bounded in probability,

• 4. $med(\mathbf {Z}|Y \in I_{h})$ exists for both the Tukey and Oja medians. See for example Donoho and Gasko (1992) who state that Tukey median always exists and Oja (1983) for a similar conclusion for the Oja median.

• 5. $med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h})$ is bounded in probability for both Tukey and Oja medians. This follows from the fact that $med(\mathbf {Z}|Y \in I_{h})$ exists and $med_{n}(\widetilde{\mathbf {Z}}|Y \in I_{h})$ converges to $med(\mathbf {Z} | Y \in I_{h})$.

Using all the above facts, the weighted covariance-type $\widehat{\mathbf {V}}$ converges to $\mathbf {V}$ according to the Frobenius norm. Consequently, the consistency of $\widehat{\mathbf {V}}$ to $\mathbf {V}$, along with the assumption that all eigenvalues of $\mathbf {V}$ are distinct, imply that the $d_{0}$ eigenvectors of $\widehat{\mathbf {V}}$, $\widehat{\varvec{\eta }}_{k}$, $k=1,...,d_{0}$, converge to the corresponding eigenvectors of $\mathbf {V}$. This is a direct consequence of the Davis–Kahan $sin\theta$ theorem (Davis and Kahan 1970). Following, using Theorem 2.4, which implies that $med(\mathbf {Z}|Y)$ is contained in the standardized e.d.r subspace spanned by $\varvec{\eta }_{0k}$, $k=1,...,d_{0}$, we can see that the $d_{0}$ eigenvectors of $\mathbf {V}$ fall into that subspace.

Since $\widehat{\varvec{\beta }}_{k}=\widehat{\varvec{\Sigma }}_{\mathbf {xx}}^{-1/2} \widehat{\varvec{\eta }}_{k}$ and, according to probability theory, $\widehat{\varvec{\Sigma }}_{\mathbf {xx}}^{-1/2}$ converges to $\varvec{\Sigma }_{\mathbf {xx}}^{-1/2}$ at $\sqrt{n}$-rate according to the Frobenius norm, then, $\widehat{\varvec{\beta }}_{k}$, $k=1,...,d_{0}$ converge to the e.d.r directions $\varvec{\beta }_{0k}$, $k=1,...,d_{0}$.