Aspects of robust canonical correlation analysis, principal components and association

Abstract

Principal component analysis (PCA) and canonical correlation analysis (CCA) are dimension-reduction techniques in which either a random vector is well approximated in a lower dimensional subspace or two random vectors from high dimensional spaces are reduced to a new pair of low dimensional vectors after applying linear transformations to each of them. In both techniques, the closeness between the higher dimensional vector and the lower representations is under concern, measuring the closeness through a robust function. Robust SM-estimation has been treated in the context of PCA and CCA showing an outstanding performance under casewise contamination, which encourages the study of asymptotic properties. We analyze consistency and asymptotic normality for the SM-canonical vectors. As a by-product of the CCA derivations, the asymptotics for PCA can also be obtained. A classical measure of robustness as the influence function is analyzed, showing the usual performance of S-estimation in different statistical models. The general ideas behind SM-estimation in either PCA or CCA are specially tailored to the context of association, rendering robust measures of association between random variables. By these means, a robust correlation measure is derived and the connection with the association measure provided by S-estimation for bivariate scatter is analyzed. On the other hand, we also propose a second robust correlation measure which is reminiscent of depth-based procedures.

Appendix

Proof of Theorem 1

Let ${\mathscr {G}}$ be as in (22), $\varvec{\varXi }_o=\varvec{\varSigma }_{{{\textbf {x}}}{{\textbf {x}}}}$, $\varvec{\varGamma }_o=\varvec{\varSigma }_{{{\textbf {y}}}{{\textbf {y}}}}$ and the sets

$$\begin{aligned} \begin{array}{rl} E_{\eta }&{}= \left\{ \begin{array}{l} ({{\textbf {A}}},{{\textbf {B}}},{{\textbf {a}}},\varvec{\varXi },\varvec{\varGamma })\in {\mathscr {O}}_{r,p}\times {\mathscr {O}}_{r,q}\times {\mathbb {R}}^{r}\times S_{p}\times S_{q}:\\ \Vert {{\textbf {A}}}-{{\textbf {A}}}_o\Vert<\eta ,\Vert {{\textbf {B}}}-{{\textbf {B}}}_o\Vert<\eta ,\Vert \varvec{\varXi }-\varvec{\varXi }_o\Vert<\eta ,\Vert \varvec{\varGamma }-\varvec{\varGamma }_o\Vert<\eta \end{array} \right\} \\ {\tilde{E}}_{\eta }&{}= \left\{ \begin{array}{l} ({{\textbf {A}}},{{\textbf {B}}},{{\textbf {a}}},\varvec{\varXi },\varvec{\varGamma })\in {\mathscr {O}}_{r,p}\times {\mathscr {O}}_{r,q}\times {\mathbb {R}}^{r}\times S_{p}\times S_{q}:\\ \Vert \varvec{\varXi }-\varvec{\varXi }_o\Vert<\eta ,\Vert \varvec{\varGamma }-\varvec{\varGamma }_o\Vert <\eta \end{array} \right\} . \end{array} \end{aligned}$$

The function $g_P({{\textbf {A}}},{{\textbf {B}}},{{\textbf {a}}},\varvec{\varXi },\varvec{\varGamma ,\sigma })= E_P\left( \frac{{{\textbf {A}}}\varvec{\varXi }^{-1/2}{{\textbf {x}}}-{{\textbf {B}}}\varvec{\varGamma }^{-1/2}{{\textbf {y}}}}{\sigma }\right)$ is continuous in ${\mathscr {G}}$ for $P=F$. Given $\epsilon >0$, by using Theorem 1 in Adrover and Donato (2015), we get that there exist $\eta >0$ and $0<\tilde{\eta }<\delta$

$$\begin{aligned} \begin{array}{ll} \inf _{{\tilde{E}}_{\eta },\sigma<\sigma _o-\epsilon } g_F({{\textbf {A}}},{{\textbf {B}}},{{\textbf {a}}},\varvec{\varXi },\varvec{\varGamma },\sigma )&{}>\delta +{\tilde{\eta }}/2 \\ \sup _{E_{\eta },\sigma >\sigma _o+\epsilon } g_F({{\textbf {A}}},{{\textbf {B}}},{{\textbf {a}}},\varvec{\varXi },\varvec{\varGamma },\sigma )&{}<\delta -{\tilde{\eta }}/2. \end{array} \end{aligned}$$

Then, if $({{\textbf {A}}},{{\textbf {B}}},{{\textbf {a}}},\varvec{\varXi },\varvec{\varGamma })\in E_{\eta }$, there exists $n_0(\epsilon )$ such that the M-scale ${\hat{\sigma }}\in [\sigma _o-\epsilon ,\sigma _o+\epsilon ]$ for all $n>n_o(\epsilon )$. If $({{\textbf {A}}},{{\textbf {B}}},{{\textbf {a}}},\varvec{\varXi },\varvec{\varGamma })\notin E_{\eta }$, there exists $\eta >0$ and $M>0$ such that for all $n>n_1(\epsilon )$ and $F_{n}\in {\mathscr {F}}_{n}$, it holds that

$$\begin{aligned} \inf _{{{\textbf {a}}}\in \left( [-M,M]\right) ^c,E^c_{\eta },\sigma \in [\sigma _o-\epsilon ,\sigma _o+\epsilon ]} g_{F_n}({{\textbf {A}}},{{\textbf {B}}},{{\textbf {a}}},\varvec{\varXi },\varvec{\varGamma },\sigma )>\delta +{\tilde{\eta }}/2. \end{aligned}$$

Consequently, we can conclude that the SM- estimators in (15) belong to a closed bounded set. Moreover, $\lim _{n\rightarrow \infty }{\hat{\sigma }}=\sigma _o$. Therefore, the Fisher consistency given in Theorem 1 in Adrover and Donato (2015) let us conclude that any convergent subsequent should converge to $({{\textbf {A}}}_o,{{\textbf {B}}}_o,{{\textbf {a}}}_o)$ and the consistency follows.

The following technical lemma is needed to prove the asymptotic normality.

Lemma A.1

Let ${{\textbf {z}}}_{1},\dots ,{{\textbf {z}}}_{n}$ be a random sample in ${\mathbb {R}}^{m}$ from an elliptical distribution F with density (18), location parameter $\varvec{\mu }_{0}$ and dispersion parameter $\varvec{\varSigma }_{0}$. Suppose that conditions C0-C14 hold. Let ${\mathscr {G}}$ be as in (22). Let $\phi _{1}$ and $\phi _{2}$ be defined as in (26). Then, there exists a function $\tilde{ \theta }:{\mathscr {H}}\rightarrow {\mathscr {G}}$ and a bounded set ${\mathscr {C}}\subset {\mathscr {G}}$ such that ${\tilde{\theta }}_{o}$ is an interior point of ${\tilde{\theta }}\left( {\mathscr {C}}\right)$ and the sets ${\mathscr {F}}_{1i} =\left\{ \phi _{1i}\left( {\textbf {z,}}{\tilde{\theta }}\left( \xi \right) \right) :\xi \in {\mathscr {C}}\right\}$, $i=1,\ldots ,rm$ and ${\mathscr {F}}_{2k} =\left\{ \phi _{2k}\left( {\textbf {z,}}{\tilde{\theta }} \left( \xi \right) \right) :\xi \in {\mathscr {C}}\right\}$, $k=1,\ldots ,r$ are Euclidean classes with envelopes $F_{1i},$ $i=1,\ldots ,m$ and $F_{2k},$ $k=1,\ldots ,r$, such that $E_{F}\left( F_{1i}\right) ^{2} <\infty$ and $E_{F}\left( F_{2k}\right) ^{2} <\infty$.

Proof of Theorem 2(i)

Given ${\varvec{\tilde{\theta }}}_{o},$ Lemma A.1 says that $\phi _{1i}\left( {\textbf {z,}}{\varvec{\tilde{\theta }}}_{o}\right) \in {\mathscr {F}} _{1i}$, $i=1,\ldots ,mr$ and $\phi _{2k}\left( {\textbf {z,}}{\varvec{\tilde{\theta }}} _{o}\right) \in {\mathscr {F}}_{2k}$, $k=1,\ldots ,r.$ Moreover, since Theorem 1 ensures the consistency of $\varvec{{\hat{\theta }}}=\left( \hat{{\textbf {{A}}}} _{SM}^{o},\hat{{\textbf {{B}}}}_{SM}^{o},\hat{{{\textbf {a}}}}_{SM}{} {\textbf {,}}\hat{{\varvec{\varSigma }}}_{{\textbf {xx}}}^{(R)},{\varvec{\hat{\varSigma }}}_{{\textbf {yy}}}^{(R)},{\hat{\sigma }}\right)$ to ${\varvec{\tilde{\theta }}}_{o},$ given $\varepsilon _{0}>0$ we can find $n_{0}$ such that for any $n\ge n_{0}$, it holds that $P\left( \phi _{1i}\left( {{\textbf {z}}}, \hat{\varvec{\theta }}\right) \in {\mathscr {F}} _{1i},\right.$ $\phi _{2k}\left( {{\textbf {z}}},{\varvec{\hat{\theta }}}\right)$ $\in$ $\left. {\mathscr {F}} _{2k}\right) >1-\varepsilon _{0}$ for all $i\in \left\{ 1,\ldots ,mr\right\}$ and $k\in \left\{ 1,\ldots ,r\right\}$. Given ${\mathscr {F}}$ a Euclidean class and $\delta >0$, set $[\delta ]=\left\{ (f_1,f_2)\in {\mathscr {F}}\times {\mathscr {F}}:\int (f_1-f_2)^2dP<\delta ^2\right\}$. Given a sequence of independent identically distributed random variables $\xi _1,\dots ,\xi _n$ such that $\xi _1\sim P$, set

$$\begin{aligned} \nu _n(f)=\frac{1}{\sqrt{n}}\left[ \sum _{i=1}^n\left( f(\xi _i)-\int fdP\right) \right] , f\in {\mathscr {F}}. \end{aligned}$$

Given $\varepsilon >0$ and $\eta >0$, Lemma 2.16 of Pakes and Pollard (1989), C12 and C13 say that there exist $\delta >0$ and $n_{1}\in {\mathbb {N}}$ such that, for all $n\ge n_{1}$, $\left( \phi _{1j}\left( {{\textbf {z}}},{\varvec{\hat{\theta }}}\right) ,\phi _{1j}\left( {{\textbf {z}}},{\varvec{\tilde{\theta }}}_{o}\right) \right) \in$ $\left[ \delta \right]$ and $\underset{n\rightarrow \infty }{\lim \sup }\ \; P\left\{ \sup _{\left[ \delta \right] }\left| \nu _{n}\left( \phi _{1j}\left( \cdot ,\varvec{{\hat{\theta }}} \right) \right) -\nu _{n}\left( \phi _{1j}\left( \cdot ,\varvec{{\tilde{\theta }}}_{o}\right) \right) \right| >\eta \right\} <\varepsilon .$ Then, we can conclude that

$$\begin{aligned}&\nu _{n}\left( \phi _{j}\left( \cdot ,\varvec{{\hat{\theta }}}\right) \right) -\nu _{n}\left( \phi _{j}\left( \cdot ,\varvec{\tilde{ \theta }}_{o}\right) \right) = o_{P}\left( 1\right) ,j=1,2, \\&\nu _{n}\left( \phi \left( \cdot ,\varvec{{\hat{\theta }}}\right) \right) -\nu _{n}\left( \phi \left( \cdot ,\varvec{{\tilde{\theta }}}_{o}\right) \right) =o_{P}(1). \end{aligned}$$

Since $\varvec{\varPhi }\left( \varvec{{\tilde{\theta }}}_{o}\right) ={{\textbf {0}}},$ by summing up and subtracting some terms we have

$$\begin{aligned} {{\textbf {0}}}= & {} \frac{1}{n}\sum _{i=1}^{n}\phi \left( {{\textbf {z}}}_{i},\varvec{{\hat{\theta }}}\right) -\varvec{\varPhi }\left( \varvec{{\tilde{\theta }}} _{o}\right) =\varvec{\varPhi }\left( \varvec{{\hat{\theta }}}\right) -\varvec{\varPhi } \left( \varvec{{\tilde{\theta }}}_{o}\right) +\frac{1}{\sqrt{n}}\nu _{n}\left( \phi \left( \cdot ,\varvec{{\hat{\theta }}}\right) \right) \\= & {} \varvec{\varPhi }\left( \varvec{{\hat{\theta }}}\right) -\varvec{ \varPhi }\left( \varvec{{\tilde{\theta }}}_{o}\right) +\frac{1}{\sqrt{n}}\nu _{n}\left( \phi \left( \cdot ,\varvec{{\tilde{\theta }}}_{o}\right) \right) + \frac{1}{\sqrt{n}}\left[ \nu _{n}\left( \phi \left( \cdot ,\varvec{{\hat{\theta }}} \right) \right) -\nu _{n}\left( \phi \left( \cdot ,\varvec{{\tilde{\theta }}} _{o}\right) \right) \right] . \end{aligned}$$

By C14 and (27), it holds that $-\frac{1}{\sqrt{n}}\nu _{n}\left( \phi \left( \cdot ,\varvec{{\tilde{\theta }}} _{o}\right) \right) {\textbf {=}}\left[ \varvec{\varOmega } +o_{P}\left( 1\right) \right] \left( \varvec{{\hat{\theta }}}_{SM}-\varvec{\theta }_{o}\right) +o_{P}\left( 1/\sqrt{n}\right) ,$ and by the Central Limit Theorem we have that $\nu _{n}\left( \phi \left( \cdot ,\varvec{{\tilde{\theta }}}_{o}\right) \right) \overset{{\mathscr {D}}}{\rightarrow } N_{r\left( m+1\right) }\left( {\textbf {0,}}{{\textbf {V}}}\right)$. Since $\varvec{\varOmega }$ is invertible, we obtain that $\sqrt{n}\left( \varvec{{\hat{\theta }}}_{SM}-\varvec{\theta }_{o}\right) \overset{{\mathscr {D}}}{\rightarrow } N_{r\left( m+1\right) } \left( {{\textbf {0}}},\varvec{\varOmega } ^{-1}{{\textbf {V}}}_{o}\left( \varvec{\varOmega }^{-1}\right) ^{t}\right) .$ Straightforward computation let us conclude the explicit form of the asymptotic dispersion matrix and the proof follows. (ii) follows closely from (i). (iii) follows from the fact that $\varvec{{\hat{v}}}_{SM,k}$ and $\varvec{{\hat{w}}}_{SM,k}$ are subvectors of $\varvec{{\hat{\theta }}}_{SM}^{*}$ given in (ii). (iv) gives a simpler form of the asymptotic covariance matrix obtained in (iii) in case of having a non-singular matrix of derivatives $\frac{\partial \varPhi _2\left( \varvec{\theta }\right) }{\partial {{\textbf {a}}}}\left( \varvec{{\tilde{\theta }}_o}\right)$. $\square$

Proof of Theorem 4

Given $F_{\varepsilon }=\left( 1-\varepsilon \right) F+\varepsilon \delta _{{{\textbf {z}}}_{0}}$, we have to look for the SM-functionals defined as $g\left( {{\textbf {D}}},{\varvec{\tilde{\varSigma }}}_{\varepsilon },{{\textbf {a}}},\sigma _{\varepsilon }\right) =E_{F_{\varepsilon }}\chi \left( \frac{\left\| {{\textbf {D}}}{\varvec{\tilde{\varSigma }}}_{\varepsilon }{} {\textbf {z-a}}\right\| ^{2} }{\sigma _{\varepsilon }\left( {{\textbf {D}}},{{\textbf {a}}}\right) }\right) =\delta .$ Then, we look for a restricted minimum ${{\textbf {D}}}\in {\mathscr {O}}_{r,m}$, ${{\textbf {t}}}_{1},\ldots ,{{\textbf {t}}}_{r}$ are the rows of ${{\textbf {D}}}$ and the Lagrangian can be expressed as

$$\begin{aligned}{} & {} L\left( {{\textbf {t}}}_{1},\ldots ,{{\textbf {t}}}_{r},{\textbf {a,}}{\varvec{\tilde{\varSigma }}} _{\varepsilon },\eta _{11},\ldots ,\eta _{rr}\right) \\{} & {} \quad =\sigma _{\varepsilon }\left( {{\textbf {D}}},{{\textbf {a}}}\right) -\sum _{j=1}^{r}\eta _{jj}\left( {{\textbf {t}}}_{j}^{t}{{\textbf {t}}}_{j}-1\right) -\sum _{1\le j<k\le r}\eta _{jk} {{\textbf {t}}}_{k}^{t}{{\textbf {t}}}_{j}, \end{aligned}$$

where ${{\textbf {t}}}_{1,\varepsilon },\ldots ,{{\textbf {t}}}_{r,\varepsilon },{{\textbf {a}}} _{\varepsilon }$ are critical points for L. The proof follows closely to that of Theorem 1 in Croux and Ruiz-Gazen (2005). $\square$

Proof of Lemma 3

The eigenvectors $\left( 1,1\right)$ and $\left( 1,-1\right)$ of ${\varvec{\tilde{\varSigma }}}$ correspond to the eigenvalues $\left( (1-b)/(1+b)\right) ^{1/2}$ and $\left( (1+b)/(1-b)\right) ^{1/2}$, respectively. Then, the quadratic forms in both definitions coincide and ${\hat{\lambda }}=b^{*}.$ $\square$

Proof of Lemma 4

(i) and (ii) are easily derived. (iii) ${\hat{\lambda }}=1$ implies that $s(1)\le s(\lambda )$ for all $\lambda \in \left[ -1,1\right]$. Let $q(\lambda )$ be as in (35). Since $0=\lim _{s\rightarrow \infty }E\chi \left( q(\lambda )/s\right) \le \delta ,$ this implies that $s(\lambda )<\infty \$and $s(1)<\infty$. Thus, $\delta \ge \lim _{\lambda \rightarrow 1^{-}}E\chi \left( q\left( \lambda \right) /s(\lambda )\right)$ and $P(U=V)\ge 1-\delta$ which says that $P\left( X=aY+b\right) \ge 1-\delta$, with $a=S(F_{X})/S(F_{Y})$ and $b=$ $-aT(F_{Y})+T(F_{X})$. In case that ${\hat{\lambda }}=-1$, we get $P(U=-V)\ge 1-\delta$, and $P\left( X=cY+d\right)$ $\ge 1-\delta$, with $c=-S(F_{X})/S(F_{Y})$ and $d=S(F_{X})T(F_{Y})/S(F_{Y})+T(F_{X})$. Thus, (iii) is proved. (iv) In case of having (X, Y) elliptically distributed with correlation $\rho ,$ Lemma 3 let us affirm that $\hat{ \lambda }=\rho .$ $\square$

Proof of Lemma 5

(i) is easily derived and (iii) follows as in Lemma 4 (iii). (ii) Since $(U\pm V)^2\ge 0$, then $-(U^2+V^2)\le UV\le (U^2+V^2)$ and $|2UV/(U^2+V^2)|\le 1$. Therefore, $\text {med}(2UV/(U^2+V^2))\in [-1,1]$. (iv) In case of having (X, Y) elliptically distributed with correlation $\rho ,$ (U, V) is elliptically distributed with density $f(u,v)=1/Kf_0((u^2+2\rho uv+v^{2})/\sqrt{1-\rho ^{2}}))$ and $K=\pi (1-\rho ^2)^{-1/2}(F_0( \infty )-F_0(0))$ with $F_0$ a primitive of $f_0$. To see that $P_{c}=P\left( 2UV/(U^{2}+V^{2})\le \rho \right) =0.5$, we perform some change of variables to get that

$$\begin{aligned} P_{c}= & {} \frac{2^{-1}(1-\rho ^{2})^{-1/2}}{K}\int I_{(-\infty ,\rho )}\left( \frac{ z^{2}-w^{2}+\rho (z^{2}+w^{2})}{z^{2}+w^{2}+\rho (z^{2}-w^{2})}\right) f_{0}\left( z^{2}+w^{2}\right) \textrm{d}z\textrm{d}w. \end{aligned}$$

Using spherical coordinates, $1+\rho \cos 2\theta \ge 0$ and the fact that $\frac{\cos 2\theta +\rho }{1+\rho \cos 2\theta }\le \rho$ if and only if and $\cos 2\theta \le 0,$ then we have

$$\begin{aligned} P_{c}= & {} \frac{2^{-1}(1-\rho ^{2})^{-1/2}}{K}\int _{0}^{2\pi }\int _{0}^{\infty }I_{(-\infty ,\rho )}\left( \frac{\cos 2\theta +\rho }{1+\rho \cos 2\theta } \right) rf_{0}\left( r^{2}\right) \textrm{d}r\textrm{d}\theta \\= & {} \frac{2^{-1}(1-\rho ^{2})^{-1/2}}{K}\int _{0}^{2\pi }\int _{0}^{\infty }I_{ \left[ \pi /4,3\pi /4\right] \cup \left[ 5\pi /4,7\pi /4\right] }\left( \theta \right) rf_{0}\left( r^{2}\right) \textrm{d}r\textrm{d}\theta \\= & {} \frac{2^{-1}\pi (1-\rho ^{2})^{-1/2}}{K}\int _{0}^{\infty }rf_{0}\left( r^{2}\right) \textrm{d}r\textrm{d}\theta =\frac{\pi (1-\rho ^{2})^{-1/2}}{2K}\left( F_{0}(\infty )-F_{0}(0)\right) =\frac{1}{2}, \end{aligned}$$

which shows that ${\hat{\lambda }}=\rho$ and the result follows. $\square$