Simultaneous canonical correlation analysis with invariant canonical loadings

Abstract

In canonical correlation analysis (CCA), the substantive interpretations of the canonical variates are of primary interest to the applied researchers. However, there are two different interpretive approaches used by different researchers—the weight-based approach and the loading-based approach, of which the latter is favored by the majority of researchers in practice. For those who choose the loading-based approach and apply CCA simultaneously to multiple samples, they may wish to test the invariance of the canonical loadings. In this paper, three covariance structure models are defined for CCA. In particular, the first model (i.e., the CCA-W model) corresponds directly with regular CCA, including the canonical correlations and canonical weights as the parameters, while the third model (i.e., the CCA-L model) is in alignment with the loading-based interpretive approach, including the canonical correlations and canonical loadings as the parameters. The CCA-L model is further extended to the unrestricted and restricted SCCA-L models, of which the latter allows one to test the invariance of the canonical loadings. A real example drawn from the sociological literature is provided to illustrate the restrictive SCCA-L model, and some strategies to calculate good starting values for the restrictive SCCA-L model are discussed.

Appendix

In the SCCA-L model, we denote the pair of canonical variates as ${\mathbf{Y}}_{1}^{(k)} = \left( {{\mathbf{A}}_{1}^{(k)} } \right)^{\prime } {\mathbf{X}}_{1}^{(k)}$ and ${\mathbf{Y}}_{2}^{(k)} = \left( {{\mathbf{A}}_{2}^{(k)} } \right)^{\prime } {\mathbf{X}}_{2}^{(k)}$ in the kth group and their individual elements $y_{1i}$ and $y_{2i}$, i = 1, 2, …, p. We also denote ${\mathbf{Y}}_{3}^{(k)} = \left( {{\mathbf{A}}_{3}^{(k)} } \right)^{\prime } {\mathbf{X}}_{2}^{(k)}$ and its individual elements $y_{3j}$, j = 1, 2, …, d. Define ${\mathbf{Y}}^{(k)} = \left[ {\begin{array}{*{20}c} {\left( {{\mathbf{Y}}_{1}^{(k)} } \right)^{\prime } } & {\left( {{\mathbf{Y}}_{2}^{(k)} } \right)^{\prime } } & {\left( {{\mathbf{Y}}_{3}^{(k)} } \right)^{\prime } } \\ \end{array} } \right]^{\prime }$. Now the observed variables ${\mathbf{X}}^{(k)}$ can be written as

$$ {\mathbf{X}}^{(k)} {\mathbf{ = B}}^{(k)} {\mathbf{Y}}^{(k)} = \left( {\begin{array}{*{20}c} {{\mathbf{B}}_{1}^{(k)} {\mathbf{Y}}_{1}^{(k)} } \\ {{\mathbf{B}}_{2}^{(k)} {\mathbf{Y}}_{2}^{(k)} + {\mathbf{B}}_{3}^{(k)} {\mathbf{Y}}_{3}^{(k)} } \\ \end{array} } \right). $$

Note that to simplify the notation, we allow ${\mathbf{B}}^{(k)}$ to absorb the scaling diagonal matrix ${\mathbf{D}}^{(k)}$ in Eq. (7).

Proposition 1

If the canonical variates are allowed to have free variance parameters (that could vary across groups) in the restricted SCCA-L model and the model is identified by setting one loading of each column of the loading matrix, the test statistic $T = (N - K)\hat{F}$ has the same $\chi^{2}$ distribution as under normality assumption as long as different pairs of canonical variates are independent of each other and are independent of ${\mathbf{Y}}_{3}^{(k)}.$

Proof

We change the identification condition of ${\mathbf{Y}}_{3}^{(k)}$ so that ${\mathbf{Y}}_{3}^{(k)}$ has a saturated covariance matrix, but ${\mathbf{B}}_{3}^{(k)}$ involves a block of identity matrix. We denote the Jacobian matrix of the covariance structure by

$$ {\varvec{\Delta}}^{(k)} = \left[ {\begin{array}{*{20}c} {{\varvec{\Delta}}_{0}^{(k)} } & {{\varvec{\Delta}}_{k}^{(k)} } \\ \end{array} } \right], $$

where ${\varvec{\Delta}}_{k}^{(k)} = \frac{{\partial \text{vec} {\varvec{\Sigma}}^{(k)} }}{{\partial {\varvec{\upvarphi }}^{(k)\prime } }}$, k = 1,2,…,K, ${\varvec{\upvarphi }}^{(k)}$ is a $(3p + d(d + 1)/2) \times 1$ vector that involves the 2p variances and p covariances of the p pairs of canonical variates in the kth group and the d(d + 1)/2 non-duplicated elements of the covariance matrix of ${\mathbf{Y}}_{3}^{(k)}$, and ${\varvec{\Delta}}_{0}^{(k)}$ is the derivative w.r.t. all other parameters in the model.

The derivative of the discrepancy function in (8) is

$$ \frac{\partial F}{{\partial {\varvec{\uptheta^{\prime}}}}} = - \sum\limits_{k = 1}^{K} {\frac{{n_{k} - 1}}{N - K}} \left( {{\mathbf{s}}^{(k)} - {\varvec{\upsigma}}^{(k)} } \right)\left( {{\varvec{\Sigma}}^{(k)} \otimes {\varvec{\Sigma}}^{(k)} } \right)^{ - 1} {\varvec{\Delta}}^{(k)} = - 2\left( {{\mathbf{s}} - {\varvec{\upsigma}}} \right)^{\prime } {\mathbf{V}{\varvec{\Delta }}}, $$

where ${\mathbf{s}}^{(k)} = \text{vec} {\mathbf{S}}^{(k)}$, ${\varvec{\upsigma}}^{(k)} = \text{vec} {\varvec{\Sigma}}^{(k)}$, s and σ are long vectors of length K(2p − d)² that join ${\mathbf{s}}^{(k)}$ and ${\varvec{\upsigma}}^{(k)}$ from K groups together, V is a block-diagonal matrix with blocks $\frac{{n_{k} - 1}}{{2\left( {N - K} \right)}}\left( {{\varvec{\Sigma}}^{(k)} \otimes {\varvec{\Sigma}}^{(k)} } \right)^{ - 1}$, and ${\varvec{\Delta}}$ is defined as

$$ {\varvec{\Delta}} = \left[ {\begin{array}{*{20}c} {{\varvec{\Delta}}_{0}^{(1)} } & {{\varvec{\Delta}}_{1}^{(1)} } & {\mathbf{0}} & \cdots \\ {{\varvec{\Delta}}_{0}^{(2)} } & {\mathbf{0}} & {{\varvec{\Delta}}_{2}^{(2)} } & \cdots \\ \vdots & \vdots & \vdots & \ddots \\ {{\varvec{\Delta}}_{0}^{(K)} } & {\mathbf{0}} & {\mathbf{0}} & \cdots \\ \end{array} \begin{array}{*{20}c} {\mathbf{0}} \\ {\mathbf{0}} \\ \vdots \\ {{\varvec{\Delta}}_{K}^{(K)} } \\ \end{array} } \right]. $$

Similar to the single group situation, standard asymptotic derivations give the expansion of the test statistic

$$ T = \left( {N - K} \right)\hat{F} = \left( {{\mathbf{s}} - {\varvec{\upsigma}}_{0} } \right)^{\prime } {\mathbf{U}}\left( {{\mathbf{s}} - {\varvec{\upsigma}}_{0} } \right) + o_{p} \left( 1 \right), $$

where the asymptotic framework allows $N \to \infty$ but fixes the proportions $\frac{{n_{k} - 1}}{N - K}$ in each of the groups. The matrix U is defined as ${\mathbf{U}} = {\mathbf{V}} - {\mathbf{V}{\varvec{\Delta }}}\left( {{\varvec{\Delta^{\prime}{\mathbf{V}}{\varvec{\Delta }}}}} \right)^{ - 1} {\varvec{\Delta^{\prime}{\mathbf{V}}}}$, evaluated at its population value. The vector ${\varvec{\upsigma}}_{0}$ is σ evaluated at the correctly specified population.

When data are nonnormally distributed but with finite fourth-order moments, Browne and Shapiro (1988, Eq. 2.7) have shown that for a single group, the asymptotic covariance matrix of $\sqrt {n_{k} - 1} \left( {{\mathbf{s}}^{(k)} - {\varvec{\upsigma}}_{0}^{(k)} } \right)$ is given by

$$ {\varvec{\Gamma}}^{(k)} = {\varvec{\Gamma}}_{N}^{(k)} + \sum\limits_{i = 1}^{p} {\left( {{\varvec{\Lambda}}_{i}^{(k)} \otimes {\varvec{\Lambda}}_{i}^{(k)} } \right){\mathbf{C}}_{i}^{(k)} \left( {{\varvec{\Lambda}}_{i}^{(k)} \otimes {\varvec{\Lambda}}_{i}^{(k)} } \right)^{\prime } } + \left( {{\varvec{\Lambda}}_{0}^{(k)} \otimes {\varvec{\Lambda}}_{0}^{(k)} } \right){\mathbf{C}}_{0}^{(k)} \left( {{\varvec{\Lambda}}_{0}^{(k)} \otimes {\varvec{\Lambda}}_{0}^{(k)} } \right)^{\prime } , $$

where ${\varvec{\Gamma}}_{N}^{(k)}$ is the asymptotic covariance matrix under the normal assumption, ${\mathbf{C}}_{i}^{(k)}$ (i = 1, 2, …, p) is the $4 \times 4$ cumulant matrix of the ith pair of canonical variates, ${\varvec{\Lambda}}_{i}^{(k)}$ (i = 1, 2, …, p) is the (2p + d) × 2 loading matrix that relates the manifest variables with the ith pair of canonical variates, ${\mathbf{C}}_{0}^{(k)}$ is the d ² × d ² cumulant matrix of ${\mathbf{Y}}_{3}^{(k)}$, and ${\varvec{\Lambda}}_{0}^{(k)}$ is the (2p + d) × d loading matrix on ${\mathbf{Y}}_{3}^{(k)}$. For K groups, the asymptotic covariance matrix ${\varvec{\Gamma}}$ of the long vector $\sqrt {N - K} \left( {{\mathbf{s}} - {\varvec{\upsigma}}_{0} } \right)$ is a block-diagonal matrix with ${\varvec{\Gamma}}^{(k)}$ as blocks: ${\varvec{\Gamma}} = {\varvec{\Gamma}}_{N} + \sum\limits_{i = 1}^{p} {{\mathbf{L}}_{i} {\mathbf{C}}_{i} {\mathbf{L^{\prime}}}_{i} } + {\mathbf{L}}_{0} {\mathbf{C}}_{0} {\mathbf{L^{\prime}}}_{0}$, where ${\varvec{\Gamma}}_{N}$ and ${\mathbf{C}}_{i}$ are block-diagonal matrices with ${\varvec{\Gamma}}_{N}^{(k)}$ and ${\mathbf{C}}_{i}^{(k)}$ as blocks and ${\mathbf{L}}_{i}$ is a block-diagonal matrix with ${\varvec{\Lambda}}_{i}^{(k)} \otimes {\varvec{\Lambda}}_{i}^{(k)}$ as blocks. Asymptotically, this test statistic has a distribution of weighted sum of $\chi^{2}$ variates, each of single degrees of freedom, where the weights are given by the eigenvalues of the matrix ${\mathbf{U}}{\varvec {\Gamma } }$. To prove asymptotic robustness, we only need to show ${\mathbf{U}}{\varvec{\Gamma }} = {\mathbf{U}}{\varvec{\Gamma }}_{N}$, for which we now show ${\mathbf{UL}}_{i} = {\mathbf{0}}$.

Note that the derivative of ${\varvec{\upsigma}}^{(k)}$ w.r.t the half-vectorized covariance matrix of the ith pair of canonical variates (or w.r.t that of ${\mathbf{Y}}_{3}^{(k)}$) is $\left( {{\varvec{\Lambda}}_{i}^{(k)} \otimes {\varvec{\Lambda}}_{i}^{(k)} } \right){\mathbf{K}}\left( {{\mathbf{K^{\prime}K}}} \right)^{ - 1}$, where ${\mathbf{K}}$ is the $4 \times 3$ transition matrix for i > 0 and is the d ² × d(d + 1)/2 transition matrix for i = 0 (Browne and Shapiro 1988). This means that $\left( {{\varvec{\Lambda}}_{i}^{(k)} \otimes {\varvec{\Lambda}}_{i}^{(k)} } \right){\mathbf{K}}\left( {{\mathbf{K^{\prime}K}}} \right)^{ - 1}$ contributes to three columns in ${\varvec{\Delta}}_{k}^{(k)}$ for i > 0 and to d(d + 1)/2 columns in ${\varvec{\Delta}}_{k}^{(k)}$ for i = 0. The relationship $\left( {{\varvec{\Lambda}}_{i}^{(k)} \otimes {\varvec{\Lambda}}_{i}^{(k)} } \right){\mathbf{K}}\left( {{\mathbf{K^{\prime}K}}} \right)^{ - 1} {\mathbf{K^{\prime}}} = {\varvec{\Lambda}}_{i}^{(k)} \otimes {\varvec{\Lambda}}_{i}^{(k)}$ shows that $({\varvec{\Lambda}}_{i}^{(k)} \otimes {\varvec{\Lambda}}_{i}^{(k)} )$ is in the column space of ${\varvec{\Delta}}_{k}^{(k)}$. Given the structure of block matrices ${\varvec{\Delta}}$ and ${\mathbf{L}}_{i}$, we see that the latter matrix is in the column space of the former matrix, which implies ${\mathbf{UL}}_{i} = {\mathbf{0}}$. This proves Proposition 1.