Simultaneous Component Analysis by Means of Tucker3

Abstract

A new model for simultaneous component analysis (SCA) is introduced that contains the existing SCA models with common loading matrix as special cases. The new SCA-T3 model is a multi-set generalization of the Tucker3 model for component analysis of three-way data. For each mode (observational units, variables, sets) a different number of components can be chosen and the obtained solution can be rotated without loss of fit to facilitate interpretation. SCA-T3 can be fitted on centered multi-set data and also on the corresponding covariance matrices. For this purpose, alternating least squares algorithms are derived. SCA-T3 is evaluated in a simulation study, and its practical merits are demonstrated for several benchmark datasets.

Appendix: Proofs of Lemma 3.1 and Lemma 3.2

Proof of Lemma 3.1

We start by proving equality (9) for SCA-P fitted via PCA as described at the beginning of Sect. 3.2. From the latter it can be verified that $\mathbf{F}_k=\mathbf{X}_k\,\mathbf{B}\,(\mathbf{B}^T\mathbf{B})^{-1}$. This implies that

$$\begin{aligned} \Vert \mathbf{X}_k-\mathbf{F}_k\,\mathbf{B}^T\Vert ^2= & {} \mathrm{trace}(\mathbf{X}_k^T\mathbf{X}_k) - \mathrm{trace}(\mathbf{B}\,(\mathbf{B}^T\mathbf{B})^{-1}{} \mathbf{B}^T\mathbf{X}_k^T\mathbf{X}_k) \\= & {} \mathrm{trace}(\mathbf{X}_k^T\mathbf{X}_k) - \mathrm{trace}(\mathbf{B}\,(\mathbf{B}^T\mathbf{B})^{-1}{} \mathbf{B}^T\mathbf{X}_k^T\mathbf{X}_k\mathbf{B}\,(\mathbf{B}^T\mathbf{B})^{-1}{} \mathbf{B}^T) \\= & {} \Vert \mathbf{X}_k\Vert ^2 - \Vert \mathbf{F}_k\,\mathbf{B}^T\Vert ^2, \end{aligned}$$

which is equivalent to (9).

Next, we consider SCA-PF2. It suffices to show that trace$(\mathbf{X}_k^T\mathbf{F}_k\,\mathbf{B}^T)=$ trace$(\mathbf{B}\,\mathbf{F}_k^T\mathbf{F}_k\,\mathbf{B}^T)$. In SCA-PF2 we can set $\mathbf{F}_k=\widetilde{\mathbf{A}}_k\,\mathbf{H}\,\widetilde{\mathbf{C}}_k$, with $\widetilde{\mathbf{A}}_k^T\widetilde{\mathbf{A}}_k=\mathbf{I}_R$, $\mathbf{H}^T\mathbf{H}=\widetilde{\mathbf{\Phi }}$, and $\widetilde{\mathbf{C}}_k$ diagonal $R\times R$ (Kiers et al., 1999; Timmerman & Kiers, 2003). In step 2 of the ALS algorithm for SCA-PF2 the objective function is minimized over $\mathbf{H}$, $\widetilde{\mathbf{C}}_k$, and $\mathbf{B}$ for fixed $\widetilde{\mathbf{A}}_k$. Analogous to Sect. 3.1, this boils down to fitting Parafac to the $R\times J\times K$ array with slices $\widetilde{\mathbf{A}}_k^T\mathbf{X}_k$, $k=1,\ldots ,K$. One iteration of the Parafac ALS algorithm is used to approximate $\widetilde{\mathbf{A}}_k^T\mathbf{X}_k\approx \mathbf{H}\,\widetilde{\mathbf{C}}_k\,\mathbf{B}^T$. Vectorizing on both sides yields $\mathrm{Vec}(\widetilde{\mathbf{A}}_k^T\mathbf{X}_k)\approx (\mathbf{B}\odot \mathbf{H})\,\tilde{\mathbf{c}}_k^\mathrm{(row)}$, with $\tilde{\mathbf{c}}_k^\mathrm{(row)}$ denoting the kth row of $\widetilde{\mathbf{C}}$ as a column vector. The OLS regression update of $\tilde{\mathbf{c}}_k^\mathrm{(row)}$ for fixed $\mathbf{H}$ and $\mathbf{B}$ implies that

$$\begin{aligned} \mathrm{trace}(\mathrm{Vec}(\widetilde{\mathbf{A}}_k^T\mathbf{X}_k)^T(\mathbf{B}\odot \mathbf{H})\,\tilde{\mathbf{c}}_k^\mathrm{(row)})= \mathrm{trace}((\tilde{\mathbf{c}}_k^\mathrm{(row)})^T(\mathbf{B}\odot \mathbf{H})^T(\mathbf{B}\odot \mathbf{H})\,\tilde{\mathbf{c}}_k^\mathrm{(row)}), \end{aligned}$$

which is equivalent to trace$(\mathbf{X}_k^T\widetilde{\mathbf{A}}_k\,\mathbf{H}\,\widetilde{\mathbf{C}}_k\,\mathbf{B}^T)=\Vert \widetilde{\mathbf{A}}_k\,\mathbf{H}\,\widetilde{\mathbf{C}}_k\,\mathbf{B}^T\Vert ^2$. This is the desired result. The proof for SCA-IND follows from the above by setting $\widetilde{\mathbf{\Phi }}=\mathbf{I}_R$ (Timmerman & Kiers, 2003).

Finally, we consider SCA-T3 fitted by the ALS algorithm in Sect.  3.1. The proof is analogous to the proof for SCA-PF2. We write $\mathbf{F}_k=\widetilde{\mathbf{A}}_k\left( \sum _r \tilde{c}_{kr}\,\mathbf{G}_r\right)$, with $\widetilde{\mathbf{A}}_k^T\widetilde{\mathbf{A}}_k=\mathbf{I}_P$. In step 2 of the ALS algorithm in Sect. 3.1 we fit Tucker2 to the $P\times J\times K$ array with slices $\widetilde{\mathbf{A}}_k^T\mathbf{X}_k$, $k=1,\ldots ,K$. One iteration of the Tucker2 ALS algorithm is used to approximate $\widetilde{\mathbf{A}}_k^T\mathbf{X}_k\approx \left( \sum _r \tilde{c}_{kr}\,\mathbf{G}_r\right) \mathbf{B}^T$. Vectorizing on both sides yields $\mathrm{Vec}(\widetilde{\mathbf{A}}_k^T\mathbf{X}_k)\approx (\mathbf{B}\otimes \mathbf{I}_P)\,[\mathrm{Vec}(\mathbf{G}_1)\;\ldots \;\mathrm{Vec}(\mathbf{G}_R)]\,\tilde{\mathbf{c}}_k^\mathrm{(row)}$. As above, the OLS regression update of $\tilde{\mathbf{c}}_k^\mathrm{(row)}$ for fixed $\mathbf{B}$ and $\mathcal{G}$ yields the result. This completes the proof. $\square$

Generally, the equality in (9) follows from an OLS regression update specifically for sample k. For SCA-ECP there is no such update and the equality does not hold. Indeed, fitting SCA-ECP to the PANAS dataset in Sect. 5.1 yields inequality in (9).

Proof of Lemma 3.2

After step 2 of the ALS algorithm for SCA-T3 (Sect. 3.1) the fitted model for $\mathbf{X}_k$ can be written as $\widetilde{\mathbf{A}}_k\,\mathbf{G}\,(\tilde{\mathbf{c}}_k^\mathrm{(row)}\otimes \mathbf{B}^T)$, with $\mathbf{G}=[\mathbf{G}_1\;\ldots \;\mathbf{G}_R]$ and $\tilde{\mathbf{c}}_k^\mathrm{(row)}$ the kth row of $\widetilde{\mathbf{C}}$ as a column vector. Analogous to (4) for Tucker3 (Sect. 2.1), we obtain

$$\begin{aligned} \mathrm{Vec}(\mathbf{X}_\mathrm{all})\approx \left[ \begin{array}{c} (\tilde{\mathbf{c}}_1^\mathrm{(row)})^T\otimes \mathbf{B}\otimes \widetilde{\mathbf{A}}_1 \\ \vdots \\ (\tilde{\mathbf{c}}_K^\mathrm{(row)})^T\otimes \mathbf{B}\otimes \widetilde{\mathbf{A}}_K \end{array}\right] \mathrm{Vec}(\mathbf{G}). \end{aligned}$$

(18)

The matrix on the right-hand side is columnwise orthonormal, since

$$\begin{aligned} \sum _{k=1}^K ((\tilde{\mathbf{c}}_k^\mathrm{(row)})^T\otimes \mathbf{B}\otimes \widetilde{\mathbf{A}}_k)^T((\tilde{\mathbf{c}}_k^\mathrm{(row)})^T\otimes \mathbf{B}\otimes \widetilde{\mathbf{A}}_k)= & {} \sum _{k=1}^K (\tilde{\mathbf{c}}_k^\mathrm{(row)}(\tilde{\mathbf{c}}_k^\mathrm{(row)})^T\otimes \mathbf{B}^T\mathbf{B}\otimes \widetilde{\mathbf{A}}_k^T\widetilde{\mathbf{A}}_k) \nonumber \\= & {} \left( \sum _{k=1}^K \tilde{\mathbf{c}}_k^\mathrm{(row)}(\tilde{\mathbf{c}}_k^\mathrm{(row)})^T\right) \otimes \mathbf{I}_Q\otimes \mathbf{I}_P \nonumber \\= & {} \widetilde{\mathbf{C}}^T\widetilde{\mathbf{C}}\otimes \mathbf{I}_Q\otimes \mathbf{I}_P \nonumber \\= & {} \mathbf{I}_R\otimes \mathbf{I}_Q\otimes \mathbf{I}_P=\mathbf{I}_{PQR}. \end{aligned}$$

(19)

The update of $\mathcal{G}$ in step 2 of the ALS algorithm for SCA-T3 in Sect. 3.1 is computed via OLS regression in (18). Indeed, this is identical to the update of $\mathcal{G}$ in the ALS algorithm for Tucker2 fitted to the $P\times J\times K$ array with slices $\widetilde{\mathbf{A}}_k^T\mathbf{X}_k$, $k=1,\ldots ,K$. After convergence of the ALS algorithm for SCA-T3 we set $\mathbf{A}_k=N_k^{1/2}\widetilde{\mathbf{A}}_k$ and $\mathbf{c}_k^\mathrm{(row)}=N_k^{-1/2}\,\tilde{\mathbf{c}}_k^\mathrm{(row)}$, $k=1,\ldots ,K$. For the blocks of the matrix in (18), we have $(\mathbf{c}_k^\mathrm{(row)})^T\otimes \mathbf{B}\otimes \mathbf{A}_k=(\tilde{\mathbf{c}}_k^\mathrm{(row)})^T\otimes \mathbf{B}\otimes \widetilde{\mathbf{A}}_k$. Since the update of $\mathbf{G}$ in (19) is done via OLS regression with orthonormal predictors, it follows that for SCA-T3 we can compute the fit percentage due to term (p, q, r) as in (11). Moreover, these terms sum up to the total fit percentage (8). This is analogous to Tucker3 (Sect. 2.1). This completes the proof. $\square$