Cluster Correspondence Analysis

Abstract

A method is proposed that combines dimension reduction and cluster analysis for categorical data by simultaneously assigning individuals to clusters and optimal scaling values to categories in such a way that a single between variance maximization objective is achieved. In a unified framework, a brief review of alternative methods is provided and we show that the proposed method is equivalent to GROUPALS applied to categorical data. Performance of the methods is appraised by means of a simulation study. The results of the joint dimension reduction and clustering methods are compared with the so-called tandem approach, a sequential analysis of dimension reduction followed by cluster analysis. The tandem approach is conjectured to perform worse when variables are added that are unrelated to the cluster structure. Our simulation study confirms this conjecture. Moreover, the results of the simulation study indicate that the proposed method also consistently outperforms alternative joint dimension reduction and clustering methods.

Appendix: GROUPALS and Cluster CA

To show the relationship between GROUPALS and cluster CA, we consider the GROUPALS objective function for the case where all variables are categorical.

$$\begin{aligned} \min \phi _{\text {groupals}}\left( {\mathbf {B},\mathbf {Z}_{K},\mathbf {G}}\right) =\frac{1}{p}\sum _{j=1}^{p}\left\| \mathbf {Z}_{K}\mathbf {G}-\mathbf {Z}_{j}\mathbf {B}_{j}\right\| ^{2}, \end{aligned}$$

subject to

$$\begin{aligned} \sum \limits _{j=1}^{p} \mathbf {B}_{j}^{^{\prime }}\mathbf {Z}_{j}^{^{\prime }}\mathbf {Z}_{j} \mathbf {B}_{j}=\mathbf {I}_{k}. \end{aligned}$$

We can solve this problem deriving the first-order conditions. These first-order conditions can be used to formulate an alternating least-squares algorithm. Thus, we fix $\mathbf {Z}_{K}$ and solve for $\mathbf {B}_{j}$ and $\mathbf {G}$ by setting up the Lagrangian:

$$\begin{aligned} \psi&=\frac{1}{p}\sum _{j=1}^{p}\hbox {trace }\left( \mathbf {Z} _{K}\mathbf {G-Z}_{j}\mathbf {B}_{j}\right) ^{^{\prime }}\left( \mathbf {Z} _{K}\mathbf {G-Z}_{j}\mathbf {B}_{j}\right) +\hbox {trace } \mathbf {L}\left( \sum _{j=1}^{p}\mathbf {B}_{j}^{^{\prime }}\mathbf {D} _{j}\mathbf {B}_{j}-\mathbf {I}_{k}\right) \\&=\hbox {trace }\mathbf {G}^{^{\prime }}\mathbf {Z}_{K}^{^{\prime } }\mathbf {Z}_{K}\mathbf {G}+\frac{1}{p}\sum _{j=1}^{p}\hbox {trace } \mathbf {B}_{j}^{^{\prime }}\mathbf {Z}_{j}^{^{\prime }}\mathbf {Z}_{j} \mathbf {B}_{j}-\frac{2}{p}\sum _{j=1}^{p}\hbox {trace }\mathbf {G} ^{^{\prime }}\mathbf {Z}_{K}{}^{^{\prime }}\mathbf {Z}_{j}\mathbf {B} _{j}\\&\quad +\hbox {trace }\mathbf {L}\left( \sum _{j=1}^{p}\mathbf {B} _{j}^{^{\prime }}\mathbf {D}_{j}\mathbf {B}_{j}-\mathbf {I}_{k}\right) \\&=\hbox {trace }\mathbf {G}^{^{\prime }}\mathbf {Z}_{K}{}^{^{\prime } }\mathbf {Z}_{K}\mathbf {G}+\frac{k}{p}-\frac{2}{p}\sum _{j=1}^{p} \hbox {trace }\mathbf {G}^{^{\prime }}\mathbf {Z}_{K}{}^{^{\prime } }\mathbf {Z}_{j}\mathbf {B}_{j}+\hbox {trace }\mathbf {L}\left( \sum _{j=1}^{p}\mathbf {B}_{j}^{^{\prime }}\mathbf {D}_{j}\mathbf {B} _{j}-\mathbf {I}_{k}\right) , \end{aligned}$$

where $\mathbf {L}$ is the matrix of Lagrange multipliers and $\mathbf {D}_j = \mathbf {Z}_j^{\prime }\mathbf {Z}_j$. Taking derivatives and equating to zero yields the first-order conditions.

For $\mathbf {G:}$

$$\begin{aligned} 2\hbox { trace }\mathbf {G}^{^{\prime }}\mathbf {Z}_{K}{}^{^{\prime } }\mathbf {Z}_{K}d\mathbf {G}&\,\mathbf {=}\,\frac{2}{p}\sum _{j=1}^{p} \hbox {trace }\mathbf {B}_{j}^{^{\prime }}\mathbf {Z}_{j}^{^{\prime } }\mathbf {Z}_{K}d\mathbf {G}\\ \mathbf {G}^{^{\prime }}\mathbf {Z}_{K}{}^{^{\prime }}\mathbf {Z}_{K}&\,\mathbf {=}\,\frac{1}{p}\sum _{j=1}^{p}\mathbf {B}_{j}^{^{\prime }}\mathbf {Z} _{j}^{^{\prime }}\mathbf {Z}_{K}\\ \mathbf {G}&\,\mathbf {=}\,\frac{1}{p}\left( \mathbf {Z}_{K}{}^{^{\prime } }\mathbf {Z}_{K}\right) ^{-1}\mathbf {Z}_{K}{}^{^{\prime }}\sum _{j=1} ^{p}\mathbf {Z}_{j}\mathbf {B}_{j}. \end{aligned}$$

For $\mathbf {B}_{j}:$

$$\begin{aligned} \frac{2}{p}\hbox {trace }\mathbf {G}^{^{\prime }}\mathbf {Z}_{K} {}^{^{\prime }}\mathbf {Z}_{j}d\mathbf {B}_{j}&=2\hbox { trace } \mathbf {LB}_{j}^{^{\prime }}\mathbf {D}_{j}d\mathbf {B}_{j}\\ \frac{1}{p}\mathbf {Z}_{j}^{^{\prime }}\mathbf {Z}_{K}\mathbf {G}&=\mathbf {D}_{j}\mathbf {B}_{j}\mathbf {L}. \end{aligned}$$

Inserting the solution for $\mathbf {G}$ we obtain

$$\begin{aligned} \frac{1}{p^{2}}\mathbf {Z}_{j}^{^{\prime }}\mathbf {Z}_{K}\left( \mathbf {Z} _{K}{}^{^{\prime }}\mathbf {Z}_{K}\right) ^{-1}\mathbf {Z}_{K}{}^{^{\prime }} \sum _{j=1}^{p}\mathbf {Z}_{j}\mathbf {B}_{j}=\mathbf {D}_{j}\mathbf {B} _{j}\mathbf {L}. \end{aligned}$$

Note that, as the constraints are symmetric, $\mathbf {L}$ is also symmetric. Furthermore, as $j=1,...,p$, we have p equations. However, defining $\mathbf {Z}=\left[ \mathbf {Z}_{1},\ldots ,\mathbf {Z} _{p}\right]$ and $\mathbf {B}=\left[ \mathbf {B}_{1}^{^{\prime }} ,\ldots ,\mathbf {B}_{p}^{^{\prime }}\right] ^{^{\prime }}$, the p equations can be expressed as

$$\begin{aligned} \frac{1}{p^{2}}\mathbf {Z}^{^{\prime }}\mathbf {Z}_{K}\left( \mathbf {Z}_{K} {}^{^{\prime }}\mathbf {Z}_{K}\right) ^{-1}\mathbf {Z}_{K}{}^{^{\prime } }\mathbf {ZB}=\mathbf {DBL}, \end{aligned}$$

where $\mathbf {D}$ is a block-diagonal matrix with as diagonal blocks $\mathbf {D}_{1},\ldots ,\mathbf {D}_{p}$.

Premultiplying both sides by $\mathbf {D}^{-1/2}$ we get

$$\begin{aligned} \frac{1}{p^{2}}\mathbf {D}^{-1/2}\mathbf {Z}^{^{\prime }}\mathbf {Z}_{K}\left( \mathbf {Z}_{K}{}^{^{\prime }}\mathbf {Z}_{K}\right) ^{-1}\mathbf {Z}_{K} {}^{^{\prime }}\mathbf {Z}\mathbf {D}^{-1/2}{\mathbf {D}}^{1/2} \mathbf {B}=\mathbf {D}^{1/2}\mathbf {BL}\text {.} \end{aligned}$$

Without loss of generality we can replace $\mathbf {L}$ by its eigendecomposition $\mathbf {U}\varvec{\Lambda }\mathbf {U}^{\prime }$ to get

$$\begin{aligned} \frac{1}{p^{2}}\mathbf {D}^{-1/2}\mathbf {Z}^{^{\prime }}\mathbf {Z}_{K}\left( \mathbf {Z}_{K}{}^{^{\prime }}\mathbf {Z}_{K}\right) ^{-1}\mathbf {Z}_{K} {}^{^{\prime }}\mathbf {Z}\mathbf {D}^{-1/2}\mathbf {D}^{1/2}\mathbf {B}=\mathbf {D} ^{1/2}\mathbf {BU}{\varvec{\Lambda }} \mathbf {U}^{^{\prime }} \end{aligned}$$

so that

$$\begin{aligned} \frac{1}{p^{2}}\mathbf {D}^{-1/2}\mathbf {Z}^{^{\prime }}\mathbf {Z}_{K}\left( \mathbf {Z}_{K}{}^{^{\prime }}\mathbf {Z}_{K}\right) ^{-1}\mathbf {Z} _{K}^{^{\prime }}\mathbf {Z}\mathbf {D}^{-1/2}\mathbf {D}^{1/2}\mathbf {B}\mathbf {U} =\mathbf {D}^{1/2}\mathbf {BU}\varvec{\Lambda }. \end{aligned}$$

Hence, letting

$$\begin{aligned} \mathbf {B}^{*}={\mathbf {D}}^{1/2}\mathbf {B}\mathbf {U} \end{aligned}$$

we see that $\mathbf {B}^{*}$ can be obtained by taking the first k orthonormal eigenvectors (corresponding to the k largest eigenvalues) of

$$\begin{aligned} \frac{1}{p^{2}}\mathbf {D}^{-1/2}\mathbf {Z}^{^{\prime }}\mathbf {Z}_{K}\left( \mathbf {Z}_{K}{}^{^{\prime }}\mathbf {Z}_{K}\right) ^{-1}\mathbf {Z}_{K} {}^{^{\prime }}\mathbf {Z}\mathbf {D}^{-1/2}. \end{aligned}$$

(23)

The appropriately standardized category quantifications become

$$\begin{aligned} \mathbf {B}=\mathbf {D}^{-1/2}\mathbf {B}^{*} \end{aligned}$$

(24)

and $\mathbf {G}$ is obtained by inserting this into the first-order condition for $\mathbf {G}$, that is,

$$\begin{aligned} \mathbf {G}\,\mathbf {=}\,\frac{1}{p}\left( \mathbf {Z}_{K}{}^{^{\prime }} \mathbf {Z}_{K}\right) ^{-1}\mathbf {Z}_{K}{}^{^{\prime }}\mathbf {ZB} . \end{aligned}$$

(25)

To find $\mathbf {Z}_{K}$, recall the original objective function:

$$\begin{aligned} \min \phi _{\text {groupals}}\left( {\mathbf {B},\mathbf {Z}_{K},\mathbf {G}}\right) =\frac{1}{p}\sum _{j=1}^{p}\left\| \mathbf {Z}_{K}\mathbf {G-Z}_{j}\mathbf {B}_{j}\right\| ^{2}. \end{aligned}$$

For fixed $\mathbf {B}_{j}$, this is equivalent to considering

$$\begin{aligned} \min \phi ^{\prime }_{\text {groupals}}\left( \mathbf {Z}_{K},\mathbf {G}\right) =\left\| \frac{1}{p}\sum _{j=1} ^{p}\mathbf {Z}_{j}\mathbf {B}_{j}-\mathbf {Z}_{K}\mathbf {G}\right\| ^{2}. \end{aligned}$$

Hence, to find $\mathbf {Z}_{K}$ we can apply K-means to the “average configuration”: $\frac{1}{p}\sum _{j=1}^{p}\mathbf {Z}_{j}\mathbf {B}_{j}$.

Note: It can easily be verified that $\mathbf {D}^{1/2}\mathbf {1}$ is an eigenvector of (23) corresponding to the eigenvalue 1. Hence, as in CA and MCA, there is a so-called trivial first solution. Discarding this solution can be achieved by centering $\mathbf {Z}$.

We can summarize the resulting GROUPALS algorithm as follows:

• 1. Generate an initial cluster allocation $\mathbf {Z}_{K}$ (e.g., by randomly assigning subjects to clusters).

• 2. Use (23), (24) and (25) to obtain $\mathbf {B}$ and $\mathbf {G}$.

• 3. Apply the K-means algorithm to the average configuration $\frac{1}{p}\sum _{j=1}^{p}\mathbf {Z}_{j}\mathbf {B}_{j}$, using $\mathbf {G}$ for the initial cluster means, to update $\mathbf {Z}_{K}$ and $\mathbf {G}$.

• 4. Return to step 2 and repeat until convergence.

Comparing this algorithm to the cluster CA algorithm in Sect. 3 shows that despite the different objectives, the two approaches lead to the same algorithm when all variables are categorical.