Graphical tools for model-based mixture discriminant analysis

Abstract

The paper introduces a methodology for visualizing on a dimension reduced subspace the classification structure and the geometric characteristics induced by an estimated Gaussian mixture model for discriminant analysis. In particular, we consider the case of mixture of mixture models with varying parametrization which allow for parsimonious models. The approach is an extension of an existing work on reducing dimensionality for model-based clustering based on Gaussian mixtures. Information on the dimension reduction subspace is provided by the variation on class locations and, depending on the estimated mixture model, on the variation on class dispersions. Projections along the estimated directions provide summary plots which help to visualize the structure of the classes and their characteristics. A suitable modification of the method allows us to recover the most discriminant directions, i.e., those that show maximal separation among classes. The approach is illustrated using simulated and real data.

Appendix

1.1 Proof of Proposition 1

Assume an EDDA mixture model with common full class covariance matrix. The last condition implies that the matrix ${\varvec{M}}_\mathsf{II }$ in Eq. (2) cancels out, so the kernel matrix simplifies to

$$\begin{aligned} {\varvec{M}}= {\varvec{M}}_\mathsf{I }{\varvec{\varSigma }}_X^{-1} {\varvec{M}}_\mathsf{I }, \end{aligned}$$

where ${\varvec{M}}_\mathsf{I }= \sum _{k=1}^K \pi _k ({\varvec{\mu }}_{k} - {\varvec{\mu }})({\varvec{\mu }}_{k} - {\varvec{\mu }}){}^{\top }= {\varvec{\varSigma }}_B$, the between-class covariance matrix. The basis of the subspace $\mathcal{S }({\varvec{\beta }})$ provided by GMMDRC is obtained as the solution of the following problem

$$\begin{aligned} {\varvec{M}}{\varvec{\beta }}_j = l_j {\varvec{\varSigma }}_X {\varvec{\beta }}_j, \end{aligned}$$

with $l_1 \ge \dots \ge l_d$, and $d = \min (p,K-1)$. Thus, ${\varvec{\beta }}_j$ is the $j$th eigenvector associated to the $j$th largest eigenvalue $l_j$ ($j=1,\ldots ,d$) of the $(p \times p)$ matrix

$$\begin{aligned} {\varvec{\varSigma }}^{-1/2}_X {\varvec{M}}{\varvec{\varSigma }}^{-1/2}_X&= {\varvec{\varSigma }}^{-1/2}_X {\varvec{M}}_\mathsf{I }{\varvec{\varSigma }}_X^{-1} {\varvec{M}}_\mathsf{I }{\varvec{\varSigma }}^{-1/2}_X \\&= \left( {\varvec{\varSigma }}^{-1/2}_X {\varvec{\varSigma }}_B {\varvec{\varSigma }}^{-1/2}_X\right) {}^{\top }\left( {\varvec{\varSigma }}^{-1/2}_X {\varvec{\varSigma }}_B {\varvec{\varSigma }}^{-1/2}_X\right) . \end{aligned}$$

The subspace estimated by SIR is obtained as the solution of

$$\begin{aligned} {\varvec{\varSigma }}_B {\varvec{\beta }}_j^{\mathsf{SIR }}\ = l_j^{\mathsf{SIR }}{\varvec{\varSigma }}_X {\varvec{\beta }}_j^{\mathsf{SIR }} \end{aligned}$$

(6)

which is given by the eigen-decomposition of ${\varvec{\varSigma }}^{-1/2}_X {\varvec{\varSigma }}_B{\varvec{\varSigma }}^{-1/2}_X$. It is easily seen that ${\varvec{\beta }}_j = {\varvec{\beta }}_j^{\mathsf{SIR }}$ and $l_j = (l_j^{\mathsf{SIR }})^2$, for $j=1,\ldots ,d$. Thus, the basis of the subspace provided by GMMDRC under model EDDA with full common class covariance matrix is equivalent to the basis estimated by SIR.

We now consider the relation of GMMDRC with LDA canonical variates. From (6), we may subtract $l_j^{\mathsf{SIR }}{\varvec{\varSigma }}_{B} {\varvec{\beta }}_{j}^{{\mathsf{SIR }}}$ from both side and, recalling the decomposition of the total variance, ${\varvec{\varSigma }}_X = {\varvec{\varSigma }}_B + {\varvec{\varSigma }}_W$, we may write

$$\begin{aligned} {\varvec{\varSigma }}_B {\varvec{\beta }}_j^{\mathsf{SIR }}- l_j^{\mathsf{SIR }}{\varvec{\varSigma }}_B {\varvec{\beta }}_j^{\mathsf{SIR }}&= l_j^{\mathsf{SIR }}{\varvec{\varSigma }}_X {\varvec{\beta }}_j^{\mathsf{SIR }}- l_j^{\mathsf{SIR }}{\varvec{\varSigma }}_B {\varvec{\beta }}_j^{\mathsf{SIR }}\\ \left( 1 - l_j^{\mathsf{SIR }}\right) {\varvec{\varSigma }}_B {\varvec{\beta }}_j^{\mathsf{SIR }}&= l_j^{\mathsf{SIR }}\left( {\varvec{\varSigma }}_X - {\varvec{\varSigma }}_B\right) {\varvec{\beta }}_j^{\mathsf{SIR }}\\ {\varvec{\varSigma }}_B {\varvec{\beta }}_j^{\mathsf{SIR }}&= l_j^{\mathsf{SIR }}/\left( 1 - l_j^{\mathsf{SIR }}\right) {\varvec{\varSigma }}_W {\varvec{\beta }}_j^{\mathsf{SIR }}. \end{aligned}$$

It is clear that $l_j^{{\mathsf{SIR }}} /(1-l_{j}^{{\mathsf{SIR }}})$ and ${\varvec{\beta }}_{j}^{{\mathsf{SIR }}}$ are, respectively, the $j$th eigenvalue and the associated eigenvector of ${\varvec{\varSigma }}_W^{-1/2} {\varvec{\varSigma }}_B {\varvec{\varSigma }}_W^{-1/2}$, the decomposition solving the Rayleigh quotient used to derive canonical variates in LDA. Thus, the basis of the subspace $\mathcal{S }({\varvec{\beta }}^\mathsf{LDA })$ is equivalent to $\mathcal{S }({\varvec{\beta }}^{\mathsf{SIR }})$, which in turn is equivalent to that provided by GMMDRC under the specific model assumption.

1.2 Proof of Proposition 2

The kernel matrix of SAVE can be written in the original scale of the variables as

$$\begin{aligned} {\varvec{M}}_\mathsf{SAVE } = \sum _{k=1}^K \omega _k \left( {\varvec{I}}_p - {\varvec{\varSigma }}_X^{-1/2} {\varvec{\varSigma }}_k {\varvec{\varSigma }}_X^{-1/2} \right) ^2. \end{aligned}$$

Recalling that ${\varvec{\varSigma }}_X = {\varvec{\varSigma }}_B + {\varvec{\varSigma }}_W$, we may write the expression within parenthesis as follows:

$$\begin{aligned} {\varvec{\varSigma }}_X^{-1/2} ({\varvec{\varSigma }}_X - {\varvec{\varSigma }}_k) {\varvec{\varSigma }}_X^{-1/2}&= {\varvec{\varSigma }}_X^{-1/2} ({\varvec{\varSigma }}_B + {\varvec{\varSigma }}_W - {\varvec{\varSigma }}_k) {\varvec{\varSigma }}_X^{-1/2} \\&= {\varvec{\varSigma }}_X^{-1/2} {\varvec{\varSigma }}_B {\varvec{\varSigma }}_X^{-1/2} + {\varvec{\varSigma }}_X^{-1/2} ({\varvec{\varSigma }}_W - {\varvec{\varSigma }}_k) {\varvec{\varSigma }}_X^{-1/2}. \end{aligned}$$

Then,

$$\begin{aligned} {\varvec{M}}_\mathsf{SAVE }&= \sum _{k=1}^K \omega _k \left( {\varvec{\varSigma }}_X^{-1/2} {\varvec{\varSigma }}_B {\varvec{\varSigma }}_X^{-1/2} + {\varvec{\varSigma }}_X^{-1/2}({\varvec{\varSigma }}_W - {\varvec{\varSigma }}_k) {\varvec{\varSigma }}_X^{-1/2} \right) ^2 \\&= {\varvec{\varSigma }}_X^{-1/2} {\varvec{\varSigma }}_B {\varvec{\varSigma }}_X^{-1} {\varvec{\varSigma }}_B {\varvec{\varSigma }}_X^{-1/2} \\&+ {\varvec{\varSigma }}_X^{-1/2} \left( \sum _{k=1}^K w_k ({\varvec{\varSigma }}_k - {\varvec{\varSigma }}_W) {\varvec{\varSigma }}_X^{-1} ({\varvec{\varSigma }}_k - {\varvec{\varSigma }}_W){}^{\top }\right) {\varvec{\varSigma }}_X^{-1/2}\\&= {\varvec{\varSigma }}_X^{-1/2} {\varvec{M}}_\mathsf{I }{\varvec{\varSigma }}_X^{-1} {\varvec{M}}_\mathsf{I }{\varvec{\varSigma }}_X^{-1/2} + {\varvec{\varSigma }}_X^{-1/2} {\varvec{M}}_\mathsf{II }{\varvec{\varSigma }}_X^{-1/2} \\&= {\varvec{\varSigma }}_X^{-1/2} ( {\varvec{M}}_\mathsf{I }{\varvec{\varSigma }}_X^{-1} {\varvec{M}}_\mathsf{I }+ {\varvec{M}}_\mathsf{II }) {\varvec{\varSigma }}_X^{-1/2}, \end{aligned}$$

where ${\varvec{M}}_\mathsf{I }$ and ${\varvec{M}}_\mathsf{II }$ are those obtained from an EDDA Gaussian mixture model with a single component for each class and different class covariance matrices (VVV).

1.3 Proof of Proposition 3

The proof is analogous to that provided for Prop. 2 in Scrucca (2010) and it is not replicated here.