Unsupervised Classification with a Family of Parsimonious Contaminated Shifted Asymmetric Laplace Mixtures

Abstract

A family of parsimonious contaminated shifted asymmetric Laplace mixtures is developed for unsupervised classification of asymmetric clusters in the presence of outliers and noise. A series of constraints are applied to a modified factor analyzer structure of the component scale matrices, yielding a family of twelve models. Application of the modified factor analyzer structure and these parsimonious constraints makes these models effective for the analysis of high-dimensional data by reducing the number of free parameters that need to be estimated. A variant of the expectation-maximization algorithm is developed for parameter estimation with convergence issues being discussed and addressed. Popular model selection criteria like the Bayesian information criterion and the integrated complete likelihood (ICL) are utilized, and a novel modification to the ICL is also considered. Through a series of simulation studies and real data analyses, that includes comparisons to well-established methods, we demonstrate the improvements in classification performance found using the proposed family of models.

Appendix. Updates Used in the AECM Presented in Section 4

1.1 A.1 E-Step in the First Alternation

In the E-step of the first alternation of iteration $(k+1)$ of the proposed AECM, the sources of missing data required to compute Eq. 31 are replaced with the expected values $z^{(k+1)}_{ig}$, $v^{(k+1)}_{ig}$, $E^{(k+1)}_{1ig}$, $E^{(k+1)}_{2ig}$, $\widetilde{E}^{(k+1)}_{1ig}, \widetilde{E}^{(k+1)}_{2ig}$, respectively, for $g=1,\ldots ,G$. Formally, these expected values can be written as follows:

$$\begin{aligned} z^{(k+1)}_{ig}&:=\mathbb {E}[Z_{ig} \mid \textbf{X}_i = \textbf{x}_i]= \frac{\pi ^{(k)}_g f_{\text {CSAL}}(\textbf{x}_i\ |\ \rho ^{(k)}_g,\eta ^{(k)}_g,\varvec{\mu }^{(k)}_g,\varvec{\alpha }^{(k)}_g,\varvec{\Sigma }^{(k)}_g)}{\sum ^G_{h=1}\pi ^{(k)}_hf_{\text {CSAL}}(\textbf{x}_i\ |\ \rho ^{(k)}_h,\eta ^{(k)}_h,\varvec{\mu }^{(k)}_h,\varvec{\alpha }^{(k)}_h,\varvec{\Sigma }^{(k)}_h)},\\ v^{(k+1)}_{ig}&:=\mathbb {E}[V_{ig} \mid \textbf{X}_i = \textbf{x}_i]=\frac{\rho ^{(k)}_gf_{\text {SAL}}(\textbf{x}_i\ |\ \varvec{\mu }^{(k)}_g,\varvec{\alpha }^{(k)}_g,\varvec{\Sigma }^{(k)}_g)}{f_{\text {CSAL}}(\textbf{x}_i\ |\ \rho ^{(k)}_g,\eta ^{(k)}_g,\varvec{\mu }^{(k)}_g,\varvec{\alpha }^{(k)}_g,\varvec{\Sigma }^{(k)}_g)},\\ E^{(k+1)}_{1ig}&:=\mathbb {E}[W_{ig} \mid \textbf{X}_i = \textbf{x}_i,Z_{ig}=1,V_{ig}=1]=\sqrt{\frac{b^{(k)}_{ig}}{a^{(k)}_g}}R_{\upsilon }\left( \sqrt{a^{(k)}_g b^{(k)}_{ig}}\right) , \\ E^{(k+1)}_{2ig}&:=\mathbb {E}[1/W_{ig} \mid \textbf{X}_i = \textbf{x}_i,Z_{ig}=1,V_{ig}=1]=\sqrt{\frac{a^{(k)}_g}{b^{(k)}_{ig}}}R_{\upsilon }\left( \sqrt{a^{(k)}_g b^{(k)}_{ig}}\right) -\frac{2\upsilon }{b^{(k)}_{ig}}, \\ \widetilde{E}^{(k+1)}_{1ig}&:=\mathbb {E}\left[ \widetilde{W}_{ig} \mid \textbf{X}_i = \textbf{x}_i,Z_{ig}=1,V_{ig}=0\right] =\sqrt{\frac{\widetilde{b}^{(k)}_{ig}}{a^{(k)}_g}}R_{\upsilon }\left( \sqrt{a^{(k)}_g \widetilde{b}^{(k)}_{ig}}\right) , \\ \widetilde{E}^{(k+1)}_{2ig}&:=\mathbb {E}\left[ 1/\widetilde{W}_{ig} \mid \textbf{X}_i = \textbf{x}_i,Z_{ig}=1,V_{ig}=0\right] =\sqrt{\frac{a^{(k)}_g}{\widetilde{b}^{(k)}_{ig}}}R_{\upsilon }\left( \sqrt{a^{(k)}_g \widetilde{b}^{(k)}_{ig}}\right) -\frac{2\upsilon }{\widetilde{b}^{(k)}_{ig}} \end{aligned}$$

where $a^{(k)}_g=2+\varvec{\alpha }^{(k)'}_g(\varvec{\Sigma }^{(k)}_g)^{-1}\varvec{\alpha }^{(k)}_g,\ b^{(k)}_{ig}=\delta (\textbf{x}_i,\varvec{\mu }^{(k)}_g\mid \varvec{\Sigma }^{(k)}_g),\ \widetilde{b}^{(k)}_{ig}=\delta (\textbf{x}_i,\varvec{\mu }^{(k)}_g\mid \eta ^{(k)}_g\varvec{\Sigma }^{(k)}_g)$, $\upsilon = (2-p)/2$, $\varvec{\Sigma }_g^{(k)} = \varvec{\Lambda }_g^{(k)}\varvec{\Lambda }_g^{(k)'} + \omega _g^{(k)}\varvec{\Delta }_g^{(k)}$, and all other terms are previously defined for Eqs. 6 – 9 in Section 2.1. The closed-form updates for $E_{1ig},\ E_{2ig},\ \widetilde{E}_{1ig},$ $\widetilde{E}_{2ig}$ follow from the conditional relationship between a standard exponential random variable and a SAL random vector given in Eq. 14.

1.2 A.2 CM-Step 1 in the First Alternation

In the first CM-step of the first alternation of iteration $(k+1)$ of the proposed AECM, the maximum likelihood estimates (MLEs) for the parameters in $\varvec{\vartheta }_{11}=\{\pi _g,\rho _g,\varvec{\mu }_g,\varvec{\alpha }_g\}_{g=1}^G$ are given by

$$\begin{aligned} {\pi }^{(k+1)}_g{} & {} = \frac{n_g^{(k)}}{n}, \end{aligned}$$

(43)

$$\begin{aligned} {\rho }^{(k+1)}_g{} & {} = \frac{n^{(k)}_{g,\text {good}}}{n^{(k)}_g}, \end{aligned}$$

(44)

$$\begin{aligned} {\varvec{\mu }}^{(k+1)}_g{} & {} = \frac{B^{(k)}\sum ^n_{i=1}a^{(k)}_{ig}\textbf{x}_i-C^{(k)}\sum ^n_{i=1}b^{(k)}_{ig}\textbf{x}_i}{B^{(k)}A^{(k)}-(C^{(k)})^2}, \end{aligned}$$

(45)

$$\begin{aligned} {\varvec{\alpha }}^{(k+1)}_g{} & {} = \frac{A^{(k)}\sum ^n_{i=1} b_{ig}^{(k)}\textbf{x}_i-C^{(k)}\sum ^n_{i=1}a^{(k)}_{ig}\textbf{x}_i}{B^{(k)}A^{(k)}-(C^{(k)})^2}, \end{aligned}$$

(46)

where

$$\begin{aligned}&a^{(k)}_{ig}=z^{(k)}_{ig}\left( v^{(k)}_{ig} E^{(k)}_{2ig} + \frac{1-v^{(k)}_{ig}}{\eta _g^{(k)}}\widetilde{E}^{(k)}_{2ig}\right) ,\quad b^{(k)}_{ig} = z^{(k)}_{ig}\left( v^{(k)}_{ig}+\frac{1-v^{(k)}_{ig}}{\sqrt{\eta _g^{(k)}}}\right) \\&A^{(k)}= \sum ^n_{i=1} z^{(k)}_{ig} \left( v^{(k)}_{ig} E^{(k)}_{2ig} + \frac{1-v^{(k)}_{ig}}{\eta _g^{(k)}}\widetilde{E}^{(k)}_{2ig} \right) , \quad \! B^{(k)} \!=\! \sum ^n_{i=1} z^{(k)}_{ig} \left( v^{(k)}_{ig} E^{(k)}_{1ig} \!+\! (1-v^{(k)}_{ig}) \widetilde{E}^{(k)}_{1ig}\right) ,\\&C^{(k)}= \sum ^n_{i=1} z^{(k)}_{ig} \left( v^{(k)}_{ig} + \frac{1-v^{(k)}_{ig}}{\sqrt{\eta _g^{(k)}}}\right) , \end{aligned}$$

and $n_g^{(k)} = \sum _{g=1}^G{z_{ig}^{(k)}}$ and $n_{g,\text {good}}^{(k)} \sum _{g=1}^G{z_{ig}^{(k)}v_{ig}^{(k)}}$ were defined for Eq. 31.

1.3 A.3 E-Step in the Second Alternation

For the single-factor analysis model defined in Eq. 18,

$$\begin{aligned} \textbf{X}= \varvec{\mu }+ \varvec{\Lambda }\textbf{u}+ \varvec{\epsilon }, \end{aligned}$$

we have $\textbf{X}\sim \text {N}_p\left( \varvec{\mu },\varvec{\Lambda }\varvec{\Lambda }'+\varvec{\Psi }\right)$ and $\textbf{u}\sim \text {N}_q\left( \textbf{0},\textbf{I}_q\right)$, where all terms are as defined for Eq. 18. So, one can show that

$$\begin{aligned} \textbf{X}\mid \textbf{U}&=\textbf{u}\sim \text {N}_p\left( \varvec{\mu }+\varvec{\Lambda }\textbf{u},\varvec{\Psi }\right) , \\ \textbf{u}\mid \textbf{X}&=\textbf{x}\sim \text {N}_q\left( \varvec{\Lambda }'\left( \varvec{\Lambda }\varvec{\Lambda }' + \varvec{\Psi }\right) ^{-1}\left( \textbf{x}-\varvec{\mu }\right) , \left( \textbf{I}_q - \varvec{\Lambda }'\left( \varvec{\Lambda }\varvec{\Lambda }' + \varvec{\Psi }\right) ^{-1}\varvec{\Lambda }\right) \right) . \end{aligned}$$

It follows that the conditional expectations for the latent factor $\textbf{U}$ and the outer product $\textbf{U}\textbf{U}'$ are given by

$$\begin{aligned} \mathbb {E}\big [\textbf{U}\mid \textbf{X}= \textbf{x}\big ]&= \varvec{\beta }(\textbf{X}-\varvec{\mu }^*) \end{aligned}$$

(47)

$$\begin{aligned} \mathbb {E}\big [\textbf{U}\textbf{U}' \mid \textbf{X}= \textbf{x}\big ]&= \textbf{I}_q +\varvec{\beta }\varvec{\Lambda }\varvec{\beta }\big [(\textbf{X}-\varvec{\mu }^*)(\textbf{X}-\varvec{\mu }^*){]}\varvec{\beta }' \end{aligned}$$

(48)

where $\varvec{\beta }=\varvec{\Lambda }'(\varvec{\Lambda }\varvec{\Lambda }'+\varvec{\Psi })^{-1}$. Using Eq. 47 and 48, we can write the expected values required to compute the expected value of Eq. 35 in the E-step of the second alternation. Formally, these expected values can be written as follows:

$$\begin{aligned} \mathbb {E}\left[ Z_{ig}V_{ig}\left( \frac{1-V_{ig}}{\sqrt{\eta _g^{(k+1)}}} \right) \textbf{U}_{ig}\mid \textbf{X}_i = \textbf{x}_i\right]&=z_{ig}^{(k+1/2)}\varvec{\beta }_g^{(k)} \left( v^{(k+1/2)}_{ig}+\frac{\left( 1-v^{(k+1/2)}_{ig}\right) }{\sqrt{\eta _g^{(k+1)}}}\right) (\textbf{x}_i-\varvec{\mu }_g^{(k+1)}) \\&\quad -z_{ig}^{(k+1/2)}\varvec{\beta }_g^{(k)}\left( v^{(k+1/2)}_{ig}E^{(k+1/2)}_{1ig}+\left( 1-v^{(k+1/2)}_{ig}\right) \widetilde{E}^{(k+1/2)}_{1ig}\right) \varvec{\alpha }^{(k+1)}_g, \\ \mathbb {E}\left[ Z_{ig}V_{ig}W^{-1}_{ig}\textbf{U}_{ig} \mid \textbf{X}_i = \textbf{x}_i\right]&=z_{ig}^{(k+1/2)}v_{ig}^{(k+1/2)}\varvec{\beta }_g^{(k)}\left( E^{(k+1/2)}_{2ig}(\textbf{x}_i-\varvec{\mu }_g^{(k+1)})-\varvec{\alpha }^{(k+1)}_g\right) , \\ \mathbb {E}\left[ Z_{ig}\left( \frac{1-V_{ig}}{\eta _g^{(k+1)}}\right) \widetilde{W}^{-1}_{ig}\textbf{U}_{ig} \mid \textbf{X}_i = \textbf{x}_i\right]&=z_{ig}^{(k+1/2)}\varvec{\beta }_g^{(k)}\left( \frac{1-v_{ig}^{(k+1/2)}}{\eta _g^{(k+1)}}\widetilde{E}^{(k+1/2)}_{2ig}(\textbf{x}_i-\varvec{\mu }_g^{(k+1)})\right. \\&\quad -\left. \frac{1-v_{ig}^{(k+1/2)}}{\sqrt{\eta _g^{(k+1)}}}\varvec{\alpha }^{(k+1)}_g\right) , \\ \mathbb {E}\left[ Z_{ig}V_{ig}W^{-1}_{ig}\textbf{U}_{ig}\textbf{U}_{ig}' \mid \textbf{X}_i = \textbf{x}_i\right]&=z_{ig}^{(k+1/2)}v^{(k+1/2)}_{ig}\bigg [\textbf{I}_q +\varvec{\beta }^{(k)}_g\varvec{\Lambda }_g^{(k)} + \varvec{\beta }_g^{(k)}\\&\quad \bigg (E^{(k+1/2)}_{2ig} (\textbf{x}_i-\varvec{\mu }_g^{(k+1)})(\textbf{x}_i-\varvec{\mu }_g^{(k+1)})'&\\&\quad -2(\textbf{x}_i-\varvec{\mu }_g^{(k+1)})(\varvec{\alpha }_g^{(k+1)})' +E^{(k+1/2)}_{1ig}\varvec{\alpha }_g^{(k+1)}(\varvec{\alpha }_g^{(k+1)})' \bigg )\left( \varvec{\beta }_g^{(k)}\right) '\bigg ], \\ \mathbb {E}\left[ Z_{ig}\left( \frac{1-V_{ig}}{\eta _g^{(k+1)}}\right) W^{-1}_{ig}\textbf{U}_{ig}\textbf{U}_{ig}' \mid \textbf{X}_i = \textbf{x}_i\right]&=z_{ig}^{(k+1/2)}\frac{(1-v^{(k+1/2)}_{ig})}{\eta _g^{(k+1)}}\Bigg [\textbf{I}_q +\varvec{\beta }^{(k)}_g\varvec{\Lambda }_g^{(k)} + \left( \varvec{\beta }_g^{(k)}\right) ' \\&\Bigg (\widetilde{E}^{(k+1/2)}_{2ig} (\textbf{x}_i-\varvec{\mu }_g^{(k+1)})(\textbf{x}_i-\varvec{\mu }_g^{(k+1)})' \!-\!2\sqrt{\eta _g^{(k+1)}}(\textbf{x}_i-\varvec{\mu }_g^{(k+1)})(\varvec{\alpha }_g^{(k+1)})' \\&+\widetilde{E}^{(k+1/2)}_{1ig}\eta _g^{(k+1)}\varvec{\alpha }_g^{(k+1)}(\varvec{\alpha }_g^{(k+1)})' \Bigg )\left( \varvec{\beta }_g^{(k)}\right) '\Bigg ], \end{aligned}$$

where $z_{ig}^{(k+1/2)}$, $v^{(k+1/2)}_{ig}$, $E^{(k+1/2)}_{1ig}$, $\widetilde{E}^{(k+1/2)}_{1ig}$, $E^{(k+1/2)}_{2ig}$, and $\widetilde{E}^{(k+1/2)}_{2ig}$ are computed using the updates given in Appendix A.1, but with the required model parameters replaced with their estimates from iteration $(k+1)$, i.e., from the first alternation of the proposed AECM detailed in Section 4.

1.4 A.4 Illustrative Example of the CM-Step in the Second Alternation

To fit the CCUU model, the CM-Step of the second alternation of the proposed AECM is as follows. First, recall that the CCUU model scale matrix has the form

$$\begin{aligned} \varvec{\Sigma }_g=\varvec{\Lambda }\varvec{\Lambda }'+\omega _g\varvec{\Delta }, \end{aligned}$$

(49)

where all terms are as defined for Eqs. 18 and 29 for $g=1,\ldots ,G$. Let

$$\begin{aligned} \textbf{S}^{(k+1)}&= \sum ^g_{g=1}\pi _g^{(k+1)}\textbf{S}_g^{(k+1)}, \\ \varvec{\Theta }^{(k+1/2)}&= \textbf{I}_q-\varvec{\beta }^{(k)}\varvec{\Lambda }^{(k)} +\varvec{\beta }^{(k)}\textbf{S}^{(k+1)}\varvec{\beta }^{(k)'}, \end{aligned}$$

with $\varvec{\beta }^{(k)}$ defined according to the imposed constraints. Maximizing Eq. 37 gives the following updating equations:

$$\begin{aligned} \varvec{\beta }_g^{(k)}&= \varvec{\Lambda }^{(k)'}\left( \varvec{\Lambda }^{(k)}\varvec{\Lambda }^{(k)'}+\omega _g^{(k)}\varvec{\Delta }^{(k)}\right) ^{-1}, \\ \varvec{\Lambda }^{(k+1)}&= \left[ \sum ^G_{g=1}\frac{n^{(k+1/2)}_g}{\omega _g^{(k)}}\textbf{S}_g^{(k+1)}\varvec{\beta }_g^{(k)}\right] \left[ \sum ^G_{g=1}\frac{n^{(k+1/2)}_g}{\omega _g^{(k)}}\varvec{\Theta }_g^{(k+1/2)}\right] ^{-1}, \\ \omega _g^{(k+1)}&= \frac{1}{p}\text {tr}\left\{ \left( \varvec{\Delta }^{(k)}\right) ^{-1}\left[ \textbf{S}_g^{(k+1)} -2\varvec{\Lambda }^{(k+1)}\varvec{\beta }_g^{(k)}\textbf{S}_g^{(k+1)}+\varvec{\Lambda }^{(k+1)}\varvec{\Theta }_g^{(k+1/2)}\varvec{\Lambda }^{(k+1)}\right] \right\} , \\ \varvec{\Delta }^{(k+1)}&= \frac{1}{\kappa }\text {diag}\left\{ \varvec{\Xi }^{(k+1/2)}\right\} , \end{aligned}$$

where

$$\begin{aligned} \varvec{\Xi }^{(k+1/2)}&= \sum ^G_{g=1}\frac{n_g^{(k+1/2)}}{\omega _g^{(k+1)}}\left[ \textbf{S}_g^{(k+1)}-2\varvec{\Lambda }^{(k+1)}\varvec{\beta }_g^{(k)}\textbf{S}_g^{(k+1)} + \varvec{\Lambda }^{(k+1)}\varvec{\Theta }_g^{(k+1/2)}\varvec{\Lambda }^{(k+1)'}\right] , \\ \kappa&= \left( \prod ^p_{j=1}\varvec{\Xi }_{jj}^{(k+1/2)}\right) ^{1/p}. \end{aligned}$$