Probabilistic assessment of model-based clustering

Abstract

Finite mixtures provide a powerful tool for modeling heterogeneous data. Model-based clustering is a broadly used grouping technique that assumes the existence of the one-to-one correspondence between clusters and mixture model components. Although there are many directions of active research in the model-based clustering framework, very little attention has been paid to studying the specific nature of detected clustering solutions. In this paper, we develop an approach for assessing the variability in classifications carried out by the Bayes decision rule. The proposed technique allows assessing significance of each assignment made. We also apply the developed instrument for identifying influential observations that have impact on the structure of the detected partitioning. The proposed diagnostic methodology is studied and illustrated on synthetic data and applied to the analysis of three well-known classification datasets.

Appendix

1.1 The proof of Proposition 2.1 from Sect. 2.3

In this section, we provide main steps needed to prove Proposition 2.1. First, note that

$$\begin{aligned} h_{ik{k^\prime }}(\varvec{\Psi })&= \log {\pi _{ik}(\varvec{\Psi }) \!- \log \pi _{i{k^\prime }}(\varvec{\Psi })}\!= \log (\tau _k \phi (\varvec{x}_i;\varvec{\mu }_k, \varvec{\Sigma }_k))\\&\quad - \log (\tau _{{k^\prime }} \phi (\varvec{x}_i;\varvec{\mu }_{{k^\prime }}, \varvec{\Sigma }_{{k^\prime }}))\\&=\log {\tau _k} \!- \log {\tau _{k^{\prime }}} \!- \frac{1}{2}\log |\varvec{\Sigma }_k| + \frac{1}{2} \log |\varvec{\Sigma }_{k^{\prime }}| - \frac{1}{2}(\varvec{x}_i-\varvec{\mu }_k)^{T}\varvec{\Sigma }_k^{-1}(\varvec{x}_i-\varvec{\mu }_k) \\&\quad + \frac{1}{2}(\varvec{x}_i-\varvec{\mu }_{k^{\prime }})^{T}\varvec{\Sigma }_{k^{\prime }}^{-1}(\varvec{x}_i-\varvec{\mu }_{k^{\prime }}). \end{aligned}$$

For $k^\prime \ne K$, we obtain

$$\begin{aligned} \frac{\partial {h_{ik{k^\prime }}(\varvec{\Psi })}}{\partial \tau _k} = \frac{1}{\tau _k}, \quad \frac{\partial {h_{ik{k^\prime }}(\varvec{\Psi })}}{\partial \tau _{k^\prime }} = -\frac{1}{\tau _{k^\prime }}, \quad \text{ and } \quad \frac{\partial {h_{ik{k^\prime }}(\varvec{\Psi })}}{\partial \tau _s} = 0 \; \text{ for } \; s \ne k, k^\prime . \end{aligned}$$

For $k^\prime = K$, we have

$$\begin{aligned} \frac{\partial {h_{ikK}(\varvec{\Psi })}}{\partial \tau _k} = \frac{\partial \log {\tau _k}}{\partial \tau _k} - \frac{\partial \log \left( 1-\tau _1-\tau _2-\cdots -\tau _{K-1}\right) }{\partial \tau _k} = \frac{1}{\tau _k} +\frac{1}{\tau _K} \end{aligned}$$

$$\begin{aligned} \quad \text{ and }\quad \frac{\partial {h_{ikK}(\varvec{\Psi })}}{\partial \tau _s} = -\frac{\partial \log \left( 1-\tau _1-\tau _2-\cdots -\tau _{K-1}\right) }{\partial \tau _s} = \frac{1}{\tau _K} \; \text{ for } \; s \ne k. \end{aligned}$$

From the table of derivatives for matrix differential calculus, we immediately obtain

$$\begin{aligned} \frac{\partial \varvec{x}_i^T \varvec{\Sigma }_s^{-1} \varvec{\mu }_s}{\partial \varvec{\mu }_s} = \varvec{\Sigma }_s^{-1}\varvec{x}_i, \quad \quad \frac{\partial \varvec{\mu }_s^T \varvec{\Sigma }_s^{-1} \varvec{\mu }_s}{\partial \varvec{\mu }_s} = 2\varvec{\Sigma }_s^{-1}\varvec{\mu }_s, \quad \quad \frac{\partial \log |\varvec{\Sigma }_s|}{\partial \varvec{\Sigma }_s} = \varvec{\Sigma }_s^{-1}, \end{aligned}$$

$$\begin{aligned} \quad \text{ and }\quad \frac{\partial (\varvec{x}_i-\varvec{\mu }_s)^{T}\varvec{\Sigma }_s^{-1}(\varvec{x}_i-\varvec{\mu }_s)}{\partial \varvec{\Sigma }_s} = -\varvec{\Sigma }_s^{-1}(\varvec{x}_i-\varvec{\mu }_s)(\varvec{x}_i-\varvec{\mu }_s)^{T} \varvec{\Sigma }_s^{-1}. \end{aligned}$$

As a result, it follows that

$$\begin{aligned} \frac{\partial {h_{ik{k^\prime }}(\varvec{\Psi })}}{\partial \varvec{\mu }_k}= & {} -\frac{1}{2}\frac{\partial (\varvec{x}_i-\varvec{\mu }_k)^{T}\varvec{\Sigma }_k^{-1}(\varvec{x}_i-\varvec{\mu }_k)}{\partial \varvec{\mu }_k} =\varvec{\Sigma }_k^{-1}(\varvec{x}_i-{\varvec{\mu }}_k),\\ \frac{\partial {h_{ik{k^\prime }}(\varvec{\Psi })}}{\partial \varvec{\mu }_{k^\prime }}= & {} \frac{1}{2}\frac{\partial (\varvec{x}_i-\varvec{\mu }_{k^{\prime }})^{T}\varvec{\Sigma }_{k^\prime }^{-1}(\varvec{x}_i-\varvec{\mu }_{k^{\prime }})}{\partial \varvec{\mu }_{k^{\prime }}} =-\varvec{\Sigma }_{k^\prime }^{-1}(\varvec{x}_{i}-\varvec{\mu }_{k^\prime }),\\&\text{ and } \quad \frac{\partial {h_{ik{k'}}(\varvec{\Psi })}}{\partial \varvec{\mu }_s} = \underline{0} \quad \text{ for } \quad s \ne k, k^\prime . \end{aligned}$$

Fig. 11

Additional illustration for the simulation study from Sect. 4.1. The display represents proportions of correct (black) and incorrect (red) assignments plotted versus the sample size for various levels of overlap denoted by $\omega$

Similarly, we obtain

$$\begin{aligned} \frac{\partial {h_{ik{k^\prime }}(\varvec{\Psi })}}{\partial \varvec{\Sigma }_{k}}&= -\frac{1}{2}\frac{\partial \log |\varvec{\Sigma }_k|}{\partial \varvec{\Sigma }_k} -\frac{1}{2}\frac{\partial (\varvec{x}_i-\varvec{\mu }_k)^{T}\varvec{\Sigma }_k^{-1}(\varvec{x}_i-\varvec{\mu }_k)}{\partial \varvec{\Sigma }_k}\\&= \frac{1}{2} \varvec{\Sigma }_k^{-1}((\varvec{x}_i-\varvec{\mu }_k)(\varvec{x}_i-\varvec{\mu }_k)^{T} \varvec{\Sigma }_k^{-1} - \varvec{I}_p),\\ \frac{\partial {h_{ik{k^\prime }}(\varvec{\Psi })}}{\partial \varvec{\Sigma }_{k^\prime }}&= \frac{1}{2}\frac{\partial \log |\varvec{\Sigma }_{k^\prime }|}{\partial \varvec{\Sigma }_{k^\prime }} + \frac{1}{2}\frac{\partial (\varvec{x}_i-\varvec{\mu }_{k^{\prime }})^{T}\varvec{\Sigma }_{k^\prime }^{-1}(\varvec{x}_i-\varvec{\mu }_{k^{\prime }})}{\partial \varvec{\Sigma }_{k^\prime }}\\&= -\frac{1}{2} \varvec{\Sigma }_{k^{\prime }}^{-1}((\varvec{x}_i-\varvec{\mu }_{k^{\prime }})(\varvec{x}_i-\varvec{\mu }_{k^{\prime }})^{T} \varvec{\Sigma }_{k^{\prime }}^{-1} - \varvec{I}_p),\\&\quad \text{ and } \quad \frac{\partial {h_{ik{k^\prime }}(\varvec{\Psi })}}{\partial \varvec{\Sigma }_s} = \mathbf {0} \quad \text{ for } \quad s \ne k, k^\prime . \end{aligned}$$

After adjusting derivatives with respect to covariance matrices for symmetry, the following gradient vector can be obtained:

$$\begin{aligned}&\nabla h_{ik{k^\prime }}(\varvec{\Psi })\\&\quad = \left( \left( \frac{\partial {h_{ik{k^\prime }}(\varvec{\Psi })}}{\partial \tau _s}\right) _{s=1, \ldots , K-1}, \left( \frac{\partial {h_{ik{k^\prime }}(\varvec{\Psi })}}{\partial {\varvec{\mu }}_s}\right) ^{T}_{s=1, \ldots , K},\ \left( \frac{\partial {h_{ik{k^\prime }}(\varvec{\Psi })}}{\partial \text{ vech }\{\varvec{\Sigma }_s\}}\right) ^{T}_{s=1, \ldots , K}\right) ^{T}. \end{aligned}$$

Now, the direct application of Delta method concludes the proof.

1.2 Additional illustrations for the simulation study from Sect. 4.1

Figure 3 reveals the relationship between proportions of correct and incorrect classifications and the overlap value. In Fig. 11, we provide an additional display devoted to plotting the proportions versus the sample size. As we can see, for a given value of overlap, the proportion of correct significant classifications improves dramatically with the increase in the sample size while the proportion of correct classifications shows just a mild improvement.