Model-based clustering with sparse covariance matrices

Abstract

Finite Gaussian mixture models are widely used for model-based clustering of continuous data. Nevertheless, since the number of model parameters scales quadratically with the number of variables, these models can be easily over-parameterized. For this reason, parsimonious models have been developed via covariance matrix decompositions or assuming local independence. However, these remedies do not allow for direct estimation of sparse covariance matrices nor do they take into account that the structure of association among the variables can vary from one cluster to the other. To this end, we introduce mixtures of Gaussian covariance graph models for model-based clustering with sparse covariance matrices. A penalized likelihood approach is employed for estimation and a general penalty term on the graph configurations can be used to induce different levels of sparsity and incorporate prior knowledge. Model estimation is carried out using a structural-EM algorithm for parameters and graph structure estimation, where two alternative strategies based on a genetic algorithm and an efficient stepwise search are proposed for inference. With this approach, sparse component covariance matrices are directly obtained. The framework results in a parsimonious model-based clustering of the data via a flexible model for the within-group joint distribution of the variables. Extensive simulated data experiments and application to illustrative datasets show that the method attains good classification performance and model quality. The general methodology for model-based clustering with sparse covariance matrices is implemented in the R package mixggm, available on CRAN.

Appendices

Appendix A: Iterative conditional fitting algorithm

The ICF algorithm (Chaudhuri et al. 2007) is employed to estimate a sparse covariance matrix given a certain structure of association. In this appendix, we present the algorithm in application to Gaussian mixture model estimation and we extend it to allow for Bayesian regularization of the covariance matrix.

Given a graph ${\mathcal {G}}_k = ({\mathcal {V}}, {\mathcal {E}}_k)$, to find the corresponding sparse covariance matrix under the constraint of being positive definite we need to maximize the objective function:

$$\begin{aligned} -\dfrac{N_k}{2} \left[ \text {tr}({\mathbf {S}}_k{{\varvec{\Sigma }}}_k^{-1}) + \log \det {{\varvec{\Sigma }}}_k \right] \quad \text {with}\quad {{\varvec{\Sigma }}}_k \in {\mathcal {C}}^+\left( {\mathcal {G}}_k \right) . \end{aligned}$$

Let us make use of the following conventions: subscript [j, h] denotes element (j, h) of a matrix, a negative index such as $-j$ denotes that row or column j has been removed, subscript $[\,,j]$ (or $[j,\,]$) denotes that column (or row) j has been selected. Moreover, we denote with s(j) the set of indexes corresponding to the variables connected to variable $X_j$ in the graph, i.e. the positions of the non zero entries in the covariance matrix for $X_j$. Following Chaudhuri et al. (2007), the ICF algorithm is implemented as follows:

• 1. Set the iteration counter $r=0$. Initialize the covariance matrix ${\hat{{{\varvec{\Sigma }}}}}^{(0)}_k = \text {diag}({\mathbf {S}}_k)$.

• 2. For $j = (1,\, \ldots ,\, V)$

  • 2a compute $\varvec{\varOmega }_k^{(r)} = ({\hat{{{\varvec{\Sigma }}}}}^{(r)}_{k[-j,-j]})^{-1}$

  • 2b compute the covariance terms estimates

    $$\begin{aligned} {\hat{{{\varvec{\Sigma }}}}}^{(r)}_{k[j,s(j)]}= & {} \left( {\mathbf {S}}_{k[j,-j]}\,\varvec{\varOmega }^{(r)}_{k[\,,s(j)]} \right) \,\\&\quad \left( \varvec{\varOmega }^{(r)}_{k[s(j),\,]} {\mathbf {S}}_{k[-j,-j]} \varvec{\varOmega }^{(r)}_{k[\,,s(j)]} \right) \end{aligned}$$

  • 2c compute $\lambda _j {=} {\mathbf {S}}_{k[j,j]} - {\hat{{{\varvec{\Sigma }}}}}^{(r)}_{k[j,s(j)]} \left( {\mathbf {S}}_{k[j,-j]}\,\varvec{\varOmega }^{(r)}_{k[\,,s(j)]} \right) ^{\!\top }$

  • 2d compute the variance term estimate

    $$\begin{aligned} {\hat{{{\varvec{\Sigma }}}}}^{(r)}_{k[j,j]} = \lambda _j + {\hat{{{\varvec{\Sigma }}}}}^{(r)}_{k[j,s(j)]} \varvec{\varOmega }^{(r)}_{k[s(j),s(j)]} {\hat{{{\varvec{\Sigma }}}}}^{(r)}_{k[s(j),j]} \end{aligned}$$

• 3. Set ${\hat{{{\varvec{\Sigma }}}}}^{(r+1)}_k = {\hat{{{\varvec{\Sigma }}}}}_k^{(r)}$, increment $r = r + 1$ and return to (2).

The algorithm stops when the increase in the objective function is less than a pre-specified tolerance. The covariance matrix in output has zero entries corresponding to the graph structure and is guaranteed of being positive definite.

In the case of Bayesian regularization, the objective function becomes:

$$\begin{aligned} - \dfrac{{\tilde{N}}_k}{2} \left[ \text {tr}(\tilde{{\mathbf {S}}}_k{{\varvec{\Sigma }}}_k^{-1}) + \log \det {{\varvec{\Sigma }}}_k \right] \quad \text {with}\quad {{\varvec{\Sigma }}}_k \in {\mathcal {C}}^+\left( {\mathcal {G}}_k \right) , \end{aligned}$$

where

$$\begin{aligned} {\tilde{N}}_k = N_k + \omega + V + 1, \quad \tilde{{\mathbf {S}}}_k = \dfrac{1}{{\tilde{N}}_k} \left[ N_k {\mathbf {S}}_k + {\mathbf {W}} \right] . \end{aligned}$$

The shape of the objective function corresponds to the one not regularized. Therefore, the same algorithm can be applied replacing $N_k$ and ${\mathbf {S}}_k$ with ${\tilde{N}}_k$ and $\tilde{{\mathbf {S}}}_k$.

Appendix B: Initialization of the S-EM algorithm

The S-EM algorithm requires two initialization steps: initialization of cluster allocations and initialization of the graph structure search. For the first task we use the Gaussian model-based hierarchical clustering approach of Scrucca and Raftery (2015), which has been shown to yield good starting points, be computationally efficient and work well in practice. For initialization of the graph structure search we use the following approach. Let ${\mathbf {R}}_k$ be the correlation matrix for component k, computed as:

$$\begin{aligned} {\mathbf {R}}_k = {\mathbf {U}}_k{\mathbf {S}}_k{\mathbf {U}}_k, \end{aligned}$$

where ${\mathbf {U}}_k$ is a diagonal matrix whose elements are ${\mathbf {S}}_{k,[j,j]}^{-1/2}$ for $j=1,\ldots ,V$, i.e. the within component sample standard deviations. A sound strategy is to initialize the search for the optimal association structure by looking at the most correlated variables. Therefore, we define the adjacency matrix ${\mathbf {A}}_k$ whose off-diagonal elements $a_{jhk}$ are given by:

$$\begin{aligned} a_{jhk} = {\left\{ \begin{array}{ll} 1 \quad \text {if}~~ |r_{jhk} |~~ \ge ~\rho ,\\ 0 \quad \text {otherwise} \end{array}\right. } \end{aligned}$$

where $r_{jhk}$ is an off-diagonal element of ${\mathbf {R}}_k$ and $\rho$ is a threshold value. In practice, we define a vector of values for $\rho$ ranging from 0.4 to 1. For each value of $\rho$, the related adjacency matrix is derived and the corresponding sparse covariance matrix is estimated using the ICF algorithm. Then the different adjacency matrices are ranked according to their value of the objective function in (5). Subsequently the structure search starts from the adjacency matrix at the top of the rank.

Appendix C: Details of simulation experiments

This appendix section describes the various simulated data scenarios considered in Sect. 5 of the paper.

Scenario 1: In this setting we consider a structure with a single block of associated variables of size $\left\lfloor {\frac{V}{2}}\right\rfloor$. The groups are differentiated by the position of the block, top corner, center and bottom corner respectively. Figure 3 displays an example of such structure for $V=20$. To generate the covariance matrices, first we generate a $V\times V$ matrix with all entries equal to 0.9 and diagonal 1. Then we use it as input of the ICF algorithm to estimate the corresponding covariance matrix with the given structure.

Scenario 2: For this scenario, the graphs are generated at random from an Erdős–Rényi model. The groups are characterized by different probabilities of connection, 0.3, 0.2 and 0.1 respectively. Figure 4 presents an example of a collection of structures of association for $V=20$. Starting from a $V\times V$ matrix with all entries equal to 0.9 and diagonal 1, we employ the ICF algorithm to estimate the corresponding sparse covariance matrix. In the simulated data experiment of Part III, we consider connection probabilities equal to 0.10, 0.05 and 0.03.

Scenario 3: This scenario is characterized by hubs, i.e. highly connected variables. Each cluster has $\frac{V}{2}$ such hubs. The graph structures and the corresponding covariance matrices are generated randomly using the R package hglasso. (Tan 2014). The three groups have different sparsity levels, respectively 0.7, 0.8 and 0.9. Figure 5 presents an example of this type of graphs for $V=20$. We point out that the method implemented in the package poses strict constraints on the covariance matrix and often some connected variables have weak correlations, making difficult to infer the association structure.

Scenario 4: Here the groups have structures of different types: block diagonal, random connections and Toeplitz type. For the first group we consider a block diagonal matrix with blocks of size 5. Regarding the second, the graph is generated at random from an Erdős–Rényi model with parameter 0.2. In both cases, we start from a $V\times V$ matrix with all entries equal to 0.9 and diagonal 1, and then we employ the ICF algorithm to estimate the corresponding sparse covariance matrices. For the Toeplitz matrix we take $\sigma _{j,\,j-1} = \sigma _{j-1,\,j} = 0.5$ for $j=2,\,\ldots ,\,V$. Figure 6 depicts an example of these graph configurations for $V=20$. In the simulated data experiment of Part III, we consider an Erdős–Rényi model with parameter 0.05 and a block diagonal matrix with 5 blocks of size 20; the Toeplitz matrix is generated as before.