A Variational Approximations-DIC Rubric for Parameter Estimation and Mixture Model Selection Within a Family Setting

Abstract

Mixture model-based clustering has become an increasingly popular data analysis technique since its introduction over fifty years ago, and is now commonly utilized within a family setting. Families of mixture models arise when the component parameters, usually the component covariance (or scale) matrices, are decomposed and a number of constraints are imposed. Within the family setting, model selection involves choosing the member of the family, i.e., the appropriate covariance structure, in addition to the number of mixture components. To date, the Bayesian information criterion (BIC) has proved most effective for model selection, and the expectation-maximization (EM) algorithm is usually used for parameter estimation. In fact, this EM-BIC rubric has virtually monopolized the literature on families of mixture models. Deviating from this rubric, variational Bayes approximations are developed for parameter estimation and the deviance information criteria (DIC) for model selection. The variational Bayes approach provides an alternate framework for parameter estimation by constructing a tight lower bound on the complex marginal likelihood and maximizing this lower bound by minimizing the associated Kullback-Leibler divergence. The framework introduced, which we refer to as VB-DIC, is applied to the most commonly used family of Gaussian mixture models, and real and simulated data are used to compared with the EM-BIC rubric.

Appendices

Appendix A: Posterior distributions for the parameters of eigen-decomposed covariance matrix

Table 6 Posterior distributions of the precision parameters as well as their corresponding parameters for 12 of the members of the GPCM family

Appendix B: Posterior expected value of the precision parameters of the eigen-decomposed covariance matrix

Table 7 Posterior expected value of the precision parameters of the eigen-decomposed covariance matrix for 12 of the members of the GPCM family

Appendix C: Mathematical details for the EEV and VEV Models

3.1 C.1 EEV Model

The mixing proportions were assigned a Dirichlet prior distribution, such that

$$ q_{\rho}(\boldsymbol{\rho})=\text{Dir}(\boldsymbol{\rho};\alpha_{1}^{(0)},\ldots,\alpha_{G}^{(0)}). $$

For the mean, a Gaussian distribution conditional on the covariance matrix was used, such that

$$ q_{\boldsymbol{\mu}}(\boldsymbol{\mu} \mid\lambda,\mathbf{A},\mathbf{D}_{1},\ldots,\mathbf{D}_{G})=\prod\limits_{g=1}^{G}\phi_{d}(\boldsymbol{\mu}_{g};\mathbf{m}_{g}^{(0)},(\beta_{g}^{(0)-1}\lambda\mathbf{D}_{g}\mathbf{A}\mathbf{D}_{g}^{\prime})). $$

For the parameters of the covariance matrix, the following priors were used: the k th diagonal elements of (λA)^− 1 were assigned a Gamma $(a^{(0)}_{k},b^{(0)}_{k})$ distribution and D_g was assigned a matrix von Mises-Fisher $(\mathbf {C}^{(0)}_{g})$ distribution. By setting τ = (λA)^− 1, its prior can be written

$$ p_{\tau}(\boldsymbol{\tau})\propto \prod\limits_{k=1}^{K}\tau_{k}^{\frac{a^{(0)}_{k}}{2}-1} \exp\left\{-\frac{b^{(0)}_{k}}{2}\tau_{k}\right\}, $$

where τ_k is the k th diagonal element of τ = (λA)^− 1.

The matrix D has a density as defined by Gupta and Nagar (2000):

$$ p(\mathbf{D}) = b(\mathbf{Q}^{(0)},\mathbf{P}_{g}^{(0)})\exp(\text{tr}\{\mathbf{Q}^{(0)}\mathbf{D}\mathbf{P}_{g}^{(0)}\mathbf{D}^{\prime}\}) [d\mathbf{D}], $$

for D ∈ O(d,d), where O(d,d) is the Stiefel manifold of d × d matrices, [dD] is the unit invariant measure on O(d,d), and A⁽⁰⁾ and $\mathbf {B}_{g}^{(0)}$ are symmetric and diagonal matrices, respectively.

The joint distribution of μ₁,…,μ_G, τ, and D is

$$ \begin{array}{@{}rcl@{}} &&p(\boldsymbol{\mu}_{1},\ldots,\boldsymbol{\mu}_{G},\boldsymbol{\tau}, \mathbf{D}) \propto\prod\limits_{g=1}^{G}|\beta_{g}^{(0)}\boldsymbol{\tau}|^{\frac{1}{2}} \exp \left\{\frac{-(\boldsymbol{\mu}_{g}-\mathbf{m}_{g}^{(0)})\beta_{g}^{(0)}\mathbf{D}_{g}^{\prime}\boldsymbol{\tau} \mathbf{D}_{g}(\boldsymbol{\mu}_{g}-\mathbf{m}_{g}^{(0)})'}{2} \right\}\\ &&\times\exp \left\{\text{tr}(\mathbf{Q}^{(0)}\mathbf{D}\mathbf{P}_{g}^{(0)}\mathbf{D}^{\prime})\right\}\prod\limits_{k=1}^{K}\tau_{k}^{\frac{a^{(0)}_{k}}{2}-1} \exp\left\{-\frac{b^{(0)}_{k}}{2}\tau_{k}\right\}. \end{array} $$

The likelihood of the data can be written

$$ \mathcal{L}(\boldsymbol{\mu}_{1},\!\ldots\!,\boldsymbol{\mu}_{G},\boldsymbol{\tau}, \mathbf{D} \mid \mathbf{y}_{1}, \!\ldots\!, \mathbf{y}_{n}) \!\propto\!\prod\limits_{i=1}^{n}\prod\limits_{g=1}^{G}|\boldsymbol{\tau}|^{{\hat{z}_{ig}}/{2}} \exp\! \left\{\!-\frac{\hat{z}_{ig}}{2}(\mathbf{y}_{i} - \boldsymbol{\mu}_{g})\mathbf{D}_{g}^{\prime}\boldsymbol{\tau} \mathbf{D}_{g}(\mathbf{y}_{i} - \boldsymbol{\mu}_{g})'\right\}.$$

Therefore, the joint posterior distribution of μ, τ, and D can be written

$$p(\boldsymbol{\mu}_{1},\ldots,\boldsymbol{\mu}_{G},\boldsymbol{\tau}, \!\mathbf{D} \!\mid \mathbf{y}_{1}, \ldots, \mathbf{y}_{n}) \!\propto\! p(\boldsymbol{\mu}_{1},\ldots,\boldsymbol{\mu}_{G},\boldsymbol{\tau}, \mathbf{D}) \mathcal{L}(\boldsymbol{\mu}_{1},\ldots,\boldsymbol{\mu}_{G},\boldsymbol{\tau}, \mathbf{D} \!\mid\! \mathbf{y}_{1}, \ldots, \mathbf{y}_{n}).$$

Thus, the posterior distribution of mean becomes

$$q_{\boldsymbol{\mu}}(\boldsymbol{\mu}_{1},\ldots,\boldsymbol{\mu}_{G} \mid \boldsymbol{\tau},\mathbf{D}_{1},\ldots,\mathbf{D}_{G})=\prod\limits_{g=1}^{G}\phi_{d}(\boldsymbol{\mu}_{g};\mathbf{m}_{g},(\beta_{g}\mathbf{D}_{g}^{\prime}\boldsymbol{\tau}\mathbf{D}_{g})^{-1}),$$

where $\beta _{g} = \beta _{g}^{(0)}+{\sum }_{i=1}^{n}\hat {z}_{ig}$ and

$$\mathbf{m}_{g} =\frac{1}{\beta_{g}}\left( \beta_{g}^{(0)} \mathbf{m}_{g}^{(0)}+ \sum\limits_{i=1}^{n}\hat{z}_{ig}\mathbf{y}_{i}\right).$$

The posterior distribution for the k th diagonal element of τ = (λA)^− 1 is

$$q_{\tau}(\tau_{k}) = \text{Gamma} (a_{k},b_{k})$$

where $a_{k}=a_{k}^{(0)}+d{\sum }_{g=1}^{G}{\sum }_{i=1}^{n}\hat {z}_{ig}=a_{k}^{(0)}+dn$ and

$$b_{k}=b_{k}^{(0)}+\sum\limits_{g=1}^{G}\left( \sum\limits_{i=1}^{n}\hat{z}_{ig}y_{ik}^{2}+\beta_{g}^{(0)}m_{gk}^{2}- \beta_{g}m_{gk}^{2}\right).$$

We have

$$ \begin{array}{@{}rcl@{}} q(\mathbf{D}_{g}|\mathbf{y};\boldsymbol{\mu}_{g},\boldsymbol{\tau})&\propto& \exp\left\{ \text{tr} \left (-\frac{1}{2}{(\boldsymbol{\mu}_{g}-\mathbf{m}_{g}^{(0)})\beta_{g}^{(0)}\mathbf{D}_{g}^{\prime}\boldsymbol{\tau} \mathbf{D}_{g}(\boldsymbol{\mu}_{g}-\mathbf{m}_{g}^{(0)})'}\right ) \right \} \\ &&\times\exp\left\{ \text{tr} \left (-\frac{1}{2}{\sum\limits_{i=1}^{n}z_{ig}(\mathbf{y}-\boldsymbol{\mu}_{g})\mathbf{D}_{g}^{\prime}\boldsymbol{\tau} \mathbf{D}_{g}(\mathbf{y}-\boldsymbol{\mu}_{g})'}+\mathbf{Q}_{g}^{(0)}\mathbf{D}_{g}\mathbf{P}_{g}^{(0)}\mathbf{D}_{g}^{\prime}\right )\right \}, \end{array} $$

which has the functional form of a Bingham matrix distribution, i.e., the form

$$\exp \left\{{\text{tr}(\mathbf{Q}_{g} \mathbf{D}_{g}\mathbf{P}_{g} \mathbf{D}_{g}^{\prime}})\right\},$$

where $\mathbf {Q}_{g} = \mathbf {Q}_{g}^{(0)}+\boldsymbol {\tau }$ and

$$ \mathbf{P}_{g} = \mathbf{P}_{g}^{(0)}-\frac{1}{2}\left[\sum\limits_{i=1}^{n}z_{ig}(\mathbf{y}-\boldsymbol{\mu}_{g})(\mathbf{y}-\boldsymbol{\mu}_{g})^{\prime}+(\boldsymbol{\mu}_{g}-\mathbf{m}_{g}^{(0)})\beta_{g}^{(0)}(\boldsymbol{\mu}_{g}-\mathbf{m}_{g}^{(0)})'\right]. $$

3.2 C.2 VEV Model

Similarly, the posterior distribution of D_g for the VEV model has the form

$$ \begin{array}{@{}rcl@{}} q(\mathbf{D}_{g}|\mathbf{y};\boldsymbol{\mu}_{g},\boldsymbol{\tau}_{g})\!\!&\propto&\!\! \exp\left\{ \text{tr} \left (-\frac{1}{2}{(\boldsymbol{\mu}_{g}-\mathbf{m}_{g}^{(0)})\beta_{g}^{(0)}\mathbf{D}_{g}^{\prime}\boldsymbol{\tau}_{g} \mathbf{D}_{g}(\boldsymbol{\mu}_{g}-\mathbf{m}_{g}^{(0)})'}\right ) \right \} \\ &&\!\!\times \exp\left\{ \text{tr} \left (-\frac{1}{2} \sum\limits_{i=1}^{n}\hat{z}_{ig}(\mathbf{y}-\!\boldsymbol{\mu}_{g})\mathbf{D}_{g}^{\prime}\boldsymbol{\tau}_{g} \mathbf{D}_{g}(\mathbf{y}\!-\boldsymbol{\mu}_{g})'+\mathbf{Q}_{g}^{(0)}\mathbf{D}_{g}\mathbf{P}_{g}^{(0)}\mathbf{D}_{g}^{\prime}\right )\right \}, \end{array} $$

which has the functional form of a Bingham matrix distribution, i.e., the form

$$\exp\left\{{\text{tr}(\mathbf{Q}_{g} \mathbf{D}_{g}\mathbf{P}_{g} \mathbf{D}_{g}^{\prime}})\right\},$$

where $\mathbf {Q}_{g}=\mathbf {Q}_{g}^{(0)}+\boldsymbol {\tau }_{g}$ and

$$\mathbf{P}_{g} = -\frac{1}{2}\left[\sum\limits_{i=1}^{n}\hat{z}_{ig}(\mathbf{y}-\boldsymbol{\mu}_{g})(\mathbf{y}-\boldsymbol{\mu}_{g})'+(\boldsymbol{\mu}_{g}-\mathbf{m}_{g}^{(0)})\beta_{g}^{(0)}(\boldsymbol{\mu}_{g}-\mathbf{m}_{g}^{(0)})'\right].$$