Bayesian inference for infinite asymmetric Gaussian mixture with feature selection

Abstract

Data clustering is a fundamental unsupervised learning approach that impacts several domains such as data mining, computer vision, information retrieval, and pattern recognition. In this work, we develop a statistical framework for data clustering which uses Dirichlet processes and asymmetric Gaussian distributions. The parameters of this framework are learned using Markov Chain Monte Carlo inference approaches. We also integrate a feature selection technique to choose the features that are most informative in order to construct an appropriate model in terms of clustering accuracy. This paper reports results based on experiments that concern dynamic textures clustering as well as scene categorization.

Appendix

Based on the hyperparameters setting chosen in Section 4, we deduce the posteriors for all of the parameters. For parameter $\alpha$, the posteriors depend only on the number of observations N and the number of components M, and not on how the distributions are distributed among the mixtures:

$$\begin{aligned} p \big (\alpha \mid k, n \big ) \propto \frac{\alpha ^{M-\frac{3}{2}}exp(-\frac{1}{2\alpha })\Gamma (\alpha )}{\Gamma (N+\alpha )} \end{aligned}$$

(32)

The complete posteriors for $\mu$, $\mu _{irr}$, $\lambda$ and r are obtained as follows:

$$\begin{aligned}&p\big ( \mu _{j k} \mid \dots \big ) \propto \, \mathcal {N}\bigg ((r\lambda + S_{lj k}\sum _{i: \phi _{ijk}=1, X_{ik} <\mu _{j k}} X_{i k} \nonumber \\&+ s_{rj k}\sum _{i: \phi _{ijk}=1, X_{ik}\ge \mu _{j k}} X_{i k}) / (r + p s_{lj k} + (n_j-p) s_{rj k}), \nonumber \\&\frac{1}{r + p s_{lj k} + (n_j - p) s_{rj k}} \bigg ) \end{aligned}$$

(33)

$$\begin{aligned}&p\big ( \mu _{j k}^{irr} \mid \dots \big ) \propto \mathcal {N}\bigg (\frac{\sum _{i, \phi _{ijk}=0} x_{ik}^{irr} S_{j k}^{irr} + r_k^{irr} \lambda _k^{irr}}{r_k^{irr} + n_j^{irr} S_{j k}^{irr} }, \nonumber \\&\frac{1}{r_k^{irr} + n_j^{irr} S_{j k}^{irr}}\bigg ) \end{aligned}$$

(34)

$$\begin{aligned} p\big (&\lambda \mid \mu _{1 k},\dots ,\mu _{M k}, r\big ) \propto \mathcal {N}\bigg (\frac{r\sum _{j=1}^M \mu _{j k} +\mu _x\sigma _x^{-2}}{\sigma _x^{-2}+Mr_k}, \nonumber \\&\frac{1}{\sigma _x^{-2}+Mr_k}\bigg ) \end{aligned}$$

(35)

$$\begin{aligned}&p\big (r \mid \mu _{1 k},\dots ,\mu _{M k}, \lambda \big ) \propto \gamma \bigg (\frac{M+1}{2}, \end{aligned}$$

(36)

$$\begin{aligned}&\frac{2}{\sigma _x^2 + \sum _{j=1}^M (\mu _{jk} - \lambda _k)^2}\bigg ) \end{aligned}$$

(37)

The complete posteriors for $s_{ljk}$, $s_{rjk}$, $s_{jk}^{irr}$, $\beta$ and w are obtained as follows:

$$\begin{aligned}&p\big (S_{lj k} \mid X, \mu _{j}, S_{rj}, \beta _l, w_l \big ) \nonumber \\&\propto \exp \bigg [-\frac{S_{lj k}\sum _{i:X_{i k}<\mu _{j k}}^n(x_{i k} - \mu _{j k})^2}{2} -\frac{w_{lk}\beta _{lk} S_{lj k}}{2}\bigg ] \end{aligned}$$

(38)

$$\begin{aligned}&p\big ( S_{j}^{irr} \mid X, \mu _{j}^{irr}, \beta ^{irr},w^{irr} \big ) \propto \Gamma \bigg (\frac{N_{jk}^{irr} \beta _k^{irr}}{2}, \nonumber \\&\frac{2}{\beta _k^{irr} w_k^{irr} + \sum _{i,\phi _{ijk} = 0} (X_{ik} - \mu _{j k}^{irr})^2}\bigg ) \end{aligned}$$

(39)

$$\begin{aligned}&p\big ( \beta _l \mid S_{l1 k},\dots ,S_{lM k}, w_l) \propto \Gamma (\frac{\beta _l}{2})^{-M}exp\bigg (-\frac{1}{2\beta _l}\bigg )\nonumber \\&(\frac{\beta _l}{2})^{\frac{M\beta _l -3 }{2}} \prod _{j=1}^{M}(w_l S_{lj k})^{\frac{\beta _l}{2}} exp\bigg (-\frac{\beta _l w_l s_{lj k}}{2}\bigg ) \end{aligned}$$

(40)

$$\begin{aligned}&p\big ( w_l \mid S_{l1 k},\dots ,S_{lM k}, \beta _l) \propto \Gamma \bigg (\frac{M\beta _l+1}{2}, \nonumber \\&\frac{2}{\sigma _y^{-2}+\beta _l \sum _{j=1}^{M}S_{lj k}}\bigg ) \end{aligned}$$

(41)

$N_{jk}^{re}$ and $N_{jk}^{irr}$ are the number of observations allocated to mixture j with feature k considered as relevant and irrelevant, respectively.

The complete posteriors for feature saliency $\phi$ with gamma parameters a and b, with $n_{jk}$ the number of feature k relevant for component j can then be expressed by:

$$\begin{aligned} p \big ( \rho _{j k} \mid \dots \big ) \propto \text {Beta} \left( a + n_{jk}, b + N - n_{jk}\right) \end{aligned}$$

(42)

$$\begin{aligned} p\big (a \mid \dots )\propto & {} a e^{-\frac{a}{2}} \bigg (\frac{\Gamma (a+b)}{\Gamma (a)}\bigg )^M \prod _{j=1}^M \rho _{jk}^{a-1 } \nonumber \\ p\big (b \mid \dots )\propto & {} b e^{-\frac{b}{2}} \bigg (\frac{\Gamma (a+b)}{\Gamma (b)}\bigg )^M \prod _{j=1}^M (1-\rho _{jk})^{a-1 } \end{aligned}$$

(43)