Bayesian approach for mixture models with grouped data

Abstract

Finite mixture modeling approach is widely used for the analysis of bimodal or multimodal data that are individually observed in many situations. However, in some applications, the analysis becomes substantially challenging as the available data are grouped into categories. In this work, we assume that the observed data are grouped into distinct non-overlapping intervals and follow a finite mixture of normal distributions. For the inference of the model parameters, we propose a parametric approach that accounts for the categorical features of the data. The main idea of our method is to impute the missing information of the original data through the Bayesian framework using the Gibbs sampling techniques. The proposed method was compared with the maximum likelihood approach, which uses the Expectation-Maximization algorithm for the estimation of the model parameters. It was also illustrated with an application to the Old Faithful geyser data.

Appendices

Appendix 1: Conditional distribution of ${\mathbf {Z}}_{i}$ given $\varvec{\Psi }$ and $D_{ik}=1$

The conditional distribution of ${\mathbf {D}}_i$ given $(\varvec{\Psi },Z_{ri}=1)$ is the multinomial distribution $Multinomial\left( 1,P_{1r}(\varvec{\Theta }_r), \ldots ,P_{Kr}(\varvec{\Theta }_r)\right)$. Hence conditionally on $\varvec{\Psi }$ and $Z_{ri} = 1$ the probability density function (pdf) of $\left( D_{i1},\ldots ,D_{iK}\right)$ can be expressed as $\{P_{kr}(\varvec{\Theta }_r)\}^{D_{ik}}$. It follows that the general expression of the pdf of $\left( D_{i1},\ldots ,D_{iK}\right) |(Z_{1i},\ldots ,Z_{Ri})$ is $\prod ^R_{r=1} [\prod ^K_{k=1}\{P_{kr}(\varvec{\Theta }_r)\}^{D_{ik}}]^{Z_{ri}}$. As a result, the joint pdf of $\left( D_{i1},\ldots ,D_{iK}\right)$ and $(Z_{1i},\ldots ,Z_{Ri})$ is given as follows:

$$\begin{aligned} f({\mathbf {D}}_{i},{\mathbf {Z}}_{i})&= \prod ^R_{r=1}\prod ^K_{k=1}\Bigl \{P_{kr}\bigl (\varvec{\Theta }_r \bigr )\Bigr \}^{D_{ik}Z_{ri}}\pi _r^{Z_{ri}}. \end{aligned}$$

Moreover, the marginal pdf of $(D_{i1} = d_{i1},\ldots ,D_{iK} = d_{iK})$ is given as

$$\begin{aligned} P(D_{i1} = d_{i1},\ldots ,D_{iK} = d_{iK})&= \sum \limits _{Z_{1i}+\ldots +Z_{Ri}=1}\prod \limits ^R_{r=1}\prod \limits ^K_{k=1} \Bigl \{P_{kr}(\varvec{\Theta }_r)\Bigr \}^{D_{ik}Z_{ri}}\pi _r^{Z_{ri}}\\&= \prod \limits ^K_{k=1}\left\{ \sum \limits ^R_{r=1}\pi _r P_{kr}(\varvec{\Theta }_r)\right\} ^{d_{ik}}. \end{aligned}$$

Therefore,

$$\begin{aligned} P(Z_{1i}&= z_{1i},\ldots ,Z_{Ri}=z_{Ri}|D_{i1}=d_{i1},\ldots ,D_{iK}= d_{iK})\\&= \prod \limits ^K_{k=1}\left[ \sum \limits ^R_{r=1}\left\{ \frac{\pi _r P_{kr}(\varvec{\Theta }_r)}{\sum ^R_{l=1}\pi _lP_{kl}(\varvec{\Theta }_l)}\right\} ^{z_{ri}} \right] ^{d_{ik}}, \end{aligned}$$

and

$$\begin{aligned} {\mathbf {Z}}_i|\varvec{\Psi },D_{ik}=1\sim \textit{Multinomial}\left( 1,\frac{\pi _1 P_{k1}(\varvec{\Theta }_1)}{\sum ^R_{r=1}\pi _r P_{kr}(\varvec{\Theta }_r)}, \ldots ,\frac{\pi _R P_{kR}(\varvec{\Theta }_R)}{\sum ^R_{r=1}\pi _r P_{kr}(\varvec{\Theta }_r)}\right) . \end{aligned}$$

Appendix 2: Joint probability density function of $(x_i,{\mathbf {D}}_{i},{\mathbf {Z}}_{i},\varvec{\Psi })$

The conditional pdf of $x_i$ given $D_{ik}=1$ and $Z_{ri}=1$ is specified as in (3). It follows that the general expression of the pdf of $x_i$ given $\varvec{\Psi }, D_{i}$ and $Z_{i}$ can be expressed as

$$\begin{aligned} f(x_i|\varvec{\Psi },{\mathbf {D}}_i,{\mathbf {Z}}_i)=\prod ^R_{r=1}\left[ \prod ^K_{k=1}\left\{ \frac{f_r(x_i |\varvec{\Theta }_r)}{P_{kr}(\varvec{\Theta }_r)}\right\} ^{D_{ik}}\right] ^{Z_{ri}}, \end{aligned}$$

and the joint pdf of $(x_i, {\mathbf {D}}_i, {\mathbf {Z}}_i, \varvec{\Psi })$ is

$$\begin{aligned} f(x_i,{\mathbf {Z}}_i,{\mathbf {D}}_i,\varvec{\Psi })&= \prod \limits ^K_{k=1}\prod \limits ^R_{r=1} \left\{ \frac{f_r(x_i|\varvec{\Theta }_r)}{P_{kr}(\varvec{\Theta }_r)}\right\} ^{Z_{ri} D_{ik}}\left\{ P_{kr}(\varvec{\Theta }_r)\right\} ^{Z_{ri}D_{ik}}\pi _r^{Z_{ri}}\\&= \prod \limits ^R_{r=1}\left\{ \prod \limits ^K_{k=1}f_r(x_i|\varvec{\Theta }_r) \right\} ^{D_{ik}Z_{ri}}\pi _r^{Z_{ri}}. \end{aligned}$$

Under the assumption that the components of the mixture distribution are normal we have

$$\begin{aligned} f_r(x|\Theta _r)=\frac{1}{\sqrt{2\pi }\sigma _r}\exp \left\{ -\frac{(x- \mu _r)^2}{2\sigma ^2_r}\right\} , \end{aligned}$$

for $r=1,\ldots ,R$. Moreover, noting that $\sum _{k=1}^{K}D_{ik}=1, i=1,\ldots ,n$, it is straightforward to see that

$$\begin{aligned}&f(x_i,{\mathbf {D}}_i,{\mathbf {Z}}_i,\varvec{\Psi }) = \prod ^n_{i=1}\prod ^R_{r=1}\left[ \prod ^K_{k=1} \left\{ f_{r}(x_{i}|\varvec{\Theta }_r)\right\} ^{D_{ik}Z_{ri}}\right] \pi _r^{Z_{ri}}\\&\quad = \prod ^n_{i=1}\prod ^R_{r=1}\left[ \prod ^K_{k=1}\Bigl (\frac{1}{\sqrt{2\pi }} \Bigr )^{D_{ik}Z_{ri}}\Bigl (\frac{1}{\sigma ^2_r}\Bigr )^{\frac{1}{2}D_{ik}Z_{ri}} \exp \left\{ -\frac{1}{2}\frac{D_{ik}Z_{ri}(x_i-\mu _r)^{2}}{\sigma ^2_r}\right\} \right] \pi _r^{Z_{ri}}\\&\quad \!=\!\Bigl (\frac{1}{\sqrt{2\pi }}\Bigr )^n \prod ^R_{r=1}\left( \frac{1}{\sigma ^2_r} \right) ^{\frac{1}{2}\sum \limits ^n_{i=1}\sum \limits ^K_{k=1}D_{ik}Z_{ri}} \exp \left\{ \!-\!\frac{1}{2}\sum \limits ^n_{i=1}\sum \limits ^K_{k=1}\frac{D_{ik} Z_{ri}(x_i-\mu _r)^2}{\sigma ^2_r}\right\} \pi _r^{\sum \limits ^n_{i=1}Z_{ri}}\\&\quad =\Bigl (\frac{1}{\sqrt{2\pi }}\Bigr )^n \prod ^R_{r=1}\left( \frac{1}{\sigma ^2_r} \right) ^{\frac{1}{2}\sum \limits ^n_{i=1}Z_{ri}}\exp \left\{ -\frac{1}{2} \sum \limits ^n_{i=1}\sum \limits ^K_{k=1}\frac{D_{ik}Z_{ri}(x_i-\mu _r)^2}{\sigma ^2_r}\right\} \pi _r^{\sum \limits ^n_{i=1}Z_{ri}}. \end{aligned}$$