Mixture models: building a parameter space

Abstract

Despite the flexibility and popularity of mixture models, their associated parameter spaces are often difficult to represent due to fundamental identification problems. This paper looks at a novel way of representing such a space for general mixtures of exponential families, where the parameters are identifiable, interpretable, and, due to a tractable geometric structure, the space allows fast computational algorithms to be constructed.

Appendix

Following the methodology in Sect. 2.1 select suitably separated grid points $\varvec{\mu }=(\mu _1,\ldots ,\mu _L)$, which are fixed throughout. Select initial proportions $\varvec{\rho }^{(0)}=(\rho _1^{(0)},\ldots ,\rho _L^{(0)})$ and local mixture parameters $\underline{\varvec{\lambda }}^{(0)} =(\varvec{\lambda }^{1,(0)},\ldots ,\varvec{\lambda }^{L,(0)})$. Suppose we have $\varvec{\rho }^{(r)}$ and $\underline{\varvec{\lambda }}^{(r)}$ at step r, where $L_r$ is the number of non-zero proportions and $L_r \le L$. To obtain the estimates at step $r+1$ run the following steps.

• 1. Calculate $\rho _l^{(r+1)}=\frac{n_l}{n}$, where $n_l= \sum \nolimits _{i=1}^{n}w^{(r+1)}_{il}$ and for $x=1,\ldots ,n;\,\, l=1,\ldots ,L_r$

  $$\begin{aligned} w^{(r+1)}_{il}=\frac{\rho _l^{(r)} g_{\mu _l}\left( x_i,\varvec{\lambda }^{l,(r)}\right) }{\sum \nolimits _{l=1}^{L_r} \rho _l^{(r)} g_{\mu _l}\left( x_i,\varvec{\lambda }^{l,(r)}\right) }. \end{aligned}$$

• 2. Choose a positive value $0 < \gamma < 1$, and check if there is any l such that $\rho _l^{(r+1)} < \gamma$.

  • (a) If yes: exclude the components corresponding to $\rho _l^{(r+1)} < \gamma$, update $L_r\rightarrow L_{r+1}$ and go back to step 1.

  • (b) If no: go to step 3.

• 3. Classify the data set into $\varvec{x}^1,\ldots \varvec{x}^{L_{r+1}}$ by assigning each $x_i$ to only one local mixture component. For each $l=1,\ldots ,L_{r+1}$, update $\varvec{\lambda }^{l,(r)}$ by

  $$\begin{aligned} \varvec{\lambda }^{l,(r+1)}= {\text {*}}{arg\,max}_{\varvec{\lambda } \in \Lambda _{\mu _l}} l_{\mu _l}(x^l,\varvec{\lambda }), \end{aligned}$$

  where $l_{\mu _l}(x^l,\cdot )$ is the log-likelihood function for the component l as defined in Marriott (2002).

Remark 1

Step 2 restricts the number of required components for fitting a data set in a way that there is enough information necessary for running inference on each local mixture component. The value, $\gamma$, has an influence on the final result of the algorithm in a similar way that an initial value affects the convergence of a general EM algorithm (Table 1).