Bayesian finite mixtures with an unknown number of components: The allocation sampler

Statistics and Computing volume 17, pages147–162(2007)Cite this article

Abstract

A new Markov chain Monte Carlo method for the Bayesian analysis of finite mixture distributions with an unknown number of components is presented. The sampler is characterized by a state space consisting only of the number of components and the latent allocation variables. Its main advantage is that it can be used, with minimal changes, for mixtures of components from any parametric family, under the assumption that the component parameters can be integrated out of the model analytically. Artificial and real data sets are used to illustrate the method and mixtures of univariate and of multivariate normals are explicitly considered. The problem of label switching, when parameter inference is of interest, is addressed in a post-processing stage.

This is a preview of subscription content, log in to check access.

References

  1. Aitkin M. 2001. Likelihood and Bayesian analysis of mixtures. Statistical Modelling 1: 287–304.

    Article  Google Scholar 

  2. Böhning D. and Seidel W. 2003. Editorial: Recent developments in mixture models. Computational Statistics and Data Analysis 41: 349–357.

    Article  MathSciNet  Google Scholar 

  3. Casella G., Robert C.P., and Wells M.T. 2000. Mixture models, latent variables and partitioned importance sampling. Tech Report 2000-03. CREST, INSEE, Paris.

  4. Carlin B.P. and Chib S. 1995. Bayesian model choice via Markov chain Monte Carlo methods. Journal of the Royal Statistical Society B 57: 473–484.

    MATH  Google Scholar 

  5. Carpaneto G. and Toth P. 1980. Algorithm 548: Solution of the assignment problem [H]. ACM Transactions on Mathematical Software 6: 104–111.

    Article  Google Scholar 

  6. Celeux G., Hurn M., and Robert C.P. 2000. Computational and inferential difficulties with mixture posterior distributions. Journal of the American Statistical Association 95: 957–970.

    MATH  Article  MathSciNet  Google Scholar 

  7. Chib S. 1995. Marginal Likelihood from the Gibbs Output. Journal of the American Statistical Association 90: 1313–1321.

    MATH  Article  MathSciNet  Google Scholar 

  8. Dellaportas P. and Papageorgiou I. 2006. Multivariate mixtures of normals with unknown number of components. Statistics and Computing 16: 57–68.

    Article  MathSciNet  Google Scholar 

  9. Dempster A.P., Laird N.M., and Rubin D.B. 1977. Maximum likelihood estimation from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society B 39: 1–38.

    MATH  MathSciNet  Google Scholar 

  10. Diebolt J. and Robert C.P. 1994. Estimation of finite mixture distributions through Bayesian sampling. Journal of the Royal Statistical Society B 56: 363–375.

    MATH  MathSciNet  Google Scholar 

  11. Fearnhead P. 2004. Particle filters for mixture models with an unknown number of components. Statistics and Computing 14: 11–21.

    Article  MathSciNet  Google Scholar 

  12. Frühwirth-Schnatter S. 2001. Markov Chain Monte Carlo Estimation of Classical and Dynamic Switching and Mixture Models. Journal of the American Statistical Association 96: 194–209.

    MATH  Article  MathSciNet  Google Scholar 

  13. Green P.J. 1995. Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82: 711–732.

    MATH  Article  MathSciNet  Google Scholar 

  14. Ishwaran H., James L.F., and Sun J. 2001. Bayesian model selection in finite mixtures by marginal density decompositions. Journal of the American Statistical Association 96: 1316–1332.

    MATH  Article  MathSciNet  Google Scholar 

  15. Jain S. and Neal R.M. 2004. A split-merge Markov Chain Monte Carlo procedure for the Dirichlet process mixture model. Journal of Computational and Graphical Statistics 13: 158–182.

    Article  MathSciNet  Google Scholar 

  16. Jasra A., Holmes C.C., and Stephens D.A. 2005. Markov Chain Monte Carlo methods and the label switching problem in Bayesian mixture modeling. Statistical Science 20: 50–67.

    MATH  Article  MathSciNet  Google Scholar 

  17. Marin J.-M., Mengersen K., and Robert C.P. 2005. Bayesian modelling and inference on mixtures of distributions. In: Dey D. and Rao C.R. (Eds.), Handbook of Statistics vol. 25, North-Holland.

  18. McLachlan G. and Peel D. 2000. Finite Mixture Models, John Wiley & Sons, New York.

    Google Scholar 

  19. Mengersen K.L. and Robert C.P. 1996. Testing for Mixtures: A Bayesian entropic approach. In: Bernardo J.M. Berger J.O., Dawid A.P. and Smith A.F.M. (Eds.), Bayesian Statistics vol. 5, Oxford University Press, pp. 255–276.

  20. Nobile A. 1994. Bayesian Analysis of Finite Mixture Distributions, Ph.D. dissertation, Department of Statistics, Carnegie Mellon University, Pittsburgh. Available at http://www.stats.gla.ac.uk/~agostino

  21. Nobile A. 2004. On the posterior distribution of the number of components in a finite mixture. The Annals of Statistics 32: 2044–2073.

    MATH  Article  MathSciNet  Google Scholar 

  22. Nobile A. 2005. Bayesian finite mixtures: a note on prior specification and posterior computation. Technical Report 05-3, Department of Statistics, University of Glasgow.

  23. Phillips D.B. and Smith A.F.M. 1996. Bayesian model comparison via jump diffusions. In: Gilks W.R., Richardson S. and Spiegelhalter D.J. (Eds.), Markov Chain Monte Carlo in Practice, Chapman & Hall, London, pp. 215–239.

    Google Scholar 

  24. Raftery A.E. 1996. Hypothesis testing and model selection. In: Gilks W.R., Richardson S., and Spiegelhalter D.J. (Eds.), Markov Chain Monte Carlo in Practice, Chapman & Hall, London, pp. 163–187.

    Google Scholar 

  25. Richardson S. and Green P.J. 1997. On Bayesian analysis of mixtures with an unknown number of components (with discussion). Journal of the Royal Statistical Society B 59: 731–792.

    MATH  Article  MathSciNet  Google Scholar 

  26. Roeder K. 1990. Density estimation with confidence sets exemplified by superclusters and voids in galaxies. Journal of the American Statistical Association 85: 617–624.

    MATH  Article  Google Scholar 

  27. Roeder K. and Wasserman L. 1997. Practical Bayesian density estimation using mixtures of normals. Journal of the American Statistical Association 92: 894–902.

    MATH  Article  MathSciNet  Google Scholar 

  28. Steele R.J., Raftery A.E., and Emond M.J. 2003. Computing normalizing constants for finite mixture models via incremental mixture importance sampling (IMIS). Tech Report 436, Dept of Statistics, U. of Washington.

  29. Stephens M. 2000a. Bayesian analysis of mixture models with an unknown number of components—an alternative to reversible jump methods. The Annals of Statistics 28: 40–74.

    MATH  Article  MathSciNet  Google Scholar 

  30. Stephens M. 2000b. Dealing with label switching in mixture models. Journal of the Royal Statistical Society B 62: 795–809.

    MATH  Article  MathSciNet  Google Scholar 

  31. Titterington D.M., Smith A.F.M., and Makov U.E. 1985. Statistical Analysis of Finite Mixture Distributions, John Wiley & Sons, New York.

    Google Scholar 

  32. Zhang Z., Chan K.L., Wu Y., and Chen C. 2004. Learning a multivariate Gaussian mixture model with the reversible jump MCMC algorithm. Statistics and Computing 14: 343–355.

    Article  MathSciNet  Google Scholar 

Download references

Author information

Affiliations

  1. Department of Statistics, University of Glasgow, Glasgow, G12 8QW, U.K

    Agostino Nobile & Alastair T. Fearnside

Authors
  1. Agostino Nobile
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alastair T. Fearnside
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Agostino Nobile.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Nobile, A., Fearnside, A.T. Bayesian finite mixtures with an unknown number of components: The allocation sampler. Stat Comput 17, 147–162 (2007). https://doi.org/10.1007/s11222-006-9014-7

Download citation

Keywords

  • Classification
  • Galaxy data
  • Iris data
  • Label switching
  • Markov chain Monte Carlo
  • Multivariate normal mixtures
  • Normal mixtures
  • Reversible jump