Advertisement

Split and Merge EM Algorithm for Improving Gaussian Mixture Density Estimates

  • Naonori Ueda
  • Ryohei Nakano
  • Zoubin Ghahramani
  • Geoffrey E. Hinton
Article

Abstract

The EM algorithm for Gaussian mixture models often gets caught in local maxima of the likelihood which involve having too many Gaussians in one part of the space and too few in another, widely separated part of the space. We present a new EM algorithm which performs split and merge operations on the Gaussians to escape from these configurations. This algorithm uses two novel criteria for efficiently selecting the split and merge candidates. Experimental results on synthetic and real data show the effectiveness of using the split and merge operations to improve the likelihood of both the training data and of held-out test data.

Keywords

Mixture Model Gaussian Mixture Model Deterministic Anneal Probabilistic Principal Component Analyzer Current Parameter Estimate 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    G. MacLachlan and K. Basford, Mixture Models: Inference and Application to Clustering, Marcel Dekker, 1988.Google Scholar
  2. 2.
    N. Kambhatla and T.K. Leen, “Classifying with Gaussian Mixtures and Clusters,” in Advances in Neural Information Processing Systems 7, Cambridge MA: MIT Press, 1995, pp. 681–687.Google Scholar
  3. 3.
    D. Ormoneit and V. Tresp, “Improved Gaussian Mixture Density Estimates Using Bayesian Penalty Terms and Network Averaging,” in Advances in Neural Information Processing Systems 8, D.S. Touretzky, G. Tesauro and T.K. Leen (eds.), Cambridge MA: MIT Press, 1996, pp. 542–548.Google Scholar
  4. 4.
    L. Rabiner and Juang Biing-Hwang, Fundamentals of Speech Recognition, PTR Prentice-Hall, 1993.Google Scholar
  5. 5.
    A.P. Dempster, N.M. Laird, and D.B. Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” Journal of Royal Statistical Society B, vol. 39, 1977, pp. 1–38.MathSciNetzbMATHGoogle Scholar
  6. 6.
    N. Ueda and R. Nakano, “Deterministic AnnealingVariant of the EM Algorithm,” Advances in Neural Information Processing Systems 7, D.S. Touretzky, G. Tesauro and T.K. Leen (eds.), Cambridge MA: MIT Press, 1995, pp. 545–552.Google Scholar
  7. 7.
    N. Ueda and R. Nakano, “Deterministic Annealing EM Algorithm,” Neural Networks, vol. 11, 1998, pp. 271–282.CrossRefGoogle Scholar
  8. 8.
    N. Ueda and R. Nakano, “A New Competitive Learning Approach Based on an Equidistortion Principle for Designing Optimal Vector Quantizers,” Neural Networks, vol. 7, no.8, 1994, pp. 1211–1227.CrossRefGoogle Scholar
  9. 9.
    M.E. Tipping and C.M. Bishop, “Mixtures of Probabilistic Principal Component Analysers,” Tech. Rep. NCRG-97-3, Aston Univ. Birmingham, UK, 1997.Google Scholar
  10. 10.
    Z. Ghahramani and G.E. Hinton, “The EM Algorithm for Mixtures of Factor Analyzers,” Tech. Report CRG-TR-96-1, Univ. of Toronto, 1997. http://www.gatsby.ucl.ac.uk/~zoubin/papers/tr-96-1.ps.gz.Google Scholar
  11. 11.
    N. Ueda, R. Nakano, Z. Ghahramani, and G.E. Hinton, “SMEM Algorithm for Mixture Models,” in Advances in Neural Information Processing Systems 11, M.S. Kearns, S.A. Solla and D.A. Cohn (eds.), Cambridge MA: MIT Press, 1999, pp. 599–605.Google Scholar
  12. 12.
    N. Ueda, R. Nakano, Z. Ghahramani, and G.E. Hinton, “SMEM Algorithm for Mixture Models,” Neural Computation, MIT Press, to appear.Google Scholar

Copyright information

© Kluwer Academic Publishers 2000

Authors and Affiliations

  • Naonori Ueda
    • 1
  • Ryohei Nakano
    • 1
  • Zoubin Ghahramani
    • 2
  • Geoffrey E. Hinton
    • 2
  1. 1.NTT Communication Science LaboratoriesKyotoJapan
  2. 2.Gatsby Computational Neuroscience UnitUniversity College LondonLondonUK

Personalised recommendations