Split and Merge EM Algorithm for Improving Gaussian Mixture Density Estimates
- 286 Downloads
The EM algorithm for Gaussian mixture models often gets caught in local maxima of the likelihood which involve having too many Gaussians in one part of the space and too few in another, widely separated part of the space. We present a new EM algorithm which performs split and merge operations on the Gaussians to escape from these configurations. This algorithm uses two novel criteria for efficiently selecting the split and merge candidates. Experimental results on synthetic and real data show the effectiveness of using the split and merge operations to improve the likelihood of both the training data and of held-out test data.
KeywordsMixture Model Gaussian Mixture Model Deterministic Anneal Probabilistic Principal Component Analyzer Current Parameter Estimate
Unable to display preview. Download preview PDF.
- 1.G. MacLachlan and K. Basford, Mixture Models: Inference and Application to Clustering, Marcel Dekker, 1988.Google Scholar
- 2.N. Kambhatla and T.K. Leen, “Classifying with Gaussian Mixtures and Clusters,” in Advances in Neural Information Processing Systems 7, Cambridge MA: MIT Press, 1995, pp. 681–687.Google Scholar
- 3.D. Ormoneit and V. Tresp, “Improved Gaussian Mixture Density Estimates Using Bayesian Penalty Terms and Network Averaging,” in Advances in Neural Information Processing Systems 8, D.S. Touretzky, G. Tesauro and T.K. Leen (eds.), Cambridge MA: MIT Press, 1996, pp. 542–548.Google Scholar
- 4.L. Rabiner and Juang Biing-Hwang, Fundamentals of Speech Recognition, PTR Prentice-Hall, 1993.Google Scholar
- 6.N. Ueda and R. Nakano, “Deterministic AnnealingVariant of the EM Algorithm,” Advances in Neural Information Processing Systems 7, D.S. Touretzky, G. Tesauro and T.K. Leen (eds.), Cambridge MA: MIT Press, 1995, pp. 545–552.Google Scholar
- 9.M.E. Tipping and C.M. Bishop, “Mixtures of Probabilistic Principal Component Analysers,” Tech. Rep. NCRG-97-3, Aston Univ. Birmingham, UK, 1997.Google Scholar
- 10.Z. Ghahramani and G.E. Hinton, “The EM Algorithm for Mixtures of Factor Analyzers,” Tech. Report CRG-TR-96-1, Univ. of Toronto, 1997. http://www.gatsby.ucl.ac.uk/~zoubin/papers/tr-96-1.ps.gz.Google Scholar
- 11.N. Ueda, R. Nakano, Z. Ghahramani, and G.E. Hinton, “SMEM Algorithm for Mixture Models,” in Advances in Neural Information Processing Systems 11, M.S. Kearns, S.A. Solla and D.A. Cohn (eds.), Cambridge MA: MIT Press, 1999, pp. 599–605.Google Scholar
- 12.N. Ueda, R. Nakano, Z. Ghahramani, and G.E. Hinton, “SMEM Algorithm for Mixture Models,” Neural Computation, MIT Press, to appear.Google Scholar