Abstract
The EM algorithm for Gaussian mixture models often gets caught in local maxima of the likelihood which involve having too many Gaussians in one part of the space and too few in another, widely separated part of the space. We present a new EM algorithm which performs split and merge operations on the Gaussians to escape from these configurations. This algorithm uses two novel criteria for efficiently selecting the split and merge candidates. Experimental results on synthetic and real data show the effectiveness of using the split and merge operations to improve the likelihood of both the training data and of held-out test data.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
G. MacLachlan and K. Basford, Mixture Models: Inference and Application to Clustering, Marcel Dekker, 1988.
N. Kambhatla and T.K. Leen, “Classifying with Gaussian Mixtures and Clusters,” in Advances in Neural Information Processing Systems 7, Cambridge MA: MIT Press, 1995, pp. 681–687.
D. Ormoneit and V. Tresp, “Improved Gaussian Mixture Density Estimates Using Bayesian Penalty Terms and Network Averaging,” in Advances in Neural Information Processing Systems 8, D.S. Touretzky, G. Tesauro and T.K. Leen (eds.), Cambridge MA: MIT Press, 1996, pp. 542–548.
L. Rabiner and Juang Biing-Hwang, Fundamentals of Speech Recognition, PTR Prentice-Hall, 1993.
A.P. Dempster, N.M. Laird, and D.B. Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” Journal of Royal Statistical Society B, vol. 39, 1977, pp. 1–38.
N. Ueda and R. Nakano, “Deterministic AnnealingVariant of the EM Algorithm,” Advances in Neural Information Processing Systems 7, D.S. Touretzky, G. Tesauro and T.K. Leen (eds.), Cambridge MA: MIT Press, 1995, pp. 545–552.
N. Ueda and R. Nakano, “Deterministic Annealing EM Algorithm,” Neural Networks, vol. 11, 1998, pp. 271–282.
N. Ueda and R. Nakano, “A New Competitive Learning Approach Based on an Equidistortion Principle for Designing Optimal Vector Quantizers,” Neural Networks, vol. 7, no.8, 1994, pp. 1211–1227.
M.E. Tipping and C.M. Bishop, “Mixtures of Probabilistic Principal Component Analysers,” Tech. Rep. NCRG-97-3, Aston Univ. Birmingham, UK, 1997.
Z. Ghahramani and G.E. Hinton, “The EM Algorithm for Mixtures of Factor Analyzers,” Tech. Report CRG-TR-96-1, Univ. of Toronto, 1997. http://www.gatsby.ucl.ac.uk/~zoubin/papers/tr-96-1.ps.gz.
N. Ueda, R. Nakano, Z. Ghahramani, and G.E. Hinton, “SMEM Algorithm for Mixture Models,” in Advances in Neural Information Processing Systems 11, M.S. Kearns, S.A. Solla and D.A. Cohn (eds.), Cambridge MA: MIT Press, 1999, pp. 599–605.
N. Ueda, R. Nakano, Z. Ghahramani, and G.E. Hinton, “SMEM Algorithm for Mixture Models,” Neural Computation, MIT Press, to appear.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Ueda, N., Nakano, R., Ghahramani, Z. et al. Split and Merge EM Algorithm for Improving Gaussian Mixture Density Estimates. The Journal of VLSI Signal Processing-Systems for Signal, Image, and Video Technology 26, 133–140 (2000). https://doi.org/10.1023/A:1008155703044
Published:
Issue Date:
DOI: https://doi.org/10.1023/A:1008155703044