Abstract
The EM algorithm is a sophisticated method for estimating statistical models with hidden variables based on the Kullback–Leibler divergence. A natural extension of the Kullback–Leibler divergence is given by a class of Bregman divergences, which in general enjoy robustness to contamination data in statistical inference. In this paper, a modification of the EM algorithm based on the Bregman divergence is proposed for estimating finite mixture models. The proposed algorithm is geometrically interpreted as a sequence of projections induced from the Bregman divergence. Since a rigorous algorithm includes a nonlinear optimization procedure, two simplification methods for reducing computational difficulty are also discussed from a geometrical viewpoint. Numerical experiments on a toy problem are carried out to confirm appropriateness of the simplifications.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Agresti A. (2002). Categorical data analysis. New York, Wiley-Interscience, pp. 538–575
Amari S. (1995). Information geometry of the EM and em algorithms for neural networks. Neural Networks 8(9): 1379–1408
Amari S., Nagaoka H. (2000). Methods of information geometry. Providence, RI: American Mathematical Society
Fujimoto, Y., Murata, N. (2006). Robust estimation for mixture of probability tables based on β-likelihood. In Proceedings of the sixth SIAM conference on data mining (pp. 519–523).
Fujisawa H., Eguchi S. (2005). Robust estimation in the normal mixture model. Journal of Statistical Planning and Inference 136, 3989–4011
Hampel F.R., Ronchetti E.M., Rousseeuw P.J., Stahel W.A. (1986). Robust statistics. New York, Wiley
Hofmann, T. (1999). Probabilistic latent semantic analysis. In Proceedings of uncertainty in artificial intelligence, UAI ’99 (pp. 289–296).
McLachlan G.J., Krishnan T. (1997). The EM algorithm and extensions. New York, John Wiley
Meng X.L., Rubin D.B. (1993). Maximum likelihood estimation via the ECM algorithm: a general framework. Biometrika 80(2): 267–278
Minami M., Eguchi S. (2002). Robust blind source separation by beta divergence. Neural Computation 14, 1859–1886
Murata N., Takenouchi T., Kanamori T., Eguchi S. (2004). Information geometry of U-Boost and Bregman divergence. Neural Computation 16(7): 1437–1481
R Development Core Team (2005). R: A language and environment for statistical computing. http://www.R-project.org. R Foundation for Statistical Computing.
Takenouchi T., Eguchi S. (2004). Robustifying AdaBoost by adding the naive error rate. Neural Computation 16(4): 767–787
Takenouchi, T. (2005). Robust boosting algorithm for multiclass classification by eta-divergence. In Proceedings of 2nd international symposium on information geometry and its applications (pp. 12–16).
Watanabe M., Yamaguchi K. (2004). EM algorithm in neural network learning. In: Murata N., Ikeda S.(eds) The EM algorithm and related statistical models. New York, Marcel Dekker, pp. 95–125
Author information
Authors and Affiliations
Corresponding author
About this article
Cite this article
Fujimoto, Y., Murata, N. A modified EM algorithm for mixture models based on Bregman divergence. AISM 59, 3–25 (2007). https://doi.org/10.1007/s10463-006-0097-x
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10463-006-0097-x