An Entropy Maximization Approach to Optimal Model Selection in Gaussian Mixtures

  • Antonio Peñalver
  • Juan M. Sáez
  • Francisco Escolano
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2905)


In this paper we address the problem of estimating the parameters of a Gaussian mixture model. Although the EM (Expectation-Maximization) algorithm yields the maximum-likelihood solution it has many problems: (i) it requires a careful initialization of the parameters; (ii) the optimal number of kernels in the mixture may be unknown before-hand. We propose a criterion based on the entropy of the pdf (probability density function) associated to each kernel to measure the quality of a given mixture model, and a modification of the classical EM algorithm to find the optimal number of kernels in the mixture. We test this method with synthetic and real data and compare the results with those obtained with the classical EM with a fixed number of kernels.


Maximum Entropy Gaussian Mixture Model Entropy Maximization Approach Fuse Kernel Neural Processing Letter 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bishop, C.: Neural Networks for Pattern Recognition. Oxford Univ. Press, Oxford (1995)Google Scholar
  2. 2.
    Blake, C.L., Merz, C.J.: UCI repository of machine learning databases. University of California, Irvine, Dept. of Information and Computer Sciences (1998)Google Scholar
  3. 3.
    Cover, T., Thomas, J.: Elements of Information Theory. J. Wiley and Sons, Chichester (1991)zbMATHCrossRefGoogle Scholar
  4. 4.
    Dempster, A., Laird, N., Rubin, D.: Maximum Likelihood estimation from incomplete data via the EM Algorithm. Journal of The Royal Statistical Society B 39, 1–38 (1977)zbMATHMathSciNetGoogle Scholar
  5. 5.
    Figueiredo, M.A.T., Leitao, J.M.N., Jain, A.K.: On Fitting Mixture Models. In: Hancock, E.R., Pelillo, M. (eds.) EMMCVPR 1999. LNCS, vol. 1654, pp. 54–69. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  6. 6.
    Figueiredo, J.M.N., Jain, A.K.: Unsupervised Selection and Estimation of Finite Mixture Models. In: Proceedings of the International Conference on Pattern Recognition. ICPR2000, Barcelona (2000)Google Scholar
  7. 7.
    Parzen, E.: On estimation of a probability density function and mode. Annals of Mathematical Statistics 33, 1065–1076 (1962)zbMATHMathSciNetCrossRefGoogle Scholar
  8. 8.
    Redner, R.A., Walker, H.F.: Mixture Densities, Maximum Likelihood, and the EM Algorithm. SIAM Review 26(2), 195–239 (1984)zbMATHMathSciNetCrossRefGoogle Scholar
  9. 9.
    Rissanen, J.: Stochastic Complexity in Statistical Inquiry. World Scientific, Singapore (1989)zbMATHGoogle Scholar
  10. 10.
    Shannon, C.: A Mathematical Theory of Comunication. The Bell System Technical Journal 27, 379–423, 623-656 (1948)zbMATHMathSciNetGoogle Scholar
  11. 11.
    Ueda, N., Nakano, R., Ghahramani, Z., Hinton, G.E.: SMEM Algorithm for Mixture Models. Neural Computation 12, 2109–2128 (2000)CrossRefGoogle Scholar
  12. 12.
    Viola, P., Wells III, W.M.: Alignment by Maximization of Mutual Information. In: 5th Intern. Conf. on Computer Vision, Cambridge, MA, pp. 16–23. IEEE, Los Alamitos (1995)Google Scholar
  13. 13.
    Viola, P., Schraudolph, N.N., Sejnowski, T.J.: Empirical Entropy Manipulation for Real-World Problems. Adv. in Neural Infor. Proces. Systems 8. MIT Press (1996)Google Scholar
  14. 14.
    Vlassis, N., Likas, A.: A Kurtosis-Based Dynamic Approach to Gaussian Mixture Modeling. IEEE Trans. Systems, Man, and Cybernetics 29(4), 393–399 (1999)CrossRefGoogle Scholar
  15. 15.
    Vlassis, N., Likas, A., Krose, B.: A Multivariate Kurtosis-Based Dynamic Approach to Gaussian Mixture Modeling. Inteligent Autonomous Systems Tech. Report (2000)Google Scholar
  16. 16.
    Vlassis, N., Likas, A.: A Greedy EM Algorithm for Gaussian Mixture Learning. Neural Processing Letters (to appear)Google Scholar
  17. 17.
    Wolpert, D., Wolf, D.: Estimating Function of Probability Distribution from a Finite Set of Samples. Phisical Review E 52(6) (1995)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Antonio Peñalver
    • 1
  • Juan M. Sáez
    • 1
  • Francisco Escolano
    • 1
  1. 1.Robot Vision Group Departamento de Ciencia de la Computación e Inteligencia ArtificialUniversidad de AlicanteSpain

Personalised recommendations