Mixtures of Kikuchi Approximations

  • Roberto Santana
  • Pedro Larrañaga
  • Jose A. Lozano
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4212)


Mixtures of distributions concern modeling a probability distribution by a weighted sum of other distributions. Kikuchi approximations of probability distributions follow an approach to approximate the free energy of statistical systems. In this paper, we introduce the mixture of Kikuchi approximations as a probability model. We present an algorithm for learning Kikuchi approximations from data based on the expectation-maximization (EM) paradigm. The proposal is tested in the approximation of probability distributions that arise in evolutionary computation.


Mixture of distributions Kikuchi approximations estimation of distribution algorithms EM 


  1. 1.
    Bishop, C.M., Lawrence, N., Jaakkola, T., Jordan, M.I.: Approximating posterior distributions in belief networks using mixtures. In: Proc. Conf. Advances in Neural Information Processing Systems 10, NIPS, pp. 416–422. MIT Press, Cambridge (1998)Google Scholar
  2. 2.
    Bosman, P.A., Thierens, D.: Multi-objective optimization with diversity preserving mixture-based iterated density estimation evolutionary algorithms. International Journal of Approximate Reasoning 31(3), 259–289 (2002)zbMATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Cook, S.A.: The complexity of theorem-proving procedures. In: Proceedings of the Third Annual ACM Symposium on Theory of Computing, pp. 151–158, Shaker Heights, Ohio (1971)Google Scholar
  4. 4.
    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society B(39), 1–38 (1977)MathSciNetGoogle Scholar
  5. 5.
    Jakulin, A., Rish, I., Bratko, I.: Kikuchi-Bayes: Factorized models for approximate classification in closed form. Technical Report RC23314 (WO408-175), IBM (August 2004)Google Scholar
  6. 6.
    Kikuchi, R.: A theory of cooperative phenomena. Physical Review 81(6), 988–1003 (1951)zbMATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    Larrañaga, P., Lozano, J.A. (eds.): Estimation of Distribution Algorithms. A New Tool for Evolutionary Computation. Kluwer Academic Publishers, Boston/Dordrecht/London (2002)zbMATHGoogle Scholar
  8. 8.
    McLachlan, G., Peel, D.: Finite Mixture Models. John Wiley & Sons, Chichester (2000)zbMATHCrossRefGoogle Scholar
  9. 9.
    Meila, M., Jordan, M.I.: Learning with mixtures of trees. Journal of Machine Learning Research 1, 1–48 (2000)CrossRefMathSciNetGoogle Scholar
  10. 10.
    Mühlenbein, H., Paaß, G.: From recombination of genes to the estimation of distributions I. Binary parameters. In: Ebeling, W., Rechenberg, I., Voigt, H.-M., Schwefel, H.-P. (eds.) PPSN 1996. LNCS, vol. 1141, pp. 178–187. Springer, Heidelberg (1996)CrossRefGoogle Scholar
  11. 11.
    Peña, J., Lozano, J.A., Larrañaga, P.: Globally multimodal problem optimization via an estimation of distribution algorithm based on unsupervised learning of Bayesian networks. Evolutionary Computation 13(1), 43–66 (2005)CrossRefGoogle Scholar
  12. 12.
    Santana, R.: A Markov network based factorized distribution algorithm for optimization. In: Lavrač, N., Gamberger, D., Todorovski, L., Blockeel, H. (eds.) ECML 2003. LNCS, vol. 2837, pp. 337–348. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  13. 13.
    Santana, R.: Estimation of distribution algorithms with Kikuchi approximations. Evolutionary Computation 13(1), 67–97 (2005)CrossRefGoogle Scholar
  14. 14.
    Santana, R., Larrañaga, P., Lozano, J.A.: Properties of Kikuchi approximations constructed from clique based decompositions. Technical Report EHU-KZAA-IK-2/05, Department of Computer Science and Artificial Intelligence, University of the Basque Country (April 2005), available from:
  15. 15.
    Santana, R., Ochoa, A., Soto, M.R.: The mixture of trees factorized distribution algorithm. In: Proceedings of the Genetic and Evolutionary Computation Conference GECCO 2001, pp. 543–550. Morgan Kaufmann Publishers, San Francisco (2001)Google Scholar
  16. 16.
    Yedidia, J.S., Freeman, W.T., Weiss, Y.: Constructing free energy approximations and generalized belief propagation algorithms. Technical Report TR-2004-040, Mitsubishi Electric Research Laboratories (May 2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Roberto Santana
    • 1
  • Pedro Larrañaga
    • 1
  • Jose A. Lozano
    • 1
  1. 1.Intelligent System Group, Department of Computer Science and Artificial IntelligenceUniversity of the Basque CountrySan SebastiánSpain

Personalised recommendations