A Maximum Entropy Approach to Sampling in EDA – The Single Connected Case

  • Alberto Ochoa
  • Robin Höns
  • Marta Soto
  • Heinz Mühlenbein
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2905)


The success of evolutionary algorithms, in particular Factorized Distribution Algorithms (FDA), for many pattern recognition tasks heavily depends on our ability to reduce the number of function evaluations.

This paper introduces a method to reduce the population size overhead. We use low order marginals during the learning step and then compute the maximum entropy joint distributions for the cliques of the graph. The maximum entropy distribution is computed by an Iterative Proportional Fitting embedded in a junction tree message passing scheme to ensure consistency.

We show for the class of single connected FDA that our method outperforms the commonly-used PLS sampling.


Bayesian Network Maximum Entropy Maximum Entropy Principle Junction Tree Learning Step 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Chow, C.K., Liu, C.N.: Approximating discrete probability distributions with dependence trees. IEEE Trans. on Information Theory IT14(3), 462–467 (1968)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, New York (1989)Google Scholar
  3. 3.
    Csiszár, I.: I-divergence geometry of probability distributions and minimization problems. Annals of Probability 3, 146–158 (1975)zbMATHCrossRefGoogle Scholar
  4. 4.
    Goldberg, D.E.: Genetic Algorithms in search, optimization, and machine learning. Addison-Wesley, Reading (1989)zbMATHGoogle Scholar
  5. 5.
    Grünwald, P.: The Minimum Description Length Principle and Reasoning under Uncertainty. PhD thesis, University of Amsterdam (1998)Google Scholar
  6. 6.
    Henrion, M.: Propagating uncertainty in Bayesian networks by probabilistic logic sampling. Uncertainty in Artificial Intelligence 2, 317–324 (1988)Google Scholar
  7. 7.
    Jaynes, E.T.: Information theory and statistical mechanics. Phys. Rev 6, 620–643 (1957)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Jaynes, E.T.: Where do we stand on maximum entropy? In: Levine, R.D., Tribus, M. (eds.) The Maximum Entropy Formalism, MIT Press, Camb. (1978)Google Scholar
  9. 9.
    Jensen, F.V., Jensen, F.: Optimal junction trees. In: Proc. of the 10th Conference on Uncertainty in Artificial Intelligence, Seattle, Wash., pp. 360–366 (1994)Google Scholar
  10. 10.
    Jiroušek, R., Přeučil, S.: On the effective implementation of the iterative proportional fitting procedure. Comput. Statistics & Data Analysis 19, 177–189 (1995)zbMATHCrossRefGoogle Scholar
  11. 11.
    Lauritzen, S.L.: Graphical Models. Clarendon Press, Oxford (1996)Google Scholar
  12. 12.
    Meyer, C.-H.: Korrektes Schließen bei unvollständiger Information. PhD thesis, Fernuniversität Hagen (1998)Google Scholar
  13. 13.
    Mühlenbein, H., Mahnig, T., Ochoa, A.: Schemata, distributions and graphical models in evolutionary optimization. Journal of Heuristics 5(2), 213–247 (1999)CrossRefGoogle Scholar
  14. 14.
    Mühlenbein, H., Paaß, G.: From recombination of genes to the estimation of distributions i. binary parameters. In: Ebeling, W., Rechenberg, I., Voigt, H.-M., Schwefel, H.-P. (eds.) PPSN 1996. LNCS, vol. 1141, pp. 178–187. Springer, Heidelberg (1996)CrossRefGoogle Scholar
  15. 15.
    Pearl, J.: Probabilistic reasoning in intelligent systems: Networks of plausible inference. Morgan Kaufmann, San Mateo (1988)Google Scholar
  16. 16.
    Soto, M.: A Singled Connected Factorized Distribution Algorithm and its cost of evaluation. PhD thesis, University of Havana, Havana, Cuba (July 2003) (In Spanish, adviser A. Ochoa)Google Scholar
  17. 17.
    Soto, M., Ochoa, A.: A Factorized Distribution Algorithm based on polytrees. In: Proceedings of the 2000 Congress on Evolutionary Computation CEC 2000, La Jolla, California, July 6-9, pp. 232–237. IEEE Press, Los Alamitos (2000)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Alberto Ochoa
    • 1
    • 2
  • Robin Höns
    • 1
  • Marta Soto
    • 2
  • Heinz Mühlenbein
    • 1
  1. 1.Fraunhofer Institute for Autonomous Intelligent SystemsSankt AugustinGermany
  2. 2.Institute of CyberneticsMathematics and PhysicsHavanaCuba

Personalised recommendations