Belief Propagation on Partially Ordered Sets

  • Robert J. McEliece
  • Muhammed Yildirim
Part of the The IMA Volumes in Mathematics and its Applications book series (IMA, volume 134)


In this paper, which is based on the important recent work of Yedidia, Freeman, and Weiss, we present a generalized form of belief propagation, viz. belief propagation on a partially ordered set (PBP). PBP is an iterative message-passing algorithm for solving, either exactly or approximately, the marginalized product density problem, which is a general computational problem of wide applicability. We will show that PBP can be thought of as an algorithm for minimizing a certain “free energy” function, and by exploiting this interpretation, we will exhibit a one-to-one correspondence between the fixed points of PBP and the stationary points of the free energy.


Free Energy Belief Propagation Helmholtz Free Energy Hasse Diagram Cluster Variation Method 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    S.M. Aji and R.J. McEliece, “The generalized distributive law,” IEEE Trans. Inform. Theory, 46(2): 325–343 (March 2000).MathSciNetzbMATHCrossRefGoogle Scholar
  2. [2]
    S.M. Aji and R.J. McEliece, “The generalized distributive law and free energy minimization,” Proc. 2001 Allerton Conf. Comm. Control and Computing (Oct. 2001).Google Scholar
  3. [3]
    T.M. Cover and J. A. Thomas, Elements of Information Theory. New York: John Wiley and Sons, 1991.zbMATHCrossRefGoogle Scholar
  4. [4]
    W.H. Fleming, Functions of Several Variables. Reading, Mass.: Addison-Wesley, 1965.zbMATHGoogle Scholar
  5. [5]
    R.A. Horn and C.R. Johnson, Matrix Analysis. Cambridge: Cambridge University Press, 1985.zbMATHGoogle Scholar
  6. [6]
    F.R. Kschichang, B.J. Frey, and H.-A. Loeliger, “Factor graphs and the sum-product algorithm,” IEEE Trans. Inform. Theory, 47(2): 498–519 (Feb. 2001).MathSciNetCrossRefGoogle Scholar
  7. [7]
    P. Pakzad and V. Anantharam, “Belief propagation and statistical physics,” Proc. 2002 Conf. Inform. Sciences and Systems, Princeton U., March 2002.Google Scholar
  8. [8]
    J. Pearl, Probabilistic Reasoning in Intelligent Systems. San Francisco: Morgan Kaufmann, 1988.Google Scholar
  9. [9]
    G.R. Shafer and P.P. Shenoy, “Probability propagation,” Ann. Mat. Art. Intell., 2: 327–352 (1990).MathSciNetzbMATHCrossRefGoogle Scholar
  10. [10]
    R.P. Stanley, Enumerative Combinatorics, Vol. 1. (Cambridge Studies in Advanced Mathematics 49) Cambridge: Cambridge University Press, 1997.zbMATHGoogle Scholar
  11. [11]
    J.S. Yedidia, “An idiosyncratic journey beyond mean field theory,” pp. 21–35 in Advanced Mean Field Methods, Theory and Practice, eds. Manfred Opper and David Saad, MIT Press, 2001.Google Scholar
  12. [12]
    J.S. Yedidia, W.T. Freeman, and Y. Weiss, “Generalized belief propagation,” pp. 689–695 in Advances in Neural Information Processing Systems 13 (2000) eds. Todd K. Leen, Thomas G. Dietterich, and Volker Tresp.Google Scholar
  13. [13]
    J.S. Yedidia, W.T. Freeman, and Y. Weiss, “Bethe free energy, Kikuchi approximations, and belief propagation algorithms,” available at
  14. [14]
    J.S. Yedidia, W.T. Freeman, and Y. Weiss, “Constructing free energy approximations and generalized belief propagation algorithms,” available at
  15. [15]
    J.M. Yeomans, Statistical Mechanics of Phase Transitions. Oxford: Oxford University Press, 1992.Google Scholar

Copyright information

© Springer-Verlag New York, Inc. 2003

Authors and Affiliations

  • Robert J. McEliece
  • Muhammed Yildirim
    • 1
  1. 1.Department of Electrical EngineeringCalifornia Institute of TechnologyPasadenaUSA

Personalised recommendations