Advertisement

On the Minima of Bethe Free Energy in Gaussian Distributions

  • Yu Nishiyama
  • Sumio Watanabe
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5097)

Abstract

Belief propagation (BP) is effective for computing marginal probabilities of a high dimensional probability distribution. Loopy belief propagation (LBP) is known not to compute precise marginal probabilities and not to guarantee its convergence. The fixed points of LBP are known to accord with the extrema of Bethe free energy. Hence, the fixed points are analyzed by minimizing the Bethe free energy.

In this paper, we consider the Bethe free energy in Gaussian distributions and analytically clarify the extrema, equivalently, the fixed points of LBP for some particular cases. The analytical results tell us a necessary condition for LBP convergence and the quantities which determine the accuracy of LBP in Gaussian distributions. Based on the analytical results, we perform numerical experiments of LBP and compare the results with analytical solutions.

Keywords

Belief Propagation Marginal Probability Single Loop LDPC Code Neural Computation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Heskes, T.: On the uniqueness of loopy belief propagation fixed points. Neural Computation 16(11), 2379–2414 (2004)MATHCrossRefGoogle Scholar
  2. 2.
    Weiss, Y.: Belief propagation and revision in networks with loops. Technical report 1616, MIT AI lab (1997)Google Scholar
  3. 3.
    Weiss, Y.: Correctness of local probability propagation in graphical models with loops. Neural Computation 12(1), 1–41 (2000)MATHCrossRefGoogle Scholar
  4. 4.
    Weiss, Y., Freeman, W.: Correctness of belief propagation in gaussian graphical models of arbitrary topology. Neural Computation 13(10), 2173–2200 (2001)MATHCrossRefGoogle Scholar
  5. 5.
    Ikeda, S., Tanaka, T., Amari, S.: Stochastic reasoning, free energy, and information geometry. Neural Computation 16(9), 1779–1810 (2004)MATHCrossRefGoogle Scholar
  6. 6.
    Ikeda, S., Tanaka, T., Amari, S.: Information geometry of turbo and low-density parity-check codes. IEEE Trans. Inf. Theory 50(6), 1097–1114 (2004)CrossRefMathSciNetGoogle Scholar
  7. 7.
    Kabashima, Y., Saad, D.: The TAP approach to intensive and extensive connectivity systems. In: Opper, M., Saad, D. (eds.) Advanced Mean Field Methods -Theory and Practice, pp. 65–84. MIT Press, Cambridge (2001)Google Scholar
  8. 8.
    Yedidia, J., Freeman, W., Weiss, Y.: Bethe free energy, kikuchi approximations, and belief propagation algorithms. Technical Report TR2001-16, Mitsubishi Electric Research Laboratories (2001)Google Scholar
  9. 9.
    Tanaka, K., Shouno, H., Okada, M.: Accuracy of the bethe approximation for hyperparameter estimation in probabilistic image processing. J.Phys. A, Math. Gen. 37(36), 8675–8696 (2004)MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Tanaka, K., Inoue, J., Titterington, D.M.: Loopy belief propagation and probabilistic image processing. In: proceedings of 2003 IEEE International workshop on Neural Networks for Signal Processing, vol. 13, pp. 383–392 (2003)Google Scholar
  11. 11.
    Nishiyama, Y., Watanabe, S.: Theoretical Analysis of Accuracy of Belief Propagation in Gaussian Models. IEICE Technical Report 107(50), 23–28 (2007)Google Scholar
  12. 12.
    Nishiyama, Y., Watanabe, S.: Theoretical Analysis of Accuracy of Gaussian Belief Propagation. In: Proceedings of International Conference on Artificial Neural Networks 2007, pp. 29–38 (2007)Google Scholar
  13. 13.
    Yuille, A.L.: CCCP algorithms to minimize the Bethe and Kikuchi free energies: convergent alternatives to belief propagation. Neural Computation 14(7), 1691–1722 (2002)MATHCrossRefGoogle Scholar
  14. 14.
    Tanaka, K.: Generalized belief propagation formula in probabilistic information processing based on gaussian graphical model. IEICE D-II, Vol. J88-D-II, No. 12, pp. 2368–2379 (in Japanese) (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Yu Nishiyama
    • 1
  • Sumio Watanabe
    • 2
  1. 1.Department of Computational Intelligence and Systems ScienceTokyo Institute of TechnologyYokohamaJapan
  2. 2.Precision and Intelligence LaboratoryTokyo Institute of TechnologyYokohamaJapan

Personalised recommendations