Generalization of Concave and Convex Decomposition in Kikuchi Free Energy

  • Yu Nishiyama
  • Sumio Watanabe
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5163)


Computing marginal probabilities from a high-dimensional distribution is generally an intractable task which arises in various probabilistic information processing. The minimum of Kikuchi free energy is known to yield more accurate marginals than that of Bethe free energy. Concave convex procedure (CCCP) proposed by Yuille is known to be an algorithm which guarantees to monotonically decrease both free energies.

In this paper, we generalize CCCP algorithm for Kikuchi free energy and present a new CCCP (NCCCP) algorithm which includes a high-dimensional free vector. The free vector enables NCCCP to be more stable and to reduce enormous computational cost underlying CCCP. Moreover it seems that there exists an optimal series of free vectors that can minimize the huge computational cost. This means that there exists an optimal concave and convex decomposition schedule in Kikuchi free energy.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Pearl, J.: Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, San Mateo (1988)Google Scholar
  2. 2.
    Murphy, K.P., Weiss, Y., Jordan, M.I.: Loopy Belief Propagation for Approximate Inference: An Empirical Study. In: 15th conference on Uncertainty in Artificial Intelligence, pp. 467–475 (1999)Google Scholar
  3. 3.
    McEliece, R.J., MacKay, D.J.C., Cheng, J.: Turbo Decoding as an Instance of Pearl’s ”Belief Propagation” Algorithm. IEEE Journal on Selected Areas in Communications 16(2), 140–152 (1998)CrossRefGoogle Scholar
  4. 4.
    MacKay, D.J.C.: Good Error-Correcting Codes Based on Very Sparse Matrices. IEEE Transactions on Information Theory 45(2), 399–431 (1999)MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Kabashima, Y.: A CDMA multiuser detection algorithm on the basis of belief propagation. J. Phys. A, Math. Gen. 36(43), 11111–11121 (2003)MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Tanaka, K.: Statistical-mechanical approach to image processing. J. Phys. A, Math. Gen. 35(37), R81–R150 (2002)CrossRefGoogle Scholar
  7. 7.
    Kabashima, Y., Saad, D.: The Belief in TAP. Neural Information Processing Systems 11 (1998)Google Scholar
  8. 8.
    Yedidia, J., Freeman, W., Weiss, Y.: Generalized Belief Propagation. Neural Information Processing Systems 13 (2000)Google Scholar
  9. 9.
    Welling, M., Teh, Y.W.: Belief Optimization for Binary Networks: A Stable Alternative to Loopy Belief Propagation. In: 17th conference on Uncertainty in Artificial Intelligence, pp. 554–561 (2001)Google Scholar
  10. 10.
    Tonosaki, Y., Kabashima, Y.: Sequential Minimization of the Bethe Free Energy of Ising Spin Systems. Interdisciplinary Information Sciences 13(1), 57–64 (2007)MATHMathSciNetGoogle Scholar
  11. 11.
    Kikuchi, R.: A Theory of Cooperative Phenomena. Phys. Rev. 81(6), 988–1003 (1951)MATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    Yuille, A.L.: CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation. Neural Computation 14(7), 1691–1722 (2002)MATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Yu Nishiyama
    • 1
  • Sumio Watanabe
    • 2
  1. 1.Department of Computational Intelligence and Systems Science, Tokyo Institute of Technology YokohamaJapan
  2. 2.Precision and Intelligence Laboratory, Tokyo Institute of Technology YokohamaJapan

Personalised recommendations