Advertisement

Message Scheduling Methods for Belief Propagation

  • Christian KnollEmail author
  • Michael Rath
  • Sebastian Tschiatschek
  • Franz Pernkopf
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9285)

Abstract

Approximate inference in large and densely connected graphical models is a challenging but highly relevant problem. Belief propagation, as a method for performing approximate inference in loopy graphs, has shown empirical success in many applications. However, convergence of belief propagation can only be guaranteed for simple graphs. Whether belief propagation converges depends strongly on the applied message update scheme, and specialized schemes can be highly beneficial. Yet, residual belief propagation is the only established method utilizing this fact to improve convergence properties. In experiments, we observe that residual belief propagation fails to converge if local oscillations occur and the same sequence of messages is repeatedly updated. To overcome this issue, we propose two novel message update schemes. In the first scheme we add noise to oscillating messages. In the second scheme we apply weight decay to gradually reduce the influence of these messages and consequently enforce convergence. Furthermore, in contrast to previous work, we consider the correctness of the obtained marginals and observe significant performance improvements when applying the proposed message update schemes to various Ising models with binary random variables.

Keywords

Residual belief propagation Asynchronous message scheduling Convergence analysis 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Cooper, G.F.: The computational complexity of probabilistic inference using Bayesian belief networks. Artificial Intelligence 42(2), 393–405 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Elidan, G., McGraw, I., Koller, D.: Residual belief propagation: Informed scheduling for asynchronous message passing. In: Conference on Uncertainty in Artificial Intelligence (UAI) (2006)Google Scholar
  3. 3.
    Frey, B.J., MacKay, D.J.: A revolution: Belief propagation in graphs with cycles. In: Neural Information Processing Systems (NIPS), pp. 479–485 (1998)Google Scholar
  4. 4.
    Georgii, H.O.: Gibbs Measures and Phase Transitions, vol. 9 (2011)Google Scholar
  5. 5.
    Goldberger, J., Kfir, H.: Serial schedules for belief-propagation: analysis of convergence time. IEEE Transactions on Information Theory 54(3), 1316–1319 (2008)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Gómez, V., Mooij, J.M., Kappen, H.J.: Truncating the loop series expansion for belief propagation. The Journal of Machine Learning Research (2007)Google Scholar
  7. 7.
    Heskes, T.: On the uniqueness of loopy belief propagation fixed points. Neural Computation 16(11), 2379–2413 (2004)CrossRefzbMATHGoogle Scholar
  8. 8.
    Ising, E.: Beitrag zur Theorie des Ferromagnetismus. Zeitschrift für Physik A Hadrons and Nuclei 31(1), 253–258 (1925)zbMATHGoogle Scholar
  9. 9.
    Jordan, M.I.: Graphical models. Statistical Science, pp. 140–155 (2004)Google Scholar
  10. 10.
    Kfir, H., Kanter, I.: Parallel versus sequential updating for belief propagation decoding. Physica A: Statistical Mechanics and its Applications 330(1)Google Scholar
  11. 11.
    Koller, D., Friedman, N.: Probabilistic Graphical Models: Principles and Techniques. MIT press (2009)Google Scholar
  12. 12.
    Lauritzen, S.L., Spiegelhalter, D.J.: Local computations with probabilities on graphical structures and their application to expert systems. Journal of the Royal Statistical Society. Series B (Methodological), 157–224 (1988)Google Scholar
  13. 13.
    Meshi, O., Jaimovich, A., Globerson, A., Friedman, N.: Convexifying the bethe free energy. In: Conference on Uncertainty in Artificial Intelligence (UAI), pp. 402–410. AUAI Press (2009)Google Scholar
  14. 14.
    Mezard, M., Montanari, A.: Information, Physics, and Computation. Oxford University Press (2009)Google Scholar
  15. 15.
    Mézard, M., Parisi, G.: The Bethe lattice spin glass revisited. The European Physical Journal B-Condensed Matter and Complex Systems 20(2), 217–233 (2001)CrossRefGoogle Scholar
  16. 16.
    Mooij, J.M.: libdai: A free and open source c++ library for discrete approximate inference in graphical models. The Journal of Machine Learning Research 11 (2010)Google Scholar
  17. 17.
    Mooij, J.M., Kappen, H.J.: Loop corrections for approximate inference on factor graphs. Journal of Machine Learning Research 8, 1113–1143 (2007)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Mooij, J.M., Kappen, H.J.: Sufficient conditions for convergence of the sum-product algorithm. IEEE Transactions on Information Theory 53(12), 4422–4437 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Representation and Reasoning Series. Morgan Kaufmann Publishers (1988)Google Scholar
  20. 20.
    Pernkopf, F., Peharz, R., Tschiatschek, S.: Introduction to Probabilistic Graphical Models (2014)Google Scholar
  21. 21.
    Pretti, M.: A message-passing algorithm with damping. Journal of Statistical Mechanics: Theory and Experiment 2005(11), P11008 (2005)CrossRefGoogle Scholar
  22. 22.
    Sharon, E., Litsyn, S., Goldberger, J.: Efficient serial message-passing schedules for LDPC decoding. IEEE Transactions on Information Theory 53(11), 4076–4091 (2007)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Sontag, D., Choe, D.K., Li, Y.: Efficiently searching for frustrated cycles in MAP inference. In: Conference on Uncertainty in Artificial Intelligence (UAI) (2012)Google Scholar
  24. 24.
    Sutton, C.A., McCallum, A.: Improved dynamic schedules for belief propagation. In: Conference on Uncertainty in Artificial Intelligence (UAI) (2007)Google Scholar
  25. 25.
    Taga, N., Mase, S.: On the convergence of belief propagation algorithm for stochastic networks with loops. Citeseer (2004)Google Scholar
  26. 26.
    Tatikonda, S.C., Jordan, M.I.: Loopy belief propagation and Gibbs measures. In: Conference on Uncertainty in Artificial Intelligence (UAI) (2002)Google Scholar
  27. 27.
    Weiss, Y.: Correctness of local probability propagation in graphical models with loops. Neural Computation 12(1), 1–41 (2000)CrossRefzbMATHGoogle Scholar
  28. 28.
    Yedidia, J.S., Freeman, W.T., Weiss, Y.: Bethe free energy, Kikuchi approximations, and belief propagation algorithms. Neural Information Processing Systems (NIPS) 13 (2001)Google Scholar
  29. 29.
    Yedidia, J.S., Freeman, W.T., Weiss, Y.: Understanding belief propagation and its generalizations. Exploring Artificial Intelligence in the New Millennium 8, 239–269 (2003)zbMATHGoogle Scholar
  30. 30.
    Yuille, A.L.: CCCP algorithms to minimize the Bethe and Kikuchi free energies: Convergent alternatives to belief propagation. Neural Computation 14(7) (2002)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Christian Knoll
    • 1
    Email author
  • Michael Rath
    • 1
  • Sebastian Tschiatschek
    • 2
  • Franz Pernkopf
    • 1
  1. 1.Signal Processing and Speech Communication LaboratoryGraz University of TechnologyGrazAustria
  2. 2.Learning and Adaptive Systems Group, Department of Computer ScienceETH ZurichZürichSwitzerland

Personalised recommendations