Regularized Gaussian belief propagation

  • Francois Kamper
  • Johan A. du Preez
  • Sarel J. Steel
  • Stephan Wagner
Article
  • 167 Downloads

Abstract

Belief propagation (BP) has been applied in a variety of inference problems as an approximation tool. BP does not necessarily converge in loopy graphs, and even if it does, is not guaranteed to provide exact inference. Even so, BP is useful in many applications due to its computational tractability. In this article, we investigate a regularized BP scheme by focusing on loopy Markov graphs (MGs) induced by a multivariate Gaussian distribution in canonical form. There is a rich literature surrounding BP on Gaussian MGs (labelled Gaussian belief propagation or GaBP), and this is known to experience the same problems as general BP on graphs. GaBP is known to provide the correct marginal means if it converges (this is not guaranteed), but it does not provide the exact marginal precisions. We show that our adjusted BP will always converge, with sufficient tuning, while maintaining the exact marginal means. As a further contribution we show, in an empirical study, that our GaBP variant can accelerate GaBP and compares well with other GaBP-type competitors in terms of convergence speed and accuracy of approximate marginal precisions. These improvements suggest that the principle of regularized BP should be investigated in other inference problems. The selection of the degree of regularization is addressed through the use of two heuristics. A by-product of GaBP is that it can be used to solve linear systems of equations; the same is true for our variant and we make an empirical comparison with the conjugate gradient method.

Keywords

Belief propagation Approximate inference Gaussian distributions Regularization Convergence 

References

  1. Aji, S., McEliece, R.: The generalized distributive law. IEEE Trans. Inform. Theory 46, 325–343 (2000)MathSciNetCrossRefMATHGoogle Scholar
  2. Bach, F., Jenatton, R., Mairal, J., Obozinski, G.: Convex optimization with sparsity-inducing norms. In: Sra, S., Nowozin, S., Wright, J. (eds.) Optimization for Machine Learning. MIT Press, Cambridge (2011)Google Scholar
  3. Bickson, D.: Gaussian Belief Propagation: Theory and Application, PhD thesis. The Hebrew University of Jerusalem (2008)Google Scholar
  4. Chandrasekaran, V., Johnson, J.K., Willsky, A.S.: Estimation in Gaussian graphical models using tractable subgraphs: a walk-sum analysis. IEEE Trans. Signal Process. 56, 1916–1930 (2008)MathSciNetCrossRefGoogle Scholar
  5. Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Annal. Stat. 32(2), 407–499 (2004)MathSciNetCrossRefMATHGoogle Scholar
  6. El-Kurdi, Y., Giannacopoulos, D., Gross, W.J.: Relaxed Gaussian belief propagation. In: Proceedings of the 2012 IEEE International Symposium on Information Theory (2012a)Google Scholar
  7. El-Kurdi, Y., Gross, W.J., Giannacopoulos, D.: Efficient implementation of Gaussian belief propagation solver for large sparse diagonally dominant linear systems. IEEE Trans. Magn. 48, 471–474 (2012b)CrossRefGoogle Scholar
  8. Frey, B., Kschischang, F.: Probability propagation and iterative decoding. In: Proceedings of the 34th Annual Allerton Conference on Communication, Control, and Computing, Allerton House, Monticello (1996)Google Scholar
  9. Gallager, R.G.: Low-Density Parity-Check Codes. MIT Press, Cambridge (1963)MATHGoogle Scholar
  10. Guo, Q., Huang, D.: EM-based joint channel estimation and detection for frequency selective channels using Gaussian message passing. IEEE Trans. Signal Process. 59, 4030–4035 (2011)MathSciNetCrossRefGoogle Scholar
  11. Guo, Q., Li, P.: LMMSE turbo equalization based on factor graphs. IEEE J. Sel. Areas Commun. 26, 311–319 (2008)CrossRefGoogle Scholar
  12. Johnson, J.K., Bickson, D., Dolev, D.: Fixing convergence of Gaussian belief propagation. In: International Symposium on Information Theory (ISIT), Seoul (2009)Google Scholar
  13. Koller, D., Friedman, N.: Probabilistic Graphical Models Principles and Techniques. MIT Press, Cambridge (2009)MATHGoogle Scholar
  14. Lauritzen, S,, Spiegelhalter, D. Local computations with probabilities on graphical structures and their application to expert systems. J. R. Stat. Soc. B 50,157–224 (1988)Google Scholar
  15. Liu, Y., Chandrasekaran, V., Anandkumar, A., Willsky, A.S.: Feedback message passing for inference in Gaussian graphical models. IEEE Trans. Signal Process. 60(8), 4135–4150 (2012)MathSciNetCrossRefGoogle Scholar
  16. Malioutov, D.M., Johnson, J.K., Willsky, A.S.: Walk-sums and belief propagation in gaussian graphical models. J. Mach. Learn. Res. 7, 2031–2064 (2006)MathSciNetMATHGoogle Scholar
  17. Montanari, A., Prabhakar, B., Tse, D.: Belief propagation based multi-user detection. In: IEEE Information Theory Workshop, Punta del Este, Uruguay (2006)Google Scholar
  18. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Francisco (1988)MATHGoogle Scholar
  19. Seeger, M.W., Wipf, D.P.: Variational Bayesian inference techniques. IEEE Signal Process. Mag. 27, 81–91 (2010)CrossRefGoogle Scholar
  20. Shachter, R.: Probabilistic inference and influence diagrams. Oper. Res. 36, 589–605 (1988)CrossRefMATHGoogle Scholar
  21. Shafer, G., Shenoy, P.: Probability propagation. Ann. Mat. Art. Intell. 2, 327–352 (1990)Google Scholar
  22. Shental, O., Siegel, P.H., Wolf. J.K., Bickson, D., Dolev, D.: Gaussian belief propagation solver for systems of linear equations. In: IEEE International Symposium on Informational Theory (ISIT), pp 1863–1867 (2008)Google Scholar
  23. Shewchuk, J.R.: An Introduction to the Conjugate Gradient Method Without the Agonizing Pain. School of Computer Science. Carnegie Mellon University, Pittsburgh, pp. 15213 (1994)Google Scholar
  24. Su, Q., Wu, Y.: On convergence conditions of Gaussian belief propagation. IEEE Int. Trans. Signal Process. 63, 1144–1155 (2015)MathSciNetCrossRefGoogle Scholar
  25. Weiss, Y.: Correctness of local probability in graphical models with loops. Neural Comput. 12, 1–41 (2000)MathSciNetCrossRefGoogle Scholar
  26. Weiss, Y., Freeman, W.T.: Correctness of belief propagation in Gaussian graphical models of arbitrary topology. Neural Comput. 13(10), 2173–2200 (2001)CrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Department of Statistics and Actuarial ScienceUniversity of StellenboschStellenboschSouth Africa
  2. 2.Department of Electrical and Electronic EngineeringUniversity of StellenboschStellenboschSouth Africa
  3. 3.Department of Mathematical SciencesUniversity of StellenboschStellenboschSouth Africa

Personalised recommendations