Relax, Compensate and Then Recover

  • Arthur Choi
  • Adnan Darwiche
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6797)


We present in this paper a framework of approximate probabilistic inference which is based on three simple concepts. First, our notion of an approximation is based on “relaxing” equality constraints, for the purposes of simplifying a problem so that it can be solved more readily. Second, is the concept of “compensation,” which calls for imposing weaker notions of equality to compensate for the relaxed equality constraints. Third, is the notion of “recovery,” where some of the relaxed equality constraints are incrementally recovered, based on an assessment of their impact on improving the quality of an approximation. We discuss how this framework subsumes one of the most influential algorithms in probabilistic inference: loopy belief propagation and some of its generalizations. We also introduce a new heuristic recovery method that was key to a system that successfully participated in a recent evaluation of approximate inference systems, held in UAI’10. We further discuss the relationship between this framework for approximate inference and an approach to exact inference based on symbolic reasoning.


Bayesian Network Equivalence Constraint Conjunctive Normal Form Factor Graph Probabilistic Inference 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aji, S.M., McEliece, R.J.: The generalized distributive law and free energy minimization. In: Proceedings of the 39th Allerton Conference on Communication, Control and Computing, pp. 672–681 (2001)Google Scholar
  2. 2.
    Bilmes, J.: Results from the evaluation of probabilistic inference systems at UAI 2006 (2006),
  3. 3.
    Chavira, M., Darwiche, A.: On probabilistic inference by weighted model counting. Artificial Intelligence 172(6-7), 772–799 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Chavira, M., Darwiche, A., Jaeger, M.: Compiling relational Bayesian networks for exact inference. International Journal of Approximate Reasoning 42(1-2), 4–20 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Choi, A., Chan, H., Darwiche, A.: On Bayesian network approximation by edge deletion. In: Proceedings of the 21st Conference on Uncertainty in Artificial Intelligence (UAI), pp. 128–135. Arlington, Virginia (2005)Google Scholar
  6. 6.
    Choi, A., Chavira, M., Darwiche, A.: Node splitting: A scheme for generating upper bounds in Bayesian networks. In: Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence (UAI), pp. 57–66 (2007)Google Scholar
  7. 7.
    Choi, A., Darwiche, A.: An edge deletion semantics for belief propagation and its practical impact on approximation quality. In: Proceedings of the 21st National Conference on Artificial Intelligence (AAAI), pp. 1107–1114 (2006)Google Scholar
  8. 8.
    Choi, A., Darwiche, A.: A variational approach for approximating Bayesian networks by edge deletion. In: Proceedings of the 22nd Conference on Uncertainty in Artificial Intelligence (UAI), pp. 80–89 (2006)Google Scholar
  9. 9.
    Choi, A., Darwiche, A.: Approximating the partition function by deleting and then correcting for model edges. In: Proceedings of the 24th Conference on Uncertainty in Artificial Intelligence (UAI), pp. 79–87 (2008)Google Scholar
  10. 10.
    Choi, A., Darwiche, A.: Focusing generalizations of belief propagation on targeted queries. In: Proceedings of the 23rd AAAI Conference on Artificial Intelligence (AAAI), pp. 1024–1030 (2008)Google Scholar
  11. 11.
    Choi, A., Darwiche, A.: Many-pairs mutual information for adding structure to belief propagation approximations. In: Proceedings of the 23rd AAAI Conference on Artificial Intelligence (AAAI), pp. 1031–1036 (2008)Google Scholar
  12. 12.
    Choi, A., Darwiche, A.: Relax then compensate: On max-product belief propagation and more. In: Proceedings of the Twenty-Third Annual Conference on Neural Information Processing Systems (NIPS), pp. 351–359 (2009)Google Scholar
  13. 13.
    Choi, A., Standley, T., Darwiche, A.: Approximating weighted Max-SAT problems by compensating for relaxations. In: Gent, I.P. (ed.) CP 2009. LNCS, vol. 5732, pp. 211–225. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  14. 14.
    Darwiche, A.: Decomposable negation normal form. Journal of the ACM 48(4), 608–647 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Darwiche, A.: Recursive conditioning. Artificial Intelligence 126(1-2), 5–41 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Darwiche, A.: A compiler for deterministic, decomposable negation normal form. In: Proceedings of the Eighteenth National Conference on Artificial Intelligence (AAAI), pp. 627–634. AAAI Press, Menlo Park (2002)Google Scholar
  17. 17.
    Darwiche, A.: A logical approach to factoring belief networks. In: Proceedings of KR, pp. 409–420 (2002)Google Scholar
  18. 18.
    Darwiche, A.: A differential approach to inference in Bayesian networks. Journal of the ACM 50(3), 280–305 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Darwiche, A., Dechter, R., Choi, A., Gogate, V., Otten, L.: Results from the probablistic inference evaluation of UAI 2008 (2008),
  20. 20.
    Darwiche, A., Marquis, P.: A knowledge compilation map. Journal of Artificial Intelligence Research 17, 229–264 (2002)MathSciNetzbMATHGoogle Scholar
  21. 21.
    Dechter, R.: Bucket elimination: A unifying framework for probabilistic inference. In: Proceedings of the 12th Conference on Uncertainty in Artificial Intelligence (UAI), pp. 211–219 (1996)Google Scholar
  22. 22.
    Dechter, R., Rish, I.: Mini-buckets: A general scheme for bounded inference. J. ACM 50(2), 107–153 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Elidan, G., Globerson, A.: Summary of the 2010 UAI approximate inference challenge (2010),
  24. 24.
    Elidan, G., McGraw, I., Koller, D.: Residual belief propagation: Informed scheduling for asynchronous message passing. In: Proceedings of the 22nd Conference in Uncertainty in Artificial Intelligence (2006)Google Scholar
  25. 25.
    Geiger, D., Meek, C., Wexler, Y.: A variational inference procedure allowing internal structure for overlapping clusters and deterministic constraints. J. Artif. Intell. Res. (JAIR) 27, 1–23 (2006)MathSciNetzbMATHGoogle Scholar
  26. 26.
    Jaakkola, T.: Tutorial on variational approximation methods. In: Saad, D., Opper, M. (eds.) Advanced Mean Field Methods, ch. 10, pp. 129–160. MIT Press, Cambridge (2001)Google Scholar
  27. 27.
    Jensen, F.V., Lauritzen, S., Olesen, K.: Bayesian updating in recursive graphical models by local computation. Computational Statistics Quarterly 4, 269–282 (1990)zbMATHGoogle Scholar
  28. 28.
    Jha, A., Suciu, D.: Knowledge compilation meets database theory: compiling queries to decision diagrams. In: Proceedings of the 14th International Conference on Database Theory, pp. 162–173 (2011)Google Scholar
  29. 29.
    Jordan, M.I., Ghahramani, Z., Jaakkola, T., Saul, L.K.: An introduction to variational methods for graphical models. Machine Learning 37(2), 183–233 (1999)CrossRefzbMATHGoogle Scholar
  30. 30.
    Lauritzen, S.L., Spiegelhalter, D.J.: Local computations with probabilities on graphical structures and their application to expert systems. Journal of Royal Statistics Society, Series B 50(2), 157–224 (1988)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Mateescu, R., Kask, K., Gogate, V., Dechter, R.: Join-graph propagation algorithms. J. Artif. Intell. Res. (JAIR) 37, 279–328 (2010)MathSciNetzbMATHGoogle Scholar
  32. 32.
    Mooij, J.M., Kappen, H.J.: Sufficient conditions for convergence of the sum-product algorithm. IEEE Transactions on Information Theory 53(12), 4422–4437 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Francisco (1988)zbMATHGoogle Scholar
  34. 34.
    Pipatsrisawat, K., Darwiche, A.: New compilation languages based on structured decomposability. In: Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (AAAI), pp. 517–522 (2008)Google Scholar
  35. 35.
    Pipatsrisawat, K., Darwiche, A.: Top-down algorithms for constructing structured DNNF: Theoretical and practical implications. In: Proceedings of the 19th European Conference on Artificial Intelligence, pp. 3–8 (2010)Google Scholar
  36. 36.
    Yedidia, J.S., Freeman, W.T., Weiss, Y.: Understanding belief propagation and its generalizations. In: Lakemeyer, G., Nebel, B. (eds.) Exploring Artificial Intelligence in the New Millennium, ch. 8, pp. 239–269. Morgan Kaufmann, San Francisco (2003)Google Scholar
  37. 37.
    Yuille, A.L.: CCCP algorithms to minimize the Bethe and Kikuchi free energies: Convergent alternatives to belief propagation. Neural Computation 14(7), 1691–1722 (2002)CrossRefzbMATHGoogle Scholar
  38. 38.
    Zhang, N.L., Poole, D.: Exploiting causal independence in Bayesian network inference. Journal of Artificial Intelligence Research 5, 301–328 (1996)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Arthur Choi
    • 1
  • Adnan Darwiche
    • 1
  1. 1.Computer Science DepartmentUniversity of CaliforniaLos AngelesUSA

Personalised recommendations