Machine Learning

, Volume 72, Issue 3, pp 205–229 | Cite as

New closed-form bounds on the partition function

  • Dvijotham Krishnamurthy
  • Soumen Chakrabarti
  • Subhasis Chaudhuri
Article
  • 548 Downloads

Abstract

Estimating the partition function is a key but difficult computation in graphical models. One approach is to estimate tractable upper and lower bounds. The piecewise upper bound of Sutton et al. is computed by breaking the graphical model into pieces and approximating the partition function as a product of local normalizing factors for these pieces. The tree reweighted belief propagation algorithm (TRW-BP) by Wainwright et al. gives tighter upper bounds. It optimizes an upper bound expressed in terms of convex combinations of spanning trees of the graph. Recently, Globerson et al. gave a different, convergent iterative dual optimization algorithm TRW-GP for the TRW objective. However, in many practical applications, particularly those that train CRFs with many nodes, TRW-BP and TRW-GP are too slow to be practical. Without changing the algorithm, we prove that TRW-BP converges in a single iteration for associative potentials, and give a closed form for the solution it finds. The closed-form solution obviates the need for complex optimization. We use this result to develop new closed-form upper bounds for MRFs with arbitrary pairwise potentials. Being closed-form, they are much faster to compute than TRW-based bounds. We also prove similar convergence results for loopy belief propagation (LBP) and use it to obtain closed-form solutions to the LBP pseudomarginals and approximation to the partition function for associative potentials. We then use recent results proved by Wainwright et al for binary MRFs to obtain closed-form lower bounds on the partition function. We then develop novel lower bounds for arbitrary associative networks. We report on experiments with synthetic and real-world graphs. Our new upper bounds are considerably tighter than the piecewise bounds in practice. Moreover, we can compute our bounds on several graphs where TRW-BP does not converge. Our novel lower bound, in spite of being closed-form and much faster to compute, outperforms more complicated popular algorithms for computing lower bounds like mean-field on densely connected graphs by wide margins although it does worse on sparsely connected graphs like chains.

Keywords

Partition function Graphical model Associative potential Approximate inference Belief propagation Variational methods 

References

  1. Borwein, J., & Lewis, A. S. (2006). Convex analysis and nonlinear optimization. Berlin: Springer. MATHGoogle Scholar
  2. Boyd, S., & Vandenberghe, L. (2004). Convex optimization. Cambridge: Cambridge University Press. MATHGoogle Scholar
  3. Globerson, A., & Jaakkola, T. (2007). Convergent propagation algorithms via oriented trees. In Proceedings of the twenty-second conference on uncertainty in AI (UAI), Vancouver, Canada, July 2007. Google Scholar
  4. Jordan, M., & Wainwright, M. (2003). Graphical models, exponential families and variational inference (Technical Report 649). Department of Statistics, U.C. Berkeley. Google Scholar
  5. Lange, K. (2004). Optimization. Berlin: Springer. MATHGoogle Scholar
  6. Lu, Q., & Getoor, L. (2003). Link-based classification. In ICML (pp. 496–503). Google Scholar
  7. Ravikumar, P., & Lafferty, J. (2004). Variational Chernoff bounds for graphical models. In UAI conference. Google Scholar
  8. Rennie, J. D. M. (2005). A class of convex functions. http://people.csail.mit.edu/jrennie/writing, May 2005.
  9. Sudderth, E., Wainwright, M., & Willsky, A. (2008). Loop series and Bethe variational bounds in attractive graphical models. In J. Platt, D. Koller, Y. Singer, & S. Roweis (Eds.), Advances in neural information processing systems 20 (pp. 1425–1432). Cambridge: MIT Press. Google Scholar
  10. Sutton, C., & McCallum, D. (2005). Piecewise training for undirected models. In Proceedings of the twenty-second conference on uncertainty in AI (UAI), Toronto, Canada, July 2005. Google Scholar
  11. Wainwright, M. J. (2002). Stochastic processes on graphs with cycles: geometric and variational approaches. PhD thesis, Massachusetts Institute of Technology, Supervisors A. S. Willsky and T. S. Jaakkola. Google Scholar
  12. Wainwright, M. J., Jaakkola, T. S., & Willsky, A. S. (2005). A new class of upper bounds on the log partition function. IEEE Transactions on Information Theory, 51(7), 2313–2335. CrossRefMathSciNetGoogle Scholar
  13. Yedidia, J. S., Freeman, W. T., & Weiss, Y. I. (2000). Generalized belief propagation. In NIPS (pp. 689–695). Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Dvijotham Krishnamurthy
    • 1
  • Soumen Chakrabarti
    • 1
  • Subhasis Chaudhuri
    • 1
  1. 1.IIT BombayMumbaiIndia

Personalised recommendations