Abstract
Estimating the partition function is a key but difficult computation in graphical models. One approach is to estimate tractable upper and lower bounds. The piecewise upper bound of Sutton et al. is computed by breaking the graphical model into pieces and approximating the partition function as a product of local normalizing factors for these pieces. The tree reweighted belief propagation algorithm (TRW-BP) by Wainwright et al. gives tighter upper bounds. It optimizes an upper bound expressed in terms of convex combinations of spanning trees of the graph. Recently, Globerson et al. gave a different, convergent iterative dual optimization algorithm TRW-GP for the TRW objective. However, in many practical applications, particularly those that train CRFs with many nodes, TRW-BP and TRW-GP are too slow to be practical. Without changing the algorithm, we prove that TRW-BP converges in a single iteration for associative potentials, and give a closed form for the solution it finds. The closed-form solution obviates the need for complex optimization. We use this result to develop new closed-form upper bounds for MRFs with arbitrary pairwise potentials. Being closed-form, they are much faster to compute than TRW-based bounds. We also prove similar convergence results for loopy belief propagation (LBP) and use it to obtain closed-form solutions to the LBP pseudomarginals and approximation to the partition function for associative potentials. We then use recent results proved by Wainwright et al for binary MRFs to obtain closed-form lower bounds on the partition function. We then develop novel lower bounds for arbitrary associative networks. We report on experiments with synthetic and real-world graphs. Our new upper bounds are considerably tighter than the piecewise bounds in practice. Moreover, we can compute our bounds on several graphs where TRW-BP does not converge. Our novel lower bound, in spite of being closed-form and much faster to compute, outperforms more complicated popular algorithms for computing lower bounds like mean-field on densely connected graphs by wide margins although it does worse on sparsely connected graphs like chains.
Article PDF
Similar content being viewed by others
References
Borwein, J., & Lewis, A. S. (2006). Convex analysis and nonlinear optimization. Berlin: Springer.
Boyd, S., & Vandenberghe, L. (2004). Convex optimization. Cambridge: Cambridge University Press.
Globerson, A., & Jaakkola, T. (2007). Convergent propagation algorithms via oriented trees. In Proceedings of the twenty-second conference on uncertainty in AI (UAI), Vancouver, Canada, July 2007.
Jordan, M., & Wainwright, M. (2003). Graphical models, exponential families and variational inference (Technical Report 649). Department of Statistics, U.C. Berkeley.
Lange, K. (2004). Optimization. Berlin: Springer.
Lu, Q., & Getoor, L. (2003). Link-based classification. In ICML (pp. 496–503).
Ravikumar, P., & Lafferty, J. (2004). Variational Chernoff bounds for graphical models. In UAI conference.
Rennie, J. D. M. (2005). A class of convex functions. http://people.csail.mit.edu/jrennie/writing, May 2005.
Sudderth, E., Wainwright, M., & Willsky, A. (2008). Loop series and Bethe variational bounds in attractive graphical models. In J. Platt, D. Koller, Y. Singer, & S. Roweis (Eds.), Advances in neural information processing systems 20 (pp. 1425–1432). Cambridge: MIT Press.
Sutton, C., & McCallum, D. (2005). Piecewise training for undirected models. In Proceedings of the twenty-second conference on uncertainty in AI (UAI), Toronto, Canada, July 2005.
Wainwright, M. J. (2002). Stochastic processes on graphs with cycles: geometric and variational approaches. PhD thesis, Massachusetts Institute of Technology, Supervisors A. S. Willsky and T. S. Jaakkola.
Wainwright, M. J., Jaakkola, T. S., & Willsky, A. S. (2005). A new class of upper bounds on the log partition function. IEEE Transactions on Information Theory, 51(7), 2313–2335.
Yedidia, J. S., Freeman, W. T., & Weiss, Y. I. (2000). Generalized belief propagation. In NIPS (pp. 689–695).
Author information
Authors and Affiliations
Corresponding author
Additional information
Editors: Walter Daelemans, Bart Goethals, Katharina Morik.
Rights and permissions
About this article
Cite this article
Krishnamurthy, D., Chakrabarti, S. & Chaudhuri, S. New closed-form bounds on the partition function. Mach Learn 72, 205–229 (2008). https://doi.org/10.1007/s10994-008-5072-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10994-008-5072-8