A Comparative Study of Modern Inference Techniques for Structured Discrete Energy Minimization Problems


Szeliski et al. published an influential study in 2006 on energy minimization methods for Markov random fields. This study provided valuable insights in choosing the best optimization technique for certain classes of problems. While these insights remain generally useful today, the phenomenal success of random field models means that the kinds of inference problems that have to be solved changed significantly. Specifically, the models today often include higher order interactions, flexible connectivity structures, large label-spaces of different cardinalities, or learned energy tables. To reflect these changes, we provide a modernized and enlarged study. We present an empirical comparison of more than 27 state-of-the-art optimization techniques on a corpus of 2453 energy minimization instances from diverse applications in computer vision. To ensure reproducibility, we evaluate all methods in the OpenGM 2 framework and report extensive results regarding runtime and solution quality. Key insights from our study agree with the results of Szeliski et al. for the types of models they studied. However, on new and challenging types of models our findings disagree and suggest that polyhedral methods and integer programming solvers are competitive in terms of runtime and solution quality over a large range of model types.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11


  1. 1.

    The inpainting-N4/8 and color-seg-N4/8-models were originally used in variational approaches together with total variation regularizers (Lellmann and Schnörr 2011). A comparison with variational models is beyond the scope of this study.

  2. 2.

    Here we consider spanning trees as subproblems such that the relaxation is equivalent to the local polytope relaxation.

  3. 3.

    This includes terminal constraints TC, multi-terminal constraints MTC, cycle inequalities CC and facet defining cycle inequalities CCFDB as well as odd-wheel constraints OWC.

  4. 4.

    Due to the increased workload compared to the experiments in Kappes et al. (2013), we switch to a homogeneous cluster and no longer use the Intel Xeon W3550 3.07GHz CPU equipped with 12 GB RAM.


  1. Achterberg, T., Koch, T., & Martin, A. (2005). Branching rules revisited. Operations Research Letters, 33(1), 42–54.

    MATH  MathSciNet  Article  Google Scholar 

  2. Alahari, K., Kohli, P., & Torr, P. H. S. (2008). Reduce, reuse and recycle: Efficiently solving multi-label MRFs. In: CVPR.

  3. Alahari, K., Kohli, P., & Torr, P. H. S. (2010). Dynamic hybrid algorithms for MAP inference in discrete MRFs. IEEE PAMI, 32(10), 1846–1857.

    Article  Google Scholar 

  4. Andres, B., Beier, T., & Kappes, J. H. (2014). OpenGM2. http://hci.iwr.uni-heidelberg.de/opengm2/.

  5. Andres, B., Beier, T., & Kappes, J. H. (2012). OpenGM: A C++ library for discrete graphical models. ArXiv e-prints. http://arxiv.org/abs/1206.0111.

  6. Andres, B., Kappes, J. H., Beier, T., Köthe, U., & Hamprecht, F. A. (2011). Probabilistic image segmentation with closedness constraints. In ICCV.

  7. Andres, B., Kappes, J. H., Beier, T., Köthe, U., & Hamprecht, F. A. (2012). The lazy flipper: Efficient depth-limited exhaustive search in discrete graphical models. In ECCV.

  8. Andres, B., Kappes, J. H., Köthe, U., Schnörr, C., & Hamprecht, F. A. (2010). An empirical comparison of inference algorithms for graphical models with higher order factors using OpenGM. In DAGM.

  9. Andres, B., Köthe, U., Kroeger, T., Helmstaedter, M., Briggman, K. L., Denk, W., & Hamprecht, F. A. (2012). 3D segmentation of SBFSEM images of neuropil by a graphical model over supervoxel boundaries. Medical Image Analysis, 16(4), 796–805. doi:10.1016/j.media.2011.11.004. http://www.sciencedirect.com/science/article/pii/S1361841511001666.

  10. Andres, B., Kröger, T., Briggman, K. L., Denk, W., Korogod, N., Knott, G., Köthe, U., & Hamprecht, F. A. (2012). Globally optimal closed-surface segmentation for connectomics. In ECCV.

  11. Batra, D., & Kohli, P. (2011). Making the right moves: Guiding alpha-expansion using local primal-dual gaps. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011 (pp. 1865–1872). IEEE.

  12. Bergtholdt, M., Kappes, J. H., Schmidt, S., & Schnörr, C. (2010). A study of parts-based object class detection using complete graphs. IJCV, 87(1–2), 93–117.

  13. Besag, J. (1986). On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society. Series B (Methodological), 48(3), 259–302. doi:10.2307/2345426.

    MATH  MathSciNet  Google Scholar 

  14. Bonato, T., Jünger, M., Reinelt, G., & Rinaldi, G. (2014). Lifting and separation procedures for the cut polytope. Mathematical Programming A, 146(1–2), 351–378. doi:10.1007/s10107-013-0688-2.

    MATH  MathSciNet  Article  Google Scholar 

  15. Boykov, Y. (2003). Computing geodesics and minimal surfaces via graph cuts. In ICCV.

  16. Boykov, Y., Veksler, O., & Zabih, R. (2001). Fast approximate energy minimization via graph cuts. IEEE PAMI, 23(11), 1222–1239. doi:10.1109/34.969114.

    Article  Google Scholar 

  17. Brandes, U., Delling, D., Gaertler, M., Görke, R., Hoefer, M., Nikoloski, Z., et al. (2008). On modularity clustering. IEEE Transactions on Knowledge and Data Engineering, 20(2), 172–188.

    Article  Google Scholar 

  18. Călinescu, G., Karloff, H., & Rabani, Y. (2000). An improved approximation algorithm for multiway cut. Journal of Computer and System Sciences, 60(3), 564–574.

    MATH  MathSciNet  Article  Google Scholar 

  19. Chekuri, C., Khanna, S., Naor, J., & Zosin, L. (2004). A linear programming formulation and approximation algorithms for the metric labeling problem. SIAM Journal of Discrete Mathematics, 18(3), 608–625.

    MathSciNet  Article  Google Scholar 

  20. Cocosco, C. A., Kollokian, V., Kwan, R. S., & Evans, A. C. (1997). Brainweb: Online interface to a 3d MRI simulated brain database. NeuroImage, 5(4), S425.

    Google Scholar 

  21. IBM, ILOG CPLEX Optimizer. http://www-01.ibm.com/software/integration/optimization/cplex-optimizer/ (2013).

  22. Delong, A., Osokin, A., Isack, H., & Boykov, Y. (2012). Fast approximate energy minimization with label costs. International Journal of Computer Vision, 96, 1–27. http://www.csd.uwo.ca/~yuri/Abstracts/ijcv10_lc-abs.shtml.

  23. Elidan, G., & Globerson, A. (2011) The probabilistic inference challenge (PIC2011). http://www.cs.huji.ac.il/project/PASCAL/.

  24. Felzenszwalb, P. F., & Huttenlocher, D. P. (2006). Efficient belief propagation for early vision. International Journal of Computer Vision, 70(1), 41–54.

    Article  Google Scholar 

  25. Fix, A., Gruber, A., Boros, E., & Zabih, R. (2011). A graph cut algorithm for higher-order Markov random fields. In ICCV. doi:10.1109/ICCV.2011.6126347.

  26. Gallagher, A. C., Batra, D., & Parikh, D. (2011). Inference for order reduction in Markov random fields. In CVPR.

  27. Globerson, A., & Jaakkola, T. (2007). Fixing max-product: Convergent message passing algorithms for MAP LP-relaxations. In NIPS.

  28. Goldberg, D. (1991). What every computer scientist should know about floating-point arithmetic. ACM Computing Surveys, 23(1), 5–48. doi:10.1145/103162.103163.

    Article  Google Scholar 

  29. Gorelick, L., Veksler, O., Boykov, Y., Ben Ayed, I., & Delong, A. (2014). Local submodular approximations for binary pairwise energies. In Computer Vision and Pattern Recognition.

  30. Gould, S., Fulton, R., & Koller, D. (2009). Decomposing a scene into geometric and semantically consistent regions. In ICCV.

  31. Guignard, M., & Kim, S. (1987). Lagrangean decomposition: A model yielding stronger Lagrangean bounds. Mathematical Programming, 39(2), 215–228.

    MATH  MathSciNet  Article  Google Scholar 

  32. Hoiem, D., Efros, A. A., & Hebert, M. (2011). Recovering occlusion boundaries from an image. IJCV, 91(3), 328–346.

    MATH  MathSciNet  Article  Google Scholar 

  33. Hutter, F., Hoos, H. H., & Stützle, T. (2005). Efficient stochastic local search for MPE solving. In L. P. Kaelbling & A. Saffiotti (Eds.), IJCAI (pp. 169–174).

  34. Jaimovich, A., Elidan, G., Margalit, H., & Friedman, N. (2006). Towards an integrated protein–protein interaction network: A relational Markov network approach. Journal of Computational Biology, 13(2), 145–164.

    MathSciNet  Article  Google Scholar 

  35. Kappes, J. H., Andres, B., Hamprecht, F. A., Schnörr, C., Nowozin, S., Batra, D., Kim, S., Kausler, B. X., Lellmann, J., Komodakis, N., & Rother, C. (2013). A comparative study of modern inference techniques for discrete energy minimization problem. In CVPR.

  36. Kappes, J. H., Beier, T., & Schnörr, C. (2014). MAP-inference on large scale higher-order discrete graphical models by fusion moves. In ECCV—International Workshop on Graphical Models in Computer Vision.

  37. Kappes, J. H., Savchynskyy, B., & Schnörr, C. (2012). A bundle approach to efficient MAP-inference by Lagrangian relaxation. In CVPR.

  38. Kappes, J. H., Speth, M., Andres, B., Reinelt, G., & Schnörr, C. (2011). Globally optimal image partitioning by multicuts. In EMMCVPR.

  39. Kappes, J. H., Speth, M., Reinelt, G., & Schnörr, C. (2013). Higher-order segmentation via multicuts. ArXiv e-prints. http://arxiv.org/abs/1305.6387.

  40. Kappes, J. H., Speth, M., Reinelt, G., & Schnörr, C. (2013). Towards efficient and exact MAP-inference for large scale discrete computer vision problems via combinatorial optimization. InCVPR.

  41. Kausler, B. X., Schiegg, M., Andres, B., Lindner, M., Leitte, H., Hufnagel, L., Koethe, U., & Hamprecht, F. A. (2012). A discrete chain graph model for 3d+t cell tracking with high misdetection robustness. In ECCV.

  42. Kernighan, B. W., & Lin, S. (1970). An efficient heuristic procedure for partitioning graphs. The Bell Systems Technical Journal, 49(2), 291–307.

    MATH  Article  Google Scholar 

  43. Kim, S., Nowozin, S., Kohli, P., & Yoo, C. D. (2011). Higher-order correlation clustering for image segmentation. In NIPS (pp. 1530–1538).

  44. Kim, T., Nowozin, S., Kohli, P., & Yoo, C. D. (2011). Variable grouping for energy minimization. In CVPR (pp. 1913–1920).

  45. Kleinberg, J., & Tardos, É. (1999). Approximation algorithms for classification problems with pairwise relationships: Metric labeling and Markov random fields. In Proceedings of the Annual IEEE Symposium on Foundations of Computer Science (FOCS).

  46. Kohli, P., Ladicky, L., & Torr, P. (2009). Robust higher order potentials for enforcing label consistency. International Journal of Computer Vision, 82(3), 302–324. doi:10.1007/s11263-008-0202-0.

    Article  Google Scholar 

  47. Koller, D., & Friedman, N. (2009). Probabilistic graphical models: Principles and techniques. Cambridge: MIT Press.

    Google Scholar 

  48. Kolmogorov, V. (2006). Convergent tree-reweighted message passing for energy minimization. PAMI, 28(10), 1568–1583.

    Article  Google Scholar 

  49. Kolmogorov, V., & Rother, C. (2006). Comparison of energy minimization algorithms for highly connected graphs. In ECCV (pp. 1–15).

  50. Kolmogorov, V., & Zabih, R. (2002). What energy functions can be minimized via graph cuts? In ECCV. http://dl.acm.org/citation.cfm?id=645317.649315.

  51. Komodakis, N., & Paragios, N. (2008). Beyond loose LP-relaxations: Optimizing MRFs by repairing cycles. In ECCV.

  52. Komodakis, N., Paragios, N., & Tziritas, G. (2011). MRF energy minimization and beyond via dual decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(3), 531–552.

    Article  Google Scholar 

  53. Komodakis, N., & Tziritas, G. (2007). Approximate labeling via graph cuts based on linear programming. IEEE PAMI, 29(8), 1436–1453. doi:10.1109/TPAMI.2007.1061.

    Article  Google Scholar 

  54. Kovtun, I. (2003). Partial optimal labeling search for a np-hard subclass of (max, +) problems. In B. Michaelis & G. Krell (Eds.), DAGM-Symposium, Lecture Notes in Computer Science (Vol. 2781, pp. 402–409). Heidelberg: Springer.

  55. Lauritzen, S. L. (1996). Graphical Models. Oxford: Oxford University Press.

    Google Scholar 

  56. Lellmann, J., & Schnörr, C. (2011). Continuous multiclass labeling approaches and algorithms. SIAM Journal of Imaging Sciences, 4(4), 1049–1096.

    MATH  Article  Google Scholar 

  57. Lempitsky, V., Rother, C., Roth, S., & Blake, A. (2010). Fusion moves for Markov random field optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(8), 1392–1405. doi:10.1109/TPAMI.2009.143.

    Article  Google Scholar 

  58. Martins, A. F. T., Figueiredo, M. A. T., Aguiar, P. M. Q., Smith, N. A., & Xing, E. P. (2011). An augmented lagrangian approach to constrained MAP inference. In ICML (pp. 169–176).

  59. Nieuwenhuis, C., Toeppe, E., & Cremers, D. (2013). A survey and comparison of discrete and continuous multi-label optimization approaches for the Potts model. International Journal of Computer Vision, 104, 223–240. doi:10.1007/s11263-013-0619-y.

    MATH  MathSciNet  Article  Google Scholar 

  60. Nowozin, S., & Lampert, C. H. (2011). Structured learning and prediction in computer vision. Foundations and Trends in Computer Graphics and Vision, 6(3–4), 185–365.

    MATH  Google Scholar 

  61. Nowozin, S., Rother, C., Bagon, S., Sharp, T., Yao, B., & Kohli, P. (2011). Decision tree fields. In ICCV (pp. 1668–1675). IEEE.

  62. Orabona, F., Hazan, T., Sarwate, A., & Jaakkola, T. (2014). On measure concentration of random maximum a-posteriori perturbations. In Proc. ICML.

  63. Osokin, A., Vetrov, D., & Kolmogorov, V. (2011). Submodular decomposition framework for inference in associative markov networks with global constraints. In CVPR (pp. 1889–1896).

  64. Otten, L., & Dechter, R. (2011). Anytime AND/OR depth-first search for combinatorial optimization. In Proceedings of the Annual Symposium on Combinatorial Search (SOCS).

  65. Papandreou, G., & Yuille, A. (2011). Perturb-and-MAP random fields: Using discrete optimization to learn and sample from energy models. In Proc. ICCV.

  66. Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. San Francisco, CA: Morgan Kaufmann Publishers Inc.

    Google Scholar 

  67. Prua, D., & Werner, T. (2013). Universality of the local marginal polytope. In CVPR (pp. 1738–1743). IEEE.

  68. Rother, C., Kolmogorov, V., Lempitsky, V. S., & Szummer, M. (2007). Optimizing binary MRFs via extended roof duality. InCVPR.

  69. Rother, C., Kumar, S., Kolmogorov, V., & Blake, A. (2005). Digital tapestry. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) (Vol. 1, pp. 589–596). IEEE Computer Society, Washington, DC, USA. doi:10.1109/CVPR.2005.130.

  70. Savchynskyy, B., Kappes, J. H., Swoboda, P., & Schnörr, C. (2013). Global MAP-optimality by shrinking the combinatorial search area with convex relaxation. In NIPS.

  71. Savchynskyy, B., & Schmidt, S. (2013). Getting feasible variable estimates from infeasible ones: MRF local polytope study. In Workshop on Inference for Probabilistic Graphical Models at ICCV 2013.

  72. Savchynskyy, B., & Schmidt, S. (2014). Getting feasible variable estimates from infeasible ones: MRF local polytope study. In Advanced structured prediction. MIT Press.

  73. Savchynskyy, B., Schmidt, S., Kappes, J. H., & Schnörr, C. (2012). Efficient MRF energy minimization via adaptive diminishing smoothing. UAI, 2012, 746–755.

    Google Scholar 

  74. Schlesinger, M. (1976). Sintaksicheskiy analiz dvumernykh zritelnikh signalov v usloviyakh pomekh (Syntactic analysis of two-dimensional visual signals in noisy conditions). Kibernetika, 4, 113–130.

    Google Scholar 

  75. Sontag, D., Choe, D. K., & Li, Y. (2012). Efficiently searching for frustrated cycles in MAP inference. In N. de Freitas & K. P. Murphy (Eds.) UAI (pp. 795–804). AUAI Press.

  76. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., et al. (2008). A comparative study of energy minimization methods for Markov random fields with smoothness-based priors. IEEE PAMI, 30(6), 1068–1080. doi:10.1109/TPAMI.2007.70844.

  77. Tarlow, D., Batra, D., Kohli, P., & Kolmogorov, V. (2011). Dynamic tree block coordinate ascent. In Proceedings of the International Conference on Machine Learning (ICML).

  78. Verma, T., & Batra, D. (2012). Maxflow revisited: An empirical comparison of maxflow algorithms for dense vision problems. In BMVC (pp. 1–12).

  79. Wainwright, M. J., Jaakkola, T., & Willsky, A. S. (2005). MAP estimation via agreement on trees: Message-passing and linear programming. IEEE Transactions on Information Theory, 51(11), 3697–3717.

    MATH  MathSciNet  Article  Google Scholar 

  80. Werner, T. (2007). A linear programming approach to max-sum problem: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(7), 1165–1179. doi:10.1109/TPAMI.2007.1036.

    Article  Google Scholar 

  81. Wesselmann, F., & Stuhl, U. (2012). Implementing cutting plane management and selection techniques. Tech. rep., University of Paderborn. http://www.optimization-online.org/DB_HTML/2012/12/3714.html.

  82. Woodford, O. J., Torr, P. H. S., Reid, I. D., & Fitzgibbon, A. W. (2009). Global stereo reconstruction under second order smoothness priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(12), 2115–2128.

    Article  Google Scholar 

  83. Yanover, C., Schueler-Furman, O., & Weiss, Y. (2008). Minimizing and learning energy functions for side-chain prediction. Journal of Computational Biology, 15(7), 899–911.

    MathSciNet  Article  Google Scholar 

  84. Yedidia, J. S., Freeman, W. T., & Weiss, Y. (2004). Constructing free energy approximations and generalized belief propagation algorithms. MERL Technical Report, 2004–040. http://www.merl.com/papers/docs/TR2004-040.

Download references


We thank Rick Szeliski and Pushmeet Kohli for inspiring discussions. This work has been supported by the German Research Foundation (DFG) within the program “Spatio- / Temporal Graphical Models and Applications in Image Analysis”, Grant GRK 1653.

Author information



Corresponding author

Correspondence to Jörg H. Kappes.

Additional information

Communicated by K. Ikeuchi.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 4053 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kappes, J.H., Andres, B., Hamprecht, F.A. et al. A Comparative Study of Modern Inference Techniques for Structured Discrete Energy Minimization Problems. Int J Comput Vis 115, 155–184 (2015). https://doi.org/10.1007/s11263-015-0809-x

Download citation


  • Discrete graphical models
  • Combinatorial optimization
  • Benchmark