International Journal of Computer Vision

, Volume 115, Issue 2, pp 155–184 | Cite as

A Comparative Study of Modern Inference Techniques for Structured Discrete Energy Minimization Problems

  • Jörg H. KappesEmail author
  • Bjoern Andres
  • Fred A. Hamprecht
  • Christoph Schnörr
  • Sebastian Nowozin
  • Dhruv Batra
  • Sungwoong Kim
  • Bernhard X. Kausler
  • Thorben Kröger
  • Jan Lellmann
  • Nikos Komodakis
  • Bogdan Savchynskyy
  • Carsten Rother


Szeliski et al. published an influential study in 2006 on energy minimization methods for Markov random fields. This study provided valuable insights in choosing the best optimization technique for certain classes of problems. While these insights remain generally useful today, the phenomenal success of random field models means that the kinds of inference problems that have to be solved changed significantly. Specifically, the models today often include higher order interactions, flexible connectivity structures, large label-spaces of different cardinalities, or learned energy tables. To reflect these changes, we provide a modernized and enlarged study. We present an empirical comparison of more than 27 state-of-the-art optimization techniques on a corpus of 2453 energy minimization instances from diverse applications in computer vision. To ensure reproducibility, we evaluate all methods in the OpenGM 2 framework and report extensive results regarding runtime and solution quality. Key insights from our study agree with the results of Szeliski et al. for the types of models they studied. However, on new and challenging types of models our findings disagree and suggest that polyhedral methods and integer programming solvers are competitive in terms of runtime and solution quality over a large range of model types.


Discrete graphical models Combinatorial optimization  Benchmark 



We thank Rick Szeliski and Pushmeet Kohli for inspiring discussions. This work has been supported by the German Research Foundation (DFG) within the program “Spatio- / Temporal Graphical Models and Applications in Image Analysis”, Grant GRK 1653.

Supplementary material

11263_2015_809_MOESM1_ESM.pdf (4 mb)
Supplementary material 1 (pdf 4053 KB)


  1. Achterberg, T., Koch, T., & Martin, A. (2005). Branching rules revisited. Operations Research Letters, 33(1), 42–54.zbMATHMathSciNetCrossRefGoogle Scholar
  2. Alahari, K., Kohli, P., & Torr, P. H. S. (2008). Reduce, reuse and recycle: Efficiently solving multi-label MRFs. In: CVPR.Google Scholar
  3. Alahari, K., Kohli, P., & Torr, P. H. S. (2010). Dynamic hybrid algorithms for MAP inference in discrete MRFs. IEEE PAMI, 32(10), 1846–1857.CrossRefGoogle Scholar
  4. Andres, B., Beier, T., & Kappes, J. H. (2014). OpenGM2.
  5. Andres, B., Beier, T., & Kappes, J. H. (2012). OpenGM: A C++ library for discrete graphical models. ArXiv e-prints.
  6. Andres, B., Kappes, J. H., Beier, T., Köthe, U., & Hamprecht, F. A. (2011). Probabilistic image segmentation with closedness constraints. In ICCV.Google Scholar
  7. Andres, B., Kappes, J. H., Beier, T., Köthe, U., & Hamprecht, F. A. (2012). The lazy flipper: Efficient depth-limited exhaustive search in discrete graphical models. In ECCV.Google Scholar
  8. Andres, B., Kappes, J. H., Köthe, U., Schnörr, C., & Hamprecht, F. A. (2010). An empirical comparison of inference algorithms for graphical models with higher order factors using OpenGM. In DAGM.Google Scholar
  9. Andres, B., Köthe, U., Kroeger, T., Helmstaedter, M., Briggman, K. L., Denk, W., & Hamprecht, F. A. (2012). 3D segmentation of SBFSEM images of neuropil by a graphical model over supervoxel boundaries. Medical Image Analysis, 16(4), 796–805. doi: 10.1016/
  10. Andres, B., Kröger, T., Briggman, K. L., Denk, W., Korogod, N., Knott, G., Köthe, U., & Hamprecht, F. A. (2012). Globally optimal closed-surface segmentation for connectomics. In ECCV.Google Scholar
  11. Batra, D., & Kohli, P. (2011). Making the right moves: Guiding alpha-expansion using local primal-dual gaps. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011 (pp. 1865–1872). IEEE.Google Scholar
  12. Bergtholdt, M., Kappes, J. H., Schmidt, S., & Schnörr, C. (2010). A study of parts-based object class detection using complete graphs. IJCV, 87(1–2), 93–117.Google Scholar
  13. Besag, J. (1986). On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society. Series B (Methodological), 48(3), 259–302. doi: 10.2307/2345426.zbMATHMathSciNetGoogle Scholar
  14. Bonato, T., Jünger, M., Reinelt, G., & Rinaldi, G. (2014). Lifting and separation procedures for the cut polytope. Mathematical Programming A, 146(1–2), 351–378. doi: 10.1007/s10107-013-0688-2.zbMATHMathSciNetCrossRefGoogle Scholar
  15. Boykov, Y. (2003). Computing geodesics and minimal surfaces via graph cuts. In ICCV.Google Scholar
  16. Boykov, Y., Veksler, O., & Zabih, R. (2001). Fast approximate energy minimization via graph cuts. IEEE PAMI, 23(11), 1222–1239. doi: 10.1109/34.969114.CrossRefGoogle Scholar
  17. Brandes, U., Delling, D., Gaertler, M., Görke, R., Hoefer, M., Nikoloski, Z., et al. (2008). On modularity clustering. IEEE Transactions on Knowledge and Data Engineering, 20(2), 172–188.CrossRefGoogle Scholar
  18. Călinescu, G., Karloff, H., & Rabani, Y. (2000). An improved approximation algorithm for multiway cut. Journal of Computer and System Sciences, 60(3), 564–574.zbMATHMathSciNetCrossRefGoogle Scholar
  19. Chekuri, C., Khanna, S., Naor, J., & Zosin, L. (2004). A linear programming formulation and approximation algorithms for the metric labeling problem. SIAM Journal of Discrete Mathematics, 18(3), 608–625.MathSciNetCrossRefGoogle Scholar
  20. Cocosco, C. A., Kollokian, V., Kwan, R. S., & Evans, A. C. (1997). Brainweb: Online interface to a 3d MRI simulated brain database. NeuroImage, 5(4), S425.Google Scholar
  21. Delong, A., Osokin, A., Isack, H., & Boykov, Y. (2012). Fast approximate energy minimization with label costs. International Journal of Computer Vision, 96, 1–27.
  22. Elidan, G., & Globerson, A. (2011) The probabilistic inference challenge (PIC2011).
  23. Felzenszwalb, P. F., & Huttenlocher, D. P. (2006). Efficient belief propagation for early vision. International Journal of Computer Vision, 70(1), 41–54.CrossRefGoogle Scholar
  24. Fix, A., Gruber, A., Boros, E., & Zabih, R. (2011). A graph cut algorithm for higher-order Markov random fields. In ICCV. doi: 10.1109/ICCV.2011.6126347.
  25. Gallagher, A. C., Batra, D., & Parikh, D. (2011). Inference for order reduction in Markov random fields. In CVPR.Google Scholar
  26. Globerson, A., & Jaakkola, T. (2007). Fixing max-product: Convergent message passing algorithms for MAP LP-relaxations. In NIPS.Google Scholar
  27. Goldberg, D. (1991). What every computer scientist should know about floating-point arithmetic. ACM Computing Surveys, 23(1), 5–48. doi: 10.1145/103162.103163.CrossRefGoogle Scholar
  28. Gorelick, L., Veksler, O., Boykov, Y., Ben Ayed, I., & Delong, A. (2014). Local submodular approximations for binary pairwise energies. In Computer Vision and Pattern Recognition.Google Scholar
  29. Gould, S., Fulton, R., & Koller, D. (2009). Decomposing a scene into geometric and semantically consistent regions. In ICCV.Google Scholar
  30. Guignard, M., & Kim, S. (1987). Lagrangean decomposition: A model yielding stronger Lagrangean bounds. Mathematical Programming, 39(2), 215–228.zbMATHMathSciNetCrossRefGoogle Scholar
  31. Hoiem, D., Efros, A. A., & Hebert, M. (2011). Recovering occlusion boundaries from an image. IJCV, 91(3), 328–346.zbMATHMathSciNetCrossRefGoogle Scholar
  32. Hutter, F., Hoos, H. H., & Stützle, T. (2005). Efficient stochastic local search for MPE solving. In L. P. Kaelbling & A. Saffiotti (Eds.), IJCAI (pp. 169–174).Google Scholar
  33. Jaimovich, A., Elidan, G., Margalit, H., & Friedman, N. (2006). Towards an integrated protein–protein interaction network: A relational Markov network approach. Journal of Computational Biology, 13(2), 145–164.MathSciNetCrossRefGoogle Scholar
  34. Kappes, J. H., Andres, B., Hamprecht, F. A., Schnörr, C., Nowozin, S., Batra, D., Kim, S., Kausler, B. X., Lellmann, J., Komodakis, N., & Rother, C. (2013). A comparative study of modern inference techniques for discrete energy minimization problem. In CVPR.Google Scholar
  35. Kappes, J. H., Beier, T., & Schnörr, C. (2014). MAP-inference on large scale higher-order discrete graphical models by fusion moves. In ECCV—International Workshop on Graphical Models in Computer Vision.Google Scholar
  36. Kappes, J. H., Savchynskyy, B., & Schnörr, C. (2012). A bundle approach to efficient MAP-inference by Lagrangian relaxation. In CVPR.Google Scholar
  37. Kappes, J. H., Speth, M., Andres, B., Reinelt, G., & Schnörr, C. (2011). Globally optimal image partitioning by multicuts. In EMMCVPR.Google Scholar
  38. Kappes, J. H., Speth, M., Reinelt, G., & Schnörr, C. (2013). Higher-order segmentation via multicuts. ArXiv e-prints.
  39. Kappes, J. H., Speth, M., Reinelt, G., & Schnörr, C. (2013). Towards efficient and exact MAP-inference for large scale discrete computer vision problems via combinatorial optimization. InCVPR.Google Scholar
  40. Kausler, B. X., Schiegg, M., Andres, B., Lindner, M., Leitte, H., Hufnagel, L., Koethe, U., & Hamprecht, F. A. (2012). A discrete chain graph model for 3d+t cell tracking with high misdetection robustness. In ECCV.Google Scholar
  41. Kernighan, B. W., & Lin, S. (1970). An efficient heuristic procedure for partitioning graphs. The Bell Systems Technical Journal, 49(2), 291–307.zbMATHCrossRefGoogle Scholar
  42. Kim, S., Nowozin, S., Kohli, P., & Yoo, C. D. (2011). Higher-order correlation clustering for image segmentation. In NIPS (pp. 1530–1538).Google Scholar
  43. Kim, T., Nowozin, S., Kohli, P., & Yoo, C. D. (2011). Variable grouping for energy minimization. In CVPR (pp. 1913–1920).Google Scholar
  44. Kleinberg, J., & Tardos, É. (1999). Approximation algorithms for classification problems with pairwise relationships: Metric labeling and Markov random fields. In Proceedings of the Annual IEEE Symposium on Foundations of Computer Science (FOCS).Google Scholar
  45. Kohli, P., Ladicky, L., & Torr, P. (2009). Robust higher order potentials for enforcing label consistency. International Journal of Computer Vision, 82(3), 302–324. doi: 10.1007/s11263-008-0202-0.CrossRefGoogle Scholar
  46. Koller, D., & Friedman, N. (2009). Probabilistic graphical models: Principles and techniques. Cambridge: MIT Press.Google Scholar
  47. Kolmogorov, V. (2006). Convergent tree-reweighted message passing for energy minimization. PAMI, 28(10), 1568–1583.CrossRefGoogle Scholar
  48. Kolmogorov, V., & Rother, C. (2006). Comparison of energy minimization algorithms for highly connected graphs. In ECCV (pp. 1–15).Google Scholar
  49. Kolmogorov, V., & Zabih, R. (2002). What energy functions can be minimized via graph cuts? In ECCV.
  50. Komodakis, N., & Paragios, N. (2008). Beyond loose LP-relaxations: Optimizing MRFs by repairing cycles. In ECCV.Google Scholar
  51. Komodakis, N., Paragios, N., & Tziritas, G. (2011). MRF energy minimization and beyond via dual decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(3), 531–552.CrossRefGoogle Scholar
  52. Komodakis, N., & Tziritas, G. (2007). Approximate labeling via graph cuts based on linear programming. IEEE PAMI, 29(8), 1436–1453. doi: 10.1109/TPAMI.2007.1061.CrossRefGoogle Scholar
  53. Kovtun, I. (2003). Partial optimal labeling search for a np-hard subclass of (max, +) problems. In B. Michaelis & G. Krell (Eds.), DAGM-Symposium, Lecture Notes in Computer Science (Vol. 2781, pp. 402–409). Heidelberg: Springer.Google Scholar
  54. Lauritzen, S. L. (1996). Graphical Models. Oxford: Oxford University Press.Google Scholar
  55. Lellmann, J., & Schnörr, C. (2011). Continuous multiclass labeling approaches and algorithms. SIAM Journal of Imaging Sciences, 4(4), 1049–1096.zbMATHCrossRefGoogle Scholar
  56. Lempitsky, V., Rother, C., Roth, S., & Blake, A. (2010). Fusion moves for Markov random field optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(8), 1392–1405. doi: 10.1109/TPAMI.2009.143.CrossRefGoogle Scholar
  57. Martins, A. F. T., Figueiredo, M. A. T., Aguiar, P. M. Q., Smith, N. A., & Xing, E. P. (2011). An augmented lagrangian approach to constrained MAP inference. In ICML (pp. 169–176).Google Scholar
  58. Nieuwenhuis, C., Toeppe, E., & Cremers, D. (2013). A survey and comparison of discrete and continuous multi-label optimization approaches for the Potts model. International Journal of Computer Vision, 104, 223–240. doi: 10.1007/s11263-013-0619-y.zbMATHMathSciNetCrossRefGoogle Scholar
  59. Nowozin, S., & Lampert, C. H. (2011). Structured learning and prediction in computer vision. Foundations and Trends in Computer Graphics and Vision, 6(3–4), 185–365.zbMATHGoogle Scholar
  60. Nowozin, S., Rother, C., Bagon, S., Sharp, T., Yao, B., & Kohli, P. (2011). Decision tree fields. In ICCV (pp. 1668–1675). IEEE.Google Scholar
  61. Orabona, F., Hazan, T., Sarwate, A., & Jaakkola, T. (2014). On measure concentration of random maximum a-posteriori perturbations. In Proc. ICML.Google Scholar
  62. Osokin, A., Vetrov, D., & Kolmogorov, V. (2011). Submodular decomposition framework for inference in associative markov networks with global constraints. In CVPR (pp. 1889–1896).Google Scholar
  63. Otten, L., & Dechter, R. (2011). Anytime AND/OR depth-first search for combinatorial optimization. In Proceedings of the Annual Symposium on Combinatorial Search (SOCS).Google Scholar
  64. Papandreou, G., & Yuille, A. (2011). Perturb-and-MAP random fields: Using discrete optimization to learn and sample from energy models. In Proc. ICCV.Google Scholar
  65. Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. San Francisco, CA: Morgan Kaufmann Publishers Inc.Google Scholar
  66. Prua, D., & Werner, T. (2013). Universality of the local marginal polytope. In CVPR (pp. 1738–1743). IEEE.Google Scholar
  67. Rother, C., Kolmogorov, V., Lempitsky, V. S., & Szummer, M. (2007). Optimizing binary MRFs via extended roof duality. InCVPR.Google Scholar
  68. Rother, C., Kumar, S., Kolmogorov, V., & Blake, A. (2005). Digital tapestry. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) (Vol. 1, pp. 589–596). IEEE Computer Society, Washington, DC, USA. doi: 10.1109/CVPR.2005.130.
  69. Savchynskyy, B., Kappes, J. H., Swoboda, P., & Schnörr, C. (2013). Global MAP-optimality by shrinking the combinatorial search area with convex relaxation. In NIPS.Google Scholar
  70. Savchynskyy, B., & Schmidt, S. (2013). Getting feasible variable estimates from infeasible ones: MRF local polytope study. In Workshop on Inference for Probabilistic Graphical Models at ICCV 2013.Google Scholar
  71. Savchynskyy, B., & Schmidt, S. (2014). Getting feasible variable estimates from infeasible ones: MRF local polytope study. In Advanced structured prediction. MIT Press.Google Scholar
  72. Savchynskyy, B., Schmidt, S., Kappes, J. H., & Schnörr, C. (2012). Efficient MRF energy minimization via adaptive diminishing smoothing. UAI, 2012, 746–755.Google Scholar
  73. Schlesinger, M. (1976). Sintaksicheskiy analiz dvumernykh zritelnikh signalov v usloviyakh pomekh (Syntactic analysis of two-dimensional visual signals in noisy conditions). Kibernetika, 4, 113–130.Google Scholar
  74. Sontag, D., Choe, D. K., & Li, Y. (2012). Efficiently searching for frustrated cycles in MAP inference. In N. de Freitas & K. P. Murphy (Eds.) UAI (pp. 795–804). AUAI Press.Google Scholar
  75. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., et al. (2008). A comparative study of energy minimization methods for Markov random fields with smoothness-based priors. IEEE PAMI, 30(6), 1068–1080. doi: 10.1109/TPAMI.2007.70844.
  76. Tarlow, D., Batra, D., Kohli, P., & Kolmogorov, V. (2011). Dynamic tree block coordinate ascent. In Proceedings of the International Conference on Machine Learning (ICML).Google Scholar
  77. Verma, T., & Batra, D. (2012). Maxflow revisited: An empirical comparison of maxflow algorithms for dense vision problems. In BMVC (pp. 1–12).Google Scholar
  78. Wainwright, M. J., Jaakkola, T., & Willsky, A. S. (2005). MAP estimation via agreement on trees: Message-passing and linear programming. IEEE Transactions on Information Theory, 51(11), 3697–3717.zbMATHMathSciNetCrossRefGoogle Scholar
  79. Werner, T. (2007). A linear programming approach to max-sum problem: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(7), 1165–1179. doi: 10.1109/TPAMI.2007.1036.CrossRefGoogle Scholar
  80. Wesselmann, F., & Stuhl, U. (2012). Implementing cutting plane management and selection techniques. Tech. rep., University of Paderborn.
  81. Woodford, O. J., Torr, P. H. S., Reid, I. D., & Fitzgibbon, A. W. (2009). Global stereo reconstruction under second order smoothness priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(12), 2115–2128.CrossRefGoogle Scholar
  82. Yanover, C., Schueler-Furman, O., & Weiss, Y. (2008). Minimizing and learning energy functions for side-chain prediction. Journal of Computational Biology, 15(7), 899–911.MathSciNetCrossRefGoogle Scholar
  83. Yedidia, J. S., Freeman, W. T., & Weiss, Y. (2004). Constructing free energy approximations and generalized belief propagation algorithms. MERL Technical Report, 2004–040.

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Jörg H. Kappes
    • 1
    Email author
  • Bjoern Andres
    • 2
  • Fred A. Hamprecht
    • 1
  • Christoph Schnörr
    • 1
  • Sebastian Nowozin
    • 3
  • Dhruv Batra
    • 4
  • Sungwoong Kim
    • 5
  • Bernhard X. Kausler
    • 1
  • Thorben Kröger
    • 1
  • Jan Lellmann
    • 6
  • Nikos Komodakis
    • 7
  • Bogdan Savchynskyy
    • 8
  • Carsten Rother
    • 8
  1. 1.Heidelberg UniversityHeidelbergGermany
  2. 2.Combinatorial Image Analysis, Max Planck Institute for InformaticsSaarbrückenGermany
  3. 3.Machine Learning and Perception, Microsoft ResearchCambridgeUnited Kingdom
  4. 4.Virginia Tech, 302 Whittemore HallBlacksburgUSA
  5. 5.Qualcomm Research Korea, 15th FL., POBA Gangnam TowerSeoulRepublic of Korea
  6. 6.DAMTP, University of CambridgeCambridgeUnited Kingdom
  7. 7.Universite Paris-Est, Ecole des Ponts ParisTech, Cité DescartesChamps-sur-MarneFrance
  8. 8.Dresden University of TechnologyDresdenGermany

Personalised recommendations