Skip to main content

Advertisement

SpringerLink
Log in

A Comparative Study of Modern Inference Techniques for Structured Discrete Energy Minimization Problems

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Szeliski et al. published an influential study in 2006 on energy minimization methods for Markov random fields. This study provided valuable insights in choosing the best optimization technique for certain classes of problems. While these insights remain generally useful today, the phenomenal success of random field models means that the kinds of inference problems that have to be solved changed significantly. Specifically, the models today often include higher order interactions, flexible connectivity structures, large label-spaces of different cardinalities, or learned energy tables. To reflect these changes, we provide a modernized and enlarged study. We present an empirical comparison of more than 27 state-of-the-art optimization techniques on a corpus of 2453 energy minimization instances from diverse applications in computer vision. To ensure reproducibility, we evaluate all methods in the OpenGM 2 framework and report extensive results regarding runtime and solution quality. Key insights from our study agree with the results of Szeliski et al. for the types of models they studied. However, on new and challenging types of models our findings disagree and suggest that polyhedral methods and integer programming solvers are competitive in terms of runtime and solution quality over a large range of model types.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. The inpainting-N4/8 and color-seg-N4/8-models were originally used in variational approaches together with total variation regularizers (Lellmann and Schnörr 2011). A comparison with variational models is beyond the scope of this study.

  2. Here we consider spanning trees as subproblems such that the relaxation is equivalent to the local polytope relaxation.

  3. This includes terminal constraints TC, multi-terminal constraints MTC, cycle inequalities CC and facet defining cycle inequalities CCFDB as well as odd-wheel constraints OWC.

  4. Due to the increased workload compared to the experiments in Kappes et al. (2013), we switch to a homogeneous cluster and no longer use the Intel Xeon W3550 3.07GHz CPU equipped with 12 GB RAM.

References

  • Achterberg, T., Koch, T., & Martin, A. (2005). Branching rules revisited. Operations Research Letters, 33(1), 42–54.

    Article  MATH  MathSciNet  Google Scholar 

  • Alahari, K., Kohli, P., & Torr, P. H. S. (2008). Reduce, reuse and recycle: Efficiently solving multi-label MRFs. In: CVPR.

  • Alahari, K., Kohli, P., & Torr, P. H. S. (2010). Dynamic hybrid algorithms for MAP inference in discrete MRFs. IEEE PAMI, 32(10), 1846–1857.

    Article  Google Scholar 

  • Andres, B., Beier, T., & Kappes, J. H. (2014). OpenGM2. http://hci.iwr.uni-heidelberg.de/opengm2/.

  • Andres, B., Beier, T., & Kappes, J. H. (2012). OpenGM: A C++ library for discrete graphical models. ArXiv e-prints. http://arxiv.org/abs/1206.0111.

  • Andres, B., Kappes, J. H., Beier, T., Köthe, U., & Hamprecht, F. A. (2011). Probabilistic image segmentation with closedness constraints. In ICCV.

  • Andres, B., Kappes, J. H., Beier, T., Köthe, U., & Hamprecht, F. A. (2012). The lazy flipper: Efficient depth-limited exhaustive search in discrete graphical models. In ECCV.

  • Andres, B., Kappes, J. H., Köthe, U., Schnörr, C., & Hamprecht, F. A. (2010). An empirical comparison of inference algorithms for graphical models with higher order factors using OpenGM. In DAGM.

  • Andres, B., Köthe, U., Kroeger, T., Helmstaedter, M., Briggman, K. L., Denk, W., & Hamprecht, F. A. (2012). 3D segmentation of SBFSEM images of neuropil by a graphical model over supervoxel boundaries. Medical Image Analysis, 16(4), 796–805. doi:10.1016/j.media.2011.11.004. http://www.sciencedirect.com/science/article/pii/S1361841511001666.

  • Andres, B., Kröger, T., Briggman, K. L., Denk, W., Korogod, N., Knott, G., Köthe, U., & Hamprecht, F. A. (2012). Globally optimal closed-surface segmentation for connectomics. In ECCV.

  • Batra, D., & Kohli, P. (2011). Making the right moves: Guiding alpha-expansion using local primal-dual gaps. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011 (pp. 1865–1872). IEEE.

  • Bergtholdt, M., Kappes, J. H., Schmidt, S., & Schnörr, C. (2010). A study of parts-based object class detection using complete graphs. IJCV, 87(1–2), 93–117.

  • Besag, J. (1986). On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society. Series B (Methodological), 48(3), 259–302. doi:10.2307/2345426.

    MATH  MathSciNet  Google Scholar 

  • Bonato, T., Jünger, M., Reinelt, G., & Rinaldi, G. (2014). Lifting and separation procedures for the cut polytope. Mathematical Programming A, 146(1–2), 351–378. doi:10.1007/s10107-013-0688-2.

    Article  MATH  MathSciNet  Google Scholar 

  • Boykov, Y. (2003). Computing geodesics and minimal surfaces via graph cuts. In ICCV.

  • Boykov, Y., Veksler, O., & Zabih, R. (2001). Fast approximate energy minimization via graph cuts. IEEE PAMI, 23(11), 1222–1239. doi:10.1109/34.969114.

    Article  Google Scholar 

  • Brandes, U., Delling, D., Gaertler, M., Görke, R., Hoefer, M., Nikoloski, Z., et al. (2008). On modularity clustering. IEEE Transactions on Knowledge and Data Engineering, 20(2), 172–188.

    Article  Google Scholar 

  • Călinescu, G., Karloff, H., & Rabani, Y. (2000). An improved approximation algorithm for multiway cut. Journal of Computer and System Sciences, 60(3), 564–574.

    Article  MATH  MathSciNet  Google Scholar 

  • Chekuri, C., Khanna, S., Naor, J., & Zosin, L. (2004). A linear programming formulation and approximation algorithms for the metric labeling problem. SIAM Journal of Discrete Mathematics, 18(3), 608–625.

    Article  MathSciNet  Google Scholar 

  • Cocosco, C. A., Kollokian, V., Kwan, R. S., & Evans, A. C. (1997). Brainweb: Online interface to a 3d MRI simulated brain database. NeuroImage, 5(4), S425.

    Google Scholar 

  • IBM, ILOG CPLEX Optimizer. http://www-01.ibm.com/software/integration/optimization/cplex-optimizer/ (2013).

  • Delong, A., Osokin, A., Isack, H., & Boykov, Y. (2012). Fast approximate energy minimization with label costs. International Journal of Computer Vision, 96, 1–27. http://www.csd.uwo.ca/~yuri/Abstracts/ijcv10_lc-abs.shtml.

  • Elidan, G., & Globerson, A. (2011) The probabilistic inference challenge (PIC2011). http://www.cs.huji.ac.il/project/PASCAL/.

  • Felzenszwalb, P. F., & Huttenlocher, D. P. (2006). Efficient belief propagation for early vision. International Journal of Computer Vision, 70(1), 41–54.

    Article  Google Scholar 

  • Fix, A., Gruber, A., Boros, E., & Zabih, R. (2011). A graph cut algorithm for higher-order Markov random fields. In ICCV. doi:10.1109/ICCV.2011.6126347.

  • Gallagher, A. C., Batra, D., & Parikh, D. (2011). Inference for order reduction in Markov random fields. In CVPR.

  • Globerson, A., & Jaakkola, T. (2007). Fixing max-product: Convergent message passing algorithms for MAP LP-relaxations. In NIPS.

  • Goldberg, D. (1991). What every computer scientist should know about floating-point arithmetic. ACM Computing Surveys, 23(1), 5–48. doi:10.1145/103162.103163.

    Article  Google Scholar 

  • Gorelick, L., Veksler, O., Boykov, Y., Ben Ayed, I., & Delong, A. (2014). Local submodular approximations for binary pairwise energies. In Computer Vision and Pattern Recognition.

  • Gould, S., Fulton, R., & Koller, D. (2009). Decomposing a scene into geometric and semantically consistent regions. In ICCV.

  • Guignard, M., & Kim, S. (1987). Lagrangean decomposition: A model yielding stronger Lagrangean bounds. Mathematical Programming, 39(2), 215–228.

    Article  MATH  MathSciNet  Google Scholar 

  • Hoiem, D., Efros, A. A., & Hebert, M. (2011). Recovering occlusion boundaries from an image. IJCV, 91(3), 328–346.

    Article  MATH  MathSciNet  Google Scholar 

  • Hutter, F., Hoos, H. H., & Stützle, T. (2005). Efficient stochastic local search for MPE solving. In L. P. Kaelbling & A. Saffiotti (Eds.), IJCAI (pp. 169–174).

  • Jaimovich, A., Elidan, G., Margalit, H., & Friedman, N. (2006). Towards an integrated protein–protein interaction network: A relational Markov network approach. Journal of Computational Biology, 13(2), 145–164.

    Article  MathSciNet  Google Scholar 

  • Kappes, J. H., Andres, B., Hamprecht, F. A., Schnörr, C., Nowozin, S., Batra, D., Kim, S., Kausler, B. X., Lellmann, J., Komodakis, N., & Rother, C. (2013). A comparative study of modern inference techniques for discrete energy minimization problem. In CVPR.

  • Kappes, J. H., Beier, T., & Schnörr, C. (2014). MAP-inference on large scale higher-order discrete graphical models by fusion moves. In ECCV—International Workshop on Graphical Models in Computer Vision.

  • Kappes, J. H., Savchynskyy, B., & Schnörr, C. (2012). A bundle approach to efficient MAP-inference by Lagrangian relaxation. In CVPR.

  • Kappes, J. H., Speth, M., Andres, B., Reinelt, G., & Schnörr, C. (2011). Globally optimal image partitioning by multicuts. In EMMCVPR.

  • Kappes, J. H., Speth, M., Reinelt, G., & Schnörr, C. (2013). Higher-order segmentation via multicuts. ArXiv e-prints. http://arxiv.org/abs/1305.6387.

  • Kappes, J. H., Speth, M., Reinelt, G., & Schnörr, C. (2013). Towards efficient and exact MAP-inference for large scale discrete computer vision problems via combinatorial optimization. InCVPR.

  • Kausler, B. X., Schiegg, M., Andres, B., Lindner, M., Leitte, H., Hufnagel, L., Koethe, U., & Hamprecht, F. A. (2012). A discrete chain graph model for 3d+t cell tracking with high misdetection robustness. In ECCV.

  • Kernighan, B. W., & Lin, S. (1970). An efficient heuristic procedure for partitioning graphs. The Bell Systems Technical Journal, 49(2), 291–307.

    Article  MATH  Google Scholar 

  • Kim, S., Nowozin, S., Kohli, P., & Yoo, C. D. (2011). Higher-order correlation clustering for image segmentation. In NIPS (pp. 1530–1538).

  • Kim, T., Nowozin, S., Kohli, P., & Yoo, C. D. (2011). Variable grouping for energy minimization. In CVPR (pp. 1913–1920).

  • Kleinberg, J., & Tardos, É. (1999). Approximation algorithms for classification problems with pairwise relationships: Metric labeling and Markov random fields. In Proceedings of the Annual IEEE Symposium on Foundations of Computer Science (FOCS).

  • Kohli, P., Ladicky, L., & Torr, P. (2009). Robust higher order potentials for enforcing label consistency. International Journal of Computer Vision, 82(3), 302–324. doi:10.1007/s11263-008-0202-0.

    Article  Google Scholar 

  • Koller, D., & Friedman, N. (2009). Probabilistic graphical models: Principles and techniques. Cambridge: MIT Press.

    Google Scholar 

  • Kolmogorov, V. (2006). Convergent tree-reweighted message passing for energy minimization. PAMI, 28(10), 1568–1583.

    Article  Google Scholar 

  • Kolmogorov, V., & Rother, C. (2006). Comparison of energy minimization algorithms for highly connected graphs. In ECCV (pp. 1–15).

  • Kolmogorov, V., & Zabih, R. (2002). What energy functions can be minimized via graph cuts? In ECCV. http://dl.acm.org/citation.cfm?id=645317.649315.

  • Komodakis, N., & Paragios, N. (2008). Beyond loose LP-relaxations: Optimizing MRFs by repairing cycles. In ECCV.

  • Komodakis, N., Paragios, N., & Tziritas, G. (2011). MRF energy minimization and beyond via dual decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(3), 531–552.

    Article  Google Scholar 

  • Komodakis, N., & Tziritas, G. (2007). Approximate labeling via graph cuts based on linear programming. IEEE PAMI, 29(8), 1436–1453. doi:10.1109/TPAMI.2007.1061.

    Article  Google Scholar 

  • Kovtun, I. (2003). Partial optimal labeling search for a np-hard subclass of (max, +) problems. In B. Michaelis & G. Krell (Eds.), DAGM-Symposium, Lecture Notes in Computer Science (Vol. 2781, pp. 402–409). Heidelberg: Springer.

  • Lauritzen, S. L. (1996). Graphical Models. Oxford: Oxford University Press.

    Google Scholar 

  • Lellmann, J., & Schnörr, C. (2011). Continuous multiclass labeling approaches and algorithms. SIAM Journal of Imaging Sciences, 4(4), 1049–1096.

    Article  MATH  Google Scholar 

  • Lempitsky, V., Rother, C., Roth, S., & Blake, A. (2010). Fusion moves for Markov random field optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(8), 1392–1405. doi:10.1109/TPAMI.2009.143.

    Article  Google Scholar 

  • Martins, A. F. T., Figueiredo, M. A. T., Aguiar, P. M. Q., Smith, N. A., & Xing, E. P. (2011). An augmented lagrangian approach to constrained MAP inference. In ICML (pp. 169–176).

  • Nieuwenhuis, C., Toeppe, E., & Cremers, D. (2013). A survey and comparison of discrete and continuous multi-label optimization approaches for the Potts model. International Journal of Computer Vision, 104, 223–240. doi:10.1007/s11263-013-0619-y.

    Article  MATH  MathSciNet  Google Scholar 

  • Nowozin, S., & Lampert, C. H. (2011). Structured learning and prediction in computer vision. Foundations and Trends in Computer Graphics and Vision, 6(3–4), 185–365.

    MATH  Google Scholar 

  • Nowozin, S., Rother, C., Bagon, S., Sharp, T., Yao, B., & Kohli, P. (2011). Decision tree fields. In ICCV (pp. 1668–1675). IEEE.

  • Orabona, F., Hazan, T., Sarwate, A., & Jaakkola, T. (2014). On measure concentration of random maximum a-posteriori perturbations. In Proc. ICML.

  • Osokin, A., Vetrov, D., & Kolmogorov, V. (2011). Submodular decomposition framework for inference in associative markov networks with global constraints. In CVPR (pp. 1889–1896).

  • Otten, L., & Dechter, R. (2011). Anytime AND/OR depth-first search for combinatorial optimization. In Proceedings of the Annual Symposium on Combinatorial Search (SOCS).

  • Papandreou, G., & Yuille, A. (2011). Perturb-and-MAP random fields: Using discrete optimization to learn and sample from energy models. In Proc. ICCV.

  • Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. San Francisco, CA: Morgan Kaufmann Publishers Inc.

    Google Scholar 

  • Prua, D., & Werner, T. (2013). Universality of the local marginal polytope. In CVPR (pp. 1738–1743). IEEE.

  • Rother, C., Kolmogorov, V., Lempitsky, V. S., & Szummer, M. (2007). Optimizing binary MRFs via extended roof duality. InCVPR.

  • Rother, C., Kumar, S., Kolmogorov, V., & Blake, A. (2005). Digital tapestry. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) (Vol. 1, pp. 589–596). IEEE Computer Society, Washington, DC, USA. doi:10.1109/CVPR.2005.130.

  • Savchynskyy, B., Kappes, J. H., Swoboda, P., & Schnörr, C. (2013). Global MAP-optimality by shrinking the combinatorial search area with convex relaxation. In NIPS.

  • Savchynskyy, B., & Schmidt, S. (2013). Getting feasible variable estimates from infeasible ones: MRF local polytope study. In Workshop on Inference for Probabilistic Graphical Models at ICCV 2013.

  • Savchynskyy, B., & Schmidt, S. (2014). Getting feasible variable estimates from infeasible ones: MRF local polytope study. In Advanced structured prediction. MIT Press.

  • Savchynskyy, B., Schmidt, S., Kappes, J. H., & Schnörr, C. (2012). Efficient MRF energy minimization via adaptive diminishing smoothing. UAI, 2012, 746–755.

    Google Scholar 

  • Schlesinger, M. (1976). Sintaksicheskiy analiz dvumernykh zritelnikh signalov v usloviyakh pomekh (Syntactic analysis of two-dimensional visual signals in noisy conditions). Kibernetika, 4, 113–130.

    Google Scholar 

  • Sontag, D., Choe, D. K., & Li, Y. (2012). Efficiently searching for frustrated cycles in MAP inference. In N. de Freitas & K. P. Murphy (Eds.) UAI (pp. 795–804). AUAI Press.

  • Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., et al. (2008). A comparative study of energy minimization methods for Markov random fields with smoothness-based priors. IEEE PAMI, 30(6), 1068–1080. doi:10.1109/TPAMI.2007.70844.

  • Tarlow, D., Batra, D., Kohli, P., & Kolmogorov, V. (2011). Dynamic tree block coordinate ascent. In Proceedings of the International Conference on Machine Learning (ICML).

  • Verma, T., & Batra, D. (2012). Maxflow revisited: An empirical comparison of maxflow algorithms for dense vision problems. In BMVC (pp. 1–12).

  • Wainwright, M. J., Jaakkola, T., & Willsky, A. S. (2005). MAP estimation via agreement on trees: Message-passing and linear programming. IEEE Transactions on Information Theory, 51(11), 3697–3717.

    Article  MATH  MathSciNet  Google Scholar 

  • Werner, T. (2007). A linear programming approach to max-sum problem: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(7), 1165–1179. doi:10.1109/TPAMI.2007.1036.

    Article  Google Scholar 

  • Wesselmann, F., & Stuhl, U. (2012). Implementing cutting plane management and selection techniques. Tech. rep., University of Paderborn. http://www.optimization-online.org/DB_HTML/2012/12/3714.html.

  • Woodford, O. J., Torr, P. H. S., Reid, I. D., & Fitzgibbon, A. W. (2009). Global stereo reconstruction under second order smoothness priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(12), 2115–2128.

    Article  Google Scholar 

  • Yanover, C., Schueler-Furman, O., & Weiss, Y. (2008). Minimizing and learning energy functions for side-chain prediction. Journal of Computational Biology, 15(7), 899–911.

    Article  MathSciNet  Google Scholar 

  • Yedidia, J. S., Freeman, W. T., & Weiss, Y. (2004). Constructing free energy approximations and generalized belief propagation algorithms. MERL Technical Report, 2004–040. http://www.merl.com/papers/docs/TR2004-040.

Download references

Acknowledgments

We thank Rick Szeliski and Pushmeet Kohli for inspiring discussions. This work has been supported by the German Research Foundation (DFG) within the program “Spatio- / Temporal Graphical Models and Applications in Image Analysis”, Grant GRK 1653.

Author information

Authors and Affiliations

  1. Heidelberg University, Speyerer Strasse 6, 69115, Heidelberg, Germany

    Jörg H. Kappes, Fred A. Hamprecht, Christoph Schnörr, Bernhard X. Kausler & Thorben Kröger

  2. Combinatorial Image Analysis, Max Planck Institute for Informatics, Campus E1.4, 66123, Saarbrücken, Germany

    Bjoern Andres

  3. Machine Learning and Perception, Microsoft Research, 21 Station Road, Cambridge, CB12FB, United Kingdom

    Sebastian Nowozin

  4. Virginia Tech, 302 Whittemore Hall, Blacksburg, VA, 24061, USA

    Dhruv Batra

  5. Qualcomm Research Korea, 15th FL., POBA Gangnam Tower, 343, Hakdong-ro, Gangnam-gu, 135-820, Seoul, Republic of Korea

    Sungwoong Kim

  6. DAMTP, University of Cambridge, Wilberforce Road, Cambridge, CB3 0WA, United Kingdom

    Jan Lellmann

  7. Universite Paris-Est, Ecole des Ponts ParisTech, Cité Descartes, 6-8 Avenue Blaise Pascal, 77455, Champs-sur-Marne, France

    Nikos Komodakis

  8. Dresden University of Technology, 01062, Dresden, Germany

    Bogdan Savchynskyy & Carsten Rother

Authors
  1. Jörg H. Kappes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bjoern Andres
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fred A. Hamprecht
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christoph Schnörr
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sebastian Nowozin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dhruv Batra
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Sungwoong Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Bernhard X. Kausler
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Thorben Kröger
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Jan Lellmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Nikos Komodakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Bogdan Savchynskyy
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Carsten Rother
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jörg H. Kappes.

Additional information

Communicated by K. Ikeuchi.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 4053 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kappes, J.H., Andres, B., Hamprecht, F.A. et al. A Comparative Study of Modern Inference Techniques for Structured Discrete Energy Minimization Problems. Int J Comput Vis 115, 155–184 (2015). https://doi.org/10.1007/s11263-015-0809-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-015-0809-x

Keywords

Search

Navigation