An Inference Algorithm for Multi-label MRF-MAP Problems with Clique Size 100

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12365)


In this paper, we propose an algorithm for optimal solutions to submodular higher order multi-label MRF-MAP energy functions which can handle practical computer vision problems with up to 16 labels and cliques of size 100. The algorithm uses a transformation which transforms a multi-label problem to a 2-label problem on a much larger clique. Earlier algorithms based on this transformation could not handle problems larger than 16 labels on cliques of size 4. The proposed algorithm optimizes the resultant 2-label problem using the submodular polyhedron based Min Norm Point algorithm. The task is challenging because the state space of the transformed problem has a very large number of invalid states. For polyhedral based algorithms the presence of invalid states poses a challenge as apart from numerical instability, the transformation also increases the dimension of the polyhedral space making the straightforward use of known algorithms impractical. The approach reported in this paper allows us to bypass the large costs associated with invalid configurations, resulting in a stable, practical, optimal and efficient inference algorithm that, in our experiments, gives high quality outputs on problems like pixel-wise object segmentation and stereo matching.


Submodular minimization Discrete optimization Hybrid methods MRF-MAP Image segmentation 

Supplementary material

504476_1_En_16_MOESM1_ESM.pdf (257 kb)
Supplementary material 1 (pdf 257 KB)


  1. 1.
    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. PAMI 34(11), 2274–2282 (2012)CrossRefGoogle Scholar
  2. 2.
    Arora, C., Maheshwari, S.: Multi label generic cuts: optimal inference in multi label multi clique MRF-MAP problems. In: CVPR, pp. 1346–1353 (2014) Google Scholar
  3. 3.
    Birchfield, S., Tomasi, C.: A pixel dissimilarity measure that is insensitive to image sampling. PAMI 20(4), 401–406 (1998)CrossRefGoogle Scholar
  4. 4.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)CrossRefGoogle Scholar
  5. 5.
    Chakrabarty, D., Jain, P., Kothari, P.: Provable submodular minimization using Wolfe’s algorithm. In: NIPS, pp. 802–809 (2014)Google Scholar
  6. 6.
    Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 833–851. Springer, Cham (2018). Scholar
  7. 7.
    Delong, A., Osokin, A., Isack, H.N., Boykov, Y.: Fast approximate energy minimization with label costs. Int. J. Comput. Vis. 96(1), 1–27 (2012). Scholar
  8. 8.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results.
  9. 9.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient belief propagation for early vision. Int. J. Comput. Vis. 70(1), 41–54 (2006). Scholar
  10. 10.
    Fix, A., Gruber, A., Boros, E., Zabih, R.: A graph cut algorithm for higher-order Markov random fields. In: ICCV, pp. 1020–1027 (2011)Google Scholar
  11. 11.
    Fix, A., Wang, C., Zabih, R.: A primal-dual algorithm for higher-order multilabel Markov random fields. In: CVPR, pp. 1138–1145 (2014)Google Scholar
  12. 12.
    Freedman, D., Drineas, P.: Energy minimization via graph cuts: settling what is possible. In: CVPR, pp. 939–946 (2005)Google Scholar
  13. 13.
    Fujishige, S., Hayashi, T., Isotani, S.: The minimum-norm-point algorithm applied to submodular function minimization and linear programming (2006)Google Scholar
  14. 14.
    Fujishige, S., Isotani, S.: A submodular function minimization algorithm based on the minimum-norm base. Pac. J. Optim. 7, 3–17 (2011)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Gallagher, A.C., Batra, D., Parikh, D.: Inference for order reduction in Markov random fields. In: CVPR, pp. 1857–1864 (2011)Google Scholar
  16. 16.
    Globerson, A., Jaakkola, T.S.: Fixing max-product: convergent message passing algorithms for map LP-relaxations. In: NIPS, pp. 553–560 (2008)Google Scholar
  17. 17.
    Gould, S.: Darwin: a framework for machine learning and computer vision research and development. JMLR 13(Dec), 3533–3537 (2012)MathSciNetGoogle Scholar
  18. 18.
    Hazan, T., Shashua, A.: Norm-product belief propagation: primal-dual message-passing for approximate inference. Inf. Theory 56(12), 6294–6316 (2010)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Ishikawa, H.: Higher-order clique reduction without auxiliary variables. In: CVPR, pp. 1362–1369 (2014)Google Scholar
  20. 20.
    Ishikawa, H.: Exact optimization for Markov random fields with convex priors. PAMI 25(10), 1333–1336 (2003)CrossRefGoogle Scholar
  21. 21.
    Ishikawa, H.: Transformation of general binary MRF minimization to the first-order case. TPAMI 33(6), 1234–1249 (2011)CrossRefGoogle Scholar
  22. 22.
    Iwata, S., Fleischer, L., Fujishige, S.: A combinatorial strongly polynomial algorithm for minimizing submodular functions. JACM 48(4), 761–777 (2001)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Jegelka, S., Bach, F., Sra, S.: Reflection methods for user-friendly submodular optimization. In: NIPS, pp. 1313–1321 (2013)Google Scholar
  24. 24.
    Kahl, F., Strandmark, P.: Generalized roof duality for pseudo-Boolean optimization. In: ICCV, pp. 255–262 (2011)Google Scholar
  25. 25.
    Kappes, J.H., et al.: A comparative study of modern inference techniques for structured discrete energy minimization problems. IJCV 115(2), 155–184 (2015). Scholar
  26. 26.
    Kohli, P., Torr, P.H., et al.: Robust higher order potentials for enforcing label consistency. IJCV 82(3), 302–324 (2009). Scholar
  27. 27.
    Koller, D., Friedman, N., Bach, F.: Probabilistic Graphical Models: Principles and Techniques. MIT Press, Cambridge (2009)Google Scholar
  28. 28.
    Kolmogorov, V.: Convergent tree-reweighted message passing for energy minimization. PAMI 28(10), 1568–1583 (2006)CrossRefGoogle Scholar
  29. 29.
    Kolmogorov, V.: Submodularity on a tree: unifying \(L^\natural \)-convex and bisubmodular functions. In: Murlak, F., Sankowski, P. (eds.) MFCS 2011. LNCS, vol. 6907, pp. 400–411. Springer, Heidelberg (2011). Scholar
  30. 30.
    Kolmogorov, V.: Minimizing a sum of submodular functions. Discrete Appl. Math. 160(15), 2246–2258 (2012)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Kolmogorov, V.: A new look at reweighted message passing. TPAMI 37(5), 919–930 (2015)CrossRefGoogle Scholar
  32. 32.
    Kolmogorov, V., Zabin, R.: What energy functions can be minimized via graph cuts? TPAMI 26(2), 147–159 (2004)CrossRefGoogle Scholar
  33. 33.
    Komodakis, N., Paragios, N.: Beyond pairwise energies: efficient optimization for higher-order MRFs. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2009, pp. 2985–2992. IEEE (2009)Google Scholar
  34. 34.
    McCormick, S.T.: Submodular function minimization (2005)Google Scholar
  35. 35.
    Murota, K.: On steepest descent algorithms for discrete convex functions. SIAM J. Optim. 14(3), 699–707 (2004)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Orlin, J.B.: A faster strongly polynomial time algorithm for submodular function minimization. Math. Program. 118(2), 237–251 (2009). Scholar
  37. 37.
    Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, Burlington (2014)zbMATHGoogle Scholar
  38. 38.
    Potetz, B., Lee, T.S.: Efficient belief propagation for higher-order cliques using linear constraint nodes. CVIU 112(1), 39–54 (2008)Google Scholar
  39. 39.
    Ramalingam, S., Russell, C., Ladicky, L., Torr, P.H.: Efficient minimization of higher order submodular functions using monotonic Boolean functions. arXiv preprint arXiv:1109.2304 (2011)
  40. 40.
    Roth, S., Black, M.J.: Fields of experts. IJCV 82(2), 205–229 (2009). Scholar
  41. 41.
    Rother, C., Kohli, P., Feng, W., Jia, J.: Minimizing sparse higher order energy functions of discrete variables. In: CVPR, pp. 1382–1389 (2009)Google Scholar
  42. 42.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47(1–3), 7–42 (2002). Scholar
  43. 43.
    Schrijver, A.: A combinatorial algorithm minimizing submodular functions in strongly polynomial time. J. Comb. Theory Ser. B 80(2), 346–355 (2000)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Schrijver, A.: Combinatorial Optimization: Polyhedra and Efficiency, vol. 24. Springer, Heidelberg (2003)zbMATHGoogle Scholar
  45. 45.
    Shanu, I., Arora, C., Maheshwari, S.: Inference in higher order MRF-map problems with small and large cliques. In: CVPR, pp. 7883–7891 (2018)Google Scholar
  46. 46.
    Shanu, I., Arora, C., Singla, P.: Min norm point algorithm for higher order MRF-MAP inference. In: CVPR, pp. 5365–5374 (2016)Google Scholar
  47. 47.
    Sontag, D., Globerson, A., Jaakkola, T.: Introduction to dual decomposition for inference. Optim. Mach. Learn. 1, 219–254 (2011)Google Scholar
  48. 48.
    Szeliski, R., et al.: A comparative study of energy minimization methods for Markov random fields with smoothness-based priors. TPAMI 30(6), 1068–1080 (2008)CrossRefGoogle Scholar
  49. 49.
    Tarlow, D., Givoni, I.E., Zemel, R.S.: HOP-MAP: efficient message passing with high order potentials. In: AISTATS (2010)Google Scholar
  50. 50.
    Windheuser, T., Ishikawa, H., Cremers, D.: Generalized roof duality for multi-label optimization: optimal lower bounds and persistency. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 400–413. Springer, Heidelberg (2012). Scholar
  51. 51.
    Woodford, O., Torr, P., Reid, I., Fitzgibbon, A.: Global stereo reconstruction under second order smoothness priors. In: CVPR, pp. 1–8 (2008)Google Scholar
  52. 52.
    Yedidia, J.S., Freeman, W.T., Weiss, Y.: Generalized belief propagation. In: Advances in Neural Information Processing Systems, pp. 689–695 (2001)Google Scholar
  53. 53.
    Zhang, J., Djolonga, J., Krause, A.: Higher-order inference for multi-class log-supermodular models. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1859–1867 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Indraprastha Institute of Information TechnologyDelhiIndia
  2. 2.Indian Institute of TechnologyDelhiIndia

Personalised recommendations