Journal of Mathematical Imaging and Vision

, Volume 58, Issue 2, pp 211–238 | Cite as

Image Labeling by Assignment

  • Freddie Åström
  • Stefania Petra
  • Bernhard Schmitzer
  • Christoph Schnörr


We introduce a novel geometric approach to the image labeling problem. Abstracting from specific labeling applications, a general objective function is defined on a manifold of stochastic matrices, whose elements assign prior data that are given in any metric space, to observed image measurements. The corresponding Riemannian gradient flow entails a set of replicator equations, one for each data point, that are spatially coupled by geometric averaging on the manifold. Starting from uniform assignments at the barycenter as natural initialization, the flow terminates at some global maximum, each of which corresponds to an image labeling that uniquely assigns the prior data. Our geometric variational approach constitutes a smooth non-convex inner approximation of the general image labeling problem, implemented with sparse interior-point numerics in terms of parallel multiplicative updates that converge efficiently.


Image labeling Assignment manifold Fisher–Rao metric Riemannian gradient flow Replicator equations Information geometry Neighborhood filters Nonlinear diffusion 

Mathematics Subject Classification

62H35 65K05 68U10 62M40 



Support by the German Research Foundation (DFG) was gratefully acknowledged, Grant GRK 1653.


  1. 1.
    Amari, S.I., Nagaoka, H.: Methods of Information Geometry. American Mathematical Society, Oxford University Press, Oxford (2000)MATHGoogle Scholar
  2. 2.
    Aujol, J.F., Gilboa, G., Chan, T., Osher, S.: Structure-texture image decomposition-modeling, algorithms, and parameter selection. Int. J. Comput. Vis. 67(1), 111–136 (2006)CrossRefMATHGoogle Scholar
  3. 3.
    Ball, K.: An elementary introduction to modern convex geometry. In: Levy, S. (ed.) Flavors of Geometry, MSRI Publ., vol. 31, pp. 1–58. Cambridge University Press (1997)Google Scholar
  4. 4.
    Bayer, D., Lagarias, J.: The nonlinear geometry of linear programming. I. Affine and projective scaling trajectories. Trans. Am. Math. Soc. 314(2), 499–526 (1989)MathSciNetMATHGoogle Scholar
  5. 5.
    Bayer, D., Lagarias, J.: The nonlinear geometry of linear programming. II. Legendre transform coordinates and central trajectories. Trans. Am. Math. Soc. 314(2), 527–581 (1989)MathSciNetMATHGoogle Scholar
  6. 6.
    Bishop, C.: Pattern Recognition and Machine Learning. Springer, Berlin (2006)MATHGoogle Scholar
  7. 7.
    Bomze, I., Budinich, M., Pelillo, M., Rossi, C.: Annealed replication: a new heuristic for the maximum clique problem. Discr. Appl. Math. 121, 27–49 (2002)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Bomze, I.M.: Regularity versus degeneracy in dynamics, games, and optimization: a unified approach to different aspects. SIAM Rev. 44(3), 394–414 (2002)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Buades, A., Coll, B., Morel, J.: A review of image denoising algorithms, with a new one. SIAM Multiscale Model. Simul. 4(2), 490–530 (2005)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Buades, A., Coll, B., Morel, J.M.: Neighborhood filters and PDEs. Numer. Math. 105, 1–34 (2006)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Cabrales, A., Sobel, J.: On the limit points of discrete selection dynamics. J. Econ. Theory 57, 407–419 (1992)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Čencov, N.: Statistical Decision Rules and Optimal Inference. American Mathematical Society, Providence (1982)Google Scholar
  13. 13.
    Chambolle, A., Cremers, D., Pock, T.: A convex approach to minimal partitions. SIAM J. Imaging Sci. 5(4), 1113–1158 (2012)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Chan, T., Esedoglu, S., Nikolova, M.: Algorithms for Finding Global Minimizers of Image Segmentation and Denoising Models. SIAM J. Appl. Math. 66(5), 1632–1648 (2006)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Hérault, L., Horaud, R.: Figure-ground discrimination: a combinatorial optimization approach. IEEE Trans. Pattern Anal. Mach. Intell. 15(9), 899–914 (1993)CrossRefGoogle Scholar
  16. 16.
    Heskes, T.: Convexity arguments for efficient minimization of the Bethe and Kikuchi free energies. J. Artif. Intell. Res. 26, 153–190 (2006)MathSciNetMATHGoogle Scholar
  17. 17.
    Hofbauer, J., Siegmund, K.: Evolutionary game dynamics. Bull. Am. Math. Soc. 40(4), 479–519 (2003)MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    Hofman, T., Buhmann, J.: Pairwise data clustering by deterministic annealing. IEEE Trans. Pattern Anal. Mach. Intell. 19(1), 1–14 (1997)CrossRefGoogle Scholar
  19. 19.
    Horst, R., Tuy, H.: Global Optimization: Deterministic Approaches, 3rd edn. Springer, Berlin (1996)CrossRefMATHGoogle Scholar
  20. 20.
    Hummel, R., Zucker, S.: On the foundations of the relaxation labeling processes. IEEE Trans. Pattern Anal. Mach. Intell. 5(3), 267–287 (1983)CrossRefMATHGoogle Scholar
  21. 21.
    Jost, J.: Riemannian Geometry and Geometric Analysis, 4th edn. Springer, Berlin (2005)MATHGoogle Scholar
  22. 22.
    Kappes, J., Andres, B., Hamprecht, F., Schnörr, C., Nowozin, S., Batra, D., Kim, S., Kausler, B., Kröger, T., Lellmann, J., Komodakis, N., Savchynskyy, B., Rother, C.: A comparative study of modern inference techniques for structured discrete energy minimization problems. Int. J. Comput. Vis. 115(2), 155–184 (2015)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Kappes, J., Savchynskyy, B., Schnörr, C.: A bundle approach to efficient MAP-inference by Lagrangian relaxation. In: Proc. CVPR (2012)Google Scholar
  24. 24.
    Kappes, J., Schnörr, C.: MAP-inference for highly-connected graphs with DC-programming. In: Pattern Recognition—30th DAGM Symposium, LNCS, vol. 5096, pp. 1–10. Springer (2008)Google Scholar
  25. 25.
    Karcher, H.: Riemannian center of mass and mollifier smoothing. Commun. Pure Appl. Math. 30, 509–541 (1977)MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Karcher, H.: Riemannian center of mass and so called karcher mean. arxiv:1407.2087 (2014)
  27. 27.
    Kass, R.: The geometry of asymptotic inference. Stat. Sci. 4(3), 188–234 (1989)MathSciNetCrossRefMATHGoogle Scholar
  28. 28.
    Kolmogorov, V.: Convergent tree-reweighted message passing for energy minimization. IEEE Trans. Pattern Anal. Mach. Intell. 28(10), 1568–1583 (2006)CrossRefGoogle Scholar
  29. 29.
    Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004)CrossRefMATHGoogle Scholar
  30. 30.
    Ledoux, M.: The Concentration of Measure Phenomenon. American Mathematical Society, Providence (2001)MATHGoogle Scholar
  31. 31.
    Lellmann, J., Lenzen, F., Schnörr, C.: Optimality bounds for a variational relaxation of the image partitioning problem. J. Math. Imaging Vis. 47(3), 239–257 (2013)MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Lellmann, J., Schnörr, C.: Continuous multiclass labeling approaches and algorithms. SIAM J. Imaging Sci. 4(4), 1049–1096 (2011)MathSciNetCrossRefMATHGoogle Scholar
  33. 33.
    Losert, V., Alin, E.: Dynamics of games and genes: discrete versus continuous time. J. Math. Biol. 17(2), 241–251 (1983)MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    Luce, R.: Individual Choice Behavior: A Theoretical Analysis. Wiley, New York (1959)MATHGoogle Scholar
  35. 35.
    Milanfar, P.: A tour of modern image filtering. IEEE Signal Process. Mag. 30(1), 106–128 (2013)CrossRefGoogle Scholar
  36. 36.
    Milanfar, P.: Symmetrizing smoothing filters. SIAM J. Imaging Sci. 6(1), 263–284 (2013)MathSciNetCrossRefMATHGoogle Scholar
  37. 37.
    Montúfar, G., Rauh, J., Ay, N.: On the fisher metric of conditional probability polytopes. Entropy 16(6), 3207–3233 (2014)MathSciNetCrossRefMATHGoogle Scholar
  38. 38.
    Nesterov, Y., Todd, M.: On the riemannian geometry defined by self-concordant barriers and interior-point methods. Found. Comput. Math. 2, 333–361 (2002)MathSciNetCrossRefMATHGoogle Scholar
  39. 39.
    Orland, H.: Mean-field theory for optimization problems. J. Phys. Lett. 46(17), 763–770 (1985)CrossRefGoogle Scholar
  40. 40.
    Pavan, M., Pelillo, M.: Dominant sets and pairwise clustering. IEEE Trans. Pattern Anal. Mach. Intell. 29(1), 167–172 (2007)CrossRefGoogle Scholar
  41. 41.
    Pelillo, M.: The dynamics of nonlinear relaxation labeling processes. J. Math. Imaging Vis. 7, 309–323 (1997)MathSciNetCrossRefGoogle Scholar
  42. 42.
    Pelillo, M.: Replicator equations, maximal cliques, and graph isomorphism. Neural Comput. 11(8), 1933–1955 (1999)CrossRefGoogle Scholar
  43. 43.
    Rosenfeld, A., Hummel, R., Zucker, S.: Scene labeling by relaxation operations. IEEE Trans. Syst. Man Cybern. 6, 420–433 (1976)MathSciNetCrossRefMATHGoogle Scholar
  44. 44.
    Singer, A., Shkolnisky, Y., Nadler, B.: Diffusion interpretation of non-local neighborhood filters for signal denoising. SIAM J. Imaging Sci. 2(1), 118–139 (2009)Google Scholar
  45. 45.
    Sutton, R., Barto, A.: Reinforcement Learning, 2nd edn. MIT Press, Cambridge (1999)Google Scholar
  46. 46.
    Swoboda, P., Shekhovtsov, A., Kappes, J., Schnörr, C., Savchynskyy, B.: Partial optimality by pruning for MAP-inference with general graphical models. IEEE Trans. Patt. Anal. Mach. Intell. 38(7), 1370–1382 (2016)Google Scholar
  47. 47.
    Wainwright, M., Jordan, M.: Graphical models, exponential families, and variational inference. Found. Trends Mach. Learn. 1(1–2), 1–305 (2008)MATHGoogle Scholar
  48. 48.
    Weickert, J.: Anisotropic Diffusion in Image Processing. B.G Teubner, Leipzig (1998)MATHGoogle Scholar
  49. 49.
    Werner, T.: A linear programming approach to max-sum problem: a review. IEEE Trans. Pattern Anal. Mach. Intell. 29(7), 1165–1179 (2007)CrossRefGoogle Scholar
  50. 50.
    Yedidia, J., Freeman, W., Weiss, Y.: Constructing free-energy approximations and generalized belief propagation algorithms. IEEE Trans. Inf. Theory 51(7), 2282–2312 (2005)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Heidelberg Collaboratory for Image ProcessingHeidelberg UniversityHeidelbergGermany
  2. 2.Mathematical Imaging GroupHeidelberg UniversityHeidelbergGermany
  3. 3.CEREMADEUniversity Paris-DauphineParisFrance
  4. 4.Image and Pattern Analysis GroupHeidelberg UniversityHeidelbergGermany

Personalised recommendations