Journal of Mathematical Imaging and Vision

, Volume 52, Issue 3, pp 436–458 | Cite as

Globally Optimal Joint Image Segmentation and Shape Matching Based on Wasserstein Modes

  • Bernhard Schmitzer
  • Christoph Schnörr


A functional for joint variational object segmentation and shape matching is developed. The formulation is based on optimal transport w.r.t. geometric distance and local feature similarity. Geometric invariance and modelling of object-typical statistical variations is achieved by introducing degrees of freedom that describe transformations and deformations of the shape template. The shape model is mathematically equivalent to contour-based approaches but inference can be performed without conversion between the contour and region representations, allowing combination with other convex segmentation approaches and simplifying optimization. While the overall functional is non-convex, non-convexity is confined to a low-dimensional variable. We propose a locally optimal alternating optimization scheme and a globally optimal branch and bound scheme, based on adaptive convex relaxation. Combining both methods allows to eliminate the delicate initialization problem inherent to many contour based approaches while remaining computationally practical. The properties of the functional, its ability to adapt to a wide range of input data structures and the different optimization schemes are illustrated and compared by numerical experiments.


Image segmentation Object matching Optimal transport Shape analysis Convex relaxation 



This work was supported by the DFG, Grant GRK 1653.


  1. 1.
    Agarwal, S., Awan, A., Roth, D.: Learning to detect objects in images via a sparse, part-based representation. IEEE Trans. Pattern Anal. Mach. Intell. 26(11), 1475–1490 (2004)CrossRefGoogle Scholar
  2. 2.
    Agueh, M., Carlier, G.: Barycenters in the wasserstein space. SIAM J. Math. Anal. 43(2), 904–924 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  3. 3.
    Ambrosio, L., Gigli, N.: A user’s guide to optimal transport. In: Modelling and Optimisation of Flows on Networks. Lect. Not. Math., vol. 2062, pp. 1–155. Springer, Heidelberg (2013)Google Scholar
  4. 4.
    Beg, M.F., Miller, M.I., Trouvé, A., Younes, L.: Computing large deformation metric mappings via geodesic flows of diffeomorphisms. Int. J. Comput. Vis. 61(2), 139–157 (2005)CrossRefGoogle Scholar
  5. 5.
    Berkels, B., Fletcher, T., Heeren, B., Rumpf, M., Wirth, B.: Discrete geodesic regression in shape space. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2013), pp. 108–122 (2013)Google Scholar
  6. 6.
    Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in computer vision. IEEE Trans. Pattern. Anal. Mach. Intell. 26(9), 1124–1137 (2004)CrossRefGoogle Scholar
  7. 7.
    Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Efficient computation of isometry-invariant distances between surfaces. SIAM J. Sci. Comput. 28, 1812–1836 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Chan, T.F., Esedoglu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM J. Appl. Math. 66(5), 1632–1648 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  9. 9.
    Charon, N., Trouvé, A.: Functional currents: a new mathematical tool to model and analyse functional shapes. J. Math. Imaging Vis. 48(3), 413–431 (2014)CrossRefzbMATHGoogle Scholar
  10. 10.
    Charpiat, G., Faugeras, O., Keriven, R.: Approximations of shape metrics and application to shape warping and empirical shape statistics. Found. Comput. Math. 5(1), 1–58 (2005)CrossRefzbMATHMathSciNetGoogle Scholar
  11. 11.
    Cohen, S.D., Guibas, L.J.: The earth mover’s distance under transformation sets. In: International Conference on Computer Vision (ICCV 1999), pp. 1076–1083 (1999)Google Scholar
  12. 12.
    Cremers, D., Kohlberger, T., Schnörr, C.: Shape statistics in kernel space for variational image segmentation. Pattern Recognit. 36(9), 1929–1943 (2003)CrossRefzbMATHGoogle Scholar
  13. 13.
    Cuturi, M., Doucet, A.: Fast computation of wasserstein barycenters. In: International Conference on Machine Learning (2014)Google Scholar
  14. 14.
    Gilboa, G., Osher, S.: Nonlocal operators with applications to image processing. Multiscale Model. Simul. 7(3), 1005–1028 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  15. 15.
    Glaunes, J., Trouve, A., Younes, L.: Diffeomorphic matching of distributions: a new approach for unlabelled point-sets and sub-manifolds matching. In Computer Vision and Pattern Recognition (CVPR 2004), vol. 2, pp. 712–718 (2004)Google Scholar
  16. 16.
    Haker, S., Zhu, L., Tannenbaum, A., Angenent, S.: Optimal mass transport for registration and warping. Int. J. Comput. Vis. 60, 225–240 (2004)CrossRefGoogle Scholar
  17. 17.
    Heeren, B., Rumpf, M., Wardetzky, M., Wirth, B.: Time-discrete geodesics in the space of shells. Comput. Gr. Forum 31(5), 1755–1764 (2012)CrossRefGoogle Scholar
  18. 18.
    Klodt, M., Cremers, D.: A convex framework for image segmentation with moment constraints. In: International Conference on Computer Vision (ICCV 2011), pp. 2236–2243 (2011)Google Scholar
  19. 19.
    Kriegl, A., Michor, P.W.: The Convenient Setting of Global Analysis. Mathematical Surveys and Monographs, vol. 53. American Mathematical Society, Providence (1997)Google Scholar
  20. 20.
    Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Res. Logist. 2, 83–97 (1955)CrossRefGoogle Scholar
  21. 21.
    Kumar, M.P., Torr, P.H.S., Zisserman, A.: OBJ CUT. In Computer Vision and Pattern Recognition (CVPR 2005), vol. 1, pp. 18–25 (2005)Google Scholar
  22. 22.
    Lellmann, J., Schnörr, C.: Continuous multiclass labeling approaches and algorithms. SIAM J. Imaging Sci. 4(4), 1049–1096 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  23. 23.
    Lempitsky, V., Blake, A., Rother, C.: Image segmentation by branch-and-mincut. In: European Conference on Computer Vision (ECCV 2008), pp. 15–29 (2008)Google Scholar
  24. 24.
    Lott, J.: Some geometric calculations on Wasserstein space. Commun. Math. Phys. 277, 423–437 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  25. 25.
    Mémoli, F.: Gromov-Wasserstein distances and the metric approach to object matching. Found. Comput. Math. 11, 417–487 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  26. 26.
    Michor, P.W., Mumford, D.: Riemannian geometries on spaces of plane curves. J. Eur. Math. Soc. 8(1), 1–48 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  27. 27.
    Mio, W., Srivastava, A., Joshi, S.: On shape of plane elastic curves. Int. J. Comput. Vis. 73(3), 307–324 (2007)CrossRefGoogle Scholar
  28. 28.
    Pele, O., Werman, W.: Fast and robust Earth Mover’s Distances. In: International Conference on Computer Vision (ICCV 2009) (2009)Google Scholar
  29. 29.
    Pock, T., Chambolle, A., Cremers, D., Bischof, H.: A convex relaxation approach for computing minimal partitions. In: Computer Vision and Pattern Recognition (CVPR 2009), pp. 810–817 (2009)Google Scholar
  30. 30.
    Schmitzer, B., Schnörr, C.: Weakly convex coupling continuous cuts and shape priors. In: Scale Space and Variational Methods (SSVM 2011), pp. 423–434 (2012)Google Scholar
  31. 31.
    Schmitzer, B., Schnörr, C.: Contour manifolds and optimal transport. (2013). preprint
  32. 32.
    Schmitzer, B., Schnörr, C.: Modelling convex shape priors and matching based on the Gromov-Wasserstein distance. J. Math. Imaging Vis. 46(1), 143–159 (2013)CrossRefzbMATHGoogle Scholar
  33. 33.
    Schmitzer, B., Schnörr, C.: Object segmentation by shape matching with Wasserstein modes. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2013), pp. 123–136 (2013)Google Scholar
  34. 34.
    Sundaramoorthi, G., Mennucci, A., Soatto, S., Yezzi, A.: A new geometric metric in the space of curves, and applications to tracking deforming objects by prediction and filtering. SIAM J. Imaging Sci. 4(1), 109–145 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  35. 35.
    Villani, C.: Optimal Transport: Old and New, vol. 338. Grundlehren der mathematischen Wissenschaften. Springer, Berlin (2009)Google Scholar
  36. 36.
    Wang, W., Slepčev, D., Basu, S., Ozolek, J.A., Rohde, G.K.: A linear optimal transportation framework for quantifying and visualizing variations in sets of images. Int. J. Comput. Vis. 101, 254–269 (2012)CrossRefGoogle Scholar
  37. 37.
    Yangel, B., Vetrov, D.: Learning a model for shape-constrained image segmentation from weakly labeled data. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2013), pp. 137–150 (2013)Google Scholar
  38. 38.
    Younes, L., Michor, P.W., Shah, J., Mumford, D.: A metric on shape space with explicit geodesics. Rend. Lincei Mat. Appl. 9, 25–57 (2008)MathSciNetGoogle Scholar
  39. 39.
    Younes, L.: Shapes and Diffeomorphisms. Applied Mathematical Sciences, vol. 171. Springer (2010)Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.University of HeidelbergHeidelbergGermany

Personalised recommendations