Globally Optimal Joint Image Segmentation and Shape Matching Based on Wasserstein Modes


A functional for joint variational object segmentation and shape matching is developed. The formulation is based on optimal transport w.r.t. geometric distance and local feature similarity. Geometric invariance and modelling of object-typical statistical variations is achieved by introducing degrees of freedom that describe transformations and deformations of the shape template. The shape model is mathematically equivalent to contour-based approaches but inference can be performed without conversion between the contour and region representations, allowing combination with other convex segmentation approaches and simplifying optimization. While the overall functional is non-convex, non-convexity is confined to a low-dimensional variable. We propose a locally optimal alternating optimization scheme and a globally optimal branch and bound scheme, based on adaptive convex relaxation. Combining both methods allows to eliminate the delicate initialization problem inherent to many contour based approaches while remaining computationally practical. The properties of the functional, its ability to adapt to a wide range of input data structures and the different optimization schemes are illustrated and compared by numerical experiments.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16


  1. 1.

    Note that \(t_{\text {r}}\) is not a gradient field and thus \(\notin T_\mu {{\mathcal {W}}_{2}}({\mathbb {R}}^2)\). One could find a corresponding gradient version by lifting the rotation field from the contour to the interior, Sect. 2.3. However the functional is also meaningful with this non-gradient mode and its effect on the template is more intuitive.


  1. 1.

    Agarwal, S., Awan, A., Roth, D.: Learning to detect objects in images via a sparse, part-based representation. IEEE Trans. Pattern Anal. Mach. Intell. 26(11), 1475–1490 (2004)

    Article  Google Scholar 

  2. 2.

    Agueh, M., Carlier, G.: Barycenters in the wasserstein space. SIAM J. Math. Anal. 43(2), 904–924 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  3. 3.

    Ambrosio, L., Gigli, N.: A user’s guide to optimal transport. In: Modelling and Optimisation of Flows on Networks. Lect. Not. Math., vol. 2062, pp. 1–155. Springer, Heidelberg (2013)

  4. 4.

    Beg, M.F., Miller, M.I., Trouvé, A., Younes, L.: Computing large deformation metric mappings via geodesic flows of diffeomorphisms. Int. J. Comput. Vis. 61(2), 139–157 (2005)

    Article  Google Scholar 

  5. 5.

    Berkels, B., Fletcher, T., Heeren, B., Rumpf, M., Wirth, B.: Discrete geodesic regression in shape space. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2013), pp. 108–122 (2013)

  6. 6.

    Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in computer vision. IEEE Trans. Pattern. Anal. Mach. Intell. 26(9), 1124–1137 (2004)

    Article  Google Scholar 

  7. 7.

    Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Efficient computation of isometry-invariant distances between surfaces. SIAM J. Sci. Comput. 28, 1812–1836 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  8. 8.

    Chan, T.F., Esedoglu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM J. Appl. Math. 66(5), 1632–1648 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  9. 9.

    Charon, N., Trouvé, A.: Functional currents: a new mathematical tool to model and analyse functional shapes. J. Math. Imaging Vis. 48(3), 413–431 (2014)

    Article  MATH  Google Scholar 

  10. 10.

    Charpiat, G., Faugeras, O., Keriven, R.: Approximations of shape metrics and application to shape warping and empirical shape statistics. Found. Comput. Math. 5(1), 1–58 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  11. 11.

    Cohen, S.D., Guibas, L.J.: The earth mover’s distance under transformation sets. In: International Conference on Computer Vision (ICCV 1999), pp. 1076–1083 (1999)

  12. 12.

    Cremers, D., Kohlberger, T., Schnörr, C.: Shape statistics in kernel space for variational image segmentation. Pattern Recognit. 36(9), 1929–1943 (2003)

    Article  MATH  Google Scholar 

  13. 13.

    Cuturi, M., Doucet, A.: Fast computation of wasserstein barycenters. In: International Conference on Machine Learning (2014)

  14. 14.

    Gilboa, G., Osher, S.: Nonlocal operators with applications to image processing. Multiscale Model. Simul. 7(3), 1005–1028 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  15. 15.

    Glaunes, J., Trouve, A., Younes, L.: Diffeomorphic matching of distributions: a new approach for unlabelled point-sets and sub-manifolds matching. In Computer Vision and Pattern Recognition (CVPR 2004), vol. 2, pp. 712–718 (2004)

  16. 16.

    Haker, S., Zhu, L., Tannenbaum, A., Angenent, S.: Optimal mass transport for registration and warping. Int. J. Comput. Vis. 60, 225–240 (2004)

    Article  Google Scholar 

  17. 17.

    Heeren, B., Rumpf, M., Wardetzky, M., Wirth, B.: Time-discrete geodesics in the space of shells. Comput. Gr. Forum 31(5), 1755–1764 (2012)

    Article  Google Scholar 

  18. 18.

    Klodt, M., Cremers, D.: A convex framework for image segmentation with moment constraints. In: International Conference on Computer Vision (ICCV 2011), pp. 2236–2243 (2011)

  19. 19.

    Kriegl, A., Michor, P.W.: The Convenient Setting of Global Analysis. Mathematical Surveys and Monographs, vol. 53. American Mathematical Society, Providence (1997)

  20. 20.

    Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Res. Logist. 2, 83–97 (1955)

    Article  Google Scholar 

  21. 21.

    Kumar, M.P., Torr, P.H.S., Zisserman, A.: OBJ CUT. In Computer Vision and Pattern Recognition (CVPR 2005), vol. 1, pp. 18–25 (2005)

  22. 22.

    Lellmann, J., Schnörr, C.: Continuous multiclass labeling approaches and algorithms. SIAM J. Imaging Sci. 4(4), 1049–1096 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  23. 23.

    Lempitsky, V., Blake, A., Rother, C.: Image segmentation by branch-and-mincut. In: European Conference on Computer Vision (ECCV 2008), pp. 15–29 (2008)

  24. 24.

    Lott, J.: Some geometric calculations on Wasserstein space. Commun. Math. Phys. 277, 423–437 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  25. 25.

    Mémoli, F.: Gromov-Wasserstein distances and the metric approach to object matching. Found. Comput. Math. 11, 417–487 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  26. 26.

    Michor, P.W., Mumford, D.: Riemannian geometries on spaces of plane curves. J. Eur. Math. Soc. 8(1), 1–48 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  27. 27.

    Mio, W., Srivastava, A., Joshi, S.: On shape of plane elastic curves. Int. J. Comput. Vis. 73(3), 307–324 (2007)

    Article  Google Scholar 

  28. 28.

    Pele, O., Werman, W.: Fast and robust Earth Mover’s Distances. In: International Conference on Computer Vision (ICCV 2009) (2009)

  29. 29.

    Pock, T., Chambolle, A., Cremers, D., Bischof, H.: A convex relaxation approach for computing minimal partitions. In: Computer Vision and Pattern Recognition (CVPR 2009), pp. 810–817 (2009)

  30. 30.

    Schmitzer, B., Schnörr, C.: Weakly convex coupling continuous cuts and shape priors. In: Scale Space and Variational Methods (SSVM 2011), pp. 423–434 (2012)

  31. 31.

    Schmitzer, B., Schnörr, C.: Contour manifolds and optimal transport. (2013). preprint

  32. 32.

    Schmitzer, B., Schnörr, C.: Modelling convex shape priors and matching based on the Gromov-Wasserstein distance. J. Math. Imaging Vis. 46(1), 143–159 (2013)

    Article  MATH  Google Scholar 

  33. 33.

    Schmitzer, B., Schnörr, C.: Object segmentation by shape matching with Wasserstein modes. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2013), pp. 123–136 (2013)

  34. 34.

    Sundaramoorthi, G., Mennucci, A., Soatto, S., Yezzi, A.: A new geometric metric in the space of curves, and applications to tracking deforming objects by prediction and filtering. SIAM J. Imaging Sci. 4(1), 109–145 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  35. 35.

    Villani, C.: Optimal Transport: Old and New, vol. 338. Grundlehren der mathematischen Wissenschaften. Springer, Berlin (2009)

  36. 36.

    Wang, W., Slepčev, D., Basu, S., Ozolek, J.A., Rohde, G.K.: A linear optimal transportation framework for quantifying and visualizing variations in sets of images. Int. J. Comput. Vis. 101, 254–269 (2012)

    Article  Google Scholar 

  37. 37.

    Yangel, B., Vetrov, D.: Learning a model for shape-constrained image segmentation from weakly labeled data. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2013), pp. 137–150 (2013)

  38. 38.

    Younes, L., Michor, P.W., Shah, J., Mumford, D.: A metric on shape space with explicit geodesics. Rend. Lincei Mat. Appl. 9, 25–57 (2008)

    MathSciNet  Google Scholar 

  39. 39.

    Younes, L.: Shapes and Diffeomorphisms. Applied Mathematical Sciences, vol. 171. Springer (2010)

Download references


This work was supported by the DFG, Grant GRK 1653.

Author information



Corresponding author

Correspondence to Bernhard Schmitzer.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Schmitzer, B., Schnörr, C. Globally Optimal Joint Image Segmentation and Shape Matching Based on Wasserstein Modes. J Math Imaging Vis 52, 436–458 (2015).

Download citation


  • Image segmentation
  • Object matching
  • Optimal transport
  • Shape analysis
  • Convex relaxation