Abstract
A functional for joint variational object segmentation and shape matching is developed. The formulation is based on optimal transport w.r.t. geometric distance and local feature similarity. Geometric invariance and modelling of object-typical statistical variations is achieved by introducing degrees of freedom that describe transformations and deformations of the shape template. The shape model is mathematically equivalent to contour-based approaches but inference can be performed without conversion between the contour and region representations, allowing combination with other convex segmentation approaches and simplifying optimization. While the overall functional is non-convex, non-convexity is confined to a low-dimensional variable. We propose a locally optimal alternating optimization scheme and a globally optimal branch and bound scheme, based on adaptive convex relaxation. Combining both methods allows to eliminate the delicate initialization problem inherent to many contour based approaches while remaining computationally practical. The properties of the functional, its ability to adapt to a wide range of input data structures and the different optimization schemes are illustrated and compared by numerical experiments.
Similar content being viewed by others
Notes
Note that \(t_{\text {r}}\) is not a gradient field and thus \(\notin T_\mu {{\mathcal {W}}_{2}}({\mathbb {R}}^2)\). One could find a corresponding gradient version by lifting the rotation field from the contour to the interior, Sect. 2.3. However the functional is also meaningful with this non-gradient mode and its effect on the template is more intuitive.
References
Agarwal, S., Awan, A., Roth, D.: Learning to detect objects in images via a sparse, part-based representation. IEEE Trans. Pattern Anal. Mach. Intell. 26(11), 1475–1490 (2004)
Agueh, M., Carlier, G.: Barycenters in the wasserstein space. SIAM J. Math. Anal. 43(2), 904–924 (2011)
Ambrosio, L., Gigli, N.: A user’s guide to optimal transport. In: Modelling and Optimisation of Flows on Networks. Lect. Not. Math., vol. 2062, pp. 1–155. Springer, Heidelberg (2013)
Beg, M.F., Miller, M.I., Trouvé, A., Younes, L.: Computing large deformation metric mappings via geodesic flows of diffeomorphisms. Int. J. Comput. Vis. 61(2), 139–157 (2005)
Berkels, B., Fletcher, T., Heeren, B., Rumpf, M., Wirth, B.: Discrete geodesic regression in shape space. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2013), pp. 108–122 (2013)
Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in computer vision. IEEE Trans. Pattern. Anal. Mach. Intell. 26(9), 1124–1137 (2004)
Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Efficient computation of isometry-invariant distances between surfaces. SIAM J. Sci. Comput. 28, 1812–1836 (2006)
Chan, T.F., Esedoglu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM J. Appl. Math. 66(5), 1632–1648 (2006)
Charon, N., Trouvé, A.: Functional currents: a new mathematical tool to model and analyse functional shapes. J. Math. Imaging Vis. 48(3), 413–431 (2014)
Charpiat, G., Faugeras, O., Keriven, R.: Approximations of shape metrics and application to shape warping and empirical shape statistics. Found. Comput. Math. 5(1), 1–58 (2005)
Cohen, S.D., Guibas, L.J.: The earth mover’s distance under transformation sets. In: International Conference on Computer Vision (ICCV 1999), pp. 1076–1083 (1999)
Cremers, D., Kohlberger, T., Schnörr, C.: Shape statistics in kernel space for variational image segmentation. Pattern Recognit. 36(9), 1929–1943 (2003)
Cuturi, M., Doucet, A.: Fast computation of wasserstein barycenters. In: International Conference on Machine Learning (2014)
Gilboa, G., Osher, S.: Nonlocal operators with applications to image processing. Multiscale Model. Simul. 7(3), 1005–1028 (2008)
Glaunes, J., Trouve, A., Younes, L.: Diffeomorphic matching of distributions: a new approach for unlabelled point-sets and sub-manifolds matching. In Computer Vision and Pattern Recognition (CVPR 2004), vol. 2, pp. 712–718 (2004)
Haker, S., Zhu, L., Tannenbaum, A., Angenent, S.: Optimal mass transport for registration and warping. Int. J. Comput. Vis. 60, 225–240 (2004)
Heeren, B., Rumpf, M., Wardetzky, M., Wirth, B.: Time-discrete geodesics in the space of shells. Comput. Gr. Forum 31(5), 1755–1764 (2012)
Klodt, M., Cremers, D.: A convex framework for image segmentation with moment constraints. In: International Conference on Computer Vision (ICCV 2011), pp. 2236–2243 (2011)
Kriegl, A., Michor, P.W.: The Convenient Setting of Global Analysis. Mathematical Surveys and Monographs, vol. 53. American Mathematical Society, Providence (1997)
Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Res. Logist. 2, 83–97 (1955)
Kumar, M.P., Torr, P.H.S., Zisserman, A.: OBJ CUT. In Computer Vision and Pattern Recognition (CVPR 2005), vol. 1, pp. 18–25 (2005)
Lellmann, J., Schnörr, C.: Continuous multiclass labeling approaches and algorithms. SIAM J. Imaging Sci. 4(4), 1049–1096 (2011)
Lempitsky, V., Blake, A., Rother, C.: Image segmentation by branch-and-mincut. In: European Conference on Computer Vision (ECCV 2008), pp. 15–29 (2008)
Lott, J.: Some geometric calculations on Wasserstein space. Commun. Math. Phys. 277, 423–437 (2008)
Mémoli, F.: Gromov-Wasserstein distances and the metric approach to object matching. Found. Comput. Math. 11, 417–487 (2011)
Michor, P.W., Mumford, D.: Riemannian geometries on spaces of plane curves. J. Eur. Math. Soc. 8(1), 1–48 (2006)
Mio, W., Srivastava, A., Joshi, S.: On shape of plane elastic curves. Int. J. Comput. Vis. 73(3), 307–324 (2007)
Pele, O., Werman, W.: Fast and robust Earth Mover’s Distances. In: International Conference on Computer Vision (ICCV 2009) (2009)
Pock, T., Chambolle, A., Cremers, D., Bischof, H.: A convex relaxation approach for computing minimal partitions. In: Computer Vision and Pattern Recognition (CVPR 2009), pp. 810–817 (2009)
Schmitzer, B., Schnörr, C.: Weakly convex coupling continuous cuts and shape priors. In: Scale Space and Variational Methods (SSVM 2011), pp. 423–434 (2012)
Schmitzer, B., Schnörr, C.: Contour manifolds and optimal transport. http://arxiv.org/abs/1309.2240 (2013). preprint
Schmitzer, B., Schnörr, C.: Modelling convex shape priors and matching based on the Gromov-Wasserstein distance. J. Math. Imaging Vis. 46(1), 143–159 (2013)
Schmitzer, B., Schnörr, C.: Object segmentation by shape matching with Wasserstein modes. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2013), pp. 123–136 (2013)
Sundaramoorthi, G., Mennucci, A., Soatto, S., Yezzi, A.: A new geometric metric in the space of curves, and applications to tracking deforming objects by prediction and filtering. SIAM J. Imaging Sci. 4(1), 109–145 (2011)
Villani, C.: Optimal Transport: Old and New, vol. 338. Grundlehren der mathematischen Wissenschaften. Springer, Berlin (2009)
Wang, W., Slepčev, D., Basu, S., Ozolek, J.A., Rohde, G.K.: A linear optimal transportation framework for quantifying and visualizing variations in sets of images. Int. J. Comput. Vis. 101, 254–269 (2012)
Yangel, B., Vetrov, D.: Learning a model for shape-constrained image segmentation from weakly labeled data. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2013), pp. 137–150 (2013)
Younes, L., Michor, P.W., Shah, J., Mumford, D.: A metric on shape space with explicit geodesics. Rend. Lincei Mat. Appl. 9, 25–57 (2008)
Younes, L.: Shapes and Diffeomorphisms. Applied Mathematical Sciences, vol. 171. Springer (2010)
Acknowledgments
This work was supported by the DFG, Grant GRK 1653.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Schmitzer, B., Schnörr, C. Globally Optimal Joint Image Segmentation and Shape Matching Based on Wasserstein Modes. J Math Imaging Vis 52, 436–458 (2015). https://doi.org/10.1007/s10851-014-0546-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10851-014-0546-8