Journal of Mathematical Imaging and Vision

, Volume 59, Issue 2, pp 161–186 | Cite as

Convex Histogram-Based Joint Image Segmentation with Regularized Optimal Transport Cost

Article
  • 256 Downloads

Abstract

We investigate in this work a versatile convex framework for multiple image segmentation, relying on the regularized optimal mass transport theory. In this setting, several transport cost functions are considered and used to match statistical distributions of features. In practice, global multidimensional histograms are estimated from the segmented image regions and are compared to reference models that are either fixed histograms given a priori, or directly inferred in the non-supervised case. The different convex problems studied are solved efficiently using primal–dual algorithms. The proposed approach is generic and enables multiphase segmentation as well as co-segmentation of multiple images.

Keywords

Optimal transport Wasserstein distance Sinkhorn distance Image segmentation Convex optimization 

Notes

Acknowledgements

The authors acknowledge support from the CNRS in the context of the “Défi Imag’In” project CAlcul des VAriations pour L’Imagerie, l’Edition et la Recherche d’Images (CAVALIERI). This study has been carried out with financial support from the French State, managed by the French National Research Agency (ANR) in the frame of the Investments for the future Programme IdEx Bordeaux (ANR- 10-IDEX-03-02). The authors would like to thank Gabriel Peyré and Marco Cuturi for sharing their preliminary work and Jalal Fadili for fruitful discussions on convex optimization.

References

  1. 1.
    Aubert, G., Barlaud, M., Faugeras, O., Jehan-Besson, S.: Image segmentation using active contours: calculus of variations or shape gradients? SIAM Appl. Math. 63(6), 2128–2154 (2003)MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Aubert, G., Kornprobst, P.: Mathematical Problems in Image Processing, volume 147 of Applied Mathematical Sciences. Springer, New York (2002)MATHGoogle Scholar
  3. 3.
    Aujol, J.-F., Aubert, G., Blanc-Féraud, L.: Wavelet-based level set evolution for classification of textured images. IEEE Trans. Image Process. 12(12), 1634–1641 (2003)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Ayed, I., Chen, H., Punithakumar, K., Ross, I., Li, S.: Graph cut segmentation with a global constraint: recovering region distribution via a bound of the bhattacharyya measure. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR’10), pp. 3288–3295 (2010)Google Scholar
  5. 5.
    Berkels, B.: An unconstrained multiphase thresholding approach for image segmentation. In: Scale Space and Variational Methods in Computer Vision (SSVM’09), vol. 5567, pp. 26–37 (2009)Google Scholar
  6. 6.
    Bonneel, N., Rabin, J., Peyré, G., Pfister, H.: Sliced and radon Wasserstein barycenters of measures. J. Math. Imaging Vis. 51(1), 22–45 (2015)MathSciNetCrossRefMATHGoogle Scholar
  7. 7.
    Bonneel, N., van de Panne, M., Paris, S., Heidrich, W.: Displacement interpolation using Lagrangian mass transport. In: ACM Transactions on Graphics (SIGGRAPH Asia’11), vol. 30(6) (2011)Google Scholar
  8. 8.
    Brown, E., Chan, T.F., Bresson, X.: Completely convex formulation of the Chan–Vese image segmentation model. Int. J. Comput. Vis. 98(1), 103–121 (2012)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Brox, T., Rousson, M., Deriche, R., Weickert, J.: Unsupervised segmentation incorporating colour, texture, and motion. Comput. Anal. Images Patterns 2756, 353–360 (2003)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Brox, T., Weickert, J.: Level set segmentation with multiple regions. IEEE Trans. Image Process. 15(10), 3213–3218 (2006)CrossRefGoogle Scholar
  11. 11.
    Brune, C.: 4D imaging in tomography and optical nanoscopy. PhD thesis, University of Münster, Germany (2010)Google Scholar
  12. 12.
    Chambolle, A.: An algorithm for mean curvature motion. Interfaces Free Boundaries Math. Model. Anal. Computat. 6(2), 195–218 (2004)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Chambolle, A.: An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 20(1), 89–97 (2004)MathSciNetMATHGoogle Scholar
  14. 14.
    Chambolle, A., Darbon, J.: On total variation minimization and surface evolution using parametric maximum flows. Int. J. Comput. Vis. 84(3), 288–307 (2009)CrossRefGoogle Scholar
  15. 15.
    Chambolle, A., Pock, T.: A first-order primal–dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40, 120–145 (2011)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Chambolle, A., Pock, T.: On the ergodic convergence rates of a first-order primal–dual algorithm. J. Math. Program. 159(1–2), 253–287 (2016)Google Scholar
  17. 17.
    Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)CrossRefMATHGoogle Scholar
  18. 18.
    Chizat, L., Schmitzer, B., Peyré, G., Vialard, F.-X.: Unbalanced optimal transport: geometry and Kantorovich formulation. arXiv:1508.05216 (2015)
  19. 19.
    Combettes, P., Condat, L., Pesquet, J.-C., Vu, B.: A forward–backward view of some primal–dual optimization methods in image recovery. In: IEEE International Conference on Image Processing (ICIP’14), pp. 4141–4145 (2014)Google Scholar
  20. 20.
    Condat, L.: Fast projection onto the simplex and the l1 ball. Math. Program. 158(1), 575–585 (2016)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Corless, R., Gonnet, G., Hare, D., Jeffrey, D., Knuth, D.: On the lambert W function. Adv. Comput. Math. 5(1), 329–359 (1996)MathSciNetCrossRefMATHGoogle Scholar
  22. 22.
    Cremers, D., Rousson, M., Deriche, R.: A review of statistical approaches to level set segmentation: integrating color, texture, motion and shape. Int. J. Comput. Vis. 72(2), 215 (2007)CrossRefGoogle Scholar
  23. 23.
    Cuturi, M.: Sinkhorn distances: lightspeed computation of optimal transport. In: Neural Information Processing Systems (NIPS’13), pp. 2292–2300 (2013)Google Scholar
  24. 24.
    Cuturi, M., Doucet, A.: Fast computation of Wasserstein barycenters. In: International Conference on Machine Learning (ICML’14), pp. 685–693 (2014)Google Scholar
  25. 25.
    Cuturi, M., Peyré, G.: A smoothed dual approach for variational Wasserstein problems. SIAM J. Imaging Sci. 9(1), 320–343 (2016)MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Delon, J.: Movie and video scale-time equalization application to flicker reduction. IEEE Trans. Image Process. 15(1), 241–248 (2006)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Digne, J., Cohen-Steiner, D., Alliez, P., de Goes, F., Desbrun, M.: Feature-preserving surface reconstruction and simplification from defect-laden point sets. J. Math. Imaging Vis. 48(2), 369–382 (2014)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Ferradans, S., Papadakis, N., Peyré, G., Aujol, J.-F.: Regularized discrete optimal transport. SIAM J. Imaging Sci. 7(3), 1853–1882 (2014)MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Frogner, C., Zhang, C., Mobahi, H., Araya-Polo, M., Poggio, T.: Learning with a wasserstein loss. arXiv: 1506.05439 (2015)
  30. 30.
    Gorelick, L., Schmidt, F.R., Boykov, Y., Delong, A., Ward, A.: Segmentation with non-linear regional constraints via line-search cuts. In: European Conference on Computer Vision (ECCV’12), pp. 583–597 (2012)Google Scholar
  31. 31.
    Gramfort, A., Peyré, G., Cuturi, M.: Fast optimal transport averaging of neuroimaging data. In: Information Processing in Medical Imaging (IPMI’15) (2015)Google Scholar
  32. 32.
    Haker, S., Zhu, L., Tannenbaum, A., Angenent, S.: Optimal mass transport for registration and warping. Int. J. Comput. Vis. 60(3), 225–240 (2004)CrossRefGoogle Scholar
  33. 33.
    Herbulot, A., Jehan-Besson, S., Duffner, S., Barlaud, M., Aubert, G.: Segmentation of vectorial image features using shape gradients and information measures. J. Math. Imaging Vis. 25(3), 365–386 (2006)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Hochbaum, D., Singh, V.: An efficient algorithm for co-segmentation. In: International Conference on Computer Vision (ICCV’09), pp. 269–276 (2009)Google Scholar
  35. 35.
    Joulin, A., Bach, F., Ponce, J.: Discriminative clustering for image co-segmentation. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR’10), pp. 1943–1950 (2010)Google Scholar
  36. 36.
    Jung, M., Peyré, G., Cohen, L.D.: Texture segmentation via non-local non-parametric active contours. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR’11), pp. 74–88 (2011)Google Scholar
  37. 37.
    Kim, J., Fisher, J.W., Yezzi, A., Cetin, M., Willsky, A.S.: A nonparametric statistical method for image segmentation using information theory and curve evolution. IEEE Trans. Image Process. 14(10), 1486–1502 (2005)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Lellmann, J., Kappes, J., Yuan, J., Becker, F., Schnörr, C.: Convex multi-class image labeling by simplex-constrained total variation. In: Scale Space and Variational Methods in Computer Vision (SSVM’09), pp. 150–162 (2009)Google Scholar
  39. 39.
    Lellmann, J., Lorenz, D.A., Schönlieb, C., Valkonen, T.: Imaging with Kantorovich–Rubinstein discrepancy. SIAM J. Imaging Sci. 7(4), 2833–2859 (2014)MathSciNetCrossRefMATHGoogle Scholar
  40. 40.
    Lombardi, D., Maitre, E.: Eulerian models and algorithms for unbalanced optimal transport. ESAIM Math. Model. Numer. Anal. 49(3), 1717–1744 (2015)MathSciNetCrossRefMATHGoogle Scholar
  41. 41.
    Lorenz, D., Pock, T.: An inertial forward–backward algorithm for monotone inclusions. J. Math. Imaging Vis. 51(2), 311–325 (2015)MathSciNetCrossRefMATHGoogle Scholar
  42. 42.
    Mendoza, C., Perez-Carrasco, J.-A., Saez, A., Acha, B., Serrano, C.: Linearized multidimensional earth-mover’s-distance gradient flows. IEEE Trans. Image Process. 22(12), 5322–5335 (2013)CrossRefGoogle Scholar
  43. 43.
    Mukherjee, L., Singh, V., Dyer, C.R.: Half-integrality based algorithms for cosegmentation of images. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR’09), pp. 2028–2035 (2009)Google Scholar
  44. 44.
    Mumford, D., Shah, J.: Optimal approximation by piecewise smooth functions and associated variational problems. Commun. Pure Appl. Math. 42, 577–685 (1989)MathSciNetCrossRefMATHGoogle Scholar
  45. 45.
    Ni, K., Bresson, X., Chan, T., Esedoglu, S.: Local histogram based segmentation using the Wasserstein distance. Int. J. Comput. Vis. 84(1), 97–111 (2009)CrossRefGoogle Scholar
  46. 46.
    Nikolova, M., Esedoglu, S., Chan, T.F.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM J. Appl. Math. 66(5), 1632–1648 (2006)MathSciNetCrossRefMATHGoogle Scholar
  47. 47.
    Osher, S., Sethian, J.A.: Fronts propagating with curvature-dependent speed: algorithms based on Hamilton–Jacobi formulations. J. Comput. Phys. 79(1), 12–49 (1988)MathSciNetCrossRefMATHGoogle Scholar
  48. 48.
    Papadakis, N., Provenzi, E., Caselles, V.: A variational model for histogram transfer of color images. IEEE Trans. Image Process. 20(6), 1682–1695 (2011)MathSciNetCrossRefGoogle Scholar
  49. 49.
    Papadakis, N., Yıldızoglu, R., Aujol, J.-F., Caselles, V.: High-dimension multi-label problems: convex or non convex relaxation? SIAM J. Imaging Sci. 6(4), 2603–2639 (2013)MathSciNetCrossRefMATHGoogle Scholar
  50. 50.
    Pele, O., Werman, M.: Fast and robust earth mover’s distances. In: IEEE International Conference on Computer Vision (ICCV’09), pp. 460–467 (2009)Google Scholar
  51. 51.
    Peyré, G., Fadili, J., Rabin, J.: Wasserstein active contours. In: IEEE International Conference on Image Processing (ICIP’12) (2012)Google Scholar
  52. 52.
    Pitié, F., Kokaram, A.C., Dahyot, R.: Automated colour grading using colour distribution transfer. Comput. Vis. Image Underst. 107, 123–137 (2007)CrossRefGoogle Scholar
  53. 53.
    Pock, T., Chambolle, A., Cremers, D., Bischof, H.: A convex relaxation approach for computing minimal partitions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR’09), pp. 810–817 (2009)Google Scholar
  54. 54.
    Pock, T., Cremers, D., Bischof, H., Chambolle, A.: Global solutions of variational models with convex regularization. SIAM J. Imaging Sci. 3(4), 1122–1145 (2010)MathSciNetCrossRefMATHGoogle Scholar
  55. 55.
    Punithakumar, K., Yuan, J., Ayed, I.B., Li, S., Boykov, Y.: A convex max-flow approach to distribution-based figure-ground separation. SIAM J. Imaging Sci. 5(4), 1333–1354 (2012)MathSciNetCrossRefMATHGoogle Scholar
  56. 56.
    Rabin, J., Delon, J., Gousseau, Y.: Transportation distances on the circle. J. Math. Imaging Vis. 41(1–2), 147–167 (2011)MathSciNetCrossRefMATHGoogle Scholar
  57. 57.
    Rabin, J., Papadakis, N.: Convex color image segmentation with optimal transport distances. In: Scale Space and Variational Methods in Computer Vision (SSVM’15), pp. 241–252 (2015)Google Scholar
  58. 58.
    Rabin, J., Peyré, G., Delon, J., Bernot, M.: Wasserstein barycenter and its application to texture mixing. In: Scale Space and Variational Methods in Computer Vision (SSVM’11), pp. 435–446 (2012)Google Scholar
  59. 59.
    Ranchin, F., Chambolle, A., Dibos, F.: Total variation minimization and graph cuts for moving objects segmentation. In: Scale Space and Variational Methods in Computer Vision (SSVM’07), pp. 743–753 (2007)Google Scholar
  60. 60.
    Rangarajan, A., Yuille, A., Mjolsness, E.: Convergence properties of the softassign quadratic assignment algorithm. Neural Comput. 11(6), 1455–1474 (1999)CrossRefGoogle Scholar
  61. 61.
    Rother, C., Minka, T., Blake, A., Kolmogorov, V.: Cosegmentation of image pairs by histogram matching-incorporating a global constraint into mrfs. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR’06), vol. 1, pp. 993–1000 (2006)Google Scholar
  62. 62.
    Rother, C., Minka, T., Blake, A., Kolmogorov, V.: Cosegmentation of image pairs by histogram matching-incorporating a global constraint into MRFs. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR’06), pp. 993–1000 (2006)Google Scholar
  63. 63.
    Rousson, M., Brox, T., Deriche, R.: Active unsupervised texture segmentation on a diffusion based feature space. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR’03), pp. 699–704 (2003)Google Scholar
  64. 64.
    Rubio, J., Serrat, J., Lopez, A., Paragios, N.: Unsupervised co-segmentation through region matching. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR’12), pp. 749–756 (2012)Google Scholar
  65. 65.
    Rubner, Y., Tomasi, C., Guibas, L.J.: The earth mover’s distance as a metric for image retrieval. Int. J. Comput. Vis. 40(2), 99–121 (2000)CrossRefMATHGoogle Scholar
  66. 66.
    Schmitzer, B.: A sparse multiscale algorithm for dense optimal transport. J. Math. Imaging Vis. 56(2), 238–259 (2016)Google Scholar
  67. 67.
    Schmitzer, B., Schnörr, C.: Modelling convex shape priors and matching based on the Gromov–Wasserstein distance. J. Math. Imaging Vis. 46(1), 143–159 (2013)MathSciNetCrossRefMATHGoogle Scholar
  68. 68.
    Schmitzer, B., Schnörr, C.: Globally optimal joint image segmentation and shape matching based on Wasserstein modes. J. Math. Imaging Vis. 52(3), 436–458 (2015)MathSciNetCrossRefMATHGoogle Scholar
  69. 69.
    Sinkhorn, R., Knopp, P.: Concerning nonnegative matrices and doubly stochastic matrices. Pac. J. Math. 21(2), 343–348 (1967)MathSciNetCrossRefMATHGoogle Scholar
  70. 70.
    Solomon, J., de Goes, F., Peyré, G., Cuturi, M., Butscher, A., Nguyen, A., Du, T., Guibas, L.: Convolutional wasserstein distances: efficient optimal transportation on geometric domains. In: ACM Transactions on Graphics (SIGGRAPH’15) (2015)Google Scholar
  71. 71.
    Swoboda, P., Schnörr, C.: Variational image segmentation and cosegmentation with the wasserstein distance. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR’13), pp. 321–334 (2013)Google Scholar
  72. 72.
    Vese, L.A., Chan, T.F.: A multiphase level set framework for image segmentation using the Mumford and Shah model. Int. J. Comput. Vis. 50(3), 271–293 (2002)CrossRefMATHGoogle Scholar
  73. 73.
    Vicente, S., Kolmogorov, V., Rother, C.: Joint optimization of segmentation and appearance models. In: IEEE International Conference on Computer Vision (ICCV’09), pp. 755–762 (2009)Google Scholar
  74. 74.
    Vicente, S., Kolmogorov, V., Rother, C.: Cosegmentation revisited: models and optimization. In: European Conference on Computer Visio (ECCV’10), pp. 465–479 (2010)Google Scholar
  75. 75.
    Vicente, S., Rother, C., Kolmogorov, V.: Object cosegmentation. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR’11), pp. 2217–2224 (2011)Google Scholar
  76. 76.
    Villani, C.: Topics in Optimal Transportation. AMS, Providence (2003)CrossRefMATHGoogle Scholar
  77. 77.
    Vu, B.C.: A splitting algorithm for dual monotone inclusions involving cocoercive operators. Adv. Comput. Math. 38(3), 667–681 (2013)MathSciNetCrossRefMATHGoogle Scholar
  78. 78.
    Yıldızoğlu, R., Aujol, J.-F., Papadakis, N.: Active contours without level sets. In: IEEE International Conference on Image Processing (ICIP’12), pp. 2549–2552 (2012)Google Scholar
  79. 79.
    Yıldızoglu, R., Aujol, J.-F., Papadakis, N.: A convex formulation for global histogram based binary segmentation. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR’13), pp. 335–349 (2013)Google Scholar
  80. 80.
    Yuan, Y., Ukwatta, E., Tai, X., Fenster, A., Schnörr, C.: A fast global optimization-based approach to evolving contours with generic shape prior. UCLA Technical Report CAM, pp. 12–38 (2012)Google Scholar
  81. 81.
    Zach, C., Gallup, D., Frahm, J., Niethammer, M.: Fast global labeling for real-time stereo using multiple plane sweeps. In: International Workshop on Vision, Modeling, and Visualization (VMV’08), pp. 243–252 (2008)Google Scholar
  82. 82.
    Zhu, S., Lee, T., Yuille, A.: Region competition: unifying snakes, region growing, energy/bayes/mdl for multi-band image segmentation. In: IEEE International Conference on Computer Vision (ICCV’95), pp. 416–423 (1995)Google Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.IMB, UMR 5251Université de BordeauxTalenceFrance
  2. 2.Normandie Univ, ENSICAEN, CNRS, GREYCCaenFrance

Personalised recommendations