Journal of Mathematical Imaging and Vision

, Volume 56, Issue 3, pp 409–429 | Cite as

Transport Between RGB Images Motivated by Dynamic Optimal Transport

  • Jan Henrik Fitschen
  • Friederike LausEmail author
  • Gabriele Steidl


We propose two models for the interpolation between RGB images based on the dynamic optimal transport model of Benamou and Brenier (Numer Math 84:375–393, 2000). While the application of dynamic optimal transport and its extensions to unbalanced transform were examined for gray-value images in various papers, this is the first attempt to generalize the idea to color images. The non-trivial task to incorporate color into the model is tackled by considering RGB images as three-dimensional arrays, where the transport in the RGB direction is performed in a periodic way. Following the approach of Papadakis et al. (SIAM J Imaging Sci 7:212–238, 2014) for gray-value images we propose two discrete variational models, a constrained and a penalized one which can also handle unbalanced transport. We show that a minimizer of our discrete model exists, but it is not unique for some special initial/final images. For minimizing the resulting functionals we apply a primal-dual algorithm. One step of this algorithm requires the solution of a four-dimensional discretized Poisson equation with various boundary conditions in each dimension. For instance, for the penalized approach we have simultaneously zero, mirror, and periodic boundary conditions. The solution can be computed efficiently using fast Sin-I, Cos-II, and Fourier transforms. Numerical examples demonstrate the meaningfulness of our model.



Funding by the DFG within the Research Training Group 1932 is gratefully acknowledged.


  1. 1.
    Ambrosio, L., Gigli, N., Savaré, G.: Gradient Flows in Metric Spaces and in the Space of Probability Measures. Springer, Heidelberg (2006)zbMATHGoogle Scholar
  2. 2.
    Angenent, S., Haker, S., Tannenbaum, A.: Minimizing flows for the Monge-Kantorovich problem. SIAM J. Math. Anal. 35, 61–97 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Aravkin, A.Y., Burke, J.V., Friedlander, M.P.: Variational properties of value functions. SIAM J. Optim. 23(3), 1689–1717 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Auslender, A., Teboulle, M.: Asymptotic Cones and Functions in Optimization and Variational Inequalities. Springer, New York (2003)zbMATHGoogle Scholar
  5. 5.
    Baiocchi, C., Buttazzo, G., Gastaldi, F., Tomarelli, F.: General existence theorems for unilateral problems in continuum mechanics. Arch. Ration. Mech. Anal. 100(2), 149–189 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Benamou, J.D.: A domain decomposition method for the polar factorization of vector-valued mappings. SIAM J. Numer. Anal. 32(6), 1808–1838 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Benamou, J.-D.: Numerical resolution of an ‘unbalanced’ mass transport problem. ESAIM Math. Model. Numer. Anal. 37(5), 851–868 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Benamou, J.-D., Brenier, Y.: A computational fluid mechanics solution to the Monge-Kantorovich mass transfer problem. Numer. Math. 84(3), 375–393 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Benning, M., Calatroni, L., Düring, B., Schönlieb, C.-B.: A primal-dual approach for a total variation Wasserstein flow. Geometric Science of Information. LNCS, pp. 413–421. Springer, Berlin (2013)CrossRefGoogle Scholar
  10. 10.
    Bertalmio, M.: Image Processing for Cinema. CRC Press, Boca Raton (2014)zbMATHGoogle Scholar
  11. 11.
    Björck, A.: Numerical Methods for Least Squares Problems. SIAM, Philadelphia (1996)CrossRefzbMATHGoogle Scholar
  12. 12.
    Brune, C.: 4D imaging in tomography and optical nanoscopy. Dissertation, PhD Thesis, Münster (Westfalen) University, (2010)Google Scholar
  13. 13.
    Burger, M., Franek, M., Schönlieb, C.-B.: Regularized regression and density estimation based on optimal transport. Appl. Math. Res. eXpress 2012(2), 209–253 (2012)MathSciNetzbMATHGoogle Scholar
  14. 14.
    Burger, M., Sawatzky, A., Steidl, G.: First order algorithms in variational image processing. ArXiv:1412.4237 (2014)
  15. 15.
    Caffarelli, L.A.: A localization property of viscosity solutions to the monge-ampere equation and their strict convexity. Ann. Math. 131(1), 129–134 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Carlier, G., Oberman, A., Oudet, E.: Numerical methods for matching for teams and Wasserstein barycenters. ArXiv:1411.3602 (2014)
  17. 17.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vision 40(1), 120–145 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Chizat, L., Schmitzer, B., Peyré, G., Vialard, F.-X.: An interpolating distance between optimal transport and Fisher-Rao. ArXiv:1506.06430 (2015)
  19. 19.
    Cullen, M.J.: Implicit finite difference methods for modelling discontinouos atmospheric flows. J. Comput. Phys. 81, 319–348 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Cullen, M.J., Purser, R.J.: An extended Lagrangian theory of semigeostrophic frontogenesis. J. Atmos. Sci. 41, 1477–1497 (1989)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Cuturi, M.: Sinkhorn distances: lightspeed computation of optimal transport. Adv. Neural Inf. Process. Syst. 26, 2292–2300 (2013)Google Scholar
  22. 22.
    Cuturi, M., Coucet, A.: Fast computation of Wasserstein barycenters. In: Proceedings of the 31st International Conference on Machine Learning, pp. 685–693 (2014)Google Scholar
  23. 23.
    Dacorogna, B., Maréchal, P.: The role of perspective functions in convexity, polyconvexity, rank-one convexity and separate convexity. J. Convex Anal. 15(2), 271–284 (2008)MathSciNetzbMATHGoogle Scholar
  24. 24.
    Dedieu, J.-P.: Cônes asymptotes d’un ensemble non convexe. Application à l’optimisation. C. R. Acad. Sci. 287, 91–103 (1977)Google Scholar
  25. 25.
    Delon, J.: Midway image equalization. J. Math. Imaging Vision 21(2), 119–134 (2004)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Ferradans, S., Papadakis, N., Peyré, G., Aujol, J.-F.: Regularized discrete optimal transport. SIAM J. Imaging Sci. 7(3), 1853–1882 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Fitschen, J.H., Laus, F., Steidl, G.: Dynamic optimal transport with mixed boundary condition for color image processing. In: International Conference on Sampling Theory and Applications (SampTA). ArXiv:1501.04840, pp. 558–562 (2015)
  28. 28.
    Frogner, C., Zhang, C., Mobahi, H., Araya, M., Poggio, T.A.: Learning with a Wasserstein loss. Adv. Neural Inf. Process. Syst. 28, 2044–2052 (2015)Google Scholar
  29. 29.
    Galerne, B., Gousseau, Y., Morel, J.-M.: Random phase textures: theory and synthesis. IEEE Trans. Image Process. 20(1), 257–267 (2011)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Haber, E., Rehman, T., Tannenbaum, A.: An efficient numerical method for the solution of the \(l_2\) optimal mass transfer problem. SIAM J. Sci. Comput. 32(1), 197–211 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Jimenez, C.: Dynamic formulation of optimal transport problems. J. Convex Anal. 15(3), 593–622 (2008)MathSciNetzbMATHGoogle Scholar
  32. 32.
    Kochengin, S.A., Oliker, V.I.: Determination of reflector surfaces from near-field scattering data. Inverse Probl. 13(2), 363–373 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Maas, J., Rumpf, M., Schönlieb, C., Simon, S.: A generalized model for optimal transport of images including dissipation and density modulation. ArXiv:1504.01988 (2015)
  34. 34.
    Nikolova, M., Steidl, G.: Fast hue and range preserving histogram specification: theory and new algorithms for color image enhancement. IEEE Trans. Image Process. 23(9), 4087–4100 (2014)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Papadakis, N., Peyré, G., Oudet, E.: Optimal transport with proximal splitting. SIAM J. Imaging Sci. 7(1), 212–238 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Patankar, S.: Numerical Heat Transfer and Fluid Flow. CRC Press, New York (1980)Google Scholar
  37. 37.
    Peyré, G., Fadili, J., Rabin, J.: Wasserstein active contours. In: 19th IEEE ICIP, pp. 2541–2544 (2012)Google Scholar
  38. 38.
    Pock, T., Chambolle, A., Cremers, D., Bischof, H.: A convex relaxation approach for computing minimal partitions. In: IEEE Conference Computer Vision and Pattern Recognition, pp. 810–817 (2009)Google Scholar
  39. 39.
    Potts, D., Steidl, G.: Optimal trigonometric preconditioners for nonsymmetric Toeplitz systems. Linear Algebra Appl. 281, 265–292 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Rabin, J., Delon, J., Gousseau, Y.: Transportation distances on the circle. J. Math. Imaging Vision 41(1–2), 147–167 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Rabin, J., Peyré, G., Delon, J., Bernot, M.: Wasserstein barycenter and its application to texture mixing. In: SSVM, pp. 435–446. Springer (2012)Google Scholar
  42. 42.
    Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)CrossRefzbMATHGoogle Scholar
  43. 43.
    Santambrogio, F.: Optimal Transport for Applied Mathematicians. Springer, New York (2015)CrossRefzbMATHGoogle Scholar
  44. 44.
    Schmitzer, B.: A sparse multi-scale algorithm for dense optimal transport. ArXiv:1510.05466 (2015)
  45. 45.
    Schmitzer, B., Schnörr, C.: A hierarchical approach to optimal transport. In: SSVM, pp. 452–464. Springer (2013)Google Scholar
  46. 46.
    Strang, G., MacNamara, S.: Functions of difference matrices are Toeplitz plus Hankel. SIAM Rev. 56, 525–546 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Swoboda, P., Schnörr, C.: Convex variational image restoration with histogram priors. SIAM J. Imaging Sci. 6(3), 1719–1735 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  48. 48.
    Teuber, T., Steidl, G., Chan, R.H.: Minimization and parameter estimation for seminorm regularization models with I-divergence constraints. Inverse Probl. 29, 1–28 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  49. 49.
    Trouvé, A., Younes, L.: Metamorphoses through Lie group action. Found. Comput. Math. 5(2), 173–198 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  50. 50.
    Villani, C.: Optimal Transport: Old and New. Springer, Berlin (2008)zbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Department of MathematicsUniversity of KaiserslauternKaiserslauternGermany

Personalised recommendations