From Mahalanobis to Bregman via Monge and Kantorovich

Abstract

In his celebrated 1936 paper on “the generalized distance in statistics,” P.C. Mahalanobis pioneered the idea that, when defined over a space equipped with some probability measure P, a meaningful distance should be P-specific, with data-driven empirical counterpart. The so-called Mahalanobis distance and the related Mahalanobis outlyingness achieve this objective in the case of a Gaussian P by mapping P to the spherical standard Gaussian, via a transformation based on second-order moments which appears to be an optimal transport in the Monge-Kantorovich sense. In a non-Gaussian context, though, one may feel that second-order moments are not informative enough, or inappropriate; moreover, they might not exist. We therefore propose a distance that fully takes the underlying P into account—not just its second-order features—by considering the potential that characterizes the optimal transport mapping P to the uniform over the unit ball, along with a symmetrized version of the corresponding Bregman divergence.

Appendix. Measure Transportation in a Nutshell

This appendix provides a very elementary presentation of the measure transportation results used in this paper. It can be skipped by readers familiar with the subject.

Starting from very practical road construction problems, Gaspard Monge (1746–1818), in his 1781 Mémoire sur la Théorie des Déblais et des Remblais, probably did not realize that he was initiating a profound mathematical theory anticipating different areas of differential geometry, linear programming, nonlinear partial differential equations, and probability. Monge’s Mémoire indeed was motivated by a very terre à terre issue, related with road construction activities: how do you best move a given pile of sand to fill up a given hole of the same total volume?

The simplest and most intuitive abstract formulation of Monge’s problem is as follows. Let P₁ and P₂ denote two probability measures over (for simplicity) $(\mathbb {R}^{d}, \mathcal {B}^{d})$. Let $L : \mathbb {R}^{2d} \to [0, \infty ]$ be a Borel-measurable loss function: L(x₁,x₂) represents the cost of transporting x₁ to x₂. Monge’s problem is an optimization problem: find a measurable transport map $T_{\mathrm {P}_{1} ; \mathrm {P}_{2}} : \mathbb {R}^{d}\to \mathbb {R}^{d}$ that achieves the infimum

$$\inf_{T}{\int}_{\mathbb{R}^{d}} L(\textbf{x}, T(\mathbf{x}))\text{dP}_{1} \qquad \text{subject to} \quad T\# \mathrm{P}_{1} = \mathrm{P}_{2}$$

where T# P₁ denotes the “push forward of P₁ by T”—more classical statistical notation for this would be $\mathrm {P}_{1}^{T\textbf {X}} = \mathrm {P}_{2}$. A map $T_{\mathrm {P}_{1} ; \mathrm {P}_{2}}$ that attains this infimum is called an “optimal transport map”, in short, an “optimal transport”, of P₁ into P₂. In the sequel, we restrict to the L² loss function $L(\textbf {x}_{1},\textbf {x}_{2})=\Vert \textbf {x}_{1}- \textbf {x}_{2} {\Vert ^{2}_{2}}$; Monge was considering the more difficult Euclidean distance loss L(x₁,x₂) = ∥x₁ −x₂∥₂.

The problem looks simple, but it is not (leading to nonlinear partial differential equations of the Monge-Ampère type). Monge himself could not solve it, and relatively little progress was made until the 1940s, when renewed interest in the topic was triggered by the contributions of Leonid Vitalievitch Kantorovich (1912-1986; Nobel Prize in Economics in 1975). The fundamental idea behind Kantorovich’s approach consists in relaxing Monge’s problem into the more general one of constructing a distribution $\gamma _{\mathrm {P}_{1}\mathrm {P}_{2}}$ on $\mathbb {R}^{d}\times \mathbb {R}^{d}$ minimizing

$${\int}_{(\textbf{x},\textbf{y})\in\mathbb{R}^{d}\times \mathbb{R}^{d}} \Vert \textbf{x}-\textbf{y}\Vert^{2} d\gamma $$

(equivalently, maximizing $\int \langle \textbf {x},\textbf {y}\rangle d\gamma$) among all distributions γ with marginals P₁ and P₂. This formulation allows for a groundbreaking duality approach, with solutions of the form

$$(\text{identity}\times T_{\mathrm{P}_{1} ; \mathrm{P}_{2}})\#\mathrm{P}_{1} $$

where $(\text {identity}\times T_{\mathrm {P}_{1} ; \mathrm {P}_{2}})$ is mapping x to $(\textbf {x}, T_{\mathrm {P}_{1} ; \mathrm {P}_{2}}(\textbf {x}))$—so that $T_{\mathrm {P}_{1} ; \mathrm {P}_{2}}$ is indeed a solution of Monge’s problem.

Among the most powerful ensuing results is the Polar Factorization Theorem by Brenier (1987, 1991; see Chapter 3 in Villani (2003)) which implies, among other things, that for L² loss, if P₁ and P₂ are absolutely continuous with finite second-order moments, the solution $T_{\mathrm {P}_{1} ; \mathrm {P}_{2}}$ of Monge’s problem exists, is (a.e.) unique, and the gradient ∇ψ of some convex (potential) function ψ—a form of multivariate monotonicity.

All this, including Monge’s problem, only makes sense under finite moments of order two, though. Brenier’s results were further enhanced by a most remarkable theorem by Robert J. McCann, who had the intuition that the problem was of a geometric rather than analytical nature. His main result (McCann, 1995) implies that, for any given (absolutely continuous—no second-order moments needed) P₁ and P₂, there exists a P₁-essentially unique element in the class of gradients of convex functions mapping P₁ to P₂. Under the existence of finite moments of order two, that mapping moreover coincides with the L²-optimal (in the Monge-Kantorovich-Brenier sense) transport of P₁ to P₂.

The subject ever since has been a very active domain of mathematical analysis, with applications in various fields, from fluid mechanics to economics (see Galichon (2016)), learning, and statistics (Carlier and Galichon (2016); Panaretos and Zemel (2016, 2018)). It was popularized recently by the French Fields medalist Cédric Villani, with two monographs (Villani 2003, 2009), where we refer to for background reading, along with the two volumes by Rachev and Rüschendorf (1998), where the scope is somewhat closer to probabilistic and statistical concerns.