A Linear Optimal Transportation Framework for Quantifying and Visualizing Variations in Sets of Images

Abstract

Transportation-based metrics for comparing images have long been applied to analyze images, especially where one can interpret the pixel intensities (or derived quantities) as a distribution of ‘mass’ that can be transported without strict geometric constraints. Here we describe a new transportation-based framework for analyzing sets of images. More specifically, we describe a new transportation-related distance between pairs of images, which we denote as linear optimal transportation (LOT). The LOT can be used directly on pixel intensities, and is based on a linearized version of the Kantorovich-Wasserstein metric (an optimal transportation distance, as is the earth mover’s distance). The new framework is especially well suited for computing all pairwise distances for a large database of images efficiently, and thus it can be used for pattern recognition in sets of images. In addition, the new LOT framework also allows for an isometric linear embedding, greatly facilitating the ability to visualize discriminant information in different classes of images. We demonstrate the application of the framework to several tasks such as discriminating nuclear chromatin patterns in cancer cells, decoding differences in facial expressions, galaxy morphologies, as well as sub cellular protein distributions.

Appendix: Particle Approximation Algorithm

Here we present the details of the particle approximation algorithm outlined in Sect. 3.1. The goal of the algorithm is to approximate the given probability measure, μ on Ω by a particle measure with approximately N particles, where N is given. For images where the estimated error of the initial approximation is much larger than the typical error over the given dataset the number of particles is increased to reduce the error, while for images where the error is much smaller than typical the number of particles is reduced in order to save time when computing the OT distance. The precise criterion and implementation of adjusting the number of particles is described in step 8 below.

In our application the measure μ represents the image. It too can be thought as a particle measure particle measure itself $\mu= \sum_{j=1}^{N_{\mu}} m_{j} \delta_{y_{j}}$ where N _μ is the number of pixels, y _j the coordinates of the pixel centers, and m _j the intensity values. Our goal however is to approximate it with a particle measure with a lot fewer particles. Below we use the fact that the restriction of the measure μ to a set V is defined as $\mu |_{V} = \sum_{i=1, x_{i} \in V }^{N_{\mu}} m_{i} \delta_{x_{i}}$. The backbone of the algorithm rests on the following mathematical observations.

Observation 1. The best approximation for fixed particle locations

Assume that x ₁,…,x _N are fixed. Consider the Voronoi diagram with centers x ₁,…,x _N . Let V _i be the cell of the Voronoi diagram that correspond to the center x _i . Then among all probability measures of the form $\mu= \sum_{i=1}^{N}m_{i}\delta_{x_{i}}$ the one that approximates μ the best is

(14)

Let us remark that if μ is a particle measure as above then μ(V _i ) is just the sum of intensities of all pixels whose centers lie in V _i .

Given a measure μ supported on Ω we define the center of mass to be

$$ x(\mu) = \frac{1}{\mu(\varOmega)} \int_\varOmega x d\mu. $$

(15)

Observation 2. The Best Local ‘Recentering’ of Particles

It is easy to prove that among all one-particle measures the one that approximates μ the best is the one set at the center of mass of μ:

$$\mu(\varOmega)\delta_{x(\mu)} = \operatorname{\mathop{\operatorname{argmin}}}\bigl\{ d_W\bigl(\mu, \mu(\varOmega) \delta_y\bigr) : y \in\mathbb{R}^n \bigr\}. $$

We can apply this to ‘recenter’ the delta measures in each Voronoi cell:

Here $\mu|_{V_{i}}$ is the restriction of measure μ to the set V _i . More precisely, for A⊂Ω, $\mu|_{V_{i}}(A) = \mu(A \cap V_{i})$.

Observation 3. The Error for the Current Approximation is Easy to Estimate

Given a particle approximation as above one can compute a good upper bound on the error. In particular

(16)

If $\mu= \sum_{j=1}^{N_{\mu}} m_{j} \delta_{y_{j}}$ the upper bound on the right-hand side becomes $\sum_{i=1}^{N} \sum_{j = 1, y_{j} \in V_{i}}^{N_{\mu}} m_{j} |y_{j} - \bar{x}_{i}|^{2}$ where $\bar{x}_{i} = (\sum_{j=1, y_{j} \in V_{i}}^{N_{\mu}} m_{j} y_{j} ) / \sum_{j=1, y_{j} \in V_{i}}^{N_{\mu}} m_{j}$. We should also note that the estimate on the right-hand side gives us very useful information on ‘local’ error in each cell V _i . This enables us to determine which cells need to be refined when needed.

Based on these observations, we use the following ‘weighted Lloyd’ algorithm to produce a particle approximation to measure μ. The idea is to use enough particles to approximate each image well (according to the transport criterion), but not more than necessary given a chosen accuracy. The steps of our algorithm are:

• 1. Distribute N particles (with N an arbitrarily chosen number), x ₁,…x _N , over the domain Ω by weighted random sampling (without replacement) the measure μ with respect to the intensity values of the image.

• 2. Compute the Voronoi diagram for centers x _i ,…,x _k . Based on Observation 1, set m _i =μ(V _i ). The measure $\mu_{N}= \sum_{i=1}^{N} \mu(V_{i}) \, \delta_{x_{i}}$ is the current approximation.

• 3. Using Observation 2, we recenter the cells by setting $x_{i,\mathit{new}} = x(\mu|_{V_{i}})$ (the center of mass of μ restricted to cell V _i ).

• 4. Repeat steps 2 and 3 until the algorithm stabilizes. (the change of error upper bound is less than 0.5 % in the sequential steps).

• 5. Repeat the steps 1 to 4 10 times for each image and choose the approximation with lowest error.

• 6. We then seek to improve the approximation in regions in the image which have not been well approximated by the step above. We do so by introducing more particles in the Voronoi cells V _i where the cost of transporting the intensities in that cell to its center of mass exceed a certain threshold. To do so, we compute the transportation error $\mathit{err}_{i} = d_{W}(\mu|_{V_{i}}, \mu(V_{i}) \delta_{x_{i}})$ in cell i. Let err be the average transportation error. We add particles to all cells where err _i >1.7err. More precisely if there are any cells where err _i >1.7err then we split the cell with the largest error into two cells V _i1,V _i2, and then recompute the Voronoi diagram to determine the centers m _i1=μ(V _i1),m _i2=μ(V _i2) of these two new cells. We repeat this splitting process until all the Voronoi cells have error less than 1.7err _i . The choice of 1.7 was based on emperical evaluation with several datasets.

• 7. In order to reduce the number of particles in areas in each image where many particles are not needed, we merge nearby particles. Two particles are merged if, by merging them, the theoretical upper bound defined in (16) is lowered. When particles of mass m ₁ and m ₂ and locations x ₁ and x ₂ are merged to a single particle of mass m=m ₁+m ₂ located at the center of mass $x = \frac{m_{1} x_{1} + m_{2} x_{2}}{m_{1}+m_{2}}$ the error in the distance squared is bounded by:

  

  We merge the particles that add the smallest error first and then continue merging until d _W (μ _N ,μ _merged )<0.15d _W (μ _N ,μ). This has the overall effect of shifting the histogram (taken over all images) of the approximation error to the right, where the inequality is used in the sense of the available upper bound on the distance (Observation 3 and the estimate above) and not the actual distance. This ensures that the final error for each image after merging is around 1.15d _W (μ _N ,μ).

• 8. Finally, while the two steps above seek to find a good particle approximation for each image, with N particles or more, the final step is designed to make the particle approximation error more uniform for all images in a given dataset. For a set of images I ₁,I ₂,…,I _M , we estimate the average transportation error, E _avg , between each digital image and its approximation, as well as standard deviation τ of the errors. We set E _small =E _avg −0.5∗τ and E _big =E _avg +0.5∗τ.

  • For images that have bigger error than E _big , we split the particles as in Step 6 until the error for those images are less than E _big .

  • For images that have smaller error than E _small , we merge nearby particles instead as in Step 7. The procedure we apply is to merge the particles that add the smallest error first and then continue merging until d _W (μ,μ _merged )≥E _small .