Adaptive Anisotropic Total Variation: Analysis and Experimental Findings of Nonlinear Spectral Properties

Abstract

Our aim is to explain and characterize the behavior of adaptive total variation (TV) regularization. TV has been widely used as an edge-preserving regularizer. However, objects are often over-regularized by TV, becoming blob-like convex structures of low curvature. This phenomenon was explained mathematically in the analysis of Andreau et al. They have shown that a TV regularizer can spatially preserve perfectly sets which are nonlinear eigenfunctions of the form $\lambda u \in \partial J_{TV}(u)$, where $\partial J_{TV}(u)$ is the TV subdifferential. For TV, these shapes are indeed convex sets of low curvature. A compelling approach to better preserve structures is to use adaptive anisotropic functionals, which adapt the regularization in an image-driven manner, with strong regularization along edges and low across them. This follows the seminal work of Weickert on anisotropic diffusion. Adaptive anisotropic TV (A$^2$TV) was successfully used in several studies in the past decade. However, there is little analysis of the type of structures which can be well preserved. In this study, we address this question by a joint methodology of mathematical derivations and experiments. We rely on a recently developed theory of Burger et al on nonlinear spectral analysis of one-homogeneous functionals. We have that eigenfunction sets, admitting $\lambda u \in \partial J_{A^2TV}(u)$, are perfectly preserved under A$^2$TV flow or minimization with $L^2$ square fidelity. We thus investigate these eigenfunctions theoretically and numerically. We prove non-convex sets can be eigenfunctions in certain conditions and provide numerical results which characterize well the relations between the degree of local anisotropy of the functional and the admitted maximal curvature. A nonlinear spectral representation is formulated, where shapes are well preserved and can be manipulated effectively. Finally, examples of possible applications related to shape manipulation and guided regularization of medical and depth data are shown.

Appendices

Appendices

A$^2$TV Curvature Condition

We would like to find the relation of the maximal curvature on the set boundary to the degree of anisotropy (the parameter $a \in (0,1]$). Let us first discuss the extreme cases. For $a=1$, we get isotropic TV; therefore, the bound should be as in Eq. (14). As a decreases, the regularization diminishes and we expect the bound to grow as $a \rightarrow 0$.

We now turn to examining more closely the dual variable $\xi ^A$ from the A$^2$TV functional weak sense definition Eq. (20). As $\xi ^A$ is coupled inside ${{\,\mathrm{div}\,}}_A$, we would like to separate the influence of the adapted anisotropic matrix A from the ${{\,\mathrm{div}\,}}$; therefore, we define

$$\begin{aligned} \tilde{\xi ^A} := A\xi ^A, \end{aligned}$$

(62)

In Fig. 14, one can see the differences between the vector fields $\tilde{\xi ^A}$ and $\xi ^A$. Note especially the smoother behavior of $\tilde{\xi ^A}$ near edges.

Fig. 14

Comparison between the vector fields $\xi ^A$ (defined by Eq. (20)) and $\tilde{\xi ^A}$, Eq. (62), (a=0.5). From left: indicator function of a non-convex set C, color-coded vector field $\xi ^A$ and the cross-sectional vertical line (in red), cross sections of $\xi ^A$ and $\tilde{\xi ^A}$. Whereas $\xi ^A$ is not smooth near the edge, $\tilde{\xi ^A}$ is smooth (Color figure online)

Let us compare the TV Definition 2 and the A$^2$TV Definition 4. For eigenfunctions, $\xi$ for TV and $\tilde{\xi ^A}$ for A$^2$TV, should have a constant divergence. We denote both vector fields by $\xi$. In two dimensions, a vector field has a constant divergence when its components are of the form,

$$\begin{aligned} \Big \{ \begin{array}{lcl} \xi _x =&{} R(x)R'(y) + t_1x \\ \xi _y =&{} -R'(x)R(y) + t_2y, \end{array} \end{aligned}$$

(63)

for some differentiable function $R(\cdot )$. Clearly, we get by construction a constant divergence: $\text {div}(\xi ) = t_1+ t_2$. For total variation the function $R(\cdot )$ is identically zero (for a disk of radius r we get $R(\cdot )=0, t_1=t_2=\frac{1}{r}$). However, our numerical observations indicate it is a more complex function for A$^2$TV. For example, in the case of ellipses, $\tilde{\xi ^A}$ has a structure similar to a third-degree polynomial (as shown in Fig. 16).

Several numerical experiments were conducted using ellipse shapes (as in Fig. 15) with various eccentricities and A$^2$TV regularization with various degrees of anisotropy parameter (a). An ellipse shape is used in order to decouple the convexity and the curvature measures. Moreover, it is simple to estimate numerically its curvature.

Fig. 15

Basic ellipse shape

Fig. 16

A$^2$TV ellipse $\tilde{\xi ^A}$ cross sections—A numerical experiment to assess the behavior of $\tilde{\xi ^A}$ as a function of a. The parameters of the ellipse are: $Ra = 100, Rb = 20$

Fig. 17

A$^2$TV ellipse $\tilde{\xi ^A}$ cross sections—An experiment to assess the behavior of $\tilde{\xi ^A}$ as a function of the ratio $r = \frac{R_b}{R_a}$. Two cross sections of $\tilde{\xi ^A}$ along the axes are shown. ($a=0.5$, $R_a = 200$)

Fig. 18

Regularization by A$^2$TV of ellipses—a numerical experiment to assess the behavior of the function $f_{cr}(a)$. Here, numerous A$^2$TV tests were performed with anisotropic parameter $a\in (0.4,1]$, radii ratios of $R_b/R_a = (0.2,1]$, $R_a = 200$

For an ellipse, as in Fig. 15, $R_a\ge R_b$, the maximum curvature, located at $(x,y)=(R_a,0)$, is

$$\begin{aligned} \kappa _\mathrm{max} = R_a/R_b^2. \end{aligned}$$

(64)

Our numerical experiments are based on the schemes detailed in Sect. 5. We compute the vector field $\tilde{\xi ^A}=(\tilde{\xi ^A_x},\tilde{\xi ^A_y})$, Eq. (62), based on the dual variable $\xi ^A$ for a given function u (see the definition of A$^2$TV, Eq. (20)). Since the vector field has two components (in 2D), we show cross sections of both $\tilde{\xi ^A_x}$ and $\tilde{\xi ^A_y}$. The first experiment is done by applying A$^2$TV on a single ellipse, $Ra = 100, Rb = 20$, with various values of the anisotropy parameter a. The resultant $\tilde{\xi ^A}$ cross sections along the x and y axes are shown in Fig. 16. One can observe that on the minor axis $\tilde{\xi ^A_y}$ has a linear solution just as in the TV case (due to its low curvature). The major axis, on the other hand, is not a linear function. The curvature on the boundary is maximal and $\tilde{\xi ^A}$ is not monotone inside the set anymore. For $a>0.5$, the anisotropy of A$^2$TV is not sufficient and one does not obtain an eigenfunction, this can be seen by the clipping of $\tilde{\xi ^A_x}$ near the value of 1. For $a\le 0.5$, we get an eigenfunction with ${{\,\mathrm{div}\,}}(\tilde{\xi ^A}) = const$, here $\tilde{\xi ^A_x}$ resembles a third-degree polynomial.

In Fig. 17, we present results of another numerical experiment. Now, a fixed anisotropic parameter $a=0.5$ is used and ellipses with various radii ratios ($r = \frac{R_b}{R_a}$) are tested. When the eccentricity of the ellipse is low ($\frac{R_b}{R_a}$ closer to 1) there is a linear profile for both axes. Actually these ellipses are eigenfunction also of TV (ratios of $r\in (0.6,1]$). As the eccentricity grows with $r\in (0.15,0.5]$, we obtain eigenfunctions unique to A$^2$TV (with $a=0.5$). Finally, we have a very narrow ellipse, $r = 0.1$, which is not an eigenfunction, where the vector field is clipped.

These experiments show a distinct systematic trend relating the parameter a to the set of ellipse eigenfunctions, which is summarized in Fig. 2. As a becomes smaller, ellipses of higher eccentricity (and curvature) become admissible eigenfunctions.

In order to estimate numerically a curvature condition for ellipses, we performed the following experiment, with the results depicted in Fig. 18. Multiple A$^2$TV flows (with different a parameter) were applied to ellipses of different eccentricities ($R_b/R_a \in (0.2,1]$).

In Fig. 18, we can see two plots. The left plot shows the eigenfunction test scores (lower is better) based on the following formula,

$$\begin{aligned} T(u) = \frac{u(0,0) - u(R_a-\epsilon ,0)}{u(0,0)}, \end{aligned}$$

(65)

where u is an ellipse image after applying a A$^2$TV flow at $t = \frac{1}{10\lambda ^A}$ where $\lambda ^A$ is taken from Eq. (35). The test score, typically in the range $T(u)\in [0,1]$, was chosen due to its numerical robustness, as oppose to indicators which take into account the entire image, and are biased by numerical errors. We rely on the fundamental property of indicator set eigenfunctions which retain a constant value within the set. The time of the flow chosen for the computation is 10% of the approximated extinction time. The value of $\epsilon = 0.05 \cdot R_a$ was used (for $R_a=200$, $\epsilon = 10$).

On the left, it can be observed that as the ratio tends to 1 (toward a disk), the shape attains a better eigenfunction score. The bar with the value 0.0017 was chosen so that the ratio 0.6 is the last one to be an eigenfunction for the TV functional (which is the theoretical threshold for TV). From the graph on the left, critical thresholds for a, $a_{cr}$ values were extracted with the rule of choosing a which gives the same test score T(u) for each ellipse ratio. The right plot shows $f_{cr}$, which represents a hypothesis that the bound for an A$^2$TV eigenfunction in the form of an ellipse has a curvature bound $\kappa _\mathrm{max}$, where

$$\begin{aligned} f_{cr} = \max \left( \kappa _\mathrm{max}/\frac{P(C)}{|C|},1\right) , \end{aligned}$$

(66)

and the set C represent an ellipse with a major axis $R_a$ and a minor axis $R_b$ as shown in Fig. 15. For each ellipse ratio, we took the critical $a_{cr}$ and applied it as a function of $f_{cr,Exp}$,

$$\begin{aligned} f_{cr,Exp}(a_{cr}) = \frac{1}{a_{cr}^3}. \end{aligned}$$

Both images in Fig. 18 present a strong indication that the upper bound dependence on the parameter a is $\frac{1}{a^3}$. Note that this is the case for ellipses, it is hard to validate whether curvature and convexity/shape structures can be completely decoupled. We summarize this relation in Conjecture 1.

A$^2$TV Disk Example

We define the following function,

$$\begin{aligned} f(X) = \Bigg \{ \begin{array}{lc} hc_0 &{} |X|\le R\\ h(c_0-1) &{} R\le |X|\le R_0 \end{array}. \end{aligned}$$

(67)

The shape of f is basically a disk of radius R and height h, within a circular domain $\varOmega$ of radius $R_0$. Here, $c_0=1-R^2/R_0^2$ was chosen so the mean value is ${\bar{f}}=0$. We define the following matrix A as in (25), with $\partial C: |X|=R$. Eq. (21) can be written as,

$$\begin{aligned} v_1= & {} \frac{X}{|X|} = \frac{X}{R} = \frac{1}{R} \begin{pmatrix} x \\ y \end{pmatrix}, \qquad v_1 \perp v_2 \Rightarrow v_2 = \frac{1}{R} \begin{pmatrix} y \\ -x \end{pmatrix}\\ V= & {} [v_1;v_2] = \frac{1}{R} \begin{pmatrix} x &{}y \\ y &{}-x \end{pmatrix},\\ A= & {} V^T \varLambda V = \frac{1}{R}\begin{pmatrix} x &{}y \\ y &{}-x \end{pmatrix}\begin{pmatrix} a &{}0 \\ 0 &{}1 \end{pmatrix}\frac{1}{R}\begin{pmatrix} x &{}y \\ y &{}-x \end{pmatrix}\\= & {} \frac{1}{R^2}\begin{pmatrix} ax^2 + y^2 &{}axy-xy \\ axy-xy &{}ay^2+x^2 \end{pmatrix} . \end{aligned}$$

In this eigenproblem, the eigenvalue of $v_1$ is a and of $v_2$ is 1. Let us assume a certain vector field $\xi ^A$ and show it is a calibrable set. On the disk boundary $\partial C$, we require,

$$\begin{aligned} \xi ^A|_{\partial C} = \frac{1}{R}\begin{pmatrix} x \\ y \end{pmatrix} = \frac{X}{R} . \end{aligned}$$

The divergence operator is applied on $A\xi ^A$; therefore, we impose continuity to get,

$$\begin{aligned} \xi ^A = {\left\{ \begin{array}{ll} a\frac{X}{R} &{}, |X|<R,\\ \frac{X}{R} &{}, |X|=R,\\ a\frac{R}{c_0}\left( \frac{1}{|X|^2}-\frac{1}{R_0^2}\right) X &{}, R_0>|X|>R.\\ \end{array}\right. } \end{aligned}$$

(68)

We compute $p = {{\,\mathrm{div}\,}}_A \xi ^A = {{\,\mathrm{div}\,}}(A\xi ^A).$ Thus, we obtain,

$$\begin{aligned} p = {\left\{ \begin{array}{ll} \frac{2a}{R} &{}, |X| \le R,\\ -a\frac{2R}{c_0R_0^2} &{}, R_0>|X|>R.\\ \end{array}\right. } \end{aligned}$$

(69)

One can validate the solution admits $\Vert \xi ^A\Vert _\infty \le 1$:

$$\begin{aligned}&\Vert \xi ^A\Vert _\infty \nonumber \\&\quad = {\left\{ \begin{array}{ll} \left\Vert a\frac{X}{R}\right\Vert _\infty< a \le 1 &{}, |X|<R\\ 1 &{}, |X|=R\\ \left\Vert a\frac{R}{c_0}\left( \frac{1}{r^2}-\frac{1}{R_0^2}\right) X\right\Vert _\infty \underbrace{<}_* a \le 1 &{}, R_0>r=|X|>R\\ \end{array}\right. } \nonumber \\&\quad \Rightarrow \Vert \xi ^A\Vert _\infty \le 1 .\nonumber \\&\quad {*} = \left\Vert a\frac{R}{c_0}\left( \frac{1}{r^2}-\frac{1}{R_0^2}\right) X\right\Vert _\infty = \left\Vert a\frac{R}{r^2}\underbrace{\left( \frac{R_0^2-r^2}{R_0^2-R^2}\right) }_{<1}X\right\Vert _\infty \nonumber \\&\quad< \left\Vert a\frac{R}{r}\frac{X}{r}\right\Vert _\infty = a\underbrace{\frac{R}{r}}_{<1}\underbrace{\left\Vert \frac{X}{r}\right\Vert _\infty }_{=1}<a. \end{aligned}$$

(70)

The eigenvalue can be computed by

$$\begin{aligned} \begin{aligned} J_{A^2TV}(f)&= \langle p,f\rangle = \langle \lambda f,f\rangle \Rightarrow \lambda = \frac{J_{A^2TV}(f)}{\left\Vert f\right\Vert ^2} = \frac{ah2\pi R}{h^2\pi R^2} \\&= a\frac{2}{hR}. \end{aligned} \end{aligned}$$