Non-convex Total Variation Regularization for Convex Denoising of Signals

Abstract

Total variation (TV) signal denoising is a popular nonlinear filtering method to estimate piecewise constant signals corrupted by additive white Gaussian noise. Following a ‘convex non-convex’ strategy, recent papers have introduced non-convex regularizers for signal denoising that preserve the convexity of the cost function to be minimized. In this paper, we propose a non-convex TV regularizer, defined using concepts from convex analysis, that unifies, generalizes, and improves upon these regularizers. In particular, we use the generalized Moreau envelope which, unlike the usual Moreau envelope, incorporates a matrix parameter. We describe a novel approach to set the matrix parameter which is essential for realizing the improvement we demonstrate. Additionally, we describe a new set of algorithms for non-convex TV denoising that elucidate the relationship among them and which build upon fast exact algorithms for classical TV denoising.

Appendix

In this Appendix, we present technical results and their proofs, which are needed for the main results of the paper.

Lemma 2

Let $y \in \mathbb {R}^N$ and ${\lambda } > 0$. Let $f \in \Gamma _0(\mathbb {R}^N)$ and $B \in \mathbb {R}^{M \times N}$. Define $g :\mathbb {R}^N \rightarrow \mathbb {R}$ as

$$\begin{aligned} g(x) = \tfrac{1}{2}\Vert y - x \Vert _2^2 - {\lambda } f^{\mathsf {M}}_{ B } (x) \end{aligned}$$

(81)

where $f^{\mathsf {M}}_B$ is the generalized Moreau envelope of f. If $B^{\mathsf {T}}\! B \preccurlyeq (1 / {\lambda } ) I$, then g is convex. If $B^{\mathsf {T}}\! B \prec (1 / {\lambda } ) I$, then g is strongly convex.

Proof

We write

$$\begin{aligned} g(x)&= \tfrac{1}{2}\Vert y - x \Vert _2^2 - {\lambda } \inf _{ v \in \mathbb {R}^N } \bigl \{ f(v) + \tfrac{1}{2}\Vert B ( x - v ) \Vert _2^2 \bigr \} \nonumber \\&= \tfrac{1}{2}\Vert y - x \Vert _2^2 - \tfrac{ {\lambda } }{2} \Vert B x \Vert _2^2\nonumber \\&\quad - {\lambda } \inf _{ v \in \mathbb {R}^N } \bigl \{ f(v) - v^{\mathsf {T}}\! B^{\mathsf {T}}\! B x + \tfrac{1}{2}\Vert B v \Vert _2^2 \bigr \} \end{aligned}$$

(82)

$$\begin{aligned}&= \tfrac{1}{2}x^{\mathsf {T}}\! (I - {\lambda } B^{\mathsf {T}}\! B) x + \tfrac{1}{2}\Vert y \Vert _2^2 - y^{\mathsf {T}}\! x\nonumber \\&\quad + {\lambda } \sup _{ v \in \mathbb {R}^N } \bigl \{ -f(v) + v^{\mathsf {T}}\! B^{\mathsf {T}}\! B x - \tfrac{1}{2}\Vert B v \Vert _2^2 \bigr \}. \end{aligned}$$

(83)

The function in the curly braces is affine in x (hence convex in x). Since the supremum of a family of convex functions (here indexed by v) is itself convex, the final term of (83) is convex in x. Hence, g is convex if $I - {\lambda } B^{\mathsf {T}}\! B$ is positive semidefinite; and g is strongly convex if $I - {\lambda } B^{\mathsf {T}}\! B$ is positive definite. $\square$

Lemma 3

In the context of Lemma 2, let $e_{\max }$ denote the maximum eigenvalue of $B^{\mathsf {T}}\! B$. If $B^{\mathsf {T}}\! B \prec (1 / {\lambda } ) I$ (that is, $e_{\max } < 1/{\lambda }$), then g in (81) is $\delta$-strongly convex with (positive) modulus of strong convexity (at least) equal to

$$\begin{aligned} \delta = 1 - {\lambda } e_{\max }. \end{aligned}$$

(84)

Proof

It follows from Definition 3 that the function g in (81) is $\delta$-strongly convex if and only if the function ${\widetilde{g}}$, defined by

$$\begin{aligned} {\widetilde{g}}(x)&= g(x) - \frac{\delta }{2} \Vert x \Vert _2^2 \end{aligned}$$

(85)

$$\begin{aligned}&= \tfrac{1}{2}x^{\mathsf {T}}\! ((1- \delta )I - {\lambda } B^{\mathsf {T}}\! B) x + \tfrac{1}{2}\Vert y \Vert _2^2 - y^{\mathsf {T}}\! x \nonumber \\&\quad + {\lambda } \sup _{ v \in \mathbb {R}^N } \bigl \{ -f(v) + v^{\mathsf {T}}\! B^{\mathsf {T}}\! B x - \tfrac{1}{2}\Vert B v \Vert _2^2 \bigr \}, \end{aligned}$$

(86)

is convex. Hence, ${\widetilde{g}}$ in (86) is convex if $(1- \delta )I - {\lambda } B^{\mathsf {T}}\! B$ is positive semidefinite. Let $e_i$ be the real nonnegative eigenvalues of $B^{\mathsf {T}}B$. We have

$$\begin{aligned}&(1- \delta )I - {\lambda } B^{\mathsf {T}}\!B \succcurlyeq 0\\&\quad \iff 1-\delta - {\lambda } e_i \geqslant 0, \ \forall \, i \in \{1,2,\ldots ,N\} \\&\quad \iff \delta \leqslant \min _i \left\{ 1 - {\lambda } e_i \right\} . \\&\quad \iff \delta \leqslant 1 - {\lambda } e_{\max } \end{aligned}$$

which completes the proof. $\square$

In this paper, we use the forward-backward splitting (FBS) algorithm which entails a constant of Lipschitz continuity. The following two lemmas regard Lipschitz continuity. Lemma 4 is a part [equivalence (i) $\Leftrightarrow$ (vi)] of Theorem 18.15 of Ref. [1]. Our use of this result follows the reasoning of Ref. [2].

Lemma 4

Let $f :\mathbb {R}^N \rightarrow \mathbb {R}$ be convex and differentiable. Then the gradient $\nabla f$ is $\rho$-Lipschitz continuous if and only if $(\rho /2) \Vert {\,\cdot \,} \Vert _2^2 - f$ is convex.

Lemma 5

Let $y \in \mathbb {R}^N$ and ${\lambda } > 0$. Let $B = C D \in \mathbb {R}^{M \times N}$ with $B^{\mathsf {T}}\! B \preccurlyeq (1 / {\lambda } ) I$. Define $f :\mathbb {R}^N \rightarrow \mathbb {R}$ as

$$\begin{aligned} f(x) = \tfrac{1}{2}\Vert y - x \Vert _2^2 - {\lambda } S_{ C } ( D x ) \end{aligned}$$

(87)

where $S_C$ is the generalized Huber function (34). Then the gradient $\nabla f$ is Lipschitz continuous with a Lipschitz constant of 1.

Proof

The proof uses Lemma 4. Since both terms in (87) are differentiable, f is differentiable. Next, we show f is convex. Using (35), we write f as

$$\begin{aligned} f(x)&= \tfrac{1}{2}\Vert y - x \Vert _2^2 - {\lambda } \min _{ v \in \mathbb {R}^{N-1} } \bigl \{ \Vert v \Vert _1 + \tfrac{1}{2}\Vert C ( D x - v) \Vert _2^2 \bigr \} \\&= \tfrac{1}{2}x^{\mathsf {T}}\! ( I - {\lambda } B^{\mathsf {T}}\! B ) x - y^{\mathsf {T}}\! x + \tfrac{1}{2}\Vert y \Vert _2^2 \\&\qquad + {\lambda } \max _{ v \in \mathbb {R}^{N-1} } \bigl \{ -\Vert v \Vert _1 - \tfrac{1}{2}\Vert C v \Vert _2^2 + v^{\mathsf {T}}\! C^{\mathsf {T}}\! B x \bigr \}. \end{aligned}$$

The first term is convex because $B^{\mathsf {T}}\! B \preccurlyeq (1 / {\lambda } ) I$. The term inside the curly braces is affine in x (hence convex in x). Since the minimum of a set of convex functions (here indexed by v) is convex, f is convex. By Lemma 4, it remains to show $(1/2) \Vert {\,\cdot \,} \Vert _2^2 - f$ is convex. We have

$$\begin{aligned} \tfrac{1}{2}\Vert x \Vert _2^2 - f(x)&= \tfrac{1}{2}\Vert x \Vert _2^2 - \tfrac{1}{2}\Vert y - x \Vert _2^2 + {\lambda } S_{ C } ( Dx ) \end{aligned}$$

(88)

$$\begin{aligned}&= -\tfrac{1}{2}\Vert y \Vert _2^2 + y^{\mathsf {T}}\! x + {\lambda } S_{ C } ( Dx ). \end{aligned}$$

(89)

By Proposition 3, the generalized Huber function is convex. Hence, the right-hand side is convex in x which completes the proof. $\square$