Averaged Deep Denoisers for Image Regularization

Abstract

Plug-and-Play (PnP) and Regularization-by-Denoising (RED) are recent paradigms for image reconstruction that leverage the power of modern denoisers for image regularization. In particular, they have been shown to deliver state-of-the-art reconstructions with CNN denoisers. Since the regularization is performed in an ad-hoc manner, understanding the convergence of PnP and RED has been an active research area. It was shown in recent works that iterate convergence can be guaranteed if the denoiser is averaged or nonexpansive. However, integrating nonexpansivity with gradient-based learning is challenging, the core issue being that testing nonexpansivity is intractable. Using numerical examples, we show that existing CNN denoisers tend to violate the nonexpansive property, which can cause PnP or RED to diverge. In fact, algorithms for training nonexpansive denoisers either cannot guarantee nonexpansivity or are computationally intensive. In this work, we construct contractive and averaged image denoisers by unfolding splitting-based optimization algorithms applied to wavelet denoising and demonstrate that their regularization capacity for PnP and RED can be matched with CNN denoisers. To our knowledge, this is the first work to propose a simple framework for training contractive denoisers using network unfolding.

Appendix

We prove Proposition 11 in this section. First, we recall some notations from Sect. 4.4.

• 1. For integers $p \leqslant q$, we denote $[p,q]=\{p,p+1,\ldots ,q\}, \, [p,q]_s=\{ps,(p+1)s,\ldots , qs\}, \, [p,q]^2=[p,q] \times [p,q]$, and $[p,q]_s^2=[p,q]_s \times [p,q]_s$.

• 2. The input to the denoiser is an image $\textbf{X}: \Omega \rightarrow \mathbb {R}$, where $\Omega =[0,q-1]^2$. We also consider the image in matrix form as $\textbf{X}\in \mathbb {R}^{q \times q}$.

• 3. We use circular shifts: for $\varvec{i}\in \Omega$ and $\varvec{\tau }\in \mathbb {Z}^2$, $\varvec{i}- \varvec{\tau }$ is defined as $\left( (i_1 - \tau _1) \ \textrm{mod} \ q, (i_2 - \tau _2) \ \textrm{mod} \ q\right)$.

• 4. We extract patches with stride s. More precisely, the starting coordinates of the patches are

  $$\begin{aligned} \mathcal {J}= [0, (q_s-1)]_s^2, \end{aligned}$$

  where $q_s = q/s$. The cardinality of $\mathcal {J}$ is $q_s^2$ (total number of patches), and each pixel belongs to $k_s^2$ patches, where $k_s = k/s$.

• 5. For $\varvec{i}\in \Omega$, the patch operator $\mathcal {P}_{\varvec{i}}: \mathbb {R}^{q \times q} \rightarrow \mathbb {R}^{p}$ is defined in (17).

We now elaborate on the definition of the adjoint in (18). For a patch vector $\varvec{z}\in \mathbb {R}^p$, if we let $\textbf{Y}=\mathcal {P}^*_{\varvec{i}}(\varvec{z})$, then $\mathcal {P}_{\varvec{i}}(\textbf{Y}) = \varvec{z}$ and $\textbf{Y}(\varvec{j}) = 0$ for all $\varvec{j}\notin \Omega _{\varvec{i}}$, where

$$\begin{aligned} \Omega _{\varvec{i}}= \left\{ \varvec{i}+ \varvec{\tau }: \ \varvec{\tau }\in [0,k-1]^2\right\} . \end{aligned}$$

This completely specifies $\textbf{Y}$. In particular, we have the following property of an adjoint: for all $\textbf{X}\in \mathbb {R}^{q \times q}$ and $\varvec{z}\in \mathbb {R}^{p}$,

$$\begin{aligned} \langle \varvec{z}, \mathcal {P}_{\varvec{i}}(\textbf{X}) \rangle _{\mathbb {R}^p} = \langle \mathcal {P}_{\varvec{i}}^*(\varvec{z}), \textbf{X}\rangle _{\mathbb {R}^{q \times q}}, \end{aligned}$$

(21)

where the inner product on the left (resp. right) is the standard Euclidean inner product on $\mathbb {R}^p$ (resp. $\mathbb {R}^{q \times q}$).

The main idea behind Proposition 11 is to express the original system of overlapping patches in terms of non-overlapping patches. In particular, let $q_k=q/k$; since we assume q to be a multiple of k, $q_k$ is an integer. Let

$$\begin{aligned} \mathcal {J}_0=[0, (q_k-1)]_k^2. \end{aligned}$$

By construction, $\mathcal {J}_0 \subseteq \mathcal {J}$ and the points in $\mathcal {J}_0$ are the starting coordinates of non-overlapping patches. It is not difficult to see that $\mathcal {J}$ is the disjoint union of (circular) shifts of $\mathcal {J}_0$:

$$\begin{aligned} \mathcal {J}= \bigcup _{\varvec{i}\in [0,k_s-1]^2} \, \big ( \mathcal {J}_0 + s \varvec{i}\big ), \end{aligned}$$

(22)

where $[0,k_s-1]:=\{0, 1, \ldots , k_s-1\}$.

Using (22), we can decompose (18) as follows:

$$\begin{aligned} \textrm{D}= \frac{1}{k_s^2} \sum _{ \varvec{i}\in [0, k_s-1]^2 } \textrm{D}_{\varvec{i}}, \end{aligned}$$

(23)

where

$$\begin{aligned} \textrm{D}_{\varvec{i}} = \sum _{\varvec{j}\in \mathcal {J}_0} \ \left( \mathcal {P}^*_{\varvec{j}+s\varvec{i}} \circ \textrm{D}_{\textrm{patch}} \circ \mathcal {P}_{\varvec{j}+s\varvec{i}}\right) . \end{aligned}$$

(24)

The proof of Proposition 11 follows from a couple of observations. The first of these is an observation about operator averaging. We will say that operator $\textrm{T}$ is L-contractive if its Lipschitz constant is at most L.

Lemma 12

Let $\textrm{T}_1, \ldots , \textrm{T}_k$ be L-contractive. Then, their average,

$$\begin{aligned} \frac{1}{k} (\textrm{T}_1 + \cdots + \textrm{T}_k), \end{aligned}$$

(25)

is L-contractive. On the other hand, if $\textrm{T}_1, \ldots , \textrm{T}_k$ are $\theta$-averaged, then (25) is a $\theta$-averaged operator.

The second observation relates the contractivity (averaged) property of the patch denoisers to the image denoiser (24).

Lemma 13

For $\varvec{i}\in [0,k_s-1]^2$, the following hold.

• 1. If $\textrm{D}_{\textrm{patch}}$ is L-contractive, then $\textrm{D}_{\varvec{i}}$ is L-contractive.

• 2. If $\textrm{D}_{\textrm{patch}}$ is $\theta$-averaged, then $\textrm{D}_{\varvec{i}}$ is $\theta$-averaged.

To establish Proposition 11, note from (23) that $\textrm{D}$ is the average of $\{\textrm{D}_{\varvec{i}}\}$. The desired conclusion follows immediately from Lemma 12 and 13.

We now give the proofs of Lemma 12 and 13.

For Lemma 12, the second part follows from [13, Proposition 4.30]; however, we give the proofs of both parts for completeness.

Proof of Lemma 12

Let $\textrm{T}=(1/k)(\textrm{T}_1 + \cdots + \textrm{T}_k)$, where $\textrm{T}_1, \ldots , \textrm{T}_k$ are L-contractive. Then, for $i=1,\ldots ,k$ and for all $\varvec{x},\varvec{x}' \in \mathbb {R}^n$,

$$\begin{aligned} \Vert \textrm{T}_i(\varvec{x}) - \textrm{T}_i(\varvec{x}') \Vert \leqslant L \, \Vert \varvec{x}- \varvec{x}' \Vert . \end{aligned}$$

(26)

We can write

$$\begin{aligned} \textrm{T}(\varvec{x}) - \textrm{T}(\varvec{x}') = \frac{1}{k} \sum _{i=1}^k \, \left( \textrm{T}_{i}(\varvec{x}) - \textrm{T}_{i}(\varvec{x}') \right) . \end{aligned}$$

Using triangle inequality and (26), we get

$$\begin{aligned} \Vert \textrm{T}(\varvec{x}) - \textrm{T}(\varvec{x}') \Vert \leqslant \frac{1}{k} \sum _{i=1}^k \, L \, \Vert \varvec{x}- \varvec{x}' \Vert = L \Vert \varvec{x}- \varvec{x}' \Vert . \end{aligned}$$

This establishes the first part of Proposition 12.

Next, suppose that $\textrm{T}_1, \ldots , \textrm{T}_k$ are $\theta$-averaged, i.e., there exists nonexpansive operators $\textrm{N}_1,\ldots ,\textrm{N}_k$ such that

$$\begin{aligned} \textrm{T}_i= (1-\theta ) \textrm{I}+\theta \textrm{N}_i \qquad (i=1,\ldots ,k). \end{aligned}$$

Then

$$\begin{aligned} \textrm{T}= \frac{1}{k}(\textrm{T}_1 + \cdots + \textrm{T}_k)= (1-\theta ) \textrm{I}+ \theta \textrm{N}, \end{aligned}$$

where $\textrm{N}=(1/k)(\textrm{N}_1 + \cdots + \textrm{N}_k)$.

We can again use triangle inequality to show that $\textrm{N}$ is nonexpansive. Thus, we have shown that $\textrm{T}$ is $\theta$-averaged, which completes the proof. $\square$

We need the following result to establish Lemma 13.

Lemma 14

For any fixed s and $\varvec{i}\in [0, k_s-1]^2$, we have the following orthogonality properties:

• 1. For all $\textbf{X}: \Omega \rightarrow \mathbb {R}$,

  $$\begin{aligned} \sum _{\varvec{j}\in \mathcal {J}_0} \ \Vert \mathcal {P}_{\varvec{j}+s\varvec{i}}(\textbf{X}) \Vert ^2 = {\Vert \textbf{X}\Vert }^2. \end{aligned}$$

  (27)

  We can equivalently write this as

  $$\begin{aligned} \sum _{\varvec{j}\in \mathcal {J}_0} \ \mathcal {P}_{\varvec{j}+s\varvec{i}}^* \mathcal {P}_{\varvec{j}+s \varvec{i}}=\textrm{I}, \end{aligned}$$

  (28)

  where $\textrm{I}$ is the identity operator on $\mathbb {R}^{q\times q}$.

• 2. For all $\{\varvec{z}_{\varvec{j}} \in \mathbb {R}^p: \, \varvec{j}\in \mathcal {J}_0\}$,

  $$\begin{aligned} \big \Vert \sum _{\varvec{j}\in \mathcal {J}_0} \ \mathcal {P}_{\varvec{j}+s \varvec{i}}^*(\varvec{z}_{\varvec{j}}) \big \Vert ^2 = \sum _{\varvec{j}\in \mathcal {J}_0}\ \Vert \varvec{z}_{\varvec{j}} \Vert ^2. \end{aligned}$$

  (29)

It is understood that the norm on the left in (27) is the Euclidean norm on $\mathbb {R}^{p}$, while the norm on the right is the Euclidean norm on $\mathbb {R}^{q \times q}$.

We can establish Lemma 14 using property (21) and the fact that $\mathcal {J}_0$ are the starting coordinates of non-overlapping patches. We will skip this straightforward calculation. We are ready to prove Lemma 13.

Proof of Lemma 13

Let $\textrm{D}_{\textrm{patch}}$ be L-contractive, i.e., for all $\varvec{z},\varvec{z}' \in \mathbb {R}^p$,

$$\begin{aligned} \Vert \textrm{D}_{\textrm{patch}}(\varvec{z}) - \textrm{D}_{\textrm{patch}}(\varvec{z}') \Vert \leqslant L \Vert \varvec{z}-\varvec{z}' \Vert . \end{aligned}$$

(30)

We have to show for all $\textbf{X}, \textbf{X}' \in \mathbb {R}^{q \times q}$,

$$\begin{aligned} \Vert \textrm{D}_{\varvec{i}}(\textbf{X}) - \textrm{D}_{\varvec{i}}(\textbf{X}') \Vert ^2 \leqslant L^2 \Vert \textbf{X}-\textbf{X}' \big \Vert ^2. \end{aligned}$$

(31)

From (27) and (29), we can write

$$\begin{aligned} \Vert \textrm{D}_{\varvec{i}}(\textbf{X}) - \textrm{D}_{\varvec{i}}(\textbf{X}') \Vert ^2 = \sum _{\varvec{j}\in \mathcal {J}_0}{\Vert \textrm{D}_{\textrm{patch}} (\varvec{z}_{j}) -\textrm{D}_{\textrm{patch}} (\varvec{z}'_{j}) \Vert }^2 , \end{aligned}$$

where $\varvec{z}_{j} = \mathcal {P}_{\varvec{j}+s\varvec{i}}(\textbf{X})$ and $\varvec{z}'_{j} = \mathcal {P}_{\varvec{j}+s\varvec{i}}(\textbf{X}')$. From (30),

$$\begin{aligned} \Vert \textrm{D}_{\textrm{patch}} (\varvec{z}_{\varvec{j}}) -\textrm{D}_{\textrm{patch}} (\varvec{z}'_{\varvec{j}}) \Vert&\leqslant L \, \Vert \varvec{z}_{\varvec{j}} -\varvec{z}'_{\varvec{j}} \Vert \\&= L \, \Vert \mathcal {P}_{\varvec{j}+s\varvec{i}}(\textbf{X}-\textbf{X}') \Vert . \end{aligned}$$

Moreover, we have from (29) that

$$\begin{aligned} \sum _{\varvec{j}\in \mathcal {J}_0} \ \Vert {\textrm{P}}_{\varvec{j}+s\varvec{i}}(\textbf{X}-\textbf{X}')) \Vert ^2 = \Vert \textbf{X}-\textbf{X}' \big \Vert ^2. \end{aligned}$$

Combining the above, we get (31). This establishes part 1 of Lemma 13.

Now, let $\textrm{D}_{\textrm{patch}}$ be $\theta$-averaged, i.e.,

$$\begin{aligned} \textrm{D}_{\textrm{patch}} = (1-\theta )\textrm{I}+ \theta \textrm{N}, \end{aligned}$$

(32)

where $\textrm{N}$ is nonexpansive. Substituting (32) in (24), and using (28), we can write

$$\begin{aligned} \textrm{D}_{\varvec{i}} = (1 - \theta ) \textrm{I}+ \theta \textrm{G}_{\varvec{i}}, \end{aligned}$$

where

$$\begin{aligned} \textrm{G}_{\varvec{i}} = \sum _{\varvec{j}\in \mathcal {J}_0} \ \left( \mathcal {P}_{\varvec{j}+s\varvec{i}}^* \circ \textrm{N}\circ \mathcal {P}_{\varvec{j}+s\varvec{i}} \right) . \end{aligned}$$

(33)

Using (27) and (29) and that $\textrm{N}$ is nonexpansive, we can show (the calculation is similar to one used earlier) that, for all $\textbf{X}, \textbf{X}' \in \mathbb {R}^{q \times q}$,

$$\begin{aligned} \Vert \textrm{G}_{\varvec{i}}(\textbf{X}) - \textrm{G}_{\varvec{i}}(\textbf{X}') \Vert ^2 \leqslant \Vert \textbf{X}- \textbf{X}' \Vert ^2. \end{aligned}$$

That is, $\textrm{G}_{\varvec{i}}$ is nonexpansive. It follows from (32) that $\textrm{D}_{\varvec{i}}$ is $\theta$-averaged. This establishes part 2 of Lemma 13. $\square$