Simultaneous Image Enhancement and Restoration with Non-convex Total Variation

Abstract

In this article, we propose a novel variational model for the joint enhancement and restoration of low-light images corrupted by blurring and/or noise. The model decomposes a given low-light image into reflectance and illumination images that are recovered from blurring and/or noise. In addition, our approach utilizes non-convex total variation regularization on all variables. This allows us to adequately denoise homogeneous regions while preserving the details and edges in both reflectance and illumination images, which leads to clean and sharp final enhanced images. To solve the non-convex model, we employ a proximal alternating minimization approach, and then an iteratively reweighted $\ell _1$ algorithm and an alternating direction method of multipliers are adopted for solving the subproblems. These techniques contribute to an efficient iterative algorithm, with its convergence proven. Experimental results demonstrate the effectiveness of the proposed model when compared to other state-of-the-art methods in terms of both visual aspect and image quality measures.

A Proof of Lemma 1

Proof

(1) Since $\mathbf{u}^{k+1}$ is an optimal solution of (18), we can obtain

$$\begin{aligned} \mathcal {E}(\mathbf{u}^{k+1},L^k,\mathbf{R}^k) + \frac{\delta }{2}\Vert \mathbf{u}^{k+1} - \mathbf{u}^k\Vert _2^2 \le \mathcal {E}(\mathbf{u}^{k},L^k,\mathbf{R}^k). \end{aligned}$$

Similarly, from problems (19) and (20), we can obtain the following inequalities

$$\begin{aligned} \mathcal {E}(\mathbf{u}^{k+1},L^{k+1},\mathbf{R}^k) + \frac{\delta }{2}\Vert L^{k+1} - L^k\Vert _2^2\le & {} \mathcal {E}(\mathbf{u}^{k+1},L^k,\mathbf{R}^k), \\ \mathcal {E}(\mathbf{u}^{k+1},L^{k+1},\mathbf{R}^{k+1}) + \frac{\delta }{2}\Vert \mathbf{R}^{k+1} - \mathbf{R}^k\Vert _2^2\le & {} \mathcal {E}(\mathbf{u}^{k+1},L^{k+1},\mathbf{R}^k). \end{aligned}$$

Summing the above inequalities leads to the following inequality

$$\begin{aligned}&\mathcal {E}(\mathbf{u}^{k+1},L^{k+1},\mathbf{R}^{k+1}) + \frac{\delta }{2}\Vert \mathbf{u}^{k+1} - \mathbf{u}^k\Vert _2^2 + \frac{\delta }{2}\Vert L^{k+1} - L^k\Vert _2^2 + \frac{\delta }{2}\Vert \mathbf{R}^{k+1} - \mathbf{R}^k\Vert _2^2\nonumber \\&\quad \le \mathcal {E}(\mathbf{u}^{k},L^k,\mathbf{R}^k),\quad \quad \forall k\ge 0. \end{aligned}$$

(53)

By summing the inequality in (53) over $k=0,\ldots ,N-1$ for a large integer N, we have

$$\begin{aligned}&\frac{\delta }{2}\sum _{k=0}^{N-1} (\Vert \mathbf{u}^{k+1} - \mathbf{u}^k\Vert _2^2 + \Vert L^{k+1} - L^k\Vert _2^2 + \Vert \mathbf{R}^{k+1} - \mathbf{R}^k\Vert _2^2)\nonumber \\&\quad \le \mathcal {E}(\mathbf{u}^{0},L^0,\mathbf{R}^0) - \mathcal {E}(\mathbf{u}^{N},L^{N},\mathbf{R}^{N}). \end{aligned}$$

(54)

Since $\inf \mathcal {E} \ge 0$, by taking $N\rightarrow \infty$, the inequality (54) becomes

$$\begin{aligned} \sum _{k=0}^{\infty } (\Vert \mathbf{u}^{k+1} - \mathbf{u}^k\Vert _2^2 + \Vert L^{k+1} - L^k\Vert _2^2 + \Vert \mathbf{R}^{k+1} - \mathbf{R}^k\Vert _2^2) < \infty . \end{aligned}$$

(2) Define

$$\begin{aligned} \mathcal {E}_1(\mathbf{u})= & {} \frac{\lambda }{2}\Vert \mathbf{f}-K\mathbf{u}\Vert _2^2 + {\mu }\,\,\langle \phi _1(|\nabla \mathbf{u}|),\mathbf{1}\rangle , \\ \mathcal {E}_2(L)= & {} \frac{\alpha _1}{2}\Vert L-\hat{L}\Vert _2^2+\alpha _2\,\langle \phi _2(|\nabla L|),\mathbf{1}\rangle ,\\ \mathcal {E}_3(\mathbf{R})= & {} \beta _1 \Vert \nabla \mathbf{R}- \mathbf{G}\Vert _1 + \beta _2\,\langle \phi _3(|\nabla \mathbf{R}|),\mathbf{1}\rangle . \end{aligned}$$

From the first-order optimality conditions of problems (18)–(20), we can obtain

$$\begin{aligned} \mathbf{0}\in & {} \delta (\mathbf{u}^k - \mathbf{u}^{k-1}) + \partial _\mathbf{u}\mathcal {E}(\mathbf{u}^k,L^{k-1},\mathbf{R}^{k-1}), \quad \forall k\ge 1,\nonumber \\ \mathbf{0}\in & {} \delta (L^k - L^{k-1}) + \partial _L\mathcal {E}(\mathbf{u}^k,L^{k},\mathbf{R}^{k-1}), \quad \forall k\ge 1,\nonumber \\ \mathbf{0}\in & {} \delta (\mathbf{R}^k - \mathbf{R}^{k-1}) + \partial _\mathbf{R}\mathcal {E}(\mathbf{u}^k,L^{k},\mathbf{R}^k), \quad \forall k\ge 1. \end{aligned}$$

(55)

Since Q is a smooth function, we have

$$\begin{aligned} \partial _\mathbf{u}\mathcal {E}(\mathbf{u}^k,L^{k-1},\mathbf{R}^{k-1})= & {} \partial \mathcal {E}_1(\mathbf{u}^k) + \nabla _\mathbf{u}Q(\mathbf{u}^k,L^{k-1},\mathbf{R}^{k-1}),\nonumber \\ \partial _L \mathcal {E}(\mathbf{u}^k,L^{k},\mathbf{R}^{k-1})= & {} \partial \mathcal {E}_2(L^k) + \nabla _L Q(\mathbf{u}^k,L^{k},\mathbf{R}^{k-1}), \nonumber \\ \partial _\mathbf{R}\mathcal {E}(\mathbf{u}^k,L^{k},\mathbf{R}^{k})= & {} \partial \mathcal {E}_3(\mathbf{R}^k) + \nabla _\mathbf{R}Q(\mathbf{u}^k,L^{k},\mathbf{R}^{k}). \end{aligned}$$

(56)

Thus, the formulas in (55) can be written as

$$\begin{aligned}&-\delta (\mathbf{u}^k - \mathbf{u}^{k-1}) - (\nabla _\mathbf{u}Q(\mathbf{u}^k,L^{k-1},\mathbf{R}^{k-1}) - \nabla _\mathbf{u}Q(\mathbf{u}^k,L^{k},\mathbf{R}^{k}))\\&\quad \in \partial \mathcal {E}_1(\mathbf{u}^k) + \nabla _\mathbf{u}Q(\mathbf{u}^k,L^{k},\mathbf{R}^{k}),\\&-\delta (L^k - L^{k-1}) - (\nabla _L Q(\mathbf{u}^k,L^{k},\mathbf{R}^{k-1}) - \nabla _L Q(\mathbf{u}^k,L^{k},\mathbf{R}^{k}))\\&\quad \in \partial \mathcal {E}_2(L^k) + \nabla _L Q(\mathbf{u}^k,L^{k},\mathbf{R}^{k}),\\&-\delta (\mathbf{R}^k - \mathbf{R}^{k-1})\in \partial \mathcal {E}_3(\mathbf{R}^k) + \nabla _\mathbf{R}Q(\mathbf{u}^k,L^{k},\mathbf{R}^{k}). \end{aligned}$$

From the subdifferential calculus for separable functions [50], we can obtain the result $(\tilde{\mathbf{u}}^{k}, \tilde{L}^{k}, \tilde{\mathbf{R}}^{k}) \in \partial \mathcal {E}(\mathbf{u}^{k},{L}^{k}, \mathbf{R}^{k})$. If $(\mathbf{u}^{k_i},{L}^{k_i}, \mathbf{R}^{k_i})$ is a bounded subsequence of $(\mathbf{u}^{k},{L}^{k}, \mathbf{R}^{k})$, $(\mathbf{u}^{k_i},{L}^{k_i-1}, \mathbf{R}^{k_i-1})$ and $(\mathbf{u}^{k_i},{L}^{k_i}, \mathbf{R}^{k_i-1})$ are also bounded. From the property (1), we have

$$\begin{aligned}&(\mathbf{u}^{k_i},{L}^{k_i-1}, \mathbf{R}^{k_i-1}) - (\mathbf{u}^{k_i},{L}^{k_i}, \mathbf{R}^{k_i}) \rightarrow \mathbf{0}\quad \text {as}\quad k_i\rightarrow \infty ,\\&(\mathbf{u}^{k_i},{L}^{k_i}, \mathbf{R}^{k_i-1}) - (\mathbf{u}^{k_i},{L}^{k_i}, \mathbf{R}^{k_i}) \rightarrow \mathbf{0}\quad \text {as}\quad k_i\rightarrow \infty . \end{aligned}$$

Since $\nabla _\mathbf{u}Q$ and $\nabla _L Q$ are Lipschitz continuous on a bounded domain, $(\tilde{\mathbf{u}}^{k_i}, \tilde{L}^{k_i}, \tilde{\mathbf{R}}^{k_i}) \rightarrow \mathbf{0}$ as $k_i\rightarrow \infty$. $\square$