Efficient Convex Optimization for Non-convex Non-smooth Image Restoration

Abstract

This work focuses on recovering images from various forms of corruption, for which a challenging non-smooth, non-convex optimization model is proposed. The model consists of a concave truncated data fidelity functional and a convex total-variational term. We introduce and study various novel equivalent mathematical models from the perspective of duality, leading to two new optimization frameworks in terms of two-stage and single-stage. The two-stage optimization approach follows a classical convex-concave optimization framework with an inner loop of convex optimization and an outer loop of concave optimization. In contrast, the single-stage optimization approach follows the proposed novel convex model, which boils down to the global optimization of all variables simultaneously. Moreover, the key step of both optimization frameworks can be formulated as linearly constrained convex optimization and efficiently solved by the augmented Lagrangian method. For this, two different implementations by projected-gradient and indefinite-linearization are proposed to build up their numerical solvers. The extensive experiments show that the proposed single-stage optimization algorithms, particularly with indefinite linearization implementation, outperform the multi-stage methods in numerical efficiency, stability, and recovery quality.

Appendices

Appendix A: Close-form Solver for Sub-problem (13)

When function g to be the truncated-${l_1}$ function with any $\alpha > 0$, its dual form is ${g^*}\left( q \right) = \alpha \left( { - 1 + q} \right) I\left( {q \in \left[ {0,1} \right] } \right) \,$. Thus, the optimization problem (13) can be reformulated as:

$$\begin{aligned} \text {min}_{0 \le q_i \le 1} \,\ {q_i} \left|h_i \varvec{u}^{k+1} - f_i\right| \, - \, \alpha \cdot {q_i} + \alpha \,, \end{aligned}$$

(38)

so, in this case, the straightforward close form solution for q is

$$\begin{aligned} {q_i}^{k+1} = \left\{ {\begin{array} {ll} {1, \,\quad \left| {h_i \varvec{u}^{k+1} - f_i} \right| \le \alpha }\\ {0, \,\quad \left| {h_i \varvec{u}^{k+1} - f_i} \right| > \alpha } \,. \end{array}} \right. \end{aligned}$$

(39)

Appendix B: Numerical Implementation of (18) and (19)

In this part, we list two different implementations for solving problems (18) and (19): the first one is implemented using the classical projected gradient-ascent method (PG), while the second one employs a indefinite linearized method (IDL) recently published in [13].

1.1 Appendix B.1: Solver by Projected Gradient-Ascent Method (PG)

Clearly, the maximization problem (18) over w results in:

$$\begin{aligned} \begin{aligned} w^{l+1}&= \text {arg}\text {max}_{w_i \in [-q_i^k, q_i^k]}\, \mathcal {L_\beta }( {w,\varvec{p}^l;\varvec{u}^l}) \\&= \text {arg}\text {max}_{w_i \in [-q_i^k, q_i^k]}\, - \sum _{i \in S} \, \left( f_i - h_i {\varvec{u}}^l \right) \cdot w_i - \frac{\beta }{2} \, \left\Vert \sum _{i \in S} w_i {h_i}^T + \textrm{Div} \varvec{p}^l\right\Vert ^2 \,, \end{aligned} \end{aligned}$$

(40)

which can be easily tackled by the following projection gradient-ascent scheme:

$$\begin{aligned} {w_i}^{l+1} \, = \, \textrm{Proj}_{w_i \in [-q_i^k, q_i^k]}\Big ( {w_i}^l + \mu \, {G_i}^l\Big ) \,, \end{aligned}$$

(41)

where $\mu > 0$ is the step-size of gradient-ascent and the associate gradient is

$$\begin{aligned} {G_i}^l = -f_i - \beta {h_i} \left( \sum _{i \in S} {w_i}^l {h_i}^T + \textrm{Div} \varvec{p}^l - \frac{\varvec{u}^l}{\beta } \right) \, . \end{aligned}$$

(42)

As we observed in practice, the total number of projection gradient ascent iterations (41) has no obvious influence on its numerical performance; so we take one iteration (41) in this work to decrease computational complexity.

The maximization problem (19) over $\varvec{p}$ is reduced as:

$$\begin{aligned} \varvec{p}^{l+1} \!=\! \text {arg}\text {max}_{\left|\varvec{p}_i\right| \le \lambda }\! \mathcal {L_\beta }( w^{l+1},\varvec{p},\varvec{u}^l) \!=\! \text {arg}\text {max}_{\left|\varvec{p}_i\right| \le \lambda } -\frac{\beta }{2} \left\Vert \textrm{Div}\varvec{p} + \sum _{i \in S} w_i^{l+1} {h_i}^T - \frac{\varvec{u}^l}{\beta } \right\Vert ^2\,, \end{aligned}$$

(43)

which can be solved by the following projection gradient-ascent scheme:

$$\begin{aligned} \varvec{p}^{l+1} = {\prod }_\lambda \left( \varvec{p}^l + \sigma \, \nabla \left( \textrm{Div}\varvec{p}^l + \sum _{i \in S} w_i^{l+1} {h_i}^T - \frac{\varvec{u}^l}{\beta } \right) \right) \,, \end{aligned}$$

(44)

where $\sigma > 0$ is the step-size of gradient-ascent, $\nabla = - \textrm{Div}^T\,$, and ${\prod }_\lambda$ is the projection onto the convex set ${C_\lambda } = \left\{ {\eta |\left| \eta \right| \le \lambda } \right\} \,$.

1.2 Appendix B.2: Solver by Indefinite Linearized Method (IDL)

A proximal linearization strategy, which adds a positive-definite proximal term to linearize the studied optimization problem and derives their closed-form solutions, was introduced in [41]. Recently, He et al. [13] showed that such positive definiteness of the added proximal term for linearization is not required for convergence, they proposed an efficient indefinite linearized algorithmic that permits a larger step-size to speed up convergence.

For problem (18), the indefinite linearized w-subproblem is

$$\begin{aligned} \begin{aligned} w^{l+1}&= \text {arg}\text {max}_{w_i \in [-q_i^k, q_i^k]} - \sum _{i \in S} \, \left( f_i - h_i {\varvec{u}}^l \right) \cdot w_i - \frac{\beta }{2} \, \left\Vert \sum _{i \in S} w_i {h_i}^T + \textrm{Div} \varvec{p}^l\right\Vert ^2\\&\quad - \frac{1}{2}\left\| {w - {w^l}} \right\| _{(\tau c_1 I - \beta H{H^T})}^2 \,. \end{aligned} \end{aligned}$$

We can easily compute its closed-form solution:

$$\begin{aligned} w^{l+1} = w^l + \frac{1}{\tau c_1} \left\{ - f + H \left[ \varvec{u}^l - \beta \, \left( \sum _{i \in S} {w_i}^l {h_i}^T + \textrm{Div} \varvec{p}^l \right) \right] \right\} \,, \text {with each}\, \left|w_i\right| \le {q_i}^{l} \,, \end{aligned}$$

(45)

where $c_1 > \beta \rho \left( {H{H^T}} \right)$ and $\tau \in \left( {0.75,1} \right) \,$.

The indefinite linearized $\varvec{p}$-subproblem for problem (19) is

$$\begin{aligned} \varvec{p}^{l+1} = \text {arg}\text {max}_{\left|\varvec{p}_i\right| \le \lambda } -\frac{\beta }{2} \left\Vert \textrm{Div}\varvec{p} + \sum _{i \in S} w_i^{l+1} {h_i}^T - \frac{\varvec{u}^l}{\beta } \right\Vert ^2 - \frac{1}{2}\left\| {\varvec{p} - {\varvec{p}^l}} \right\| _{(\tau c_2 I - \beta {\textrm{Div}^T}\textrm{Div})}^2 \,. \end{aligned}$$

So, we have its closed-form solution:

$$\begin{aligned} \varvec{p}^{l+1} = \varvec{p}^l - \frac{1}{\tau c_2} \nabla \left[ \varvec{u}^l - \beta \, \left( \sum _{i \in S} {w_i}^{l+1} {h_i}^T + \textrm{Div} \varvec{p}^l \right) \right] \,, \text { with each}\, \left|p_i\right| \le \lambda \,, \end{aligned}$$

(46)

where $c_2 > \beta \rho \left( \textrm{Div}^T \textrm{Div}\right)$ and $\tau \in \left( {0.75,1} \right) \,$.

Appendix C: Numerical Implementation of (28) and (29)

Just as the previous “Appendix B”, we can implement two different numerical solvers for (28) and (29), by the classical projected gradient-ascent method (PG) and the indefinite linearized method (IDL).

1.1 Appendix C.1: Solver by Projected Gradient-Descent Method (PG)

The projection gradient-descent scheme for solving (28) is

$$\begin{aligned} \tilde{\varvec{x}}^k \,=\, \textrm{Proj}_{{\tilde{\varvec{x}}} \in \Omega } \left( \tilde{\varvec{x}}^{k-1} - \nu \, {G_i}^{k} \right) \,, \end{aligned}$$

(47)

where $\nu > 0$ is the step-size of gradient-descent, and the associate gradient is

$$\begin{aligned} {G_i}^{k} \,=\, \hat{f} \,-\, A^T \, {\varvec{\lambda }}^k \,+\, \beta \, A^T \left[ A\left( \tilde{\varvec{x}}^{k-1} - \varvec{x^k}\right) \right] \,. \end{aligned}$$

(48)

In essence, recall the definition of $\varvec{x}\,$, the iterative formulas for updating variables w and $\varvec{p}$ are as follows:

$$\begin{aligned} \begin{aligned} \tilde{w}^k \,=\, \tilde{w}^{k-1} - \mu \left[ \left( f - {{\lambda }_{+}}^k + {{\lambda }_{-}}^k \right) + \beta \left( 2\tilde{w}^{k-1} - A\varvec{x}^k_2 + A\varvec{x}^k_3 \right) \right] \\ - \mu \, \beta H \left( \sum _{i \in S} \tilde{w_i}^{k-1} {h_i}^T + \textrm{Div} \tilde{\varvec{p}}^{k-1} - A\varvec{x}^k_1 - \frac{{\varvec{u}^k}}{\beta } \right) \,, \end{aligned} \end{aligned}$$

(49)

and

$$\begin{aligned} \tilde{\varvec{p}}^k = {\prod }_\lambda \left( \tilde{\varvec{p}}^{k-1} + \sigma \, \beta \,\nabla \left( \sum _{i \in S} \tilde{w_i}^{k-1} {h_i}^T + \textrm{Div} \tilde{\varvec{p}}^{k-1} - A\varvec{x}^k_1 - \frac{\varvec{u}^k}{\beta } \right) \right) \,, \end{aligned}$$

(50)

where $\mu > 0$ is the step-size for updating $w\,$, $\sigma >0$ is the step-size for updating $\varvec{p}\,$, ${\prod }_\lambda$ is the projection onto the convex set ${C_\lambda } = \left\{ {\eta \, | \, \left| \eta \right| \le \lambda } \right\} \,$, and

$$\begin{aligned}&{A\varvec{x}}^k_1 = \left( 1-v \right) {A\varvec{x}}^{k-1}_1 + v \, \big ( \sum _{i \in S} \tilde{w_i}^{k-1} {h_i}^T + \textrm{Div} \tilde{\varvec{p}}^{k-1} \big ) \,, \end{aligned}$$

(51a)

$$\begin{aligned}&{A\varvec{x}}^k_2 = \left( 1-v \right) {A\varvec{x}}^{k-1}_2 + v \, \tilde{w}^{k-1} + v \, \left( q^k - \tilde{q}^{k-1} \right) \,, \end{aligned}$$

(51b)

$$\begin{aligned}&{A\varvec{x}}^k_3 = \left( 1-v \right) {A\varvec{x}}^{k-1}_3 - v \, \tilde{w}^{k-1} + v \, \left( q^k - \tilde{q}^{k-1} \right) \,. \end{aligned}$$

(51c)

For the truncated-$l_1$ function, its dual function is ${g^*}\left( q \right) = \alpha \left( { - 1 + q} \right) I\left( {q \in \left[ {0,1} \right] } \right) \,$, and the optimization problem (29) can then be reformulated as:

$$\begin{aligned} \tilde{q}^k \, = \, \text {arg}\text {min}_{\varvec{0}\le q \le \varvec{1}} \, \alpha \,-\, \alpha q^{T} \hat{I} \,-\, q^{T} B^T {{\varvec{\lambda }}^k} \,+\, \frac{\beta }{2} \left\Vert B\left( q - q^k\right) + A\left( \tilde{\varvec{x}}^k - \varvec{x^k}\right) \right\Vert ^2 \,. \end{aligned}$$

(52)

Since $B = \left( \varvec{0} ,\, {\hat{I}} ,\, {\hat{I}}\right) ^{T}\,$ is a variant of the identity matrix I, the above problem (52) has a closed-form solution

$$\begin{aligned} \tilde{q}^k \,=\, {\prod }_{[\varvec{0},\varvec{1}]} \left( {q}^k \,+\, \frac{A\varvec{x}^k_2 + A\varvec{x}^k_3}{2} \,+\, \frac{ \alpha + {{\lambda }_{+}}^k + {{\lambda }_{-}}^k }{2\beta } \right) \,, \end{aligned}$$

(53)

where ${\prod }_{[\varvec{0},\varvec{1}]}$ is the projection onto the convex set ${C_{[\varvec{0},\varvec{1}]} } = \left\{ {\eta \, | \, \varvec{0} \le \eta \le \varvec{1} } \right\} \,$, $A\varvec{x}^k_2$ and $A\varvec{x}^k_3$ are as shown in ().

1.2 Appendix C.2: Solver by Indefinite Linearized Method (IDL)

We now implement the indefinite linearized method (IDL) to solve sub-problem (28). Its corresponding indefinite linearized version is

$$\begin{aligned} \tilde{\varvec{x}}^k \, = \, \text {arg}\text {min}_{\varvec{x} \in \Omega }\, \left\langle {\hat{\varvec{f}}}, \varvec{x} \right\rangle \, - \, \varvec{x}^{T} A^T {{\varvec{\lambda }}^k} \, + \, \frac{\beta }{2} \left\Vert A \left( \varvec{x} - \varvec{x^k}\right) \right\Vert ^2 + \frac{1}{2}\left\| {\varvec{x} - {\varvec{x}^k}} \right\| _M^2 \,, \end{aligned}$$

(54)

where the indefinite matrix M in (54) is taken as

$$\begin{aligned} M = \tau cI - \beta {A^T}A, \ \ \text {with} \ \ c > \beta \rho \left( {{A^T}A} \right) \ \ \text {and} \ \ \tau \in \left( {0.75,1} \right) . \end{aligned}$$

So, the proximal optimization problem (54) can be reduced to

$$\begin{aligned} \begin{aligned} \tilde{\varvec{x}}^k \,&= \, \text {arg}\text {min}_{\varvec{x} \in \Omega }\, \left\langle {\hat{\varvec{f}}}, \varvec{x} \right\rangle \, - \, \varvec{x}^{T} A^T {{\varvec{\lambda }}^k} \, + \, \frac{\beta }{2} \left\Vert A \left( \varvec{x} - \varvec{x^k}\right) \right\Vert ^2 + \frac{1}{2}\left\| {\varvec{x} - {\varvec{x}^k}} \right\| _{(\tau cI - \beta {A^T}A)}^2 \,, \\&= \, \text {arg}\text {min}_{\varvec{x} \in \Omega }\, \frac{\tau c}{2} \left\Vert {\varvec{x}} - {\varvec{x}}^{k} + \frac{1}{\tau c} \left( \hat{\varvec{f}} - A^T {{\varvec{\lambda }}^k} \right) \right\Vert ^2 \,, \end{aligned} \end{aligned}$$

whose closed-form solution is

$$\begin{aligned} \tilde{\varvec{x}}^k \, = \, {\varvec{x}}^{k} - \frac{1}{\tau c} \left( \hat{\varvec{f}} - A^T {{\varvec{\lambda }}^k} \right) \,. \end{aligned}$$

(55)

Thus, the iterative formula for updating the variables w and $\varvec{p}$ are

$$\begin{aligned} \tilde{w}^k \, = \, {w}^k \,-\, \frac{1}{\tau c_1} \left( f - {{\lambda }_{+}}^k + {{\lambda }_{-}}^k - \sum _{i \in S} {h_i}{\varvec{u}}^k \right) \,, \end{aligned}$$

(56)

and

$$\begin{aligned} \tilde{\varvec{p}}^k \, = \, {\varvec{p}}^k \,-\, \frac{1}{\tau c_2} \, \nabla {\varvec{u}}^k \,, \quad \ \text {with each}\, \left|p_i\right| \le \lambda \,, \end{aligned}$$

(57)

where $c_1 > \beta \rho \left( {H{H^T}} \right) \,$, $c_2 > \beta \rho \left( \textrm{Div}^T \textrm{Div}\right) \,$, and $\tau \in \left( {0.75,1} \right) \,$.

Clearly, the coefficient matrix B in the optimization problem (29) is indeed a rearrangement of the identity matrix. Hence, the optimization of q using the indefinite linearized technique is the same as (53).