On the acceleration of optimal regularization algorithms for linear ill-posed inverse problems

Abstract

Accelerated regularization algorithms for ill-posed problems have received much attention from researchers of inverse problems since the 1980s. The current optimal theoretical results indicate that some regularization algorithms, e.g. the $\nu$-method and the Nesterov method, are such that under conventional source conditions the optimal convergence rates can be obtained with approximately the square root of the iterations of those needed for the benchmark (i.e. the Landweber iteration). In this paper, we propose a new class of regularization algorithms with parameter n, called the Acceleration Regularization of order n (AR$^n$). Theoretically, we prove that, for an arbitrary number $n>-1$, AR$^n$ can attach the optimal convergence rates with approximately the $n+1$ root of the iterations needed for the benchmark method. Moreover, unlike the existing accelerated regularization algorithms, AR$^n$s have no saturation restriction. Some symplectic iterative regularizing algorithms are developed for the numerical realization of AR$^n$. Finally, numerical experiments with integral equations and inverse problems in partial differential equations demonstrate that, at least for $n\le 2$, the numerical behavior of AR$^n$ matches our theoretical findings, also breaking the practical acceleration capability of all existing regularization algorithms.

Appendices

Appendix A: Proof of Proposition 1

If $\lambda <t^{-(n+1)}$, it is clear that $\psi \in \mathcal {S}_{C}$ with $C=1$. Now, suppose that $\lambda \in [t^{-(n+1)},\Vert A\Vert ^2]$ (without loss of generality, we can assume that $t> \Vert A\Vert ^{-\frac{2}{n+1}}$). Let $\psi \in \mathcal {S}_{C}$ and $\varphi$ be covered by $\psi$. Then, we have

$$\begin{aligned} \frac{\psi (\lambda )}{\varphi (t^{-(n+1)})} \le C e^{\frac{\lambda }{n+1} t^{n+1}} \frac{\varphi \left( t^{-(n+1)}\right) }{\varphi (t^{-(n+1)})} \le \frac{C}{{\underline{c}}} e^{\frac{\lambda }{n+1} t^{n+1}} \frac{\psi (\lambda )}{\varphi (\lambda )}, \end{aligned}$$

which implies that $\varphi \in \mathcal {S}_{C'}$ with $C'=C/{\underline{c}}$.

Furthermore, $\psi \in \mathcal {S}^{g}_{\zeta }$ and that assumption that $\varphi$ is covered by $\psi$ implies that

$$\begin{aligned} e^{\frac{2\lambda }{n+1} t^{n+1}} \zeta \left( \sqrt{\frac{\varphi (\lambda )}{\varphi (t^{-(n+1)})}} \right) \ge e^{\frac{2\lambda }{n+1} t^{n+1}} \zeta \left( \sqrt{\frac{\psi (\lambda )}{\psi (t^{-(n+1)})}} \right) \ge 1, \end{aligned}$$

from which we can determine the relation $\varphi \in \mathcal {S}^{g}_{\zeta }$.

If the function $\lambda \mapsto \frac{\psi (\lambda )}{\varphi (\lambda )}$ is increasing, (25) is certainly satisfied with $\underline{c}=1$. Moreover, when $\varphi (\lambda )\in \mathcal {S}_{C}$ is concave, the function $\xi (\lambda ):= \frac{\lambda }{\varphi (\lambda )}$ is monotonically increasing, as $\xi '= (\varphi -\lambda \varphi ')/\varphi ^2>0$. Consequently, $\varphi$ is covered by the function $\lambda$.

Appendix B: Proof of Lemma 1

(i) The relation $\varphi ^H_p\in \mathcal {S}_{C}$ follows from the following inequality:

$$\begin{aligned} \sup _{\lambda >0} \lambda ^p e^{-\frac{\lambda }{n+1} t^{n+1} } =_{\lambda =p(n+1)t^{-(n+1)}} e^{ -p} p^p (n+1)^p t^{-p(n+1)}. \end{aligned}$$

(ii) First, the inequality $\varphi ^L_{\mu }(\lambda _1\lambda _2) \le g(\lambda _1) \varphi ^L_{\mu }(\lambda _2)$ follows from

$$\begin{aligned} \log \left( \frac{1}{\lambda _1\lambda _2}\right) = \log \left( \frac{1}{\lambda _1}\right) + \log \left( \frac{1}{\lambda _2}\right) \ge g^{-1/\mu }(\lambda _1) \log \left( \frac{1}{\lambda _2}\right) \end{aligned}$$

with $g(\lambda _1)=1+\lambda _1$.

Since $\varphi ^L_{\mu }(\cdot )$ is concave, according to Proposition 1 it is covered by function $\varphi _1(\lambda )=\lambda$. Consequently, it is easy to show that $\varphi ^L_{\mu }\in \mathcal {S}_{C}$ with $C= e (n+1)$, and $\varphi ^L_\mu \in \mathcal {S}^{g}_{\zeta }$ with $g(\lambda )=1+\lambda$ and $\zeta (\lambda )=e^{-\frac{2\lambda ^{2}}{n+1}}$.

Appendix C Proof of Theorem 6

We first investigate the convergence-rate results of the best worst-case error in a special situation.

Lemma 4

Suppose $\omega (\lambda ) > 0$ for all $\lambda > 0$. If we choose for every $\delta > 0$ the parameter $t_\delta > 0$ such that

$$\begin{aligned} \Vert x(t_\delta )-x^\dagger \Vert = t^{(n+1)/2}_\delta \delta , \end{aligned}$$

(C1)

then there exists a constant $C_7 > 0$ such that

$$\begin{aligned} \sup _{{\tilde{y}}\in \bar{B}_\delta (y)}\inf _{t>0} \Vert x(t, {\tilde{y}})-x^\dagger \Vert \le (1+C_g) \delta t^{(n+1)/2}_\delta \, \, \text {~for all~}\, \delta > 0, \end{aligned}$$

(C2)

where $C_g$ is the constant defined in (19).

Moreover, there exists a constant $C_A > 0$ such that, for all $\delta > 0$,

$$\begin{aligned} \sup _{{\tilde{y}}\in \bar{B}_\delta (y)}\inf _{t>0} \Vert x(t, {\tilde{y}})-x^\dagger \Vert \ge C_A \delta t^{(n+1)/2}_\delta . \end{aligned}$$

(C3)

Proof

First, we note that the function

$$\begin{aligned} \xi _\delta (t) = t^{-(n+1)} \Vert x(t)-x^\dagger \Vert ^2= \int ^{\Vert A\Vert ^2}_0 t^{-(n+1)} e^{-\frac{2\lambda }{n+1} t^{n+1}} d \omega (\lambda ) \end{aligned}$$

is continuous and strictly decreasing, and satisfies $\lim _{t\rightarrow \infty } \xi _{\delta }(t) = 0$ and $\lim _{t\rightarrow 0} \xi _{\delta }(t) = \infty$. Therefore, we find for every $\delta > 0$ a unique value $t_{\delta } = \xi ^{-1}_{\delta }(\delta )$. Consequently, $t_\delta$ is the unique solution of equation (C1).

Let ${\tilde{y}}\in \bar{B}_\delta (y)$ be fixed. Using the inequality (19) together with the identity $A g(t,A^*A)=g(t,AA^*)A$, it follows that

$$\begin{aligned} \left\| x(t;{\tilde{y}})-x(t)\right\| ^{2}= \left\langle {\tilde{y}}-y,g^{2}(t,AA^*)AA^*({\tilde{y}}-y)\right\rangle\le & {} \delta ^{2} \max _{\lambda > 0}\lambda g^{2}(t,\lambda ) \nonumber \\ {}\le & {} C_g^{2}\delta ^{2} t^{n+1}. \end{aligned}$$

(4)

From this estimate, we obtain, with the triangular inequality and with the definition (C1) of $t_\delta$,

$$\begin{aligned} \sup _{{\tilde{y}}\in \bar{B}_\delta (y)} \inf _{t>0} \Vert x(t;{\tilde{y}})-x^\dagger \Vert \le \inf _{t>0} \left( \Vert x(t)-x^\dagger \Vert + C_g\delta t^{(n+1)/2} \right) \le (1+C_g)\delta t^{(n+1)/2}_\delta , \end{aligned}$$

which is the upper bound (C2).

For the lower bound (C3), we write, similarly,

$$\begin{aligned} \Vert x(t;{\tilde{y}})-x^\dagger \Vert ^2= & {} \Vert x(t)-x^\dagger \Vert ^2 + \Vert x(t;{\tilde{y}})-x(t)\Vert ^2 + 2 \langle x(t;{\tilde{y}})-x(t),x(t)-x^\dagger \rangle \nonumber \\= & {} \Vert x(t)-x^\dagger \Vert ^2 + \langle {\tilde{y}}-y,\,g^2(t,AA^*)AA^*({\tilde{y}}-y)\rangle \nonumber \\{} & {} + 2 \langle g(t,AA^*) ({\tilde{y}}-y), \, r(t,AA^*)A x_0 + g(t,AA^*)AA^*y-y\rangle .\nonumber \\ \end{aligned}$$

(5)

Note that for every $t > 0$ there exists a large-enough number $T_\delta > t_\delta$ such that $[T^{-(n+1)}_\delta ,t^{-(n+1)}_\delta ]$ contains at least one eigenvalue of $AA^*$ and the inequality $1- e^{- \frac{T^{-(n+1)}_{\delta }}{n+1} t^{n+1}_\delta } \ge C_A$ holds with a fixed small positive number $C_A$, dependent on only the distribution of eigenvalues $\{ \sigma ^2_j \}^{\infty }_{j=1}$ of $AA^*$. For instance, if , we set $T_\delta = \kappa _A t_\delta$, and, consequently, we can choose $C_A = 1 - e^{-1/[(n+1)\kappa ^{(n+1)}_A]}$. To that end, we let

$$\begin{aligned} z_\delta = \sum _{j:~ \sigma ^2_j \in [T^{-1}_\delta ,t^{-1}_\delta ]} \langle r(t,AA^*)A x_0 + g(t_\delta ,AA^*)AA^*y-y, v_j \rangle v_j, \end{aligned}$$

(6)

where $\{\sigma ^2_j; v_j\}_{j=1}^\infty$ is the eigen-system of the compact operator $AA^*$. Through the choice of $T_\delta$, an element $y^\delta _1\in \mathcal {Y}$ exists such that

$$\begin{aligned} {\tilde{z}}_\delta := \sum _{j:~ \sigma ^2_j \in [T^{-1}_\delta ,t^{-1}_\delta ]} \langle y^\delta _1, v_j \rangle v_j \ne 0. \end{aligned}$$

(7)

Let us first consider the case where $z_\delta \ne 0$. Then, if we set ${\tilde{y}} = y + \delta \frac{z_\delta }{\Vert z_\delta \Vert }$, the equation (5) becomes

$$\begin{aligned} \Vert x(t,{\tilde{y}})-x^\dagger \Vert ^2= & {} \Vert x(t)-x^\dagger \Vert ^2 + \frac{\delta ^2}{\Vert z_\delta \Vert ^2} \langle z_\delta , g^2(t,AA^*)AA^*z_\delta \rangle \\ {}{} & {} + \frac{2\delta }{\Vert z_\delta \Vert } \langle g(t,AA^*)z_\delta ,z_\delta \rangle . \end{aligned}$$

Thus, we may drop the last term as it is non-negative, which gives us the lower bound

$$\begin{aligned} \sup _{{\tilde{y}}\in \bar{B}_\delta (y)} \inf _{t>0} \Vert x(t,{\tilde{y}})-x^\dagger \Vert \ge \inf _{t>0} \left( \Vert x(t)-x^\dagger \Vert ^2+\delta ^2 \min _{\lambda \in [T^{-(n+1)}_\delta ,t^{-(n+1)}_\delta ]} \lambda g^2(t,\lambda )\right) ^{1/2}. \end{aligned}$$

Now, from

$$\begin{aligned} \lambda g^2(t,\lambda )= \frac{\left( 1-e^{ - \frac{\lambda }{n+1} t^{n+1} } \right) ^2}{\lambda } \ge \frac{\left( 1- e^{- \frac{T^{-(n+1)}_{\delta }}{n+1} t^{n+1}}\right) ^2}{t^{-(n+1)}_\delta } \end{aligned}$$

for all $\lambda \in [T^{-(n+1)}_\delta ,t^{-(n+1)}_\delta ]$, we can estimate further that

$$\begin{aligned} \sup _{{\tilde{y}}\in \bar{B}_\delta (y)} \inf _{t>0} \Vert x(t,{\tilde{y}})-x^\dagger \Vert \ge \inf _{t>0} \left( \Vert x(t)-x^\dagger \Vert ^2 + \delta ^2 t^{n+1}_\delta \left( 1- e^{- \frac{T^{-(n+1)}_{\delta }}{n+1} t^{n+1}}\right) ^2 \right) ^{1/2}. \end{aligned}$$

Now, since the first term is decreasing in t, and the second term is increasing in t, we can estimate the expression for $t > t_\delta$ from below using the second term at $t = t_\delta$, and for $t \le t_\delta$ using the first term at $t = t_\delta$:

$$\begin{aligned}{} & {} \sup _{{\tilde{y}}\in \bar{B}_\delta (y)} \inf _{t>0} \Vert x(t,{\tilde{y}})-x^\dagger \Vert \\{} & {} \quad \ge \min \left\{ \delta t^{(n+1)/2}_\delta , \delta t^{(n+1)/2}_\delta \left( 1- e^{- \frac{T^{-(n+1)}_{\delta }}{n+1} t_\delta ^{n+1}}\right) \right\} \ge C_A \delta t^{(n+1)/2}_\delta , \end{aligned}$$

which yields the required inequality (C3).

If $z_\delta$, as defined by (6), happens to vanish, the same argument works with the non-zero element ${\tilde{z}}_\delta$ by (7) since the last term in (5) is zero for ${\tilde{y}}=y+\delta \frac{{\tilde{z}}_\delta }{\Vert {\tilde{z}}_\delta \Vert }$. $\square$

From Lemma 4, we now obtain an equivalence relation between the noisy and the noise-free convergence rates.

Proof

Using $\varphi \in \mathcal {S}^{g}_\zeta$, we have $\phi (\gamma t)\le g(\gamma ^{-(n+1)}) \phi (t)$, which implies that

$$\begin{aligned} {\tilde{\phi }}(\gamma t) \le \gamma ^{-(n+1)/2} \sqrt{ g(\gamma ^{-(n+1)})} {\tilde{\phi }}(t), \end{aligned}$$

and so, by setting ${\tilde{g}}(\gamma ) = \gamma ^{-(n+1)/2} \sqrt{ g(\gamma ^{-(n+1)})}, \delta = {\tilde{\phi }}(t)$ and ${\tilde{\gamma }} = {\tilde{g}}(\gamma )$, we obtain, together with the monotonic decreasing of ${\tilde{\phi }}(\cdot )$,

$$\begin{aligned} {\tilde{g}}^{-1}({\tilde{\gamma }}){\tilde{\phi }}^{-1}(\delta ) \ge {\tilde{\phi }}^{-1}({\tilde{\gamma }} \delta ). \end{aligned}$$

Thus, we have

$$\begin{aligned} \psi ({\tilde{\gamma }} \delta ) = {\tilde{\gamma }}\delta [{\tilde{\phi }}^{-1}({\tilde{\gamma }}\delta )]^{\frac{n+1}{2}} \le {\tilde{\gamma }}\delta [{\tilde{g}}^{-1}({\tilde{\gamma }}){\tilde{\phi }}^{-1}(\delta )]^{\frac{n+1}{2}} = h({\tilde{\gamma }})\psi (\delta ), \end{aligned}$$

(8)

where $h({\tilde{\gamma }}) = {\tilde{\gamma }}[{\tilde{g}}^{-1}({\tilde{\gamma }})]^{\frac{n+1}{2}}$.

The equivalence of (b) and (c) follows from Theorem 5. It is sufficient to show the equivalence of (a) and (b).

In the case where $\Vert x(t) - x^{\dagger }\Vert = 0$ for all $t \ge C_6$ for some $C_6 > 0$, the inequality (44) is trivially fulfilled for some ${\tilde{c}} > 0$. Moreover, according to (4), by picking $t=C_6$, we obtain

$$\begin{aligned} \sup _{{\tilde{y}}\in \bar{B}_\delta (y)} \inf _{t>0} \Vert x(t;{\tilde{y}})-x^\dagger \Vert \le \inf _{t>0} \left( \Vert x(t)-x^\dagger \Vert + C_g\delta t^{(n+1)/2} \right) \le C_g C_6^{(n+1)/2} \delta , \end{aligned}$$

which implies the inequality (43) for some constant $c > 0$, since

$$\begin{aligned} \Vert x(t)-x^{\dagger }\Vert ^{2}= & {} \int ^{\Vert A\Vert ^2}_0 e^{-\frac{2\lambda }{n+1} t^{n+1}} d \omega (\lambda ) \\ {}\ge & {} \int ^{t^{-(n+1)}}_0 e^{-\frac{2\lambda }{n+1} t^{n+1}} d \omega (\lambda ) \ge e^{-\frac{2}{n+1}} \omega (t^{-(n+1)}), \end{aligned}$$

and we have, according to the definition of the function $\psi$ and the decreasing property of ${\tilde{\phi }}(\cdot )$, $\psi (\delta ) \ge a\delta$ for all $\delta \in (0,\delta _{0})$ for some constants $a > 0$ and $\delta _{0} > 0$.

Thus, we may assume that $\Vert x(t) - x^{\dagger }\Vert > 0$ for all $t > 0$.

First, let (44) hold. According to Theorem 5, the inequality (45) holds. Consequently, for arbitrary $\delta > 0$, we use the parameter $t_{\delta }$ defined in (C1). Then, the inequality (45) implies that

$$\begin{aligned} \delta t^{(n+1)/2}_{\delta } \le {\tilde{c}} \sqrt{\phi (t_{\delta })}. \end{aligned}$$

Consequently,

$$\begin{aligned} {\tilde{\phi }}^{-1} \left( \frac{\delta }{\sqrt{{\tilde{c}}}} \right) \ge t_{\delta } \text {~or~} \delta t^{(n+1)/2}_{\delta } \le \delta \left[ {\tilde{\phi }}^{-1} \left( \frac{\delta }{\sqrt{{\tilde{c}}}} \right) \right] ^{(n+1)/2}= \sqrt{{\tilde{c}}} \psi \left( \frac{\delta }{\sqrt{{\tilde{c}}}} \right) \end{aligned}$$

and therefore, using the inequality (C2) obtained in Lemma 4, we find, with (8),

$$\begin{aligned} \sup _{{\tilde{y}} \in {\tilde{B}}_{\delta }(y)} \inf _{t > 0} \Vert x(t;{\tilde{y}}) - x^{\dagger } \Vert\le & {} (1+C_g) \delta t^{(n+1)/2}_{\delta } \\ {}\le & {} (1+C_g){\tilde{c}} \psi \left( \frac{\delta }{{\tilde{c}}} \right) \le (1+C_g) {\tilde{c}}h({\tilde{c}}^{-1}) \psi (\delta ), \end{aligned}$$

which is the estimate (43) with $c = (1+C_g){\tilde{c}}h({\tilde{c}}^{-1})$.

Conversely, if (43) holds, we can use the inequality (C3) of Lemma 4 to obtain, from the condition (43),

$$\begin{aligned} C_A \delta t^{(n+1)/2}_\delta \le c\psi (\delta ). \end{aligned}$$

(9)

Thus, with the definition of $\psi$, we have

$$\begin{aligned} \left( \frac{C_A}{c}\right) ^{2/(n+1)} t_{\delta } \le {\tilde{\phi }}^{-1}(\delta ), \end{aligned}$$

or, equivalently,

$$\begin{aligned} \delta \le {\tilde{\phi }}\left( \left( \frac{C_A}{c}\right) ^{2/(n+1)}t_{\delta } \right) = \frac{c}{C_A} t^{-(n+1)/2}_{\delta } \sqrt{\phi \left( \left( \frac{C_A}{c}\right) ^{2/(n+1)}t_{\delta } \right) }. \end{aligned}$$

So, finally, we obtain, with the inequality $\phi (\gamma t)\le g(\gamma ^{-(n+1)}) \phi (t)$,

$$\begin{aligned} \left\| x(t_\delta )-x^{\dagger }\right\| = \delta t^{(n+1)/2}_{\delta } \le \frac{c}{C_A} \sqrt{\phi \left( \left( \frac{C_A}{c}\right) ^{2/(n+1)}t_{\delta } \right) } \le \frac{c}{C_A} \sqrt{g\left( \frac{c^2}{C^2_A}\right) } \sqrt{\phi (t_{\delta })}, \end{aligned}$$

and, since this holds for every $\delta$, (45) holds with ${\tilde{c}} = \frac{c}{C_A}\sqrt{g\left( \frac{c^2}{C^2_A}\right) }$. Consequently, (44) holds. $\square$