Applying FISTA to optimization problems (with or) without minimizers

Abstract

Beck and Teboulle’s FISTA method for finding a minimizer of the sum of two convex functions, one of which has a Lipschitz continuous gradient whereas the other may be nonsmooth, is arguably the most important optimization algorithm of the past decade. While research activity on FISTA has exploded ever since, the mathematically challenging case when the original optimization problem has no minimizer has found only limited attention. In this work, we systematically study FISTA and its variants. We present general results that are applicable, regardless of the existence of minimizers.

Appendices

Appendices

1.1 Appendix A

For the sake of completeness, we provide the following proof of Lemma 2.1 based on [20, Problem 3.2.43].

Proof of Lemma 2.1

Because due to the assumption that , it is sufficient to establish that

(126)

Indeed, since $\tau _{n}\rightarrow {+}\infty$, there exists $N \in \mathbb {N}^{*}$ such that

(127)

Now, set , and . Then, on the one hand, since is increasing and positive, we have , and is therefore an increasing sequence in $\mathbb {R}_{+}$; moreover, due to (127), . On the other hand, because $\tau _{n}\rightarrow {+}\infty$, we have $\sigma _{n} =\tau _{n+1}^{2}-\tau _{1}^{2} \rightarrow {+}\infty$. Altogether, since

(128)

by the fact that is increasing, we see that . It follows that the partial sums of do not satisfy the Cauchy property. Hence, since , we obtain

$$\begin{aligned} \sum _{n \geqslant N}\frac{\tau _{n+1}^{2}-\tau _{n}^{2}}{\tau _{n+1}^{2}-\tau _{1}^{2}} = \sum _{n \geqslant N}\frac{\xi _{n}}{\sigma _{n}} = {+}\infty . \end{aligned}$$

(129)

Consequently, in the light of (127),

(130)

and (126) follows. $\square$

1.2 Appendix B

Proof of Lemma 2.2

Let us argue by contradiction. Towards this goal, assume that $\varliminf \beta _{n} \in ] 0, {+}\infty ]$ and fix $\beta \in ] 0 , \varliminf \beta _{n} [$. Then, there exists $N \in \mathbb {N}^{*}$ such that , and hence, because , we have , it follows that $\sum _{n \geqslant N}\alpha _{n}\beta _{n} \geqslant \sum _{n \geqslant N} \beta \alpha _{n} = {+}\infty$, which violates our assumption. To sum up, $\varliminf \beta _{n} = 0$. $\square$

1.3 Appendix C

The following self-contained proof of Lemma 2.3 follows [17, Lemma 3.1] in the case $\chi =1$; however, we do not require the error sequence to be positive.

Proof of Lemma 2.3

• (i): Set $\alpha :=\varliminf _{n}\alpha _{n} \in \left[ \inf _{n \in \mathbb {N}^{*}}\alpha _{n}, {+}\infty \right]$ and let be a subsequence of that converges to $\alpha$. We first show that $\alpha < {+}\infty$. Since , it follows from (9) that . Thus, ; in particular, . Hence, since $\alpha _{k_{n}}\rightarrow \alpha$ and $\sum _{n \in \mathbb {N}^{*}}\varepsilon _{n}$ converges, it follows that $\alpha \leqslant \alpha _{1} + \sum _{k \in \mathbb {N}}\varepsilon _{k } < {+}\infty$, as claimed. In turn, to establish the convergence of , it suffices to verify that $\varlimsup _{n}\alpha _{n} \leqslant \varliminf _{n} \alpha _{n}$. Towards this goal, let $\delta$ be in $]0, {+}\infty [$. Then, on the one hand, Cauchy’s criterion ensures the existence of $k_{n_{0}} \in \mathbb {N}^{*}$ such that $\alpha _{k_{n_{0} } } - \alpha \leqslant \delta /2$ and that . On the other hand, because , (9) implies that . Altogether, , from which we deduce that $\varlimsup _{n} \alpha _{n} \leqslant \alpha + \delta$. Consequently, since $\delta$ is arbitrarily chosen in $]0, {+}\infty [$, it follows that $\varlimsup _{n}\alpha _{n} \leqslant \alpha = \varliminf _{n}\alpha _{n}$, and therefore, converges to $\alpha$.

• (ii): We derive from (9) that . Hence, since $\sum _{n \in \mathbb {N}^{*}}\varepsilon _{n}$ is convergent and, by (i), $\lim _{n}\alpha _{n} =\alpha$, letting $N \rightarrow {+}\infty$ yields $\sum _{n \in \mathbb {N}} \beta _{n} \leqslant \alpha _{1} - \alpha + \sum _{n \in \mathbb {N}}\varepsilon _{n } < {+}\infty$, and so $\sum _{n \in \mathbb {N}^{*}}\beta _{n} < {+}\infty$, as required.

$\square$

1.4 Appendix D

Proof of Lemma 2.4

Indeed, since

(131a)

(131b)

we readily obtain the conclusion. $\square$

1.5 Appendix E

Proof of Lemma 2.5

“$\Rightarrow$”: Since is a decreasing sequence in $\mathbb {R}_{+}$ and $\sum _{n \in \mathbb {N}^{*}}\alpha _{n} < {+}\infty$, it follows that $n\alpha _{n}\rightarrow 0$ (see, e.g., [20, Problem 3.2.35]). Invoking the assumption that $\sum _{n \in \mathbb {N}^{*}}\alpha _{n} < {+}\infty$ once more, we infer from Lemma 2.4 that , as desired.

“$\Leftarrow$”: A consequence of Lemma 2.4. $\square$

1.6 Appendix F

Proof of Lemma 3.1

This is similar to the one found in [11, Lemma 2.3] and included for completeness; see also [15, Lemma 3.1]. Fix . On the one hand, by (A1) and (A3) in Assumption 1.1, ${\nabla {f} }$ is Lipschitz continuous with constant $\gamma ^{-1}$, from which, the Descent Lemma (see, e.g., [8, Lemma 2.64]), and the convexity of f we infer that

(132a)

(132b)

(132c)

On the other hand, because , [8, Proposition 12.26] asserts that

(133a)

(133b)

Altogether, upon adding (132) and (133), it follows that

(134a)

(134b)

(134c)

which yields (18). $\square$

1.7 Appendix G

Proof of (50)

Recall that $\lim (\tau _n/n)=1/2$. In turn, because $(\forall n\in \mathbb {N}^{*})$ $\tau _n^2=\tau _{n+1}^2-\tau _{n+1}$, it follows that

$$\begin{aligned} \frac{n(\tau _n-\tau _{n+1})}{\tau _{n+1}}= & {} \frac{n(\tau _n^2-\tau _{n+1}^2)}{\tau _{n+1}(\tau _n+\tau _{n+1})} =\frac{-n\tau _{n+1}}{\tau _{n+1}(\tau _n+\tau _{n+1})}\nonumber \\= & {} \frac{-1}{\displaystyle \frac{\tau _n}{n}+\frac{\tau _{n+1}}{n+1}\frac{n+1}{n}} \rightarrow \frac{-1}{\frac{1}{2}+\frac{1}{2}}=-1 \end{aligned}$$

(135)

and therefore that

$$\begin{aligned} n\Bigg (\frac{\tau _n-1}{\tau _{n+1}}-1+\frac{3}{n}\Bigg ) =\frac{n(\tau _n-\tau _{n+1})}{\tau _{n+1}} -\frac{n+1}{\tau _{n+1}}\frac{n}{n+1}+3 \rightarrow -1-2+3=0. \end{aligned}$$

(136)

Hence, (50) holds. $\square$