Fast Gradient Descent for Convex Minimization Problems with an Oracle Producing a (δ, L)-Model of Function at the Requested Point

Abstract

A new concept of $(\delta ,L)$-model of a function that is a generalization of the Devolder–Glineur–Nesterov $(\delta ,L)$-oracle is proposed. Within this concept, the gradient descent and fast gradient descent methods are constructed and it is shown that constructs of many known methods (composite methods, level methods, conditional gradient and proximal methods) are particular cases of the methods proposed in this paper.

APPENDIX

In this paper, we essentially used the fact that we are solving the auxiliary problem with an error not exceeding $\widetilde \delta$ using the concept of Definition 4. It was shown that the $\widetilde \delta$-solution in the sense of Definition 4 implies the $\widetilde \delta$-optimal solution. The converse is generally not true; however, we try to give fairly general examples in which the converse result holds. The trivial case is $\widetilde \delta = 0$. In this case, the first-order optimality criterion implies that these two definitions of $\widetilde \delta$-solution are equivalent.

Assume that the following problem is being solved:

$${{\alpha }_{k}}\psi (x) + V(x,{{x}_{k}}) \to \mathop {\min}\limits_{x \in Q} ,$$

((26))

where $\psi (x)$ is a convex function and $V(x,{{x}_{k}})$ is a strongly convex function with the strong convexity constant equal to one. The auxiliary problem in the iterations of optimization methods often has this form. Certainly, there are cases in which this problem can be solved analytically, e.g., when the main problem is the smooth optimization without constraints and with the Euclidean prox-structure $V(x,y) = \tfrac{1}{2}\left\| {x - y} \right\|_{2}^{2}$. If problem (26) can be solved only numerically, then various approaches depending on the problem can be used.

Consider the case when

$$\psi (x) + V(x,{{x}_{k}}) = \sum\limits_{i = 1}^n \left[ {{{\psi }_{i}}({{x}_{i}}) + {{V}_{i}}({{x}_{i}})} \right].$$

Under this condition, problem (26) is separable. Therefore, it is sufficient to solve $n$ one-dimensional problems each of which can be solved using the bisection method [33] in time $\mathcal{O}\left( {\ln\left( {\tfrac{1}{\varepsilon }} \right)} \right)$, where $\epsilon$ is the error with respect to the function.

If we additionally assume that $\psi (x)$ has an $L$-Lipschitzian gradient in the norm $\left\| \; \right\|$, then two approaches can be used. If $V(x,{{x}_{k}})$ has an $L$-Lipschitzian gradient in the norm $\left\| \; \right\|$, then the problem can be solved in linear time $\mathcal{O}\left( {\ln\left( {\tfrac{1}{\varepsilon }} \right)} \right)$ [1]. If $V(x,{{x}_{k}})$ does not have an $L$-Lipschitzian gradient in the norm $\left\| \; \right\|$, then $V(x,{{x}_{k}})$ in problem (26) can be considered as a composite one. In this case, in order to obtain a linear convergence rate, the restart technique [14, 34] can be used.