Oracle complexity of second-order methods for smooth convex optimization

Abstract

Second-order methods, which utilize gradients as well as Hessians to optimize a given function, are of major importance in mathematical optimization. In this work, we prove tight bounds on the oracle complexity of such methods for smooth convex functions, or equivalently, the worst-case number of iterations required to optimize such functions to a given accuracy. In particular, these bounds indicate when such methods can or cannot improve on gradient-based methods, whose oracle complexity is much better understood. We also provide generalizations of our results to higher-order methods.

Appendix A: An improved second-order oracle complexity bound for strongly convex functions

In this section, we show how the A-NPE algorithm of [10], which is a second-order method analyzed for smooth convex functions, can be used to yield near-optimal performance if the function is also strongly convex. Rather than directly adapting their analysis, which is non-trivial, we use a simple restarting scheme, which allows one to convert an algorithm for the convex setting, to an algorithm in the strongly convex setting⁷.

Our algorithm is described as follows: In the first phase, we apply a generic restarting scheme (based on [3, Subsction 4.2]), where we repeatedly run A-NPE for a bounded number of steps, followed by restarting the algorithm, running it from the last iterate obtained. By strong convexity, we show that each such epoch reduces the suboptimality by a constant factor. Once we reach a point sufficiently close to the global optimum, we switch to the second phase, where we use the cubic-regularized Newton method to get a quadratic convergence rate.

To formalize this, let us first analyze the convergence rate of the first phase. We assume that we use the algorithm described in [10, Subsection 7.4]⁸. By [10, Theorem 6.4 and Theorem 3.10], we have that the t’th iterate satisfies

$$\begin{aligned} \Vert \mathbf {w}_t-\mathbf {w}_*\Vert \le D ~~\text {and }~~ f(\mathbf {w}_t) - f(\mathbf {w}^*) \le \frac{c \mu _2 \Vert \mathbf {w}_1-\mathbf {w}^*\Vert ^3}{t^{7/2}}, \end{aligned}$$

where $\mu _2$ is the Lipschitz constant of $\nabla ^2 f$, $\mathbf {w}_1$ is the initialization point, $\mathbf {w}^*$ is the unique minimizer (due to strong convexity) of f, D bounds $\Vert \mathbf {w}_1-\mathbf {w}^*\Vert$ from above, and $c>0$ is some universal constant. Since f is also assumed to be $\lambda$-strongly convex, we have $\frac{\lambda }{2}\Vert \mathbf {w}_1 - \mathbf {w}^*\Vert ^2 \le f(\mathbf {w}_1) - f(\mathbf {w}^*)$, hence $f(\mathbf {w}_t)-f(\mathbf {w}^*)$ is at most

$$\begin{aligned} \frac{c \mu _2 \Vert \mathbf {w}_1-\mathbf {w}^*\Vert ^3}{t^{7/2}} \le \frac{2c \mu _2 \Vert \mathbf {w}_1-\mathbf {w}^*\Vert (f(\mathbf {w}_1)-f(\mathbf {w}^*))}{\lambda t^{7/2}} \le \frac{2c \mu _2 D(f(\mathbf {w}_1)-f(\mathbf {w}^*))}{\lambda t^{7/2}}. \end{aligned}$$

Thus, running the algorithm for $\tau = \left( \frac{4c\mu _2 D}{\lambda }\right) ^{2/7}$ iterations, we see that $f(\mathbf {w}_t) - f(\mathbf {w}^*) \le {(f(\mathbf {w}_1)-f(\mathbf {w}^*))}/{2}$. Now, since the distance from $\mathbf {w}_t$ to $\mathbf {w}^*$ is also smaller than D, we may initialize the algorithm at the last iterate returned by the previous run and run it for $\tau$ iterations to reduce $f(\mathbf {w}_t) - f(\mathbf {w}^*)$ in, yet again, a factor of 2. Applying the algorithm for T iterations (and restarting the algorithmic parameters after every $\tau$ iterations) yields $f(\mathbf {w}_T)-f(\mathbf {w}^*) \le \frac{f(\mathbf {w}_1)-f(\mathbf {w}^*) }{2^{T/\tau }}$. Equivalently, to obtain an $\epsilon$-optimal solution, we need at most $\left( \frac{4c\mu _2 D}{\lambda }\right) ^{2/7}\log _2\left( \frac{f(\mathbf {w}_1)-f(\mathbf {w}^*) }{\epsilon }\right)$ oracle calls (note that this restarting scheme can be applied also on uniform convex functions of any order, as defined in, e.g., [19]).

Next, after performing a number of iterations sufficiently large to obtain high accuracy solutions, we proceed to the second phase of the algorithm where cubic-regularized Newton steps are applied (see [16]). According to that analysis, after reducing the optimization error to below $\lambda ^3/4\mu _2^2$, the number of cubic-regularized Newton steps required to achieve an $\epsilon$-suboptimal solution is $\mathcal {O}\left( \log \log _2\left( \frac{\lambda ^3}{\mu _2^2\epsilon }\right) \right)$. Thus, using the $\mu _1$-Lipschitzness of the gradient to bound $f(\mathbf {w}_1)-f(\mathbf {w}^*)$ from above by $\mu _1 D^2/2$, we get that the overall number of iterations is at most $\mathcal {O}\left( \left( \frac{\mu _2 D}{\lambda }\right) ^{2/7}\log _2\left( \frac{\mu _1 \mu _2^2D^2 }{\lambda ^3}\right) + \log \log _2\left( \frac{\lambda ^3}{\mu _2^2\epsilon }\right) \right)$.