Zeroth-order optimization with orthogonal random directions

Abstract

We propose and analyze a randomized zeroth-order optimization method based on approximating the exact gradient by finite differences computed in a set of orthogonal random directions that changes with each iteration. A number of previously proposed methods are recovered as special cases including spherical smoothing, coordinate descent, as well as discretized gradient descent. Our main contribution is proving convergence guarantees as well as convergence rates under different parameter choices and assumptions. In particular, we consider convex objectives, but also possibly non-convex objectives satisfying the Polyak-Łojasiewicz (PL) condition. Theoretical results are complemented and illustrated by numerical experiments.

Auxiliary lemmas

Here we collect the main auxiliary results used in the convergence analysis of Algorithm 1.

Proof of Lemma 1

Let $k\in \mathbb {N}$, denoting by $p^{(j)}$ the j-th column of $P_k$, we want to estimate

$$\begin{aligned} \left\Vert \nabla _{\left( P_k, h_k\right) } f(x)-P_k^\top \nabla f(x)\right\Vert {}^2 = {\sum _{j=1}^{\ell }} \left( \left[ \nabla _{\left( P_k, h_k\right) } f(x)\right] _j-\langle \nabla f(x), p^{(j)} \rangle \right) ^2.\qquad \end{aligned}$$

(9.1)

To get an upper-bound for the term in parenthesis, we use H.1. As $\nabla f$ is $\lambda$-Lipschitz, the Descent Lemma 7 holds: for every $x\in \mathbb {R}^d$,

$$\begin{aligned} \left| f(x+h_k p_j) - f(x) - h_k\langle \nabla f(x), p^{(j)} \rangle \right| \le {\frac{\lambda h_k^2}{2}}\left\Vert p^{(j)}\right\Vert ^2. \end{aligned}$$

(9.2)

Then, rearranging,

$$\begin{aligned} \begin{aligned} \left| \left[ \nabla _{\left( P_k, h_k\right) } f(x)\right] _j-\langle \nabla f(x),p^{(j)} \rangle \right|&\le {\frac{\lambda h_k}{2}}\left\Vert p^{(j)}\right\Vert ^2 \quad \text {a.s.} \end{aligned} \end{aligned}$$

(9.3)

Note that P.1 implies $\Vert p^{(j)}\Vert ^2\overset{a.s.}{=}d/\ell$ and so (9.1) and (9.3) yield

$$\begin{aligned} \begin{aligned} \left\Vert \nabla _{\left( P_k, h_k\right) } f(x)-P_k^\top \nabla f(x)\right\Vert {}&\le \sqrt{{\sum _{j=1}^{\ell } } \left( {\frac{\lambda h_k d}{2\ell }}\right) ^2} = {\frac{\lambda h_k d}{2\sqrt{\ell }}} \quad \text { a.s.}. \end{aligned} \end{aligned}$$

$\square$

Lemma 5

Let $\left( a_k\right)$ and $\left( c_k\right)$ be non-negative sequences with $\lim _k c_k =0$ and let $\eta \in \left( 0,1\right)$. If for every $k\in \mathbb {N}$

$$\begin{aligned} a_{k+1}\le \left( 1-\eta \right) a_k +c_k, \end{aligned}$$

(9.4)

then $\lim _k a_k =0$. Moreover, if $c_k=c/k^t$ for some $c\ge 0$ and $t>0$, then we have $\limsup _k \ k^t a_k\le {c}/{\eta }.$

Proof

Iterating the inequality in (9.4), we get

$$\begin{aligned} \begin{aligned} a_{k}&\le (1-\eta )^k a_0 \\&\quad + \left( c_{k-1}+(1-\eta ) c_{k-2} + (1-\eta )^2 c_{k-3} +...+(1-\eta )^{k-1}c_0 \right) \\&\le a_0 + \left( \sup _{j\ge 0} c_j \right) {\sum _{j\ge 0}^{}} (1-\eta )^j = : {\tilde{C}} <+\infty , \end{aligned} \end{aligned}$$

where we used the fact that $\left( c_k\right)$ is convergent (and thus bounded) and that $\eta \in \left( 0,1\right)$. In particular, the sequence $\left( a_k\right)$ is bounded and so its $\limsup$ is a real number. Then, again from the hypothesis that $a_{k+1}\le (1-\eta ) a_k +c_k$ and the existence of $\lim _k c_k$, we have $\limsup _k a_{k} \le \limsup _k \left[ (1-\eta ) a_k + c_k \right] = (1-\eta ) \limsup _k a_k + \lim _k c_k.$ So, as $\lim _k c_k =0$ and $\limsup _k a_k \in \left[ 0,+\infty \right)$, $\eta \limsup _k a_k\le 0$. Finally, as $\eta \in \left( 0,1\right)$, $\limsup _k a_k\le 0$, and so, as $a_k\ge 0$, $\lim _k a_k = 0$. Now assume that $c_k=c/k^t$ for some $c\ge 0$ and $t>0$.

Let $c'>c$. Then, following the proof of [15, Lemma 4], there exists $k_0\in \mathbb {N}$ such that for every $k\ge k_0$ it holds

$$\begin{aligned} \frac{c}{k^t}\le \frac{c'}{\eta }\left[ \frac{1}{(k+1)^t}-(1-\eta )\frac{1}{k^t}\right] . \end{aligned}$$

Using the previous inequality in (9.4), we get

$$\begin{aligned} a_{k+1}-\frac{c'}{\eta (k+1)^t}\le (1-\eta )\left[ a_{k}-\frac{c'}{\eta k^t}\right] . \end{aligned}$$

(9.5)

First suppose that there exists ${\bar{k}}\ge k_0$ such that $a_{{\bar{k}}}\le \frac{c'}{\eta {\bar{k}}^{t}}$. Using (9.5), it is easy to see by recursion that, for every $k\ge {\bar{k}}$,

$$\begin{aligned} a_{k}\le \frac{c'}{\eta k^{t}}. \end{aligned}$$

Now suppose the opposite; namely, that for every $k\ge k_0$

$$\begin{aligned} a_k-\frac{c'}{\eta k^t}>0. \end{aligned}$$

Then, iterating (9.5), we get

$$\begin{aligned} a_{k}-\frac{c'}{\eta k^t}\le (1-\eta )^{k-1}\left[ a_{1}-\frac{c'}{\eta }\right] . \end{aligned}$$

In any of the two previous cases,

$$\begin{aligned} \limsup _k \ k^t a_k\le \limsup _k \left\{ \frac{c'}{\eta }+k^t(1-\eta )^{k-1}\left[ a_{1}-\frac{c'}{\eta }\right] \right\} =\frac{c'}{\eta }. \end{aligned}$$

Finally, since the previous inequality holds for every $c'>c$, we get the claim. $\square$

Remark 23

Consider a function $f: {\mathbb {R}}^d \rightarrow {\mathbb {R}}$ with $\lambda$-Lipschitz gradient and at least one minimizer. Applying the recursion of Algorithm 1 to such a function with an arbitrary starting point $x_0 \in {\mathbb {R}}^d$ we get that $\left\Vert x_k-x_*\right\Vert ^2$, $\langle P_kP_k^\top \nabla f_k, x_k-x_* \rangle$, and $\left\Vert \nabla f_k\right\Vert ^2$ are bounded for all $k>0$. In particular, $\left\Vert \nabla f_k\right\Vert ^2$, $\left\Vert x_k-x_*\right\Vert ^2$, and $\langle P_kP_k^\top \nabla f_k, x_k-x_* \rangle$ are all integrable. To see this note first that $\nabla f_k$ and $x_k$ are both measurable. Recall that for finite $x_k$, $\lambda$-Lipschitz gradient of f implies that $\left\Vert \nabla f_k\right\Vert ^2 \le \lambda ^2 \left\Vert x_k-x_*\right\Vert ^2$. Choose an arbitrary finite $x_0$ to begin the recursion. Then,

$$\begin{aligned} \left\Vert x_1 -x_*\right\Vert ^2&\le 2\left\Vert x_0-x_*\right\Vert ^2 + 2\alpha _0^2\left\Vert P_0\nabla _{P_0} f_0\right\Vert ^2 \\&\le C_1 + C_2\left\Vert P_0\nabla _{P_0} f_0 - P_0P_0^\top \nabla f_0 + P_0P_0^\top \nabla f_0\right\Vert ^2 \\ \mathrm {Lemma }\,\,~1&\le C_1 + C_2\left( C_3 + \left( {\frac{d}{\ell }}\right) ^2 \left\Vert \nabla f_0\right\Vert ^2\right) , \end{aligned}$$

where $C_1, C_2, C_3$ are fixed and finite, the values are unimportant but can be calculated. Repeated recursion reveals that $\left\Vert x_k-x_*\right\Vert ^2$ is bounded for all $k>0$. Further, by $\lambda$-Lipschitz gradient, $\left\Vert \nabla f_k\right\Vert ^2 \le \lambda ^2 \left\Vert x_k-x_*\right\Vert ^2$. The claim on the inner product follows from Young’s inequality and the previous results. Finally, since $\left\Vert \nabla f_k\right\Vert ^2$, $\left\Vert x_k-x_*\right\Vert ^2$, and $\langle P_kP_k^\top \nabla f_k, x_k-x_* \rangle$ are bounded, and measurable, they are integrable.