New analysis and results for the Frank–Wolfe method

Abstract

We present new results for the Frank–Wolfe method (also known as the conditional gradient method). We derive computational guarantees for arbitrary step-size sequences, which are then applied to various step-size rules, including simple averaging and constant step-sizes. We also develop step-size rules and computational guarantees that depend naturally on the warm-start quality of the initial (and subsequent) iterates. Our results include computational guarantees for both duality/bound gaps and the so-called FW gaps. Lastly, we present complexity bounds in the presence of approximate computation of gradients and/or linear optimization subproblem solutions.

Appendix

Proposition 7.1

Let $B_k^w$ and $B_k^m$ be as defined in Sect. 2. Suppose that there exists an open set $\hat{Q} \subseteq E$ containing $Q$ such that $\phi (x,\cdot )$ is differentiable on $\hat{Q}$ for each fixed $x \in P$, and that $h(\cdot )$ has the minmax structure (4) on $\hat{Q}$ and is differentiable on $\hat{Q}$. Then it holds that:

$$\begin{aligned} B_k^w \ge B_k^m \ge h^*. \end{aligned}$$

Furthermore, it holds that $B_k^w = B_k^m$ in the case when $\phi (x,\cdot )$ is linear in the variable $\lambda$.

Proof

It is simple to show that $B_k^m \ge h^*$. At the current iterate $\lambda _k \in Q$, define $x_k \in \arg \min \limits _{x \in P}\phi (x, \lambda _k)$. Then from the definition of $h(\lambda )$ and the concavity of $\phi (x_k, \cdot )$ we have:

$$\begin{aligned} h(\lambda )&\le \phi (x_k, \lambda ) \le \phi (x_k, \lambda _k) + \nabla _\lambda \phi (x_k, \lambda _k)^T(\lambda - \lambda _k) \nonumber \\&= h(\lambda _k) + \nabla _\lambda \phi (x_k, \lambda _k)^T(\lambda - \lambda _k), \end{aligned}$$

(55)

whereby $\nabla _\lambda \phi (x_k, \lambda _k)$ is a subgradient of $h(\cdot )$ at $\lambda _k$. It then follows from the differentiability of $h(\cdot )$ that $\nabla h(\lambda _k) = \nabla _\lambda \phi (x_k, \lambda _k)$, and this implies from (55) that:

$$\begin{aligned} \phi (x_k, \lambda ) \le h(\lambda _k) + \nabla h(\lambda _k)^T(\lambda - \lambda _k). \end{aligned}$$

(56)

Therefore we have:

$$\begin{aligned} B_k^m = f(x_k) = \max _{\lambda \in Q}\{\phi (x_k, \lambda )\} \le \max _{\lambda \in Q}\{h(\lambda _k) + \nabla h(\lambda _k)^T(\lambda - \lambda _k)\} = B_k^w. \end{aligned}$$

If $\phi (x,\lambda )$ is linear in $\lambda$, then the second inequality in (55) is an equality, as is (56).

Proposition 7.2

Let $C_{h, Q}, \mathrm {Diam}_Q$, and $L_{h,Q}$ be as defined in Sect. 2. Then it holds that $C_{h, Q} \le L_{h,Q}(\mathrm {Diam}_Q)^2$.

Proof

Since $Q$ is convex, we have $\lambda + \alpha (\tilde{\lambda }- \lambda ) \in Q$ for all $\lambda , \tilde{\lambda } \in Q$ and for all $\alpha \in [0,1]$. Since the gradient of $h(\cdot )$ is Lipschitz, from the fundamental theorem of calculus we have:

$$\begin{aligned} h(\lambda + \alpha (\tilde{\lambda }- \lambda ))&= h(\lambda ) + \nabla h(\lambda )^T(\alpha (\tilde{\lambda }- \lambda )) \\&+\int \limits _0^1[\nabla h(\lambda + t \alpha (\tilde{\lambda }- \lambda )) - \nabla h (\lambda )]^T(\alpha (\tilde{\lambda }- \lambda )) dt \\&\ge h(\lambda ) + \nabla h(\lambda )^T(\alpha (\tilde{\lambda }- \lambda )) \\&-\int \limits _0^1 \Vert \nabla h(\lambda + t \alpha (\tilde{\lambda }- \lambda )) - \nabla h (\lambda )\Vert _*(\alpha )\Vert \tilde{\lambda }- \lambda \Vert dt \\&\ge h(\lambda ) \!+\! \nabla h(\lambda )^T(\alpha (\tilde{\lambda }- \lambda )) \!-\!\int \limits _0^1 L_{h, Q} \Vert (t \alpha )(\tilde{\lambda }- \lambda )\Vert ( \alpha )\Vert \tilde{\lambda }\!-\! \lambda \Vert dt \\&= h(\lambda ) + \nabla h(\lambda )^T(\alpha (\tilde{\lambda }- \lambda )) - \frac{\alpha ^2}{2}L_{h, Q}\Vert (\tilde{\lambda }- \lambda )\Vert ^2 \\&\ge h(\lambda ) + \nabla h(\lambda )^T(\alpha (\tilde{\lambda }- \lambda )) - \frac{\alpha ^2}{2}L_{h, Q}(\mathrm {Diam}_Q)^2, \end{aligned}$$

whereby it follows that $C_{h, Q} \le L_{h,Q}(\mathrm {Diam}_Q)^2$.

Proposition 7.3

For $k\ge 0$ the following inequality holds:

$$\begin{aligned} \sum _{i=0}^k \frac{i+1}{i+2} \le \frac{(k+1)(k+2)}{k+4}. \end{aligned}$$

Proof

The inequality above holds at equality for $k=0$. By induction, suppose the inequality is true for some given $k\ge 0$, then

$$\begin{aligned} \sum _{i=0}^{k+1} \frac{i+1}{i+2}&= \sum _{i=0}^{k} \frac{i+1}{i+2} + \frac{k+2}{k+3} \nonumber \\&\le \frac{(k+1)(k+2)}{k+4} + \frac{k+2}{k+3} \nonumber \\&= (k+2)\left[ \frac{k^2+5k+7}{k^2 + 7k +12}\right] . \end{aligned}$$

(57)

Now notice that

$$\begin{aligned}&(k^2 +5k+7)(k+5) = k^3 + 10k^2 +32k + 35 < k^3 + 10k^2 +\,33k + 36 \\&\quad = (k^2 + 7k +12)(k+3), \end{aligned}$$

which combined with (57) completes the induction.$\square$

Proposition 7.4

For $k\ge 1$ let $\bar{\alpha }:= 1-\frac{1}{\root k \of {k+1}}$. Then the following inequalities holds:

• (i) $\displaystyle \frac{\ln (k+1)}{k} \ge \bar{\alpha }$, and

• (ii) $(k+1)\bar{\alpha }\ge 1$.

Proof

To prove (i), define $f(t):= 1-e^{-t}$, and noting that $f(\cdot )$ is a concave function, the gradient inequality for $f(\cdot )$ at $t=0$ is

$$\begin{aligned} t \ge 1-e^{-t}. \end{aligned}$$

Substituting $t=\frac{\ln (k+1)}{k}$ yields

$$\begin{aligned} \frac{\ln (k+1)}{k} = t \ge 1-e^{-t} = 1 - e^{-\frac{\ln (k+1)}{k}} = 1- \frac{1}{\root k \of {k+1}} = \bar{\alpha }. \end{aligned}$$

Note that (ii) holds for $k = 1$, so assume now that $k \ge 2$. To prove (ii) for $k \ge 2$, substitute $t = -\frac{\ln (k+1)}{k}$ into the gradient inequality above to obtain $-\frac{\ln (k+1)}{k} \ge 1 - (k+1)^{\frac{1}{k}}$ which can be rearranged to:

$$\begin{aligned} (k+1)^{\frac{1}{k}} \ge 1 + \frac{\ln (k+1)}{k} \ge 1 + \frac{\ln (e)}{k} = 1 + \frac{1}{k} = \frac{k+1}{k}. \end{aligned}$$

(58)

Inverting (58) yields:

$$\begin{aligned} (k+1)^{-\frac{1}{k}} \le \frac{k}{k+1} = 1 - \frac{1}{k+1}. \end{aligned}$$

(59)

Finally, rearranging (59) and multiplying by $k+1$ yields (ii).$\square$

Proposition 7.5

For any integers $\ell , k$ with $2 \le \ell \le k$, the following inequalities hold:

$$\begin{aligned} \ln \left( \frac{k+1}{\ell }\right) \le \sum _{i = \ell }^k\frac{1}{i} \le \ln \left( \frac{k}{\ell - 1}\right) , \end{aligned}$$

(60)

and

$$\begin{aligned} \frac{k - \ell + 1}{(k+1)\ell } \le \sum _{i = \ell }^k\frac{1}{i^2} \le \frac{k - \ell + 1}{k(\ell -1)}. \end{aligned}$$

(61)

Proof

(60) and (61) are specific instances of the following more general fact: if $f(\cdot ): [1, \infty ) \rightarrow \mathbb {R}_+$ is a monotonically decreasing continuous function, then

$$\begin{aligned} \int _{\ell }^{k+1}f(t)dt \le \sum _{i = \ell }^kf(i) \le \int _{\ell - 1}^kf(t)dt. \end{aligned}$$

(62)

It is easy to verify that the integral expressions in (62) match the bounds in (60) and (61) for the specific choices of $f(t) = \frac{1}{t}$ and $f(t) = \frac{1}{t^2}$, respectively.$\square$