Linear convergence of Frank–Wolfe for rank-one matrix recovery without strong convexity

Abstract

We consider convex optimization problems which are widely used as convex relaxations for low-rank matrix recovery problems. In particular, in several important problems, such as phase retrieval and robust PCA, the underlying assumption in many cases is that the optimal solution is rank-one. In this paper we consider a simple and natural sufficient condition on the objective so that the optimal solution to these relaxations is indeed unique and rank-one. Mainly, we show that under this condition, the standard Frank–Wolfe method with line-search (i.e., without any tuning of parameters whatsoever), which only requires a single rank-one SVD computation per iteration, finds an $\epsilon$-approximated solution in only $O(\log {1/\epsilon })$ iterations (as opposed to the previous best known bound of $O(1/\epsilon )$), despite the fact that the objective is not strongly convex. We consider several variants of the basic method with improved complexities, as well as an extension motivated by robust PCA, and finally, an extension to nonsmooth problems.

Appendix

1.1 A Proof of Lemma 2

The lemma is an adaptation of Lemma 3 in [16] (which considers optimization over trace-norm balls). We restate and prove a slightly more general version of the lemma.

Lemma 10

Let $f:\mathbb {S}^n\rightarrow \mathbb {R}$ be $\beta$-smooth and convex. Let ${\mathbf {X}}^*\in \mathcal {S}_n$ be an optimal solution of rank r to the optimization problem $\min _{{\mathbf {X}}\in \mathcal {S}_n}f({\mathbf {X}})$. Let $\lambda _1,\dots ,\lambda _n$ denote the eigenvalues of $\nabla {}f({\mathbf {X}}^*)$ in non-increasing order. Let $\zeta$ be a non-negative scalar. It holds that

$$\begin{aligned} \text {rank}(\varPi _{(1+\zeta )\mathcal {S}_n}[{\mathbf {X}}^*-\beta ^{-1}\nabla {}f({\mathbf {X}}^*)])> r \quad \Longleftrightarrow \quad \zeta > r\beta {}(\lambda _{n-r}-\lambda _n), \end{aligned}$$

where $(1+\zeta )\mathcal {S}_n = \{(1+\zeta ){\mathbf {X}}~|~{\mathbf {X}}\in \mathcal {S}_n\}$, and $\varPi _{(1+\zeta )\mathcal {S}_n}[\cdot ]$ denotes the Euclidean projection onto the convex set $(1+\zeta )\mathcal {S}_n$.

Proof

Let us write the eigen-decomposition of ${\mathbf {X}}^*$ as ${\mathbf {X}}^*=\sum _{i=1}^r\lambda _i^*{\mathbf {v}}_i{\mathbf {v}}_i^{\top }$. It follows from the optimality of ${\mathbf {X}}^*$ that for all $i\in [r]$, ${\mathbf {v}}_i$ is also an eigenvector of $\nabla {}f({\mathbf {X}}^*)$ which corresponds to the smallest eigenvalue $\lambda _n$ (see Lemma 7 in [16]). Thus, if we let $\rho _1,\dots ,\rho _n$ denote the eigenvalues (in non-increasing order) of ${\mathbf {Y}}:= {\mathbf {X}}^*-\beta ^{-1}\nabla {}f({\mathbf {X}}^*)$, it holds that

$$\begin{aligned} \forall i\in [r]: \quad \rho _i= & {} \lambda _i^* - \beta ^{-1}\lambda _n;\\ \forall i>r: \quad \rho _i= & {} \lambda _i^* - \beta ^{-1}\lambda _{n-i+1}. \end{aligned}$$

Recall that $\sum _{i=1}^r\lambda _i^* =1$ and $\lambda _{r+1}^* = 0$.

It is well known that for any matrix ${\mathbf {M}}\in \mathbb {S}^n$ with eigen-decomposition ${\mathbf {M}}=\sum _{i=1}^n\sigma _i{\mathbf {u}}_i{\mathbf {u}}_i^{\top }$, the projection of ${\mathbf {M}}$ onto the set $(1+\zeta )\mathcal {S}_n$, for any $\zeta \ge 0$ is given by

$$\begin{aligned} \varPi _{(1+\zeta )\mathcal {S}_n}[{\mathbf {M}}] = \sum _{i=1}^n\max \{0,~\sigma _i-\sigma \}{\mathbf {u}}_i{\mathbf {u}}_i^{\top }, \end{aligned}$$

where $\sigma \in \mathbb {R}$ is the unique scalar such that $\sum _{i=1}^n\max \{0,~\sigma _i-\sigma \} = 1+\zeta$.

Now, we can see that $\text {rank}(\varPi _{(1+\zeta )\mathcal {S}_n}[{\mathbf {Y}}]) \le r$ if and only if $\sigma \ge \rho _{r+1} = -\beta ^{-1}\lambda _{n-r}$. Thus, if $\text {rank}(\varPi _{(1+\zeta )\mathcal {S}_n}[{\mathbf {Y}}]) \le r$ then it must hold that $\sigma \ge -\beta ^{-1}\lambda _{n-r}$ which implies that

$$\begin{aligned} 1 + \zeta&= \sum _{i=1}^n\max \{0,~\rho _i-\sigma \} \nonumber \\&= \sum _{i=1}^r\max \{0,~\rho _i-\sigma \} \le \sum _{i=1}^r\max \{0,~\rho _i-(-\beta ^{-1}\lambda _{n-r})\} \nonumber \\&= \sum _{i=1}^r(\rho _i-(-\beta ^{-1}\lambda _{n-r})) = \sum _{i=1}^r(\lambda _i +\beta (\lambda _{n-r}-\lambda _n)) = 1 + \beta {}r(\lambda _{n-r}-\lambda _{n}). \end{aligned}$$

(37)

However, (37) can hold only if $\zeta \le \beta {}r(\lambda _{n-r}-\lambda _n)$. Thus, we have $\text {rank}(\varPi _{(1+\zeta )\mathcal {S}_n}[{\mathbf {Y}}]) \le r \Longrightarrow \zeta \le \beta {}r(\lambda _{n-r}-\lambda _n)$.

On the other-hand, if $\text {rank}(\varPi _{(1+\zeta )\mathcal {S}_n}[{\mathbf {Y}}]) > r$ then it must hold that $\sigma < -\beta ^{-1}\lambda _{n-r}$ which, using the same arguments as above, implies that

$$\begin{aligned} 1 + \zeta&= \sum _{i=1}^n\max \{0,~\rho _i-\sigma \} \nonumber \\&> \sum _{i=1}^r\max \{0,~\rho _i-(-\beta ^{-1}\lambda _{n-r})\} =1 + \beta {}r(\lambda _{n-r}-\lambda _{n}). \end{aligned}$$

(38)

We see that (38) can hold only if $\zeta > \beta {}r(\lambda _{n-r}-\lambda _n)$. Thus, we also have $\text {rank}(\varPi _{(1+\zeta )\mathcal {S}_n}[{\mathbf {Y}}])> r \Longrightarrow \zeta > \beta {}r(\lambda _{n-r}-\lambda _n)$, and the lemma follows. $\square$

1.2 B Proof of Lemma 3

We first restate the lemma and then prove it.

Lemma 11

Let $f:\mathbb {S}^n\rightarrow \mathbb {R}$ be $\beta$-smooth and convex. Suppose that Assumption 1 holds w.r.t. $f(\cdot )$ with some parameter $\delta >0$. Let ${\tilde{f}}:\mathbb {S}^n\rightarrow \mathbb {R}$ be differentiable and convex, and suppose that $\sup _{{\mathbf {X}}\in \mathcal {S}_n}\Vert {\nabla {}f({\mathbf {X}}) - \nabla {\tilde{f}}({\mathbf {X}})}\Vert _F \le \nu$, for some $\nu > 0$. Then, for $\nu < \frac{1}{2}(1+\frac{2\beta }{\delta })^{-1}\delta$, Assumption 1 holds w.r.t. the function ${\tilde{f}}(\cdot )$ with parameter ${\tilde{\delta }} = \delta - 2\nu (1+\frac{2\beta }{\delta }) > 0$.

Proof

Let ${\mathbf {X}}^*$ and ${\tilde{{\mathbf {X}}}}^*$ denote minimizers of $f(\cdot )$ and ${\tilde{f}}(\cdot )$ over $\mathcal {S}_n$, respectively. Since Assumption 1 holds w.r.t. $f(\cdot )$, using the quadratic growth result of Lemma 4 we have that

$$\begin{aligned} \Vert {{\tilde{{\mathbf {X}}}}^* - {\mathbf {X}}^*}\Vert _F^2&\le \frac{2}{\delta }\left( {f({\tilde{{\mathbf {X}}}}^*)-f({\mathbf {X}}^*)}\right) \underset{(a)}{\le } \frac{2}{\delta }\langle {{\tilde{{\mathbf {X}}}}^*-{\mathbf {X}}^*, \nabla {}f({\tilde{{\mathbf {X}}}}^*)}\rangle \\&= \frac{2}{\delta }\left( {\langle {{\tilde{{\mathbf {X}}}}^*-{\mathbf {X}}^*, \nabla {}{\tilde{f}}({\tilde{{\mathbf {X}}}}^*)}\rangle + \langle {{\tilde{{\mathbf {X}}}}^*-{\mathbf {X}}^*, \nabla {}f({\tilde{{\mathbf {X}}}}^*)-\nabla {}{\tilde{f}}({\tilde{{\mathbf {X}}}}^*)}\rangle }\right) \\&\underset{(b)}{\le } \frac{2}{\delta } \langle {{\tilde{{\mathbf {X}}}}^*-{\mathbf {X}}^*, \nabla {}f({\tilde{{\mathbf {X}}}}^*) - \nabla {}{\tilde{f}}({\tilde{{\mathbf {X}}}}^*)\rangle } \underset{(c)}{\le } \frac{2\nu }{\delta }\Vert {{\tilde{{\mathbf {X}}}}^*-{\mathbf {X}}^*}\Vert _F, \end{aligned}$$

where (a) follows from convexity of $f(\cdot )$, (b) follows from optimality of ${\tilde{{\mathbf {X}}}}^*$ w.r.t. ${\tilde{f}}(\cdot )$, and (c) follows from the Cauchy-Schwarz inequality and the assumption of the lemma that $\sup _{{\mathbf {X}}\in \mathcal {S}_n}\Vert {\nabla {}f({\mathbf {X}}) - \nabla {\tilde{f}}({\mathbf {X}})}\Vert _F \le \nu$.

Thus, we get that $\Vert {{\tilde{{\mathbf {X}}}}^*-{\mathbf {X}}^*}\Vert _F \le \frac{2\nu }{\delta }$.

Using Weyl’s inequality for the eigenvalues we have that

$$\begin{aligned} \lambda _{n}(\nabla {}{\tilde{f}}({\tilde{{\mathbf {X}}}}^*))&\le \lambda _n(\nabla {}f({\mathbf {X}}^*)) + \Vert {\nabla {}{\tilde{f}}({\tilde{{\mathbf {X}}}}^*) - \nabla {}f({\mathbf {X}}^*)}\Vert _F \nonumber \\&= \lambda _{n-1}(\nabla {}f({\mathbf {X}}^*)) - \delta + \Vert {\nabla {}{\tilde{f}}({\tilde{{\mathbf {X}}}}^*) - \nabla {}f({\mathbf {X}}^*)}\Vert _F \nonumber \\&\le \lambda _{n-1}(\nabla {}{\tilde{f}}({\tilde{{\mathbf {X}}}}^*)) - \delta + 2\Vert {\nabla {}{\tilde{f}}({\tilde{{\mathbf {X}}}}^*) - \nabla {}f({\mathbf {X}}^*)}\Vert _F. \end{aligned}$$

(39)

Using the smoothness of $f(\cdot )$ and the assumption $\sup _{{\mathbf {X}}\in \mathcal {S}_n}\Vert {\nabla {}f({\mathbf {X}}) - \nabla {\tilde{f}}({\mathbf {X}})}\Vert _F \le \nu$, we have that

$$\begin{aligned} \Vert {\nabla {}{\tilde{f}}({\tilde{{\mathbf {X}}}}^*) - \nabla {}f({\mathbf {X}}^*)}\Vert _F&\le \Vert {\nabla {}{\tilde{f}}({\tilde{{\mathbf {X}}}}^*) - \nabla {}f({\tilde{{\mathbf {X}}}}^*)}\Vert _F + \Vert {\nabla {}f({\tilde{{\mathbf {X}}}}^*) - \nabla {}f({\mathbf {X}}^*)}\Vert _F \nonumber \\&\le \nu + \beta \Vert {{\tilde{{\mathbf {X}}}}^*-{\mathbf {X}}^*}\Vert _F \le \nu \left( {1+\frac{2\beta }{\delta }}\right) . \end{aligned}$$

(40)

Plugging-in (40) into (39) and rearranging we obtain

$$\begin{aligned} \lambda _{n-1}(\nabla {}{\tilde{f}}({\tilde{{\mathbf {X}}}}^*)) - \lambda _{n}(\nabla {}{\tilde{f}}({\tilde{{\mathbf {X}}}}^*)) \ge \delta - 2\nu \left( {1+\frac{2\beta }{\delta }}\right) . \end{aligned}$$

Thus, Assumption 1 indeed holds w.r.t. ${\tilde{f}}(\cdot )$ whenever $\nu < \frac{1}{2}(1+\frac{2\beta }{\delta })^{-1}\delta$. $\square$