Convergence rate of block-coordinate maximization Burer–Monteiro method for solving large SDPs

Abstract

Semidefinite programming (SDP) with diagonal constraints arise in many optimization problems, such as Max-Cut, community detection and group synchronization. Although SDPs can be solved to arbitrary precision in polynomial time, generic convex solvers do not scale well with the dimension of the problem. In order to address this issue, Burer and Monteiro (Math Program 95(2):329–357, 2003) proposed to reduce the dimension of the problem by appealing to a low-rank factorization and solve the subsequent non-convex problem instead. In this paper, we present coordinate ascent based methods to solve this non-convex problem with provable convergence guarantees. More specifically, we prove that the block-coordinate maximization algorithm applied to the non-convex Burer–Monteiro method globally converges to a first-order stationary point with a sublinear rate without any assumptions on the problem. We further show that this algorithm converges linearly around a local maximum provided that the objective function exhibits quadratic decay. We establish that this condition generically holds when the rank of the factorization is sufficiently large. Furthermore, incorporating Lanczos method to the block-coordinate maximization, we propose an algorithm that is guaranteed to return a solution that provides $1-{\mathcal {O}}\left( 1/r\right)$ approximation to the original SDP without any assumptions, where r is the rank of the factorization. This approximation ratio is known to be optimal (up to constants) under the unique games conjecture, and we can explicitly quantify the number of iterations to obtain such a solution.

Appendices

Proof of Corollary 1

Similar to the proof of Theorem 1, from Proposition 1, we have

$$\begin{aligned} f(\varvec{\sigma }^{k+1}) - f(\varvec{\sigma }^k)&= 2 \left( \Vert {g_{i_k}^k}\Vert - \langle \sigma _{i_k}^k, g_{i_k}^k \rangle \right) , \nonumber \\&= \frac{2 \Vert {g_{i_k}^k}\Vert \left( \Vert {g_{i_k}^k}\Vert - \langle \sigma _{i_k}^k, g_{i_k}^k \rangle \right) }{\Vert {g_{i_k}^k}\Vert } ,\nonumber \\&\ge \frac{ \Vert {g_{i_k}^k}\Vert ^2 - \langle \sigma _{i_k}^k, g_{i_k}^k \rangle ^2 }{\Vert {g_{i_k}^k}\Vert }, \end{aligned}$$

(47)

where the inequality follows since $\Vert {g_{i_k}^k}\Vert \ge {\langle \sigma _{i_k}^k,g_{i_k}^k \rangle }$, for all $\sigma _{i_k}^k \in {{\mathbb {R}}}^{n \times r}$. Letting $\mathbb {E}_k$ denote the expectation over $i_k$ given $\sigma ^k$, we have

$$\begin{aligned} \mathbb {E}_k f(\varvec{\sigma }^{k+1}) - f(\varvec{\sigma }^k) \ge \sum _{i=1}^n p_i \frac{ \Vert {g_i^k}\Vert ^2 - \langle \sigma _i^k, g_i^k \rangle ^2 }{\Vert {g_i^k}\Vert }. \end{aligned}$$

In particular, when $p_i=\frac{1}{n}$, for all $i \in [n]$ (i.e., for uniform sampling case), we have

$$\begin{aligned} \mathbb {E}_k f(\varvec{\sigma }^{k+1}) - f(\varvec{\sigma }^k) \ge \frac{1}{n \Vert {{\varvec{A}}}\Vert _1} \, \sum _{i=1}^n \left( \Vert {g_i^k}\Vert ^2 - \langle \sigma _i^k, g_i^k \rangle ^2 \right) , \end{aligned}$$

since $\Vert {g_i^k}\Vert \le \Vert {{\varvec{A}}}\Vert _1$, for all $i \in [n]$ by (11). Therefore, we have

$$\begin{aligned} \mathbb {E}_k f(\varvec{\sigma }^{k+1}) - f(\varvec{\sigma }^k) \ge \frac{\Vert \mathrm {grad} f(\varvec{\sigma }^k)\Vert _{\mathrm {F}}^2}{2n \Vert {{\varvec{A}}}\Vert _1}. \end{aligned}$$

(48)

On the other hand, when $p_i=\frac{\Vert {g_i^k}\Vert }{\sum _{j=1}^n \Vert {g_j^k}\Vert }$ (i.e., for importance sampling case), we have

$$\begin{aligned} \mathbb {E}_k f(\varvec{\sigma }^{k+1}) - f(\varvec{\sigma }^k) \ge \frac{ \sum _{i=1}^n \Vert {g_i^k}\Vert ^2 - \langle \sigma _i^k, g_i^k \rangle ^2 }{\sum _{j=1}^n \Vert {g_j^k}\Vert } = \frac{ \Vert \mathrm {grad} f(\varvec{\sigma }^k)\Vert _{\mathrm {F}}^2}{2 \sum _{j=1}^n \Vert {g_j^k}\Vert }. \end{aligned}$$

Letting $\Vert {{\varvec{A}}}\Vert _{1,1} = \sum _{i,j=1}^n |{\varvec{A}}_{ij}|$ denote the $L_{1,1}$ norm of matrix ${\varvec{A}}$, we observe that $\sum _{j=1}^n \Vert {g_j^k}\Vert \le \Vert {{\varvec{A}}}\Vert _{1,1}$, which in the above inequality yields

$$\begin{aligned} \mathbb {E}_k f(\varvec{\sigma }^{k+1}) - f(\varvec{\sigma }^k) \ge \frac{\Vert \mathrm {grad} f(\varvec{\sigma }^k)\Vert _{\mathrm {F}}^2}{2\Vert {{\varvec{A}}}\Vert _{1,1}}. \end{aligned}$$

(49)

In order to prove (13), which corresponds to uniform sampling case, we assume the contrary that $\mathbb {E}\Vert \mathrm {grad} f(\varvec{\sigma }^k)\Vert _{\mathrm {F}}^2 > \epsilon$, for all $k\in [K-1]$. Then, using the boundedness of f, we get

$$\begin{aligned} f^* - f(\varvec{\sigma }^0)\ge & {} \mathbb {E}f(\varvec{\sigma }^K) - f(\varvec{\sigma }^0) = \sum _{k=0}^{K-1} \mathbb {E}\left[ f(\varvec{\sigma }^{k+1}) - f(\varvec{\sigma }^k) \right] \\= & {} \sum _{k=0}^{K-1} \mathbb {E}\left[ \mathbb {E}_k f(\varvec{\sigma }^{k+1}) - f(\varvec{\sigma }^k) \right] . \end{aligned}$$

Using the expected functional ascent of BCM in (48) above, we get

$$\begin{aligned} f^* - f(\varvec{\sigma }^0) \ge \sum _{k=0}^{K-1} \frac{\mathbb {E}\Vert \mathrm {grad} f(\varvec{\sigma }^k)\Vert _{\mathrm {F}}^2}{2n\Vert {{\varvec{A}}}\Vert _1} > \frac{K \epsilon }{2n\Vert {{\varvec{A}}}\Vert _1}, \end{aligned}$$

(50)

where the last inequality follows by the assumption. Then, by contradiction, the algorithm returns a solution with $\mathbb {E}\Vert \mathrm {grad} f(\varvec{\sigma }^k)\Vert _{\mathrm {F}}^2 \le \epsilon$, for some $k\in [K-1]$, provided that

$$\begin{aligned} K \ge \frac{2n\Vert {{\varvec{A}}}\Vert _1 (f^* - f(\varvec{\sigma }^0))}{\epsilon }. \end{aligned}$$

The proof of (14), which corresponds to importance sampling case, can be obtained by using (49) (instead of (48)) in (50), and hence is omitted.

Rest of the Proof of Theorem 2

In order to quantify how close $\varvec{\sigma }^0$ and $\varvec{\sigma }$ should be so that this convergence rate holds, we need to derive explicit bounds on the higher order terms in (21) and (23), which we do in the following. The Taylor expansion of $\varvec{\sigma }^k$ around $\varvec{\sigma }$ yields

$$\begin{aligned} \sigma _i^k&= \sigma _i \cos (\Vert {u_i}\Vert t) + \frac{u_i}{\Vert {u_i}\Vert } \sin (\Vert {u_i}\Vert t), \\&= \sigma _i \left[ \sum _{\ell =0}^\infty \frac{(-1)^\ell }{(2\ell )!} \left( \Vert {u_i}\Vert t \right) ^{2\ell } \right] + \frac{u_i}{\Vert {u_i}\Vert } \left[ \sum _{\ell =0}^\infty \frac{(-1)^\ell }{(2\ell +1)!} \left( \Vert {u_i}\Vert t \right) ^{2\ell +1} \right] . \end{aligned}$$

Using this expansion, we can compute $f(\varvec{\sigma }^k) = \sum _{i,j=1}^n A_{ij} {\langle \sigma _i^k, \sigma _j^k \rangle }$. The first three terms in the expansion are already given in (22) as follows

$$\begin{aligned} f(\varvec{\sigma }^k) = f(\varvec{\sigma }) + t^2 \sum _{i=1}^n \left( {\langle u_i,v_i \rangle } - \Vert {u_i}\Vert ^2 \Vert {g_i}\Vert \right) + \beta _f, \end{aligned}$$

(51)

where $\beta _f$ represents the higher order terms. In order to find an upper bound on $|\beta _f|$, we use the Cauchy-Schwarz inequality in the higher order terms in the expansion of $f(\varvec{\sigma }^k)$, which yields the following bound

$$\begin{aligned} |\beta _f| \le \sum _{i,j=1}^n |A_{ij}| \left( \sum _{\ell =3}^\infty \frac{t^\ell }{\ell !} ( \Vert {u_i}\Vert + \Vert {u_j}\Vert )^\ell \right) . \end{aligned}$$

As $\Vert {\varvec{u}}\Vert _{\mathrm {F}}=1$, we have $\Vert {u_i}\Vert \le 1$ for all $i\in [n]$, which implies

$$\begin{aligned} |\beta _f| \le \sum _{i,j=1}^n |A_{ij}| \left( \sum _{\ell =3}^\infty \frac{t^\ell }{\ell !} 2^\ell \right) , \end{aligned}$$

where we note that t denotes the geodesic distance between $\varvec{\sigma }^k$ and $[{\bar{\varvec{\sigma }}}]$ as highlighted in (19). Assuming that $t\le 1$, we obtain the following upper bound

$$\begin{aligned} |\beta _f| \le t^3 n \Vert {{\varvec{A}}}\Vert _1 \left( \sum _{\ell =3}^\infty \frac{2^\ell }{\ell !} \right) . \end{aligned}$$

Using the inequality $\sum _{\ell =3}^\infty \frac{2^\ell }{\ell !} = e^2-5 \le 5/2$ above, we get

$$\begin{aligned} |\beta _f| \le \frac{5n \Vert {{\varvec{A}}}\Vert _1t^3 }{2}. \end{aligned}$$

Plugging this value back in (51), we obtain

$$\begin{aligned} f(\varvec{\sigma }^k) \le f(\varvec{\sigma }) + t^2 \sum _{i=1}^n \left( {\langle u_i,v_i \rangle } - \Vert {u_i}\Vert ^2 \Vert {g_i}\Vert \right) + \frac{5n \Vert {{\varvec{A}}}\Vert _1t^3 }{2}. \end{aligned}$$

(52)

Considering the same expansion for $\Vert {\mathrm {grad}}f(\varvec{\sigma }^k)\Vert _{\mathrm {F}}^2 = 2 \sum _{i=1}^n (\Vert {g_i^k}\Vert ^2 - {\langle \sigma _i^k,g_i^k \rangle }^2)$, we get the following (see (21)):

$$\begin{aligned} \Vert {\mathrm {grad}}f(\varvec{\sigma }^k)\Vert _{\mathrm {F}}^2 = 2 t^2 \sum _{i=1}^n \left( \Vert {u_i}\Vert \Vert {g_i}\Vert - {\langle \frac{u_i}{\Vert {u_i}\Vert },v_i \rangle } \right) ^2 + \beta _g, \end{aligned}$$

(53)

where $\beta _g$ represents the higher order terms. Upper bounding each higher order terms using the Cauchy-Schwarz inequality as follows, we obtain

$$\begin{aligned} |\beta _g|\le & {} 2 \sum _{i=1}^n \left[ \sum _{j,m=1}^n |A_{ij}| |A_{im}| \left( \sum _{\ell =3}^\infty \frac{t^\ell }{\ell !} ( \Vert {u_j}\Vert + \Vert {u_m}\Vert )^\ell \right) \right. \\&\left. + \sum _{j,m=1}^n |A_{ij}| |A_{im}| \left( \sum _{\begin{array}{c} \ell ,s=0 \\ \ell +s\ge 3 \end{array}}^\infty \frac{t^{\ell +s}}{\ell ! s!} ( \Vert {u_i}\Vert + \Vert {u_j}\Vert )^{\ell +s} \right) \right] . \end{aligned}$$

Using the fact that $\Vert {u_i}\Vert \le 1$ for all $i\in [n]$, we get the following upper bound

$$\begin{aligned} |\beta _g| \le 2 \sum _{i=1}^n \left[ \sum _{j,m=1}^n |A_{ij}| |A_{im}| \left( \sum _{\ell =3}^\infty \frac{t^\ell }{\ell !} 2^\ell \right) + \sum _{j,m=1}^n |A_{ij}| |A_{im}| \left( \sum _{\begin{array}{c} \ell ,s=0 \\ \ell +s\ge 3 \end{array}}^\infty \frac{t^{\ell +s}}{\ell ! s!} 2^{\ell +s} \right) \right] . \end{aligned}$$

Using the upper bound $\sum _{j,m=1}^n |A_{ij}| |A_{im}| \le \Vert {{\varvec{A}}}\Vert _1^2$ above, we obtain

$$\begin{aligned} |\beta _g| \le 2 \Vert {{\varvec{A}}}\Vert _1^2 \sum _{i=1}^n \left[ \sum _{\ell =3}^\infty \frac{t^\ell }{\ell !} 2^\ell + \sum _{\begin{array}{c} \ell ,s=0 \\ \ell +s\ge 3 \end{array}}^\infty \frac{t^{\ell +s}}{\ell ! s!} 2^{\ell +s} \right] . \end{aligned}$$

Introducing a change of variables in the last sum, we get

$$\begin{aligned} |\beta _g|&\le 2 \Vert {{\varvec{A}}}\Vert _1^2 \sum _{i=1}^n \left[ \sum _{\ell =3}^\infty \frac{t^\ell }{\ell !} 2^\ell + \sum _{\ell =3}^\infty \frac{t^\ell }{\ell !} 2^\ell \left( \sum _{s=0}^\ell \frac{\ell !}{s!(\ell -s)!} \right) \right] ,\\&= 2 \Vert {{\varvec{A}}}\Vert _1^2 \sum _{i=1}^n \left[ \sum _{\ell =3}^\infty \frac{t^\ell }{\ell !} \left( 2^\ell + 4^\ell \right) \right] . \end{aligned}$$

Assuming that $t\le 1$, we obtain the following upper bound

$$\begin{aligned} |\beta _g| \le 2 \Vert {{\varvec{A}}}\Vert _1^2 t^3 \sum _{i=1}^n \left[ \sum _{\ell =3}^\infty \frac{1}{\ell !} \left( 2^\ell + 4^\ell \right) \right] . \end{aligned}$$

Using the inequality $\sum _{\ell =3}^\infty \frac{2^\ell +4^\ell }{\ell !} = e^2+e^4-18 \le 44$ above, we get

$$\begin{aligned} |\beta _g| \le 88 n \Vert {{\varvec{A}}}\Vert _1^2 t^3. \end{aligned}$$

Plugging this value back in (53), we obtain

$$\begin{aligned} \Vert {\mathrm {grad}}f(\varvec{\sigma }^k)\Vert _{\mathrm {F}}^2 \ge 2 t^2 \sum _{i=1}^n \left( \Vert {u_i}\Vert \Vert {g_i}\Vert - {\langle \frac{u_i}{\Vert {u_i}\Vert },v_i \rangle } \right) ^2 - 88 n \Vert {{\varvec{A}}}\Vert _1^2 t^3. \end{aligned}$$

(54)

Using the same bounding technique as in (24), we get

$$\begin{aligned} \Vert \mathrm {grad} f(\varvec{\sigma }^k)\Vert _{\mathrm {F}}^2&\ge \frac{\mu }{n} \left( f({\bar{\varvec{\sigma }}}) - f(\varvec{\sigma }^k) - \frac{5n \Vert {{\varvec{A}}}\Vert _1t^3 }{2} \right) - 88 n \Vert {{\varvec{A}}}\Vert _1^2 t^3, \\&= \frac{\mu }{n} \left( f({\bar{\varvec{\sigma }}}) - f(\varvec{\sigma }^k) \right) - t^3 \Vert {{\varvec{A}}}\Vert _1 \left( 3\mu + 88 n \Vert {{\varvec{A}}}\Vert _1 \right) . \end{aligned}$$

Therefore, in order for (25) to hold, we need

$$\begin{aligned} t^3 \Vert {{\varvec{A}}}\Vert _1 \left( 3\mu + 88 n \Vert {{\varvec{A}}}\Vert _1 \right) \le \frac{\mu }{2n} \left( f({\bar{\varvec{\sigma }}}) - f(\varvec{\sigma }^k) \right) , \end{aligned}$$

which can be equivalently rewritten as follows

$$\begin{aligned} t^3 \le \frac{\mu (f({\bar{\varvec{\sigma }}}) - f(\varvec{\sigma }^k))}{2n \Vert {{\varvec{A}}}\Vert _1 \left( 3\mu + 88 n \Vert {{\varvec{A}}}\Vert _1 \right) }. \end{aligned}$$

As $f(\varvec{\sigma }^k)$ is a monotonically non-decreasing sequence, then as soon as $\varvec{\sigma }^0$ is sufficiently close to $[{\bar{\varvec{\sigma }}}]$ in the sense that

$$\begin{aligned} \mathrm {dist}(\varvec{\sigma }^0, [{\bar{\varvec{\sigma }}}]) \le \left( \frac{\mu (f({\bar{\varvec{\sigma }}}) - f(\varvec{\sigma }^k))}{2n \Vert {{\varvec{A}}}\Vert _1 \left( 3\mu + 88 n \Vert {{\varvec{A}}}\Vert _1 \right) } \right) ^{1/3}, \end{aligned}$$

then the linear convergence rate presented in (26) holds.

Proof of Theorem 7

Before presenting the proof of Theorem 7, we first introduce the following theorem that characterizes the convergence rate of the Lanczos method with random initialization.

Theorem 8

[28, Theorem 4.2] Let ${\varvec{A}}\in {{\mathbb {R}}}^{n \times n}$ be a positive semidefinite matrix, $b\in {{\mathbb {R}}}^n$ be an arbitrary vector and $\lambda _L^\ell ({\varvec{A}},b)$ denote the output of the Lanczos algorithm after $\ell$ iterations when applied to find the leading eigenvalue of ${\varvec{A}}$ (denoted by $\lambda _1({\varvec{A}})$) with initialization b. In particular,

$$\begin{aligned} \lambda _L^\ell ({\varvec{A}},b) = \max \left\{ \frac{{\langle x,{\varvec{A}}x \rangle }}{{\langle x,x \rangle }} : 0 \ne x \in \mathrm {span}(b,\dots ,{\varvec{A}}^{\ell -1}b) \right\} . \end{aligned}$$

Assume that b is uniformly distributed over the set $\{b\in {{\mathbb {R}}}^n : \Vert {b}\Vert =1\}$ and let $\epsilon \in [0,1)$. Then, the probability that the Lanczos algorithm does not return an $\epsilon$-approximation to the leading eigenvalue of ${\varvec{A}}$ exponentially decreases as follows

$$\begin{aligned} \mathbb {P}\left( \lambda _L^\ell ({\varvec{A}},b)< (1-\epsilon ) \lambda _1({\varvec{A}}) \right) {\left\{ \begin{array}{ll} \le 1.648 \sqrt{n} e^{-\sqrt{\epsilon }(2\ell -1)}, &{} \text {if } 0<\ell <n(r-1), \\ = 0, &{} \text {if } \ell \ge n(r-1). \end{array}\right. } \end{aligned}$$

Using this result, Theorem 7 is proven as follows. Since the tangent space $T_{\varvec{\sigma }}{\mathcal {M}}_r$ has dimension $n(r-1)$, then we can define a symmetric matrix (where we drop the notational dependency on $\varvec{\sigma }$ for simplicity) ${\varvec{H}}\in {{\mathbb {R}}}^{n(r-1) \times n(r-1)}$ that represents the linear operator ${\mathrm {Hess}}f(\varvec{\sigma })$ in the basis $\{ {\varvec{u}}^1,\dots ,{\varvec{u}}^{n(r-1)} \}$ such that $\mathrm {span}({\varvec{u}}^1,\dots ,{\varvec{u}}^{n(r-1)}) = T_{\varvec{\sigma }}{\mathcal {M}}_r$. In particular, letting $H_{ij} = {\langle {\varvec{u}}^i, {\mathrm {Hess}}f(\varvec{\sigma })[{\varvec{u}}^j] \rangle }$ yields the desired matrix ${\varvec{H}}$ and the Lanczos algorithm is run to find the leading eigenvalue of this matrix. Here, it is important to note that ${\varvec{H}}$ is not a psd matrix, so it is required to shift ${\varvec{H}}$ with a large enough multiple of the identity matrix so that the resulting matrix is guaranteed to be positive semidefinite. In particular, by inspecting the definition of ${\mathrm {Hess}}f(\varvec{\sigma })$ in (5), it is easy to observe that $\Vert {{\mathrm {Hess}}f(\varvec{\sigma })}\Vert _{\text {op}} \le 4\Vert {{\varvec{A}}}\Vert _1$. Therefore, it is sufficient to run the Lanczos algorithm to find the leading eigenvalue of ${\widetilde{{\varvec{H}}}} = {\varvec{H}}+4\Vert {{\varvec{A}}}\Vert _1 {\varvec{I}}$, where ${\varvec{I}}$ denotes the appropriate sized identity matrix. On the other hand, we initialize the Lanczos algorithm with a random vector ${\varvec{u}}$ of unit norm (i.e., $\Vert {\varvec{u}}\Vert _{\mathrm {F}}=1$) in the tangent space $T_{\varvec{\sigma }}{\mathcal {M}}_r$. Notice that ${\varvec{u}}$ can equivalently be represented as a vector $b\in {{\mathbb {R}}}^{n(r-1)}$ in the basis $\{ {\varvec{u}}^1,\dots ,{\varvec{u}}^{n(r-1)} \}$ as ${\varvec{u}}= \sum _{i=1}^{n(r-1)} b_i {\varvec{u}}^i$ such that $\Vert {b}\Vert =1$. Then, by Theorem 8, we have

$$\begin{aligned} \mathbb {P}\left( \lambda _L^\ell ({\widetilde{{\varvec{H}}}},b) < (1-\epsilon ) \lambda _1({\widetilde{{\varvec{H}}}}) \right) \le 1.648 \sqrt{n(r-1)} e^{-\sqrt{\epsilon }(2\ell -1)}. \end{aligned}$$

Letting $\lambda _1({\varvec{H}})$ denote the leading eigenvalue of ${\varvec{H}}$, we run the Lanczos algorithm to obtain a vector $b^*$ such that $\Vert {b^*}\Vert =1$ and ${\langle b^*, {\varvec{H}}b^* \rangle } \ge \lambda _1({\varvec{H}})/2$. Thus, we want $\mathbb {P}\left( \lambda _L^\ell ({\widetilde{{\varvec{H}}}},b) < 4\Vert {{\varvec{A}}}\Vert _1 + \lambda _1({\varvec{H}})/2 \right)$ to be small. Setting $\epsilon ^* = \frac{\lambda _1({\varvec{H}})}{16\Vert {{\varvec{A}}}\Vert _1}$, we can observe that

$$\begin{aligned} \left( 1-\epsilon ^* \right) \lambda _1({\widetilde{{\varvec{H}}}})&= \left( 1-\frac{\lambda _1({\varvec{H}})}{16\Vert {{\varvec{A}}}\Vert _1} \right) \left( 4\Vert {{\varvec{A}}}\Vert _1 + \lambda _1({\varvec{H}}) \right) , \\&= 4\Vert {{\varvec{A}}}\Vert _1 + \frac{3\lambda _1({\varvec{H}})}{4} - \frac{(\lambda _1({\varvec{H}}))^2}{16\Vert {{\varvec{A}}}\Vert _1}, \\&\ge 4\Vert {{\varvec{A}}}\Vert _1 + \frac{\lambda _1({\varvec{H}})}{2}, \end{aligned}$$

where the inequality follows since $\lambda _1({\varvec{H}}) \le 4\Vert {{\varvec{A}}}\Vert _1$. Consequently, we have

$$\begin{aligned}&\mathbb {P}\left( \lambda _L^\ell ({\widetilde{{\varvec{H}}}},b)< 4\Vert {{\varvec{A}}}\Vert _1 + \lambda _1({\varvec{H}})/2 \right) \\&\quad \le \mathbb {P}\left( \lambda _L^\ell ({\widetilde{{\varvec{H}}}},b) < (1-\epsilon ^*) \lambda _1({\widetilde{{\varvec{H}}}}) \right) \le 1.648 \sqrt{n(r-1)} e^{-\sqrt{\epsilon ^*}(2\ell -1)}. \end{aligned}$$

By Theorem 6, we know that the Lanczos method is called at most $\left\lceil 675 n \Vert {{\varvec{A}}}\Vert _1^2 / \varepsilon ^2 \right\rceil$ times to search for an $\varepsilon$-approximate concave point and for any non-desired solution we have $\lambda _1({\varvec{H}}) \ge \varepsilon$ by the definition of $\varepsilon$-approximate concave point. Then, by using a union bound over all calls to the Lanczos method, we conclude that when the Lanczos method is run for $\ell$ iterations, we have the following guarantee

$$\begin{aligned}&\mathbb {P}\left( \text {Algorithm 2+3 fails to return an} \varepsilon \text {-approximate concave point} \right) \\&\quad \le \left\lceil \frac{675 n \Vert {{\varvec{A}}}\Vert _1^2}{\varepsilon ^2} \right\rceil 1.648 \sqrt{n(r-1)} e^{-\sqrt{\frac{\varepsilon }{16\Vert {{\varvec{A}}}\Vert _1}}(2\ell -1)}. \end{aligned}$$

In order to set this probability to some $\delta \in (0,1)$, we let

$$\begin{aligned} \ell ^*= & {} \left\lceil \left( \frac{1}{2} + 2 \sqrt{\frac{\Vert {{\varvec{A}}}\Vert _1}{\varepsilon }} \right) \log \left( \frac{\left\lceil \frac{675 n \Vert {{\varvec{A}}}\Vert _1^2}{\varepsilon ^2} \right\rceil 1.648 \sqrt{n(r-1)}}{\delta } \right) \right\rceil \\= & {} \widetilde{{\mathcal {O}}}\left( \sqrt{\frac{\Vert {{\varvec{A}}}\Vert _1}{\varepsilon }} \log \left( \frac{n\sqrt{n(r-1)}}{\delta } \right) \right) , \end{aligned}$$

where tilde is used to hide poly-logarithmic factors in $\Vert {{\varvec{A}}}\Vert _1 / \varepsilon$. Since the Lanczos algorithm is guaranteed to return the leading eigenvalue with probability 1 in at most $n(r-1)$ iterations, then running each Lanczos subroutine for $\min (\ell ^*,n(r-1))$ iterations, it is guaranteed that Algorithm 2+3 returns an $\varepsilon$-approximate concave point with probability at least $1-\delta$.