Towards Accelerated Rates for Distributed Optimization over Time-Varying Networks

Abstract

We study the problem of decentralized optimization with strongly convex smooth cost functions. This paper investigates accelerated algorithms under time-varying network constraints. In our approach, nodes run a multi-step gossip procedure after taking each gradient update, thus ensuring approximate consensus at each iteration. The outer cycle is based on accelerated Nesterov scheme. Both computation and communication complexities of our method have an optimal dependence on global function condition number $\kappa _g$. In particular, the algorithm reaches an optimal computation complexity $O(\sqrt{\kappa _g}\log (1/\varepsilon ))$.

The research of A. Rogozin was partially supported by RFBR 19-31-51001 and was partially done in Sirius (Sochi). The research of A. Gasnikov was partially supported by the Ministry of Science and Higher Education of the Russian Federation (Goszadaniye) 075-00337-20-03, project no. 0714-2020-0005.

Appendices

Supplementary Material

A Proof of Theorem 3

First, note that Algorithm 2 comes down to the following iterative procedure in $\mathbb {R}^d$.

$$\begin{aligned} \overline{y}^{k+1}&= \frac{\alpha ^{k+1}\overline{u}^k + A^k\overline{x}^k}{A^{k+1}} \\ \overline{u}^{k + 1}&= \mathop {\text {arg min}}\limits _{\overline{z}\in \mathbb {R}^d}\left\{ \alpha ^{k+1} \left( \left\langle \frac{1}{n}\sum _{i=1}^n \nabla f(y_i^{k+1}), \overline{z} - \overline{y}^{k+1} \right\rangle + \frac{\mu }{2}\left\| \overline{z} - \overline{y}^{k+1} \right\| ^2 \right) + \frac{1+ A^k\mu }{2}\left\| \overline{z} - \overline{u}^k \right\| ^2 \right\} \\ \overline{x}^{k+1}&= \frac{\alpha ^{k+1}\overline{u}^{k+1} + A^k\overline{x}^k}{A^{k+1}} \end{aligned}$$

1.1 A.1 Outer Loop

Initially we recall basic properties of coefficients $A^k$ which immediately follow from Lemma 3.7 in [28] (for details see the full technical report of this paper [34]).

Lemma 2

For coefficients $A^k$, it holds

Lemma 3

Provided that consensus accuracy is $\delta '$, i.e. $\left\| \mathbf{U}^{j} - \overline{\mathbf{U}}^j \right\| ^2\leqslant \delta ' \text { for } j = 1, \ldots , k$, we have

$$\begin{aligned} f(\overline{x}^k) - f(x^*)&\leqslant \frac{\left\| \overline{u}^0 - x^* \right\| ^2}{2A^k} + \frac{2\sum _{j=1}^{k} A^j\delta }{A^k} \\ \left\| \overline{u}^k - x^* \right\| ^2&\leqslant \frac{\left\| \overline{u}^0 - x^* \right\| ^2}{1 + A^k\mu } + \frac{4\sum _{j=1}^{k} A^j\delta }{1 + A^k\mu } \end{aligned}$$

where $\delta$ is given in (5).

Proof

First, assuming that $\left\| \mathbf{U}^{j} - \overline{\mathbf{U}}^j \right\| ^2\leqslant \delta '$, we show that $\mathbf{Y}^j, \mathbf{U}^j, \mathbf{X}^j$ lie in $\sqrt{\delta '}$-neighborhood of $\mathcal {C}$ by induction. At $j=0$, we have $\left\| \mathbf{X}^0 - \overline{\mathbf{X}}^0 \right\| = \left\| \mathbf{U}^0 - \overline{\mathbf{U}}^0 \right\| = 0$. Using $A^{j+1} = A^j + \alpha ^j$, we get an induction pass $j\rightarrow j+1$.

$$\begin{aligned} \left\| \mathbf{Y}^{j+1} - \overline{\mathbf{Y}}^{j+1} \right\|&\leqslant \frac{\alpha ^{j+1}}{A^{j+1}}\left\| \mathbf{U}^j - \overline{\mathbf{U}}^j \right\| + \frac{A^j}{A^{j+1}} \left\| \mathbf{X}^j - \overline{\mathbf{X}}^j \right\| \leqslant \sqrt{\delta '} \\ \left\| \mathbf{X}^{j+1} - \overline{\mathbf{X}}^{j+1} \right\|&\leqslant \frac{\alpha ^{j+1}}{A^{j+1}}\left\| \mathbf{U}^{j+1} - \overline{\mathbf{U}}^{j+1} \right\| + \frac{A^j}{A^{j+1}} \left\| \mathbf{X}^j - \overline{\mathbf{X}}^j \right\| \leqslant \sqrt{\delta '} \end{aligned}$$

Therefore, $g(\overline{y}) = \frac{1}{n}\sum _{i=1}^n \nabla f(y_i)$ is a gradient from $(\delta , L, \mu )$-model of f, and the desired result directly follows from Theorem 3.4 in [28].

1.2 A.2 Consensus Subroutine Iterations

We specify the number of iteration required for reaching accuracy $\delta '$ in the following Lemma, which is proved in the extended version of this paper [34].

Lemma 4

Let consensus accuracy be maintained at level $\delta '$, i.e. $\left\| \mathbf{U}^j - \overline{\mathbf{U}}^j \right\| ^2\leqslant \delta ' \text { for } j = 1, \ldots , k$ and let Assumption 2 hold. Define

$$\begin{aligned} \sqrt{D} := \left( \frac{2{L_{l}}}{\sqrt{L\mu }} + 1 \right) \sqrt{\delta '} + \frac{{L_{l}}}{\mu } \sqrt{n} \left( \left\| \overline{u}^0 - x^* \right\| ^2 + \frac{8{\delta '}}{\sqrt{L\mu }}\right) ^{1/2} + \frac{2\left\| \nabla F(\mathbf{X}^*) \right\| }{\sqrt{L\mu }} \end{aligned}$$

Then it is sufficient to make $T_k = T = \frac{\tau }{2\lambda }\log \frac{D}{\delta '}$ consensus iterations in order to ensure $\delta '$-accuracy on step $k+1$, i.e. $\left\| \mathbf{U}^{k+1} - \overline{\mathbf{U}}^{k+1} \right\| ^2\leqslant \delta '$.

1.3 A.3 Putting the Proof Together

Let us show that choice of number of subroutine iterations $T_k = T$ yields

$$\begin{aligned}&f(\overline{x}^k) - f(x^*)\leqslant \frac{\left\| \overline{u}^0 - x^* \right\| ^2}{2A^k} + \frac{2\sum _{j=1}^k A^j\delta }{A^k} \end{aligned}$$

by induction. At $k=0$, we have $\left\| \mathbf{U}^0 - \overline{\mathbf{U}}^0 \right\| = 0$ and by Lemma 3 it holds

$$\begin{aligned} f(\overline{x}^1) - f(x^*) \leqslant \frac{\left\| \overline{u}^0 - x^* \right\| ^2}{2A^1} + \frac{2A^1\delta }{A^1}. \end{aligned}$$

For induction pass, assume that $\left\| \mathbf{U}^j - \overline{\mathbf{U}}^j \right\| ^2\leqslant \delta '$ for $j = 0,\ldots , k$. By Lemma 4, if we set $T_k = T$, then $\left\| \mathbf{U}^{k+1} - \overline{\mathbf{U}}^{k+1} \right\| ^2\leqslant \delta '$. Applying Lemma 3 again, we get

$$\begin{aligned}&f(\overline{x}^{k+1}) - f(x^*)\leqslant \frac{\left\| \overline{u}^0 - x^* \right\| ^2}{2A^{k+1}} + \frac{2\sum _{j=1}^{k+1} A^j\delta }{A^{k+1}} \end{aligned}$$

Recalling a bound on $A^k$ from Lemma 2 gives

Here in we used the definition of $L, \mu$ in (10): $L = 2{L_{g}},~ \mu = \frac{{\mu _{g}}}{2}$. For $\varepsilon$-accuracy:

It is sufficient to choose

Let us estimate the term $\frac{D}{\delta '}$ under $\log$.

where $D_1, D_2$ are defined in 11. Finally, the total number of iterations is