Dualize, Split, Randomize: Toward Fast Nonsmooth Optimization Algorithms

Abstract

We consider minimizing the sum of three convex functions, where the first one F is smooth, the second one is nonsmooth and proximable and the third one is the composition of a nonsmooth proximable function with a linear operator L. This template problem has many applications, for instance, in image processing and machine learning. First, we propose a new primal–dual algorithm, which we call PDDY, for this problem. It is constructed by applying Davis–Yin splitting to a monotone inclusion in a primal–dual product space, where the operators are monotone under a specific metric depending on L. We show that three existing algorithms (the two forms of the Condat–Vũ algorithm and the PD3O algorithm) have the same structure, so that PDDY is the fourth missing link in this self-consistent class of primal–dual algorithms. This representation eases the convergence analysis: it allows us to derive sublinear convergence rates in general, and linear convergence results in presence of strong convexity. Moreover, within our broad and flexible analysis framework, we propose new stochastic generalizations of the algorithms, in which a variance-reduced random estimate of the gradient of F is used, instead of the true gradient. Furthermore, we obtain, as a special case of PDDY, a linearly converging algorithm for the minimization of a strongly convex function F under a linear constraint; we discuss its important application to decentralized optimization.

Appendices

Appendix

1.1 A Lemmas A.1 and A.2

We state Lemma A.1 and Lemma A.2, which are used in the proofs of Theorem 5.1 and Theorem 5.2, respectively.

To simplify the notations, we use the following convention: when a set appears in an equation while a single element is expected, e.g. $\partial R (x^k)$, this means that the equation holds for some element in this nonempty set.

Lemma A.1

Assume that F is $\mu _F$-strongly convex, for some $\mu _F \ge 0$, and that $(g^k)_{k\in {\mathbb {N}}}$ satisfies Assumption 1. Then, the iterates of the Stochastic PD3O Algorithm satisfy

$$\begin{aligned}&{\mathbb {E}}_k \Vert v^{k+1} - v^\star \Vert _P^2 + \kappa \gamma ^2{\mathbb {E}}_k\sigma _{k+1}^2 \le \Vert v^k - v^\star \Vert _P^2 + \kappa \gamma ^2\left( 1-\rho +\frac{\beta }{\kappa }\right) \sigma _k^2 \nonumber \\&\quad -2\gamma (1-\gamma (\alpha +\kappa \delta )) D_F(x^{k},x^\star )-\gamma \mu _F\Vert x^{k}-x^\star \Vert ^2 \nonumber \\&\quad -2\gamma \langle \partial R(x^{k}) - \partial R(x^\star ),x^{k} - x^\star \rangle \nonumber \\&\quad -2\gamma {\mathbb {E}}_k\langle \partial H^*(d^{k+1}) - \partial H^*(d^\star ),d^{k+1} - d^\star \rangle \nonumber \\&\quad -\gamma ^2{\mathbb {E}}_k\big \Vert P^{-1}A(u^{k+1})+P^{-1}B(z^{k})- \left( P^{-1}A(u^{\star })+P^{-1}B(z^{\star })\right) \big \Vert _P^2. \end{aligned}$$

(22)

Proof

Applying Lemma 3.2 for $\text {DYS}(P^{-1}A,P^{-1}B,P^{-1}C)$ using the norm induced by P, we have

$$\begin{aligned} \Vert v^{k+1} - v^\star \Vert _P^2&={} \Vert v^k - v^\star \Vert _P^2 -2\gamma \langle P^{-1}B(z^{k}) - P^{-1}B(z^\star ),z^{k} - z^\star \rangle _P\\&\quad -2\gamma \langle P^{-1}C(z^{k}) - P^{-1}C(z^\star ),z^{k} - z^\star \rangle _P\\&\quad +\gamma ^2\Vert P^{-1}C(z^{k}) - P^{-1}C(z^\star )\Vert _P^2\\&\quad -2\gamma \langle P^{-1}A(u^{k+1}) - P^{-1}A(u^\star ),u^{k+1} - u^\star \rangle _P\\&\quad -\gamma ^2\Vert P^{-1}A(u^{k+1})+P^{-1}B(z^{k}) - \left( P^{-1}A(u^{\star })+P^{-1}B(z^{\star })\right) \Vert _P^2\\&= \Vert v^k - v^\star \Vert _P^2 -2\gamma \langle B(z^{k}) - B(z^\star ),z^{k} - z^\star \rangle \\&\quad +\gamma ^2\Vert P^{-1}C(z^{k}) - P^{-1}C(z^\star )\Vert _P^2\\&\quad -2\gamma \langle C(z^{k}) - C(z^\star ),z^{k} - z^\star \rangle -2\gamma \langle A(u^{k+1}) - A(u^\star ),u^{k+1} - u^\star \rangle \\&\quad -\gamma ^2\Vert P^{-1}A(u^{k+1})+P^{-1}B(z^{k}) - \left( P^{-1}A(u^{\star })+P^{-1}B(z^{\star })\right) \Vert _P^2. \end{aligned}$$

Using $A(u^{k+1}) = \big ( L^* d^{k+1} , -L s^{k+1}+\partial H^*(d^{k+1})\big ), B(z^k) = \big (\partial R(x^{k}) ,0 \big ), C(z^k) = \big ( g^{k+1},0\big )$ and $A(u^{\star }) = \big ( L^* d^\star , -L s^\star +\partial H^*(d^\star )\big ), B(z^\star ) = \big (\partial R(x^\star ) ,0 \big ), C(z^\star ) = \big ( \nabla F(x^\star ),0\big )$, we have

$$\begin{aligned} \Vert v^{k+1} - v^\star \Vert _P^2&={} \Vert v^k - v^\star \Vert _P^2 -2\gamma \langle \partial R(x^{k}) - \partial R(x^\star ),x^{k} - x^\star \rangle \\&\quad +\gamma ^2\Vert g^{k+1} - \nabla F(x^\star )\Vert ^2\\&\quad -2\gamma \langle g^{k+1} - \nabla F(x^\star ),x^{k} - x^\star \rangle -2\gamma \langle \partial H^*(d^{k+1})\\&\quad - \partial H^*(d^\star ),d^{k+1} - d^\star \rangle \\&\quad -\gamma ^2\Vert P^{-1}A(u^{k+1})+P^{-1}B(z^{k})- \left( P^{-1}A(u^{\star })+P^{-1}B(z^{\star })\right) \Vert _P^2. \end{aligned}$$

Taking conditional expectation w.r.t. ${\mathscr {F}}_k$ and using Assumption 1,

$$\begin{aligned} {\mathbb {E}}_k \Vert v^{k+1} - v^\star \Vert _P^2&\le {} \Vert v^k - v^\star \Vert _P^2 -2\gamma \langle \partial R(x^{k}) - \partial R(x^\star ),x^{k} - x^\star \rangle \\&\quad -2\gamma \langle \nabla F(x^{k}) - \nabla F(x^\star ),x^{k} - x^\star \rangle \\&\quad -2\gamma {\mathbb {E}}_k\langle \partial H^*(d^{k+1}) - \partial H^*(d^\star ),d^{k+1} - d^\star \rangle \\&\quad +\gamma ^2 \left( 2\alpha D_F(x^{k},x^\star ) + \beta \sigma _k^2\right) \\&\quad -\gamma ^2{\mathbb {E}}_k\Vert P^{-1}A(u^{k+1})+P^{-1}B(z^{k}) \\&\quad - \left( P^{-1}A(u^{\star })+P^{-1}B(z^{\star })\right) \Vert _P^2. \end{aligned}$$

Using strong convexity of F,

$$\begin{aligned} {\mathbb {E}}_k \Vert v^{k+1} - v^\star \Vert _P^2 \le {}&\Vert v^k - v^\star \Vert _P^2 -\gamma \mu _F\Vert x^{k}-x^\star \Vert ^2-2\gamma D_F(x^{k},x^\star )\\&\quad +\gamma ^2 \left( 2\alpha D_F(x^{k},x^\star ) + \beta \sigma _k^2\right) \\&\quad -2\gamma \langle \partial R(x^{k}) - \partial R(x^\star ),x^{k} - x^\star \rangle \\&\quad -2\gamma {\mathbb {E}}_k\langle \partial H^*(d^{k+1}) - \partial H^*(d^\star ),d^{k+1} - d^\star \rangle \\&\quad -\gamma ^2{\mathbb {E}}_k\Vert P^{-1}A(u^{k+1})+P^{-1}B(z^{k})\\&\quad - \left( P^{-1}A(u^{\star })+P^{-1}B(z^{\star })\right) \Vert _P^2. \end{aligned}$$

Using Assumption 1,

$$\begin{aligned}&{\mathbb {E}}_k \Vert v^{k+1} - v^\star \Vert _P^2 + \kappa \gamma ^2{\mathbb {E}}_k\sigma _{k+1}^2 \le \Vert v^k - v^\star \Vert _P^2 + \kappa \gamma ^2\left( 1-\rho +\frac{\beta }{\kappa }\right) \sigma _k^2\\&\quad -\gamma \mu _F\Vert x^{k}-x^\star \Vert ^2\\&\quad -2\gamma (1-\gamma (\alpha +\kappa \delta )) D_F(x^{k},x^\star )-2\gamma \langle \partial R(x^{k}) \\&\quad - \partial R(x^\star ),x^{k} - x^\star \rangle \\&\quad -2\gamma {\mathbb {E}}_k\langle \partial H^*(d^{k+1}) - \partial H^*(d^\star ),d^{k+1} - d^\star \rangle \\&\quad -\gamma ^2{\mathbb {E}}_k\big \Vert P^{-1}A(u^{k+1})+P^{-1}B(z^{k}) \\&\quad - \left( P^{-1}A(u^{\star })+P^{-1}B(z^{\star })\right) \big \Vert _P^2. \end{aligned}$$

$\square$

Lemma A.2

Suppose that $(g^k)_{k\in {\mathbb {N}}}$ satisfies Assumption 1. Then, the iterates of the Stochastic PDDY Algorithm satisfy

$$\begin{aligned}&{\mathbb {E}}_k \Vert v^{k+1} - v^\star \Vert _P^2 + \kappa \gamma ^2{\mathbb {E}}_k\sigma _{k+1}^2 \le \Vert v^k - v^\star \Vert _P^2 + \kappa \gamma ^2\left( 1-\rho +\frac{\beta }{\kappa }\right) \sigma _k^2\\&\quad -2\gamma (1-\gamma (\alpha +\kappa \delta )) D_F(x^{k},x^\star )-2\gamma \langle \partial H^*(y^{k}) \\&\quad - \partial H^*(y^\star ),y^{k} - y^\star \rangle \\&\quad -2\gamma {\mathbb {E}}_k\langle \partial R(s^{k+1}) - \partial R(s^\star ),s^{k+1} - s^\star \rangle . \end{aligned}$$

Proof

Applying Lemma 3.2 for $\text {DYS}(P^{-1}B,P^{-1}A,P^{-1}C)$ using the norm induced by P, we have

$$\begin{aligned} \Vert v^{k+1} - v^\star \Vert _P^2&={}\Vert v^k - v^\star \Vert _P^2 -2\gamma \langle P^{-1}A(z^{k}) - P^{-1}A(z^\star ),z^{k} - z^\star \rangle _P\\&\quad -2\gamma \langle P^{-1}C(z^{k}) - P^{-1}C(z^\star ),z^{k} - z^\star \rangle _P\\&\quad +\gamma ^2\Vert P^{-1}C(z^{k}) - P^{-1}C(z^\star )\Vert _P^2\\&\quad -2\gamma \langle P^{-1}B(u^{k+1}) - P^{-1}B(u^\star ),u^{k+1} - u^\star \rangle _P\\&\quad -\gamma ^2\Vert P^{-1}B(u^{k+1})+P^{-1}A(z^{k}) - \left( P^{-1}B(u^{\star })+P^{-1}A(z^{\star })\right) \Vert _P^2\\&={} \Vert v^k - v^\star \Vert _P^2-2\gamma \langle A(z^{k}) - A(z^\star ),z^{k} - z^\star \rangle -2\gamma \langle C(z^{k}) \\&\quad - C(z^\star ),z^{k} - z^\star \rangle \\&\quad -2\gamma \langle B(u^{k+1}) - B(u^\star ),u^{k+1} - u^\star \rangle \\&\quad +\gamma ^2\Vert P^{-1}C(z^{k}) - P^{-1}C(z^\star )\Vert _P^2\\&\quad -\gamma ^2\Vert P^{-1}B(u^{k+1})+P^{-1}A(z^{k}) - \left( P^{-1}B(u^{\star })+P^{-1}A(z^{\star })\right) \Vert _P^2. \end{aligned}$$

Using $A(z^{k}) = \big ( L^* y^k ,-L x^k+\partial H^*(y^k)\big ), B(u^{k+1}) = \big (\partial R(s^{k+1}) ,0 \big ), C(z^k) = \big ( g^{k+1},0\big )$ and $A(z^{\star }) = \big ( L^* y^\star , -L x^\star +\partial H^*(y^\star )\big ), B(u^\star ) = \big (\partial R(s^\star ) ,0 \big ), C(z^\star ) = \big ( \nabla F(x^\star ),0\big )$, we have,

$$\begin{aligned} \Vert v^{k+1} - v^\star \Vert _P^2&\le \Vert v^k - v^\star \Vert _P^2-2\gamma \langle \partial H^*(y^{k}) - \partial H^*(y^\star ),y^{k} - y^\star \rangle \\&\quad +\gamma ^2\Vert g^{k+1} - \nabla F(x^\star )\Vert ^2\\&\quad -2\gamma \langle g^{k+1} - \nabla F(x^\star ),x^{k} - x^\star \rangle -2\gamma \langle \partial R(s^{k+1}) \\&\quad - \partial R(s^\star ),s^{k+1} - s^\star \rangle . \end{aligned}$$

Applying the conditional expectation w.r.t. ${\mathscr {F}}_k$ and using Assumption 1,

$$\begin{aligned} {\mathbb {E}}_k \Vert v^{k+1} - v^\star \Vert _P^2&\le \Vert v^k - v^\star \Vert _P^2-2\gamma \langle \partial H^*(y^{k}) - \partial H^*(y^\star ),y^{k} - y^\star \rangle \\&\quad -2\gamma \langle \nabla F(x^{k}) - \nabla F(x^\star ),x^{k} - x^\star \rangle \\&\quad +\gamma ^2 \left( 2\alpha D_F(x^{k},x^\star ) + \beta \sigma _k^2\right) \\&\quad -2\gamma {\mathbb {E}}_k\langle \partial R(s^{k+1}) - \partial R(s^\star ),s^{k+1} - s^\star \rangle . \end{aligned}$$

Using the convexity of F,

$$\begin{aligned} {\mathbb {E}}_k \Vert v^{k+1} - v^\star \Vert _P^2&\le \Vert v^k - v^\star \Vert _P^2-2\gamma \langle \partial H^*(y^{k}) - \partial H^*(y^\star ),y^{k} - y^\star \rangle \\&\quad -2\gamma D_F(x^{k},x^\star )\\&\quad -2\gamma {\mathbb {E}}_k\langle \partial R(s^{k+1}) - \partial R(s^\star ),s^{k+1} - s^\star \rangle \\&\quad +\gamma ^2 \left( 2\alpha D_F(x^{k},x^\star ) + \beta \sigma _k^2\right) . \end{aligned}$$

Using Assumption 1,

$$\begin{aligned}&{\mathbb {E}}_k \Vert v^{k+1} - v^\star \Vert _P^2 + \kappa \gamma ^2{\mathbb {E}}_k\sigma _{k+1}^2 \le \Vert v^k - v^\star \Vert _P^2 + \kappa \gamma ^2\left( 1-\rho +\frac{\beta }{\kappa }\right) \sigma _k^2\\&\quad -2\gamma (1-\gamma (\alpha +\kappa \delta )) D_F(x^{k},x^\star )\\&\quad -2\gamma \langle \partial H^*(y^{k}) - \partial H^*(y^\star ),y^{k} - y^\star \rangle \\&\quad -2\gamma {\mathbb {E}}_k\langle \partial R(s^{k+1}) - \partial R(s^\star ),s^{k+1} - s^\star \rangle . \end{aligned}$$

$\square$

B Linear Convergence Results

In this section, we provide linear convergence results for the stochastic PD3O and the stochastic PDDY algorithms, in addition to Theorem 6.2. For an operator splitting method like DYS$({\tilde{A}},{\tilde{B}},{\tilde{C}})$ to converge linearly, it is necessary that ${\tilde{A}}+{\tilde{B}}+{\tilde{C}}$ is strongly monotone. But this is not sufficient, and in general,

to converge linearly, DYS$({\tilde{A}},{\tilde{B}},{\tilde{C}})$ requires the stronger assumption that ${\tilde{A}}$ or ${\tilde{B}}$ or ${\tilde{C}}$ is strongly monotone, and in addition that ${\tilde{A}}$ or ${\tilde{B}}$ is cocoercive [28]. The PDDY algorithm is equivalent to DYS$(P^{-1}B,P^{-1}A,P^{-1}C)$ and the PD3O algorithm is equivalent to DYS$(P^{-1}A,P^{-1}B,P^{-1}C)$, see Sect. 4. However, $P^{-1}A$, $P^{-1}B$ and $P^{-1}C$ are not strongly monotone. In spite of this, we will prove linear convergence of the (stochastic) PDDY and PD3O algorithms.

Thus, for both algorithms, we will make the assumption that $P^{-1}A+P^{-1}B+P^{-1}C$ is strongly monotone. This is equivalent to assuming that $M = A+B+C$ is strongly monotone; that is, that $F+R$ is strongly convex and H is smooth. For instance, the Chambolle–Pock algorithm [11, 13], which is a particular case of the PD3O and the PDDY algorithms, requires R strongly convex and H smooth to converge linearly, in general. In fact, for primal–dual algorithms to converge linearly on Problem (1), for any L, it seems unavoidable that $F+R$ is strongly convex and that the dual term $H^*$ is strongly convex too, because the algorithm needs to be contractive in both the primal and the dual spaces. This means that H must be smooth. We can remark that if H is smooth, it is tempting to use its gradient instead of its proximity operator. We can then use the proximal gradient algorithm to solve Problem (1) with $\nabla (F+H\circ L)(x)=\nabla F(x) + L^*\nabla H (Lx)$. However, in practice, it is often faster to use the proximity operator instead of the gradient, see a recent analysis of this topic in [18].

For the PD3O algorithm, we will add a cocoercivity assumption, as suggested by the general linear convergence theory of DYS. More precisely, we will assume that R is smooth, so that $P^{-1}B$ is cocoercive. Our result on the PD3O is therefore an extension of [77][Theorem 3] to the stochastic setting. For the PDDY algorithm, this assumption is not needed to prove linear convergence, which is an advantage over the PD3O algorithm.

We denote by $\Vert \cdot \Vert _{\gamma ,\tau }$ the norm induced by $\frac{\gamma }{\tau }I - \gamma ^2 L L^*$ on ${\mathcal {Y}}$.

Theorem B.1

(Linear convergence of the Stochastic PD3O Algorithm) Suppose that Assumption 1 holds. Suppose that H is $1/\mu _{H^*}$-smooth, for some $\mu _{H^*} >0$, F is $\mu _F$-strongly convex, for some $\mu _F\ge 0$, and R is $\mu _R$-strongly convex, for some $\mu _R\ge 0$, with $\mu :=\mu _F + 2\mu _R >0$. Also, suppose that R is $\lambda$-smooth, for some $\lambda >0$. Suppose that the parameters $\gamma >0$ and $\tau >0$ satisfy $\gamma \le 1/(\alpha +\kappa \delta )$, for some $\kappa > \beta /\rho$, and $\gamma \tau \Vert L\Vert ^2 < 1$. Define, for every $k\in {\mathbb {N}}$,

$$\begin{aligned} V^k :=\Vert p^{k} - p^\star \Vert ^2+ \left( 1+2\tau \mu _{H^*}\right) \Vert y^{k} - y^\star \Vert _{\gamma ,\tau }^2 + \kappa \gamma ^2 \sigma _{k}^2, \end{aligned}$$

(23)

and

$$\begin{aligned} r :=\max \left( 1-\frac{\gamma \mu }{(1+\gamma \lambda )^2},\left( 1-\rho +\frac{\beta }{\kappa }\right) ,\frac{1}{1+2\tau \mu _{H^*}}\right) . \end{aligned}$$

(24)

Then, for every $k\in {\mathbb {N}}$, ${\mathbb {E}}V^{k} \le r^k V^0$.

Proof

We first use Lemma A.1 along with the strong convexity of $R,H^*$. Note that $y^{k} = q^k$ and therefore $q^{k+1} = q^k + d^{k+1} - q^{k} = d^{k+1}$. We have

$$\begin{aligned}&{\mathbb {E}}_k \Vert p^{k+1} - p^\star \Vert ^2 + {\mathbb {E}}_k \Vert q^{k+1} - q^\star \Vert _{\gamma ,\tau }^2 + 2\gamma \mu _{H^*}{\mathbb {E}}_k\Vert q^{k+1} - q^\star \Vert ^2 + \kappa \gamma ^2{\mathbb {E}}_k\sigma _{k+1}^2 \\&\le \Vert p^{k} - p^\star \Vert ^2 + \Vert q^{k} - q^\star \Vert _{\gamma ,\tau }^2 -\gamma \mu \Vert x^{k}-x^\star \Vert ^2 + \kappa \gamma ^2\left( 1-\rho +\frac{\beta }{\kappa }\right) \sigma _k^2\\&\quad -2\gamma (1-\gamma (\alpha +\kappa \delta )) D_F(x^{k},x^\star ). \end{aligned}$$

Noting that for every $q \in {\mathcal {Y}}$, $\Vert q\Vert _{\gamma ,\tau }^2 = \frac{\gamma }{\tau }\Vert q\Vert ^2 - \gamma ^2\Vert L^* q\Vert ^2 \le \frac{\gamma }{\tau }\Vert q\Vert ^2$, and taking $\gamma \le 1/(\alpha +\kappa \delta )$, we have

$$\begin{aligned}&{\mathbb {E}}_k \Vert p^{k+1} - p^\star \Vert ^2+ \left( 1+2\tau \mu _{H^*}\right) {\mathbb {E}}_k\Vert q^{k+1} - q^\star \Vert _{\gamma ,\tau }^2 + \kappa \gamma ^2{\mathbb {E}}_k\sigma _{k+1}^2\\&\le \Vert p^{k} - p^\star \Vert ^2 + \Vert q^{k} - q^\star \Vert _{\gamma ,\tau }^2 -\gamma \mu \Vert x^{k}-x^\star \Vert ^2 + \kappa \gamma ^2\left( 1-\rho +\frac{\beta }{\kappa }\right) \sigma _k^2. \end{aligned}$$

Finally, since R is $\lambda$-smooth, $\Vert p^k - p^\star \Vert ^2 \le (1+2 \gamma \lambda + \gamma ^2 \lambda ^2)\Vert x^{k}-x^\star \Vert ^2$. Indeed, in this case, applying Lemma 3.2 with ${\tilde{A}} =0$, ${\tilde{C}} = 0$ and ${\tilde{B}} = \nabla R$, we obtain that if $x^k = {{\,\mathrm{prox}\,}}_{\gamma R}(p^k)$ and $x^\star = {{\,\mathrm{prox}\,}}_{\gamma R}(p^\star )$, then

$$\begin{aligned} \Vert x^k - x^\star \Vert ^2&= \Vert p^k - p^\star \Vert ^2-2\gamma \langle \nabla R(x^k) - \nabla R(x^\star ),x^k - x^\star \rangle \\&\quad - \gamma ^2\Vert \nabla R(x^k) - \nabla R(x^\star )\Vert ^2\\&\ge \Vert p^k - p^\star \Vert ^2-2\gamma \lambda \Vert x^k - x^\star \Vert ^2 - \gamma ^2\lambda ^2\Vert x^{k} - x^\star \Vert ^2. \end{aligned}$$

Hence,

$$\begin{aligned}&{\mathbb {E}}_k \Vert p^{k+1} - p^\star \Vert ^2+ \left( 1+2\tau \mu _{H^*}\right) {\mathbb {E}}_k\Vert q^{k+1} - q^\star \Vert _{\gamma ,\tau }^2 + \kappa \gamma ^2{\mathbb {E}}_k\sigma _{k+1}^2 \\&\quad \le \Vert p^{k} - p^\star \Vert ^2 + \Vert q^{k} - q^\star \Vert _{\gamma ,\tau }^2 -\frac{\gamma \mu }{(1+\gamma \lambda )^2}\Vert p^{k}-p^\star \Vert ^2\\&\quad + \kappa \gamma ^2\left( 1-\rho +\frac{\beta }{\kappa }\right) \sigma _k^2. \end{aligned}$$

Thus, by setting $V^k$ as in (23) and r as in (24), we have ${\mathbb {E}}_k V^{k+1} \le r V^k$. $\square$

Thus, under smoothness and strong convexity assumptions, Theorem B.1 implies linear convergence of the dual variable $y^k$ to $y^\star$, with convergence rate given by r. Since $\Vert x^k - x^\star \Vert \le \Vert p^k - p^\star \Vert$, it also implies linear convergence of the variable $x^k$ to $x^\star$, with same rate.

If $g^{k+1} = \nabla F(x^k)$, the Stochastic PD3O Algorithm reverts to the PD3O Algorithm and Theorem B.1 provides a convergence rate similar to Theorem 3 in [77]. In this case, by taking $\kappa = 1$, we obtain

$$\begin{aligned} r = \max \left( 1-\gamma \frac{\mu _F + 2\mu _R}{(1+\gamma \lambda )^2},\frac{1}{1+2\tau \mu _{H^*}}\right) , \end{aligned}$$

whereas Theorem 3 in [77] provides the rate

$$\begin{aligned} \max \left( 1-\gamma \frac{2(\mu _F+\mu _R) - \gamma \alpha \mu _F}{(1+\gamma \lambda )^2},\frac{1}{1+2\tau \mu _{H^*}}\right) \end{aligned}$$

(the reader might not recognize the rate given in Theorem 3 of [77] because of some typos in Eqn. 39 of [77]).

Theorem B.2

(Linear convergence of the Stochastic PDDY Algorithm) Suppose that Assumption 1 holds. Also, suppose that H is $1/\mu _{H^*}$-smooth and R is $\mu _R$-strongly convex, for some $\mu _R >0$ and $\mu _{H^*} >0$. Suppose that the parameters $\gamma >0$ and $\tau >0$ satisfy $\gamma \le 1/(\alpha +\kappa \delta )$, for some $\kappa > \beta /\rho$, $\gamma \tau \Vert L\Vert ^2 < 1$, and $\gamma ^2 \le \frac{\mu _{H^*}}{\Vert L\Vert ^2 \mu _R}$. Define $\eta :=2\left( \mu _{H^*} -\gamma ^2\Vert L\Vert ^2\mu _R\right) \ge 0$ and, for every $k\in {\mathbb {N}}$,

$$\begin{aligned} V^k :=(1+\gamma \mu _R)\Vert p^{k} - p^\star \Vert ^2 + (1+\tau \eta )\Vert y^{k} - y^\star \Vert _{\gamma ,\tau }^2 + \kappa \gamma ^2\sigma _{k}^2, \end{aligned}$$

(25)

and

$$\begin{aligned} r :=\max \left( \frac{1}{1+\gamma \mu _R},1-\rho +\frac{\beta }{\kappa },\frac{1}{1+\tau \eta }\right) . \end{aligned}$$

(26)

Then, for every $k\in {\mathbb {N}}$, ${\mathbb {E}}V^{k} \le r^k V^0$.

Proof

We first use Lemma A.2 along with the strong convexity of R and $H^*$. Note that $y^{k} = q^{k+1}$. We have

$$\begin{aligned} {\mathbb {E}}_k \Vert v^{k+1} - v^\star \Vert _P^2 + \kappa \gamma ^2{\mathbb {E}}_k\sigma _{k+1}^2&\le \Vert v^k - v^\star \Vert _P^2 + \kappa \gamma ^2\left( 1-\rho +\frac{\beta }{\kappa }\right) \sigma _k^2\\&\quad -2\gamma \mu _{H^*}{\mathbb {E}}_k \Vert q^{k+1} - q^\star \Vert ^2\\&\quad -2\gamma \mu _{R}{\mathbb {E}}_k\Vert s^{k+1} - s^\star \Vert ^2. \end{aligned}$$

Note that $s^{k+1} = p^{k+1} - \gamma L^* y^k$. Therefore, $s^{k+1} - s^\star = (p^{k+1} - p^\star ) - \gamma L^* (y^k - y^\star )$. Using Young’s inequality $-\Vert a+b\Vert ^2 \le -\frac{1}{2}\Vert a\Vert ^2 + \Vert b\Vert ^2$, we have $-{\mathbb {E}}_k\Vert s^{k+1} - s^\star \Vert ^2 \le -\frac{1}{2}{\mathbb {E}}_k\Vert p^{k+1} - p^\star \Vert ^2 + \gamma ^2\Vert L\Vert ^2 {\mathbb {E}}_k\Vert q^{k+1} - q^\star \Vert ^2$. Hence, using $\tau \Vert q\Vert _{\gamma ,\tau }^2 \le \gamma \Vert q\Vert ^2$,

$$\begin{aligned} {\mathbb {E}}_k \Vert v^{k+1} - v^\star \Vert _P^2 + \kappa \gamma ^2{\mathbb {E}}_k\sigma _{k+1}^2 \le {}&\Vert v^k - v^\star \Vert _P^2 + \kappa \gamma ^2\left( 1-\rho +\frac{\beta }{\kappa }\right) \sigma _k^2\\&-2\gamma \left( \mu _{H^*} -\gamma ^2\Vert L\Vert ^2\mu _R\right) {\mathbb {E}}_k \Vert q^{k+1}- q^\star \Vert ^2 \\&- \gamma \mu _{R}{\mathbb {E}}_k\Vert p^{k+1} - p^\star \Vert ^2\\ \le {}&\Vert v^k - v^\star \Vert _P^2 + \kappa \gamma ^2\left( 1-\rho +\frac{\beta }{\kappa }\right) \sigma _k^2\\&-2\tau {\mathbb {E}}_k \Vert q^{k+1}- q^\star \Vert _{\gamma ,\tau }^2\left( \mu _{H^*} -\gamma ^2\Vert L\Vert ^2\mu _R\right) \\&- \gamma \mu _{R}{\mathbb {E}}_k\Vert p^{k+1} - p^\star \Vert ^2. \end{aligned}$$

Set $\eta :=2\left( \mu _{H^*} -\gamma ^2\Vert L\Vert ^2\mu _R\right) \ge 0$. Then,

$$\begin{aligned}&(1+\gamma \mu _R){\mathbb {E}}_k \Vert p^{k+1} - p^\star \Vert ^2 + (1+\tau \eta ){\mathbb {E}}_k \Vert q^{k+1} - q^\star \Vert _{\gamma ,\tau }^2 + \kappa \gamma ^2{\mathbb {E}}_k\sigma _{k+1}^2\\&\quad \le \Vert v^k - v^\star \Vert _P^2 + \kappa \gamma ^2\left( 1-\rho +\frac{\beta }{\kappa }\right) \sigma _k^2. \end{aligned}$$

Thus, by setting $V^k$ as in (25) and r as in (26), we have ${\mathbb {E}}_k V^{k+1} \le r V^k.$ $\square$

C PriLiCoSGD and Application to Decentralized Optimization

In decentralized optimization, a network of computing agents aims at jointly minimizing an objective function by performing local computations and exchanging information along the edges [1, 46, 66, 67]. It is a particular case of linearly constrained optimization, as detailed below.

First, let us set $W:=L^*L$ and $c :=L^*b$. Replacing the variable $y^k$ by the variable $a^k :=L^*y^k$ in LiCoSGD, we can write the algorithm using W and c instead of L, $L^*$ and b, with primal variables in ${\mathcal {X}}$ only. This yields the new algorithm PriLiCoSGD, shown above, to minimize F(x) subject to $Wx=c$. The convergence results for LiCoSGD apply to PriLiCoSGD, with $(a^k)_{k\in {\mathbb {N}}}$ converging to $a^\star =-\nabla F(x^\star )$.

We can apply PriLiCoSGD to decentralized optimization as follows. Consider a connected undirected graph $G = (V,E)$, where $V = \{1,\ldots ,N\}$ is the set of nodes and E the set of edges. Consider a family $(f_i)_{i \in V}$ of $\mu$-strongly convex and $\nu$-smooth functions $f_i$, for some $\mu \ge 0$ and $\nu >0$. The problem is:

$$\begin{aligned} \min _{x \in {\mathcal {X}}} \,\sum _{i \in V} f_i(x). \end{aligned}$$

(27)

Consider a gossip matrix of the graph G; that is, a $N \times N$ symmetric positive semidefinite matrix ${\widehat{W}} = ({\widehat{W}}_{i,j})_{i,j \in V}$, such that $\ker ({\widehat{W}}) = \mathop {\mathrm {span}}\nolimits ([1\ \cdots \ 1]^\mathrm {T})$ and ${\widehat{W}}_{i,j} \ne 0$ if and only if $i=j$ or $\{i,j\} \in E$ is an edge of the graph. ${\widehat{W}}$ can be the Laplacian matrix of G, for instance. Set $W :={\widehat{W}} \otimes I$, where $\otimes$ is the Kronecker product; then decentralized communication in the network G is modeled by an application of the positive self-adjoint linear operator W on ${\mathcal {X}}^V$. Moreover, $W(x_1,\ldots ,x_N) = 0$ if and only if $x_1 = \ldots = x_N$. Therefore, Problem (27) is equivalent to the lifted problem

$$\begin{aligned} \min _{{\tilde{x}} \in {\mathcal {X}}^V} F({\tilde{x}}) \quad \text {such that} \quad W {\tilde{x}} = 0, \end{aligned}$$

(28)

where for every ${\tilde{x}}=(x_1,\ldots ,x_N) \in {\mathcal {X}}^V$, $F({\tilde{x}}) = \sum _{i=1}^N f_i(x_i)$. Let us apply PriLiCoSGD to Problem (28); we obtain the Decentralized Stochastic Optimization Algorithm (DESTROY). It generates the sequence $({\tilde{x}}^k)_{k\in {\mathbb {N}}}$, where ${\tilde{x}}^k = (x_1^k,\ldots ,x_N^k) \in {\mathcal {X}}^V$. The update of each $x_i^k$ consists in evaluating $g_i^{k+1}$, an estimate of $\nabla f_i(x_i^k)$ satisfying Assumption 1,

and communication steps involving $x_j^k$, for every neighbor j of i. For instance, the variance-reduced estimator $g_i^k$ can be the loopless SVRG estimator seen in Proposition 5.1, when $f_i$ is itself a sum of functions, or a compressed version of $\nabla f_i$ [4, 50, 65, 75].

As an application of the convergence results for LiCoSGD, we obtain the following results for DESTROY. Theorem 4.1 becomes:

Theorem C.1

(Convergence of DESTROY, deterministic case $g_i^{k+1}=\nabla f_i(x_i^k)$) Suppose that $\gamma \in (0,2/\nu )$ and that $\tau \gamma \Vert {\widehat{W}}\Vert < 1$. Then, in DESTROY, each $(x_i^k)_{k\in {\mathbb {N}}}$ converges to the same solution $x^\star$ to the problem (27) and each $(a_i^k)_{k\in {\mathbb {N}}}$ converges to $a_i^\star =-\nabla f_i(x^\star )$.

Theorem 6.1 can be applied to the stochastic case, stating ${\mathcal O}(1/k)$ convergence of the Lagrangian gap, by setting ${\mathcal {Y}}={\mathcal {X}}$ and $L=L^* = W^{1/2}$. Similarly, Theorem 6.2 yields linear convergence of DESTROY in the strongly convex case $\mu >0$, with $L^*L$ replaced by W and $\Vert L\Vert ^2$ replaced by $\Vert W\Vert =\Vert {\widehat{W}}\Vert$. In particular, in the deterministic case, with $\gamma =1/\nu$ and $\tau \gamma =\aleph /\Vert W\Vert$ for some fixed $\aleph \in (0,1)$, $\varepsilon$-accuracy is reached after ${\mathcal O}\Big (\max \big (\frac{\nu }{\mu },\frac{ \Vert W\Vert }{\omega (W)}\big )\log \big (\frac{1}{\varepsilon }\big )\Big )$ iterations. This rate is better or equivalent to the one of recently proposed decentralized algorithms, like EXTRA, DIGing, NIDS, NEXT, Harness, Exact Diffusion, see Table 1 of [76, 49][Theorem 1] and [1]. With a stochastic gradient, the rate of our algorithm is also better than [53][Equation 99].

In follow-up papers, the authors used Nesterov acceleration to propose accelerated versions of DESTROY [46] and PriLiCoSGD [64].