Non-stationary Douglas–Rachford and alternating direction method of multipliers: adaptive step-sizes and convergence

Abstract

We revisit the classical Douglas–Rachford (DR) method for finding a zero of the sum of two maximal monotone operators. Since the practical performance of the DR method crucially depends on the step-sizes, we aim at developing an adaptive step-size rule. To that end, we take a closer look at a linear case of the problem and use our findings to develop a step-size strategy that eliminates the need for step-size tuning. We analyze a general non-stationary DR scheme and prove its convergence for a convergent sequence of step-sizes with summable increments in the case of maximally monotone operators. This, in turn, proves the convergence of the method with the new adaptive step-size rule. We also derive the related non-stationary alternating direction method of multipliers. We illustrate the efficiency of the proposed methods on several numerical examples.

Appendix: The proof of Theorem 5.1

Let us assume that we apply (12) to solve the optimality condition (25) of the dual problem (24). From (12), i.e.,

$$\begin{aligned} y^{n+1} = J_{t_{n}A}\left( (1+\kappa _n)J_{t_{n-1}B}y^{n} - \kappa _n y^{n}\right) + \kappa _n\left( y^{n} -J_{t_{n-1}B}y^{n}\right) , \end{aligned}$$

we define $w^{n+1} := J_{t_{n-1}B}y^{n}$ and $z^{n+1} := J_{t_{n}A}( (1 + \kappa _n)w^{n+1} - \kappa _ny^n)$ to obtain

$$\begin{aligned} {\left\{ \begin{array}{ll} w^{n+1} := J_{t_{n-1}B}y^{n}\\ z^{n+1} := J_{t_{n}A}\left( (1 + \kappa _n)w^{n+1} - \kappa _n y^n\right) \\ y^{n+1} = z^{n+1} + \kappa _n\left( y^n- w^{n+1}\right) . \end{array}\right. } \end{aligned}$$

Shifting up this scheme by one index and changing the order, we obtain

$$\begin{aligned} {\left\{ \begin{array}{ll} z^{n} = J_{t_{n-1}A}\left( \left( 1 + \kappa _{n-1}\right) w^{n} - \kappa _{n-1}y^{n-1}\right) \\ y^{n} = z^{n} + \kappa _{n-1}\left( y^{n-1} - w^{n}\right) \\ w^{n+1} = J_{t_{n-1}B}y^n = J_{t_{n-1}B}\left( z^n + \kappa _{n-1}\left( y^{n-1} - w^n\right) \right) . \end{array}\right. } \end{aligned}$$

Let $(1 + \kappa _{n-1})w^{n} - \kappa _{n-1}y^{n-1} = x^n + w^n$. This gives $x^n = \kappa _{n-1}(w^n - y^{n-1})$ and hence, $z^n + \kappa _{n-1}(y^{n-1} - w^n) = z^n - x^n$ and $x^{n+1} = \kappa _n(w^{n+1} - y^{n}) = \kappa _n(w^{n+1} - z^n + x^n)$. Substituting these into the above expression of the DR scheme, we obtain

$$\begin{aligned} \left\{ \begin{array}{ll} z^{n} = J_{t_{n-1}A}(x^{n} + w^n)\\ w^{n+1} = J_{t_{n-1}B}(z^{n} - x^n)\\ x^{n+1} = \kappa _n(x^n + w^{n+1} - z^{n}), \end{array}\right. \end{aligned}$$

(28)

where $x^n = \kappa _{n-1}(w^{n} - y^{n-1})$.

From $z^{n} = J_{t_{n-1}A}(w^{n} + x^n)$, we have $z^n = (I + t_{n-1}A)^{-1}(w^n + x^n)$ or

$$\begin{aligned} 0 \in z^n - w^n - x^n + t_{n-1}\left( D\nabla {\varphi ^{*}}\left( D^Tz^n\right) - c\right) . \end{aligned}$$

Let $u^{n+1} \in \nabla {\varphi ^{*}}(D^Tz^n)$, which implies $D^Tz^n \in \partial {\varphi }(u^{n+1})$. Hence, we have $z^n - w^n - x^n + t_{n-1}(Du^{n+1} - c) = 0$, therefore $D^Tz^n = D^T(w^n + x^n - t_{n-1}(Du^{n+1} - c)) \in \partial {\varphi }(u^{n+1})$. This condition leads to

$$\begin{aligned} 0 \in D^T\left( t_{n-1}\left( Du^{n+1} - c\right) - x^n - w^n\right) + \partial {\varphi }\left( u^{n+1}\right) . \end{aligned}$$

This is the optimality condition of

$$\begin{aligned} u^{n+1} = \mathop {{{\,\mathrm{argmin}\,}}}\limits _u\left\{ \varphi (u) + \frac{t_{n-1}}{2}\Vert Du - c - t_{n-1}^{-1}(x^n + w^n)\Vert ^2\right\} . \end{aligned}$$

Similarly, from $w^{n+1} = J_{t_{n-1}B}(z^n - x^n)$, if we define $v^{n+1} \in \nabla {\psi ^{*}}(E^Tw^{n+1})$, then we can also derive that

$$\begin{aligned} v^{n+1} = \mathop {{{\,\mathrm{argmin}\,}}}\limits _v\left\{ \psi (v) + \frac{t_{n-1}}{2}\Vert Ev + t_{n-1}^{-1}(x^n - z^n)\Vert ^2\right\} . \end{aligned}$$

From the line $z^n - w^n - x^n + t_{n-1}(Du^{n+1} - c) = 0$ above, we can write $x^n - z^n = t_{n-1}(Du^{n+1} - c) - w^n$. Substituting this expression into the above step, we obtain

$$\begin{aligned} v^{n+1} = \displaystyle \mathop {{{\,\mathrm{argmin}\,}}}\limits _v\left\{ \psi (v) - \langle w^n, Ev\rangle + \tfrac{t_{n-1}}{2}\Vert Ev + Du^{n+1} - c\Vert ^2\right\} . \end{aligned}$$

This is the second line of (26).

Next, from $w^{n+1} - z^n + x^n + t_{n-1}Ev^{n+1} = 0$, we have $w^{n} = z^{n-1} - x^{n-1} - t_{n-2}Ev^{n}$. This implies $Ev^n = -t_{n-2}^{-1}(x^{n-1} + w^n - z^{n-1})$. From the last line of (28), we have $x^n = \kappa _{n-1}(x^{n-1} + w^n - z^{n-1})$. Combine these two lines, we get $Ev^n = -\tfrac{1}{\kappa _{n-1}t_{n-2}}x^n = -\frac{1}{t_{n-1}}x^n$ due to the update rule (9): $t_{n-1} = \kappa _{n-1}t_{n-2}$. Substituting $Ev^n = -\frac{1}{t_{n-1}}x^n$ into the u-subproblem, we obtain

$$\begin{aligned} u^{n+1} = \mathop {{{\,\mathrm{argmin}\,}}}\limits _u\left\{ \varphi (u) - \langle w^n, Du\rangle + \frac{t_{n-1}}{2}\Vert Du + Ev^n - c\Vert ^2\right\} . \end{aligned}$$

This is the first line of (26).

Now, since $z^n = w^n - t_{n-1}(Du^{n+1} - c) + x^n$, and $w^{n+1} = z^n - x^n - t_{n-1}Ev^{n+1}$, combining these expressions, we obtain $w^{n+1} = w^n - t_{n-1}(Du^{n+1} + Ev^{n+1} - c)$. This is the last line of (26).

Finally, we derive the update rule for $t_n$. Indeed, note that $y^n = z^n - x^n$, and $z^n - w^n - x^n + t_{n-1}(Du^{n+1} - c) = 0$. These relations show that $y^n = w^n - t_{n-1}(Du^{n+1} - c)$. Moreover, we also have $w^{n+1} = J_{t_{n-1}B}(z^n - x^n) = J_{t_{n-1}B}(y^n)$. In this case, we have $J_{t_{n-1}B}(y^n) - y^n = w^{n+1} - w^n + t_{n-1}(Du^{n+1} - c) = -t_{n-1}(Du^{n+1} + Ev^{n+1} - c) + t_{n-1}(Du^{n+1} - c) = -t_{n-1}Ev^{n+1}$. Hence, we can compute $\kappa _n$ as

$$\begin{aligned} \kappa _n := \frac{\Vert J_{t_{n-1}B}(y^n)\Vert }{\Vert y^n - J_{t_{n-1}B}(y^n)\Vert } = \frac{\Vert w^{n+1}\Vert }{t_{n-1}\Vert Ev^{n+1}\Vert }. \end{aligned}$$

Using the fact that $t_n := \kappa _nt_{n-1}$, we show that $t_n := \frac{\Vert w^{n+1}\Vert }{\Vert Ev^{n+1}\Vert }$, which is the last line of (26) after projecting and weighting as in Sect. 4. Since $\left\{ w^n\right\}$ is equivalent to the sequence $\left\{ u^n\right\}$ in the DR scheme (2) [or equivalently, (12)] applying to the dual optimality condition (25) of the dual problem (24), the last conclusion is a direct consequence of Theorem 3.2. $\square$