Douglas–Rachford splitting and ADMM for nonconvex optimization: accelerated and Newton-type linesearch algorithms

Abstract

Although the performance of popular optimization algorithms such as the Douglas–Rachford splitting (DRS) and the ADMM is satisfactory in convex and well-scaled problems, ill conditioning and nonconvexity pose a severe obstacle to their reliable employment. Expanding on recent convergence results for DRS and ADMM applied to nonconvex problems, we propose two linesearch algorithms to enhance and robustify these methods by means of quasi-Newton directions. The proposed algorithms are suited for nonconvex problems, require the same black-box oracle of DRS and ADMM, and maintain their (subsequential) convergence properties. Numerical evidence shows that the employment of L-BFGS in the proposed framework greatly improves convergence of DRS and ADMM, making them robust to ill conditioning. Under regularity and nondegeneracy assumptions at the limit point, superlinear convergence is shown when quasi-Newton Broyden directions are adopted.

Appendix A: Auxiliary results

This appendix contains some auxiliary results needed for the convergence analysis of Sect. 4. As shown in [53, Eq. (3.4)], the DRE can be expressed in terms of the forward-backward envelope $\varphi _{\gamma }^{\textsc {fb}}$ [36, 47, 54] as

$$\begin{aligned} \varphi _{\gamma }^{\textsc {dr}}(s) =\varphi _{\gamma }^{\textsc {fb}}(u) {}\,{:}{=}\,{} \min _{w\in \mathbb {R}^p}\left\{ \varphi _{1}(u) + \varphi _{2}(w) + \langle \nabla \varphi _{1}(u),w-u\rangle + \tfrac{1}{2\gamma }\Vert w-u\Vert ^2 \right\}, \end{aligned}$$

(A.1a)

where $u=\hbox {prox}\, _{\gamma \varphi _{1}}(s)$ and the minimum is attained at any $v\in \hbox {prox}\, _{\gamma \varphi _{2}}(2u-s)$. Equivalently,

$$\begin{aligned} \varphi _{\gamma }^{\textsc {dr}}(s) =\varphi _{1}(u) {}-{} \tfrac{\gamma }{2}\Vert \nabla \varphi _{1}(u)\Vert ^2 + \varphi _{2}^\gamma (u-\gamma \nabla {\varphi_1}(u)). \end{aligned}$$

(A.1b)

Fact A.1

( [53, Prop. 3.3]). Suppose that Assumption I holds and let $\gamma <\nicefrac {1}{L_{\varphi _{1}}}$ be fixed. Then, for all $s\in \mathbb {R}^p$ and $(u,v)\in \text {DRS} _{\gamma }(s)$ it holds that

$$\begin{aligned} \varphi (v) + \tfrac{1-\gamma L_{\varphi _{1}}}{2\gamma }\Vert v-u\Vert ^2 {}\le {} \varphi _{\gamma }^{\textsc {dr}}(s) {}\le {} \varphi (u). \end{aligned}$$

$\square$

Lemma A.2

Suppose that Assumption I holds. Then, for all $s,{\bar{u}}\in \mathbb {R}^p$ and $\gamma <\nicefrac {1}{L_{\varphi _{1}}}$

$$\begin{aligned} \varphi _{\gamma }^{\textsc {dr}}(s)-\varphi ({\bar{u}}) {}\le {} \tfrac{1+\gamma L_{\varphi _{1}}}{2\gamma } \Vert \hbox {prox}\, _{\gamma \varphi _{1}}(s)-{\bar{u}}\Vert ^2. \end{aligned}$$

Proof

Let $u\,{:}{=}\,\hbox {prox}\, _{\gamma \varphi _{1}}(s)$ for brevity. By plugging $w={\bar{u}}$ into (A.1a) we obtain

where the second inequality uses the known quadratic upper bound [8, Prop. A.24] for functions with Lipschitz-continuous gradient. $\square$

Lemma A.3

Suppose that Assumption I* holds and let $\gamma >1\nicefrac {}{\mu _{\varphi _{1}}}$ be fixed. Then,

$$\begin{aligned} \inf \varphi = -\inf \psi = -\inf \psi _{\gamma _{*}}^{\textsc {dr}}= \sup \varphi _{\gamma }^{\textsc {dr}}\end{aligned}$$

(A.2)

where $\gamma _*=\nicefrac 1\gamma$. Moreover, for any $s\in \mathbb {R}^p$ it holds that

$$\begin{aligned} \tfrac{1}{2\gamma }\Vert x_\star -v\Vert ^2 + \tfrac{\gamma \mu _{\varphi _{1}}-1}{2\gamma }\Vert x_\star -u\Vert ^2 {}\le {} \inf \varphi {}-{} \varphi _{\gamma }^{\textsc {dr}}(s) = \psi _{\gamma _{*}}^{\textsc {dr}}(s_*) {}-{} \inf \psi \end{aligned}$$

(A.3)

where $x_\star$ is the unique minimizer of $\varphi$, $s_*=-\nicefrac s\gamma$, and $(u,v)=\text {DRS} _{\gamma }(s)$.

Proof

Due to strong convexity, the set of primal solutions $\hbox {arg}\,\hbox {min} \varphi$ is a singleton, ensuring strong duality $\inf \varphi =-\inf \psi$ through [3, Thm. 5.2.1(b)-(c)]. Since $\psi _{1}$ is $\nicefrac {1}{\mu _{\varphi _{1}}}$-smooth, it follows from Fact 2.1(i) that $\inf \psi _{\gamma _{*}}^{\textsc {dr}}=\inf \psi$ for every $\gamma _*<\mu _{\varphi _{1}}$; combined with the identity $\varphi _{\gamma }^{\textsc {dr}}(s)=-\psi _{\gamma _{*}}^{\textsc {dr}}(-s\nicefrac {}\gamma )$ holding for $\gamma _*=1\nicefrac {}\gamma$ (cf. Theorem 2.3), (A.2) is obtained. Let now $s\in \mathbb {R}^p$ be fixed and consider $(u,v)=\text {DRS} _{\gamma }(s)$. From the inclusion $\tfrac{s-u}{\gamma }\in \partial \varphi _{1}(u)$ (cf. (1.10)) and strong convexity of $\varphi _{1}$ one has

$$\begin{aligned} \varphi _{1}(x_\star ) {}\ge {}&\varphi _{1}(u) + \tfrac{1}{\gamma }\langle s-u,x_\star -u\rangle + \tfrac{\mu _{\varphi _{1}}}{2}\Vert x_\star -u\Vert ^2. \end{aligned}$$

Similarly, since $\tfrac{2u-s-v}{\gamma }\in \partial \varphi _{2}(v)$ and $\varphi _{2}$ is convex, one has

$$\begin{aligned} \varphi _{2}(x_\star ) {}\ge {}&\varphi _{2}(v) + \tfrac{1}{\gamma }\langle 2u-s-v,x_\star -v\rangle . \end{aligned}$$

Summing the two inequalities yields

$$\begin{aligned} \inf \varphi {}\ge {}&\varphi _{1}(u) + \varphi _{2}(v) + \tfrac{1}{\gamma }\langle s-u,x_\star -u\rangle + \tfrac{1}{\gamma }\langle 2u-s-v,x_\star -v\rangle + \tfrac{\mu _{\varphi _{1}}}{2}\Vert x_\star -u\Vert ^2 \\ ={}&\varphi _{1}(u) + \varphi _{2}(v) + \tfrac{1}{\gamma }\langle s-u,v-u\rangle + \tfrac{1}{\gamma }\langle u-v,x_\star -v\rangle + \tfrac{\mu _{\varphi _{1}}}{2}\Vert x_\star -u\Vert ^2 \\ ={}&\varphi _{1}(u) + \varphi _{2}(v) + \tfrac{1}{\gamma }\langle s-u,v-u\rangle + \tfrac{1}{2\gamma }\Vert u-v\Vert ^2 + \tfrac{1}{2\gamma }\Vert x_\star -v\Vert ^2 + \tfrac{\gamma \mu _{\varphi _{1}}-1}{2\gamma }\Vert x_\star -u\Vert ^2 \\ ={}&\varphi _{\gamma }^{\textsc {dr}}(s) + \tfrac{1}{2\gamma }\Vert x_\star -v\Vert ^2 + \tfrac{\gamma \mu _{\varphi _{1}}-1}{2\gamma }\Vert x_\star -u\Vert ^2. \end{aligned}$$

The claim now follows from the identity $\psi _{\gamma _{*}}^{\textsc {dr}}(s_*)=-\varphi _{\gamma }^{\textsc {dr}}(s)$ shown in Theorem 2.3. $\square$