A Note on the Equivalence of Operator Splitting Methods

Abstract

This note provides a comprehensive discussion of the equivalences between some splitting methods. We survey known results concerning these equivalences which have been studied over the past few decades. In particular, we provide simplified proofs of the equivalence of the ADMM and the Douglas–Rachford method and the equivalence of the ADMM with intermediate update of multipliers and the Peaceman–Rachford method.

Appendices

Appendices

13.1.1 Appendix 1

Let A: X → X be linear. Define

(13.45)

Recall that a linear operator A: X → X is monotone if (∀x ∈ X) 〈x, Ax〉≥ 0, and is strictly monotone if $(\forall x\in X\smallsetminus \{0\})$ 〈x, Ax〉 > 0. Let h: X →ℝ and let x ∈ X. We say that h is Fréchet differentiable at x if there exists a linear operator Dh(x): X →ℝ, called the Fréchet derivative of h at x, such that ; and h is Fréchet differentiable on X if it is Fréchet differentiable at every point in X.

The following lemma is a special case of [5, Proposition 17.36].

Lemma 13.3

Let A: X → X be linear, strictly monotone, self-adjoint and invertible. Then the following hold:

• (i) q _A and $q_{A^{-1}}$ are strictly convex, continuous, Fréchet differentiable. Moreover, $(\nabla q_A,\nabla q_{A^{-1}} )=(A,A^{-1})$.

• (ii) $q_{A}^*=q_{A^{-1}}$.

Proof

Note that, likewise A, A ⁻¹ is linear, strictly monotone, self-adjoint (since (A ⁻¹)^∗ = (A ^∗)⁻¹ = A ⁻¹) and invertible. Moreover, ${\operatorname {ran}} A={\operatorname {ran}} A^{-1}=X$. (i): This follows from [5, Example 17.11 and Proposition 17.36(i)] applied to A and A ⁻¹ respectively. (ii): It follows from [5, Proposition 17.36(iii)], [28, Theorem 4.8.5.4] and the invertibility of A that $q_{A}^*=q_{A^{-1}}+\iota _{{\operatorname {ran}} A} =q_{A^{-1}}+\iota _{X}=q_{A^{-1}}$. □

Proposition 13.3

Let L: Y → X be linear. Suppose that L ^∗ L is invertible. $g\colon Y \to \left ]-\infty ,+\infty \right ]$ be convex, lower semicontinuous, and proper. Then the following hold:

• (i) $\ker L=\{0\}$.

• (ii) L ^∗ L is strictly monotone.

• (iii) ${\operatorname {dom}}(q_{L^*L}+g)^*=X$.

• (iv) $\partial (q_{L^*L}+g)=\nabla q_{L^*L}+\partial g=L^*L+\partial g$.

• (v) $(q_{L^*L}+g^*)^*$ is Fréchet differentiable on X.

• (vi) (L ^∗ L + ∂g ^∗)⁻¹ is single-valued and ${\operatorname {dom}} (L^*L+\partial g^*)^{-1}=X$.

• (vii) ${\operatorname {Prox}}_{g^*\circ L^*}={\operatorname {Id}}-L(L^*L+\partial g)^{-1}L^*$.

• (viii) ${\operatorname {Prox}}_{(g^*\circ L^*)^{*}}=L(L^*L+\partial g)^{-1}L^*$.

Proof

(i): Using [5, Fact 2.25(vi)] and the assumption that L ^∗ L is invertible we have $\ker L=\ker L^*L=\{0\}$. (ii): Using (i) we have $(\forall x\in X\smallsetminus \{0\})$ , hence L ^∗ L is strictly monotone. (iii): By (ii) and Lemma 13.3 (i) applied with A replaced by L ^∗ L we have ${\operatorname {dom}} q_{L^*L}={\operatorname {dom}} q^*_{L^*L}=X$, hence

$$\displaystyle \begin{aligned} {\operatorname{dom}} q_{L^*L}- {\operatorname{dom}} g=X- {\operatorname{dom}} g=X. \end{aligned} $$

(13.46)

It follows from (13.46), [1, Corollary 2.1] and Lemma 13.3 (ii)&(i) that ${\operatorname {dom}} (q_{L^*L}+g)^*={\operatorname {dom}} q_{L^*L}^*+{\operatorname {dom}} g^{*} ={\operatorname {dom}} q_{(L^*L)^{-1}}+{\operatorname {dom}} g^*=X+{\operatorname {dom}} g^*=X$. (iv): Combine (13.46), [1, Corollary 2.1] and Lemma 13.3 (i). (v): Since $q_{L^*L}$ is strictly convex, so is $q_{L^*L}+g$, which in view of [5, Proposition 18.9] and (iii) implies that $(q_{L^*L}+g)^*$ is Fréchet differentiable on $X={\operatorname {int}} {\operatorname {dom}} (q_{L^*L}+g)^*$. (vi): Using (iv), Fact 13.8 (i) applied with f replaced by $q_{L^*L}+g$, (v) and [5, Proposition 17.31(i)] we have $(L^*L+\partial g)^{-1}=(\partial (q_{L^*L}+g))^{-1} =\partial (q_{L^*L}+g)^* =\{\nabla (q_{L^*L}+g)^*\}$ is single-valued with ${\operatorname {dom}} (L^*L+\partial g)^{-1}=X$.

(vii): Let $x\in X={\operatorname {dom}} (L^*L+\partial g)^{-1}$ and let y ∈ X such that y = x − L(L ^∗ L + ∂g)⁻¹ L ^∗ x. Then using (vi) we have

(13.47)

Consequently, L ^∗ y + L ^∗ Lu = L ^∗ x ∈ L ^∗ Lu + ∂g(u), hence L ^∗ y ∈ ∂g(u), equivalently, in view of Fact 13.8 (i) applied with f replaced by g, u ∈ (∂g)⁻¹(L ^∗ y) = ∂g ^∗(L ^∗ y). Combining with (13.47) we learn that

$$\displaystyle \begin{aligned} x\in y+L\circ(\partial g^{*})\circ L^* (y). \end{aligned} $$

(13.48)

Note that [5, Fact 2.25(vi) and Fact 2.26] implies that ${\operatorname {ran}} L^*={\operatorname {ran}} L^*L=X$, hence $0\in {\operatorname {sri}}({\operatorname {dom}} g^*-{\operatorname {ran}} L^*)$. Therefore one can apply [5, Corollary 16.53(i)] to re-write (13.48) as $x\in ({\operatorname {Id}}+\partial (g^*\circ L^*))y$. Therefore, $y={\operatorname {Prox}}_{g^*\circ L^*}x$ by [5, Proposition 16.44]. (viii): Apply Fact 13.8 (ii) with f replaced by g ^∗∘ L ^∗. □

13.1.2 Appendix 2

Lemma 13.4

Let $g\colon Y \to \left ]-\infty ,+\infty \right ]$ be convex, lower semicontinuous, and proper. Consider the following statements:

• (i) g is strongly convex.

• (ii) g ^∗ is Fréchet differentiable and ∇g ^∗ is Lipschitz continuous.

• (iii) g ^∗∘ L ^∗ is Fréchet differentiable and ∇(g ^∗∘ L ^∗) = L ∘ (∇g ^∗) ∘ L ^∗ is Lipschitz continuous.

• (iv) (g ^∗∘ L ^∗)^∗ is strongly convex.

Then (i)⇔(ii)⇒(iii)⇔(iv).

Proof

(i)⇔(ii): See [5, Theorem 18.15]. (ii)⇒(iii): Clearly g ^∗∘ L ^∗ is Fréchet differentiable. Now let (x, y) ∈ X × X and suppose that β > 0 is a Lipschitz constant of ∇g ^∗. It follows from [5, Corollary 16.53] that . (iii)⇔(iv): Use the equivalence of (i) and (ii) applied with g replaced by (g ^∗∘ L ^∗)^∗. □

13.1.3 Appendix 3

We start by recalling the following well-known fact.

Fact 13.8

Let $f\colon X\to \left ]-\infty ,+\infty \right ]$ be convex, lower semicontinuous and proper and let γ > 0. Then the following hold:

• (i) (∂f)⁻¹ = ∂f ^∗.

• (ii) ${\operatorname {Prox}}_{\gamma f}+{\operatorname {Prox}}_{(\gamma f)^*}={\operatorname {Id}}$.

Proof

(i): See, e.g., [27, Remark on page 216] or [23, Théorème 3.1].

(ii): See, e.g., [5, Theorem 14.3(iii)]. □

Lemma 13.5

Let γ > 0. The Douglas–Rachford method given in (13.9) applied to the ordered pair (γf, γg) with a starting point x ₀ ∈ X to solve (13.8) can be rewritten as:

$$\displaystyle \begin{aligned} y_n&= {\operatorname{Prox}}_{\gamma f} x_{n} {} \end{aligned} $$

(13.49a)

$$\displaystyle \begin{aligned} x_{n+1}&=y_{n}-{\operatorname{Prox}}_{(\gamma g)^*}(2y_n-x_{n}). {} \end{aligned} $$

(13.49b)

Proof

Using (13.9a), (13.10), and Fact 13.8 (ii) applied with f replaced by g we have

$$\displaystyle \begin{aligned} x_{n+1}&= x_n-{\operatorname{Prox}}_{\gamma f }x_n+{\operatorname{Prox}}_{\gamma g}(2{\operatorname{Prox}}_{\gamma f} x_n-x_n) =x_n-y_n+{\operatorname{Prox}}_{\gamma g}(2y_n-x_n) \\ &=x_n-y_n+2y_n-x_n-{\operatorname{Prox}}_{{(\gamma g)}^*}(2y_n-x_n) =y_{n}-{\operatorname{Prox}}_{(\gamma g)^*}(2y_n-x_{n}), \end{aligned} $$

(13.50)

and the conclusion follows. □

13.1.4 Appendix 4

Proposition 13.4

Let (x, y, z) ∈ X × Y × Z and let B and $\widetilde {f}$ be defined as in (13.39) and (13.41). Then the following hold:

• (i) B ^∗ y = (A ^∗ y, C ^∗ y).

• (ii) ${\operatorname {dom}} \widetilde {f}={\operatorname {dom}} f\times \{0\}$.

• (iii) $(\forall (x,z)\in {\operatorname {dom}} \widetilde {f})$ we have z = 0 and B(x, z) = Ax.

• (iv) $B({\operatorname {dom}} \widetilde {f})=A({\operatorname {dom}} f)$.

• (v) $0\in {\operatorname {sri}} ({\operatorname {dom}} g-B({\operatorname {dom}} \widetilde {f}))$.

• (vi) ${\operatorname {argmin}}(\widetilde {f}+g\circ B)={\operatorname {argmin}} (f+g\circ A)\times \{0\} \neq \varnothing$.

• (vii) ${\operatorname {Prox}}_{\widetilde {f}}(x,z)=({\operatorname {Prox}}_f x,0)$.

• (viii) ${\operatorname {Prox}}_{(\tau g\circ B)^*}=\tau B^*{\operatorname {Prox}}_{\sigma g^*}(\sigma B)$.

Proof

(i): This clearly follows from (13.39). (ii): It follows from (13.41) that ${\operatorname {dom}}\widetilde {f}={\operatorname {dom}} f\times {\operatorname {dom}}\iota _{\{0\}}={\operatorname {dom}} f\times \{0\}$. (iii): The claim that z = 0 follows from (ii). Now combine with (13.39). (iv): Combine (ii) and (iii). (v): Combine (iv) and (13.34). (vi): We have

$$\displaystyle \begin{aligned} {\operatorname{argmin}}(\widetilde{f}+g\circ B) &={\operatorname{zer}} (\partial \widetilde{f}+B^* \circ \partial g\circ B) {} \end{aligned} $$

(13.51a)

(13.51b)

(13.51c)

where (13.51a) follows from (v) and (13.4) applied with (f, g, L) replaced by $(g,\widetilde {f},B)$, and (13.51b) follows from (13.39) and (13.41). Therefore, $(x,z)\in {\operatorname {argmin}}(\widetilde {f}+g\circ B)$ ⇔ [z = 0 and $x\in {\operatorname {zer}}(\partial f+A^* \circ \partial g\circ A)$] ⇔ $(x,z)\in {\operatorname {argmin}}(f+g\circ A) \times \{0\}$. Now combine with (13.4). (vii): Combine (13.41) and [5, Proposition 23.18]. (viii): Indeed, Proposition 13.3 (viii) implies

(13.52a)

(13.52b)

□