Analysis of the Alternating Direction Method of Multipliers for Nonconvex Problems

Abstract

This work investigates the theoretical performance of the alternating-direction method of multipliers (ADMM) as it applies to nonconvex optimization problems, and in particular, problems with nonconvex constraint sets. The alternating direction method of multipliers is an optimization method that has largely been analyzed for convex problems. The ultimate goal is to assess what kind of theoretical convergence properties the method has in the nonconvex case, and to this end, theoretical contributions are twofold. First, this work analyzes the method with local optimal solution of the ADMM subproblems, which contrasts with much analysis that requires global solutions of the subproblems. Such a consideration is important to practical implementations. Second, it is established that the method still satisfies a local convergence result. The work concludes with some more detailed discussion of how the analysis relates to previous work.

Appendices

Appendix: A Regularity of overall problem and subproblems

This section establishes the claim that Assumption 2 implies Assumption 4; that is, that a solution of the main problem (P) which satisfies conditions including the second-order sufficient conditions implies that the subproblems also have solutions satisfying similar conditions. This is established through the following lemmata. This next result is a modification of [5, Lemma 3.2.1].

Lemma A.1

Let $\mathbf {H} \in \mathbb {R}^{n \times n}$ be a symmetric matrix and let $\mathbf {C} \in \mathbb {R}^{p \times n}$ and $\mathbf {D} \in \mathbb {R}^{p^{\prime } \times n}$. Assume that H is positive definite on the nullspace of $\begin {bmatrix} \mathbf {C} \\ \mathbf {D} \end {bmatrix}$: z^THz > 0 for all z≠0 with Cz = 0 and Dz = 0. Then there exists ρ^∗ such that for all ρ > ρ^∗,

$$ \mathbf{z}^{\mathrm{T}} (\mathbf{H} + \rho \mathbf{D}^{\mathrm{T}} \mathbf{D}) \mathbf{z} > 0 $$

for all z≠0 with Cz = 0.

Proof

Assume the contrary. Then for all $k \in \mathbb {N}$, there exists z^k≠0 such that (z^k)^T(H + kD^TD)z^k ≤ 0 and Cz^k = 0. Assume without loss of generality that $\left \| \mathbf {z}^{k} \right \| = 1$ (we can scale z^k as necessary). Since $\left (\mathbf {z}^{k} \right )_{k}$ is in a compact set, we have a subsequence converging to some point $\bar {\mathbf {z}}$ with $\left \| {\bar {\mathbf {z}}} \right \| = 1$ and $\mathbf {C}\bar {\mathbf {z}} = \mathbf {0}$. Taking the limit superior of (z^k)^T(H + kD^TD)z^k ≤ 0 over this subsequence, we get:

$$ \bar{\mathbf{z}}^{\mathrm{T}} \mathbf{H} \bar{\mathbf{z}} + \underset{k}{\limsup} k (\mathbf{z}^{k})^{\mathrm{T}} \mathbf{D}^{\mathrm{T}} \mathbf{D} \mathbf{z}^{k} \le 0. $$

(18)

Since (z^k)^TD^TDz^k ≥ 0 for all k, we must have (the subsequence) {(z^k)^TD^TDz^k} converging to zero, or else the limsup would be infinite. Thus, $(\mathbf {D}\bar {\mathbf {z}})^{\mathrm {T}} \mathbf {D} \bar {\mathbf {z}} = 0$, which implies $\mathbf {D}\bar {\mathbf {z}} = \mathbf {0}$. But by hypothesis, this means $\bar {\mathbf {z}}^{\mathrm {T}} \mathbf {H}\bar {\mathbf {z}} > 0$. Combined with the fact that $\limsup _{k} k (\mathbf {z}^{k})^{\mathrm {T}} \mathbf {D}^{\mathrm {T}} \mathbf {D} \mathbf {z}^{k}$ must be nonnegative (since each term is nonnegative), this contradicts Inequality (18). □

Lemma A.2

Let Assumption 2 hold. Then for all sufficiently large ρ, Assumption 4 holds.

Proof

If (x^∗,y^∗,μ^∗,λ^∗) is a KKT point of the overall problem (P), then we have:

$$ \begin{array}{@{}rcl@{}} \nabla f(\mathbf{x}^{*}) + \nabla \mathbf{c}(\mathbf{x}^{*}) \boldsymbol{\mu}^{*} + \mathbf{A}^{\mathrm{T}} \boldsymbol{\lambda}^{*} &=& \mathbf{0}, \\ \mathbf{c}(\mathbf{x}^{*}) &=& \mathbf{0}. \end{array} $$

Since we have Ax^∗ + By^∗ = b, we can add A^T(ρ(Ax^∗ + By^∗−b)) to the first equation to get:

$$ \begin{array}{@{}rcl@{}} \nabla f(\mathbf{x}^{*}) + \nabla \mathbf{c}(\mathbf{x}^{*}) \boldsymbol{\mu}^{*} + \mathbf{A}^{\mathrm{T}} (\boldsymbol{\lambda}^{*} + \rho (\mathbf{A}\mathbf{x}^{*} + \mathbf{B}\mathbf{y}^{*} - \mathbf{b})) &=& \mathbf{0}, \\ \mathbf{c}(\mathbf{x}^{*}) &=& \mathbf{0}, \end{array} $$

which we recognize as the KKT conditions of the subproblem when y^k = y^∗ and λ^k = λ^∗. (see Eq. ??).

Differentiability of f and c and the linear independence constraint qualification for the subproblems follow directly from the conditions of Assumption 2. It remains to show that the second-order sufficient conditions hold. Let H^∗ equal the Hessian of the Lagrangian of Problem (P) at the given KKT point; that is, let:

$$ \mathbf{H}^{*} = \begin{bmatrix} \mathbf{H}_{xx}(\mathbf{x}^{*},\boldsymbol{\mu}^{*}) & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{bmatrix} $$

(where H_xx is defined in Assumption 2). By Assumption 2, for any z satisfying z≠0, Cz = 0, we must have z^TH^∗z > 0 (where, again, C is defined in Assumption 2). Noting the form of C, by Lemma 3.1 this means that there exists ρ^∗ such that for all ρ > ρ^∗,

$$ \mathbf{z}^{\mathrm{T}} (\mathbf{H}^{*} + \rho [\mathbf{A} \mathbf{B}]^{\mathrm{T}} [\mathbf{A} \mathbf{B}]) \mathbf{z} > 0, $$

for all z with z≠0 and $\left [\nabla \mathbf {c}(\mathbf {x}^{*})^{\mathrm {T}} \mathbf {0} \right ] \mathbf {z} = \mathbf {0}$. In particular, this means that for any $\mathbf {z}_{x} \in \mathbb {R}^{n}$ satisfying z_x≠0 and $\nabla \mathbf {c}(\mathbf {x}^{*})^{\mathrm {T}} \mathbf {z}_{x} = \mathbf {0}$, we have z = (z_x,0) satisfies:

$$ 0 < \mathbf{z} ^{\mathrm{T}} \left( \mathbf{H}^{*} + \rho [\mathbf{A} \mathbf{B}]^{\mathrm{T}} [\mathbf{A} \mathbf{B}]\right) \mathbf{z} = \mathbf{z}_{x}^{\mathrm{T}} \left( \mathbf{H}_{xx}(\mathbf{x}^{*},\boldsymbol{\mu}^{*}) + \rho \mathbf{A} ^{\mathrm{T}} \mathbf{A}\right) \mathbf{z}_{x}. $$

Finally, we note that $\mathbf {H}_{xx}(\mathbf {x}^{*},\boldsymbol {\mu }^{*}) + \rho \mathbf {A}^{\mathrm {T}} \mathbf {A}$ is the Hessian of the Lagrangian of the subproblem (SP) evaluated at (x^∗,μ^∗). □

Appendix: B A result in parametric optimization

Required by the proof of Lemma 3.1, the following is a technical result, although it relies on standard and straightforward results. It is a modification of a classic sufficiency result for local optimality in the parametric setting, stating that there is a minimum size neighborhood on which local optimality holds, for all problems in a perturbed family.

Lemma B.1

Let h : (z,p)↦h(z,p) be a real-valued mapping (on $\mathbb {R}^{n_{z}} \times \mathbb {R}^{n_{p}}$) such that h is twice-continuously differentiable with respect to z on some open set D_z, for all p in some open set D_p. In addition, assume that $\nabla _{zz}^{2} h$ is continuous on D_z × D_p. Assume that for all p ∈ D_p, there exists z^∗(p) ∈ D_z such that $\nabla _{z} h(\mathbf {z}^{*}(\mathbf {p}),\mathbf {p}) = \mathbf {0}$ and $\nabla _{zz}^{2} h(\mathbf {z}^{*}(\mathbf {p}),\mathbf {p})$ is positive definite. Then for any $\bar {\mathbf {p}} \in D_{p}$, there exist positive constants 𝜖 and δ such that for all $\mathbf {p} \in N_{\delta }(\bar {\mathbf {p}})$, if $\left \| {\mathbf {z}^{*}(\mathbf {p}) - \mathbf {z}^{*}(\bar {\mathbf {p}})} \right \| \le \epsilon$, then z^∗(p) minimizes h(⋅,p) on the neighborhood $N_{\epsilon }(\mathbf {z}^{*}(\mathbf {p}))$.

Proof

That z^∗(p) minimizes h(⋅,p), for all p, follows from the standard second-order sufficient conditions for unconstrained minimization; see, for instance [5, Prop. 1.1.3]. The challenge is to show that the radius of the neighborhood on which it is a minimizer is constant with respect to p. Choose $\bar {\mathbf {p}} \in D_{p}$. Since $\nabla _{zz}^{2} h(\mathbf {z}^{*}(\bar {\mathbf {p}}),\bar {\mathbf {p}})$ is positive definite and $\nabla _{zz}^{2} h$ is continuous, for all (z,p) sufficiently close to $(\mathbf {z}^{*}(\bar {\mathbf {p}}),\bar {\mathbf {p}})$, $\nabla _{zz}^{2} h(\mathbf {z},\mathbf {p})$ is positive definite. In particular, we can choose $\epsilon ^{\prime }$, δ so that:

$$ K = \left\{ (\mathbf{z},\mathbf{p}) : \left\| \mathbf{z} - \mathbf{z}^{*}(\bar{\mathbf{p}}) \right\| \le \epsilon^{\prime}, \left\| \mathbf{p} - \bar{\mathbf{p}} \right\| \le \delta \right\} \subset D_{z} \times D_{p} $$

and $\nabla _{zz}^{2} h(\mathbf {z},\mathbf {p})$ is positive definite for all (z,p) ∈ K. Since the eigenvalues of a matrix depend continuously on the elements of a matrix ([5, Proposition A.14]), we have that the eigenvalues of $\nabla _{zz}^{2} h(\mathbf {z},\mathbf {p})$ (and in particular the minimum eigenvalue) are continuous and positive for all (z,p) ∈ K, and since K is compact, we can choose a constant λ > 0 which is a lower bound on the minimum eigenvalue for all (z,p) ∈ K.

Now choose any $\mathbf {p} \in N_{\delta }(\bar {\mathbf {p}})$ and assume $\left \| \mathbf {z}^{*}(\mathbf {p}) - \mathbf {z}^{*}(\bar {\mathbf {p}}) \right \| \le \frac {\epsilon ^{\prime }}{2}$. Consider a Taylor expansion of h(⋅,p) at z^∗(p): for any s such that $\left \| \mathbf {s} \right \| \le \frac {\epsilon ^{\prime }}{2}$, there exists α_s,p ∈ (0, 1) such that:

$$ h(\mathbf{z}^{*}(\mathbf{p})+\mathbf{s},\mathbf{p}) = h(\mathbf{z}^{*}(\mathbf{p}),\mathbf{p}) + \frac{1}{2} \mathbf{s}^{\mathrm{T}} \nabla_{zz}^{2} h(\mathbf{z}^{*}(\mathbf{p}) + \alpha_{s,p}\mathbf{s},\mathbf{p}) \mathbf{s} $$

where the linear term may be ignored because $\nabla _{z} h(\mathbf {z}^{*}(\mathbf {p}),\mathbf {p}) = \mathbf {0}$. Note that (z^∗(p) + α_s,ps,p) ∈ K, no matter what the specific value of α_s,p is. Consequently, we can use the lower bound λ on the minimum eigenvalue of the Hessian to see that:

$$ h(\mathbf{z}^{*}(\mathbf{p})+\mathbf{s},\mathbf{p}) - h(\mathbf{z}^{*}(\mathbf{p}),\mathbf{p}) \ge \frac{1}{2} \lambda \left\| \mathbf{s} \right\|^{2} $$

(see for instance [5, Prop. A.18]). Define $\epsilon \equiv \frac {\epsilon ^{\prime }}{2}$. The right-hand side of the above inequality is nonnegative, showing that z^∗(p) is a minimizer of h(⋅,p) on the neighborhood $N_{\epsilon }(\mathbf {z}^{*}(\mathbf {p}))$. □