A Dynamic Alternating Direction of Multipliers for Nonconvex Minimization with Nonlinear Functional Equality Constraints

Abstract

This paper studies the minimization of a broad class of nonsmooth nonconvex objective functions subject to nonlinear functional equality constraints, where the gradients of the differentiable parts in the objective and the constraints are only locally Lipschitz continuous. We propose a specific proximal linearized alternating direction method of multipliers in which the proximal parameter is generated dynamically, and we design an explicit and tractable backtracking procedure to generate it. We prove subsequent convergence of the method to a critical point of the problem, and global convergence when the problem’s data are semialgebraic. These results are obtained with no dependency on the explicit manner in which the proximal parameter is generated. As a byproduct of our analysis, we also obtain global convergence guarantees for the proximal gradient method with a dynamic proximal parameter under local Lipschitz continuity of the gradient of the smooth part of the nonlinear sum composite minimization model.

Appendices

Appendix A: Subdifferential Calculus and the Lagrangian

Model (M) is nonconvex and nonsmooth in general, and so we turn to subdifferential calculus for nonconvex functions. Specifically, we recall the definition of subdifferentials, critical points, and a few related results. For a more detailed presentation, see [22, 23, 26].

Definition A.1

(Subdifferentials) Let ${\mathbb {E}}$ be an Euclidean vector space, $\psi : {\mathbb {E}} \rightarrow (-\infty , \infty ]$ be a proper and lsc function, and $z\in {{\,\mathrm{dom}\,}}\psi$.

• (i) The Fréchet (regular) subdifferential of $\psi$ at z, denoted by ${\hat{\partial }}\psi (z)$, is the set of all vectors $v\in {\mathbb {E}}$ satisfying

  $$\begin{aligned} \psi (x)\ge \psi (z)+\langle v,x-z\rangle + o\left( \left\| {x-z} \right\| \right) \text {.} \end{aligned}$$

  (A.1)

• (ii) The Mordukhovich (limiting) subdifferential of $\psi$ at z, denoted by $\partial \psi (z)$, is the set of all vectors $v\in {\mathbb {E}}$ such that there exist sequences $\{z^{k}\}_{k\in {\mathbb {N}}}$ and $\{v^{k}\}_{k\in {\mathbb {N}}}$ where $z^{k}\rightarrow z$, $\psi (z^{k})\rightarrow \psi (z)$, $v^{k}\in {\hat{\partial }}\psi (z^{k})$, and $v^{k}\rightarrow v$.

• (iii) The horizon subdifferential of $\psi$ at z, denoted by $\partial ^{\infty }\psi {z}$, has a similar definition to that in (ii), but instead of $v^{k}\rightarrow v$ we have $t^{k}v^{k}\rightarrow v$ for some real sequence $t^{k}\searrow 0$.

For $z\notin {{\,\mathrm{dom}\,}}\psi$ we set ${\hat{\partial }}\psi (z)=\partial \psi (z)=\partial ^{\infty }\psi (z)=\emptyset$.

Note that (A.1) is equivalent to

$$\begin{aligned} \liminf _{\underset{x\ne z}{x\rightarrow z}}\frac{\psi (x)-\psi (z)-\langle v,x-z\rangle }{\left\| {x-z} \right\| }\ge 0\text {.} \end{aligned}$$

Remark A.1

(Closedness of graph of $\partial \psi$) We note, as in [7, Remark 1(ii)], that given a convergent sequence $(x^{k}, w^{k})\xrightarrow [k\rightarrow \infty ]{}(x, w)$, such that $w^{k}\in \partial \psi (x^{k})$ and $\lim _{k\rightarrow \infty }\psi (x^{k})=\psi (x)$, it holds that $w\in \partial \psi (x)$.

Proposition A.1

(see Exercise 8.8(c), p.304 in [26]) Let $\psi :{\mathbb {R}}^{d}\rightarrow (-\infty , \infty ]$ be an extended valued function and $\phi :{\mathbb {R}}^{d}\rightarrow {\mathbb {R}}$ be a smooth function. Then,

$$\begin{aligned} \partial (\psi +\phi )(x)=\partial \psi (x) + \nabla \phi (x),\quad \forall x\in {\mathbb {R}}^{d}, \end{aligned}$$

where for $x\notin {{\,\mathrm{dom}\,}}\psi$, we note that $\emptyset +\nabla \phi (x)=\emptyset$.

Definition A.2

(Critical points) Let ${\mathbb {E}}$ be an Euclidean vector space, $\psi : {\mathbb {E}} \rightarrow (-\infty , \infty ]$ be a proper and lsc function. Then, the set of critical points of $\psi$ is defined by

$$\begin{aligned} {{\,\mathrm{crit}\,}}\psi {:=}\{x: 0\in \partial \psi (x)\}. \end{aligned}$$

Appendix B: Lipschitz and Local Lipschitz Continuity

We review the notion of Lipschitz and local Lipschitz continuity in the context of model (M). Thus, we restrict our discussion to Euclidean vector spaces and in the following ${\mathbb {X}}$ and ${\mathbb {Y}}$ denote such spaces.

We begin with the basic definition.

Definition B.1

(Lipschitz and locally Lipschitz continuity) Let $S\subseteq {\mathbb {X}}$ be a nonempty set and $\phi :S\rightarrow {\mathbb {Y}}$ be a continuous mapping over S. Then,

• (i) $\phi$ is L-Lipschitz continuous over S with $L\ge 0$ if the following holds

  $$\begin{aligned} \left\| {\phi (x) - \phi (z)} \right\| \le L\left\| {x-z} \right\| ,\quad \forall x,z\in S. \end{aligned}$$

  (B.1)

• (ii) $\phi$ is locally Lipschitz continuous over S if for every $z\in S$ there exist $\varepsilon (z)>0$, $L_{\varepsilon }(z)\ge 0$, and a neighborhood ${\mathcal {N}}_{\varepsilon }(y){:=}\{x\in S: \left\| {x-z} \right\| <\varepsilon (z)\}$, such that $\phi$ is $L_{\varepsilon }(z)$-Lipschitz continuous over ${\mathcal {N}}_{\varepsilon }(z)$, i.e.,

  $$\begin{aligned} \left\| {\phi (x) - \phi (z)} \right\| \le L_{\varepsilon }(z)\left\| {x-z} \right\| ,\quad \forall x,z\in {\mathcal {N}}_{\varepsilon }(z). \end{aligned}$$

  (B.2)

When (i) or (ii) hold with $S\equiv {\mathbb {X}}$ then $\phi$ is referred to as L-Lipschitz continuous or locally Lipschitz continuous, respectively. In addition, note that when $\phi (x)-\phi (z)$ is a matrix then $\left\| {\phi (x)-\phi (z)} \right\|$ is the spectral norm.

We also recall the following characterization of locally Lipschitz continuous mappings.

Proposition B.1

(Local Lipschitz continuity and compact sets (Theorem 2.1.6 in [13])) Let $S\subseteq {\mathbb {X}}$ be a nonempty set. Then, a mapping $\phi :{\mathbb {X}}\rightarrow {\mathbb {Y}}$ is locally Lipschitz continuous over S if and only if for every nonempty and compact set $C\subseteq S$ there exists $L_{C}\ge 0$ such that $\phi$ is $L_{C}$-Lipschitz continuous over C, i.e.,

$$\begin{aligned} \left\| {\phi (x) - \phi (z)} \right\| \le L_{C}\left\| {x-z} \right\| ,\quad \forall x,z\in C. \end{aligned}$$

(B.3)

We conclude with a proposition which deals with Lipschitz continuity properties of $C^{1}$ mappings.

Proposition B.2

(Differential mappings and Lipschitz continuity) Let $\phi :{\mathbb {X}}\rightarrow {\mathbb {Y}}$ be a $C^{1}$ mapping. Then the following claims hold.

• (i) $\phi$ is locally Lipschitz continuous;

• (ii) Let $B\subseteq {\mathbb {X}}$ be a closed ball, i.e., $B=\{x:\left\| {x-z} \right\| \le r\}$, for some $z\in {\mathbb {X}}$ and $r\in (0, \infty ]$, and assume that $\phi$ is $L_{B}$-Lipschitz continuous over B, with $L_{B}\ge 0$. Then,

  $$\begin{aligned} \left\| {\nabla \phi (x)} \right\| \le L_{B},\;\forall x\in B. \end{aligned}$$

Proof

For the first claim, see, e.g., [12, Corollary, p. 32]. The second claim can be easily obtained using the definition of the Gâteaux derivative, see [12, p. 30], and the continuity of $\nabla \phi$. $\square$

Appendix C: Proof of Lemma 4.1

Proof of Lemma 4.1

We prove that $\nabla \varphi$ is locally Lipschitz continuous by utilizing the local Lipschitz continuity characterization stated in Proposition B.1, i.e., given a nonempty and compact set $C\subset {\mathbb {R}}^{n}\times {\mathbb {R}}^{m}\times {\mathbb {R}}^{q}$ we prove that there exists $L_{C}\in {\mathbb {R}}_{+}$ such that for every $\omega =(u,v,y), {\hat{\omega }}=({\hat{u}}, {\hat{v}}, {\hat{y}})\in C$, we have

$$\begin{aligned} \left\| {\nabla \varphi ({\hat{\omega }}) -\nabla \varphi (\omega )} \right\| \le L_{C}\left\| {{\hat{\omega }}-\omega } \right\| . \end{aligned}$$

(C.1)

With $\nabla \varphi (\omega )=\left( \nabla _{u}\varphi (\omega ), \nabla _{v}\varphi (\omega ), \nabla _{y}\varphi (\omega )\right)$, we intend to prove that there exist $L_{C,u}$, $L_{C,v}$, and $L_{C,y}\in {\mathbb {R}}_{+}$, such that

$$\begin{aligned} \left\| {\nabla _{u}\varphi ({\hat{\omega }}) -\nabla _{u}\varphi (\omega )} \right\|&\le L_{C,u}\left\| {{\hat{\omega }}-\omega } \right\| , \end{aligned}$$

(C.2)

$$\begin{aligned} \left\| {\nabla _{v}\varphi ({\hat{\omega }}) -\nabla _{v}\varphi (\omega )} \right\|&\le L_{C,v}\left\| {{\hat{\omega }}-\omega } \right\| , \end{aligned}$$

(C.3)

$$\begin{aligned} \left\| {\nabla _{y}\varphi ({\hat{\omega }}) -\nabla _{y}\varphi (\omega )} \right\|&\le L_{C,y}\left\| {{\hat{\omega }}-\omega } \right\| . \end{aligned}$$

(C.4)

and as

$$\begin{aligned}&\left\| {\nabla \varphi ({\hat{\omega }}) -\nabla \varphi (\omega )} \right\| \le \left\| {\nabla _{u}\varphi ({\hat{\omega }}) -\nabla _{u}\varphi (\omega )} \right\| \\&+\left\| {\nabla _{v}\varphi ({\hat{\omega }}) -\nabla _{v} \varphi (\omega )} \right\| +\left\| {\nabla _{y}\varphi ({\hat{\omega }}) -\nabla _{y}\varphi (\omega )} \right\| , \end{aligned}$$

we can set $L_{C} = L_{C,u}+L_{C,v}+L_{C,y}$.

We begin with $\nabla _{u}\varphi$ (cf. (4.1)) and note that for every $(u,v,y),({\hat{u}},{\hat{v}},{\hat{y}})\in {\mathbb {R}}^{n}\times {\mathbb {R}}^{m}\times {\mathbb {R}}^{q}$, we have

$$\begin{aligned} \begin{aligned}&\left\| {\nabla _{u}\varphi ({\hat{u}},{\hat{v}}, {\hat{y}})-\nabla _{u}\varphi (u,v, y)} \right\| \\&\,=\!\left\| \nabla g({\hat{u}}) \!+\! \nabla F({\hat{u}})^{T}\left( y+\rho (F({\hat{u}})\!-\!G{\hat{v}})\right) -\nabla g(u) - \nabla F(u)^{T}\left( y+\rho (F(u)-Gv)\right) \right\| \\&\le \big \Vert {\nabla g({\hat{u}})-\nabla g(u)} \big \Vert +\big \Vert \nabla F({\hat{u}})^{T}\left( {\hat{y}}+\rho (F({\hat{u}})-G{\hat{v}})\right) - \nabla F(u)^{T}\left( y+\rho (F(u)-Gv)\right) \big \Vert , \end{aligned} \end{aligned}$$

(C.5)

where the inequality is due to the triangle inequality.

Next, let $C_{u}$ be the projection of C on ${\mathbb {R}}^{n}$ and $B_{u}$ be a compact ball such that $C_{u}\subseteq B_{u}$. Then, as $\nabla g$ is locally Lipschitz continuous and $B_{u}$ is compact, we can apply Proposition B.1 and obtain that there exists $L_{g}\ge 0$ such that

$$\begin{aligned} \left\| {\nabla g({\hat{u}})-\nabla g(u)} \right\| \le L_{g}\left\| {{\hat{u}}-u} \right\| ,\quad \forall u, {\hat{u}}\in B_{u}. \end{aligned}$$

Next, recall that F is $C^{1}$ and hence, by Proposition B.2(i), F is locally Lipschitz continuous. In addition, $\nabla F$ is also locally Lipschitz continuous. Thus, there exist $l_{F} \ge 0$ and $L_{F} \ge 0$, such that for every $u,{\hat{u}}\in B_{u}$, we have

$$\begin{aligned} \left\| {F({\hat{u}})-F(u)} \right\| \le l_{F}\left\| {{\hat{u}}-u} \right\| ,\ \left\| {\nabla F({\hat{u}})-\nabla F(u)} \right\| \le L_{F}\left\| {{\hat{u}}-u} \right\| ,\ \text {and}\ \left\| {\nabla F(u)} \right\| \le l_{F}, \end{aligned}$$

where the first two inequalities are due to Proposition B.1 and the last is due to Proposition B.2(ii). Applying the above inequalities, we obtain that, for every $u,{\hat{u}}\in B_{u}$, $v,{\hat{v}}\in {\mathbb {R}}^{m}$, and $y,{\hat{y}}\in {\mathbb {R}}^{q}$, we have

$$\begin{aligned} \begin{aligned}&\left\| {\nabla F({\hat{u}})^{T}\left( {\hat{y}}+\rho (F({\hat{u}})-G{\hat{v}})\right) - \nabla F(u)^{T}\left( y+\rho (F(u)-Gv)\right) } \right\| \\ {}&\,=\big \Vert \left( \nabla F({\hat{u}})^{T}-\nabla F(u)^{T}\right) \left( y+\rho (F(u)-Gv)\right) \\&\qquad + \nabla F({\hat{u}})^{T}\left( {\hat{y}}-y+\rho (F({\hat{u}})-F(u))-\rho G({\hat{v}}-v)\right) \big \Vert \\&\,\le \left\| {\nabla F({\hat{u}})-\nabla F(u)} \right\| \big \Vert {y + \rho (F(u)-Gv)}\big \Vert \\&\qquad + \left\| {\nabla F({\hat{u}})}\right\| \left( \left\| {{\hat{y}}-y} \right\| +\rho \left\| {F({\hat{u}})-F(u)} \right\| +\rho \big \Vert {G} \big \Vert \big \Vert {{\hat{v}}-v} \big \Vert \right) \\&\,\le \left( L_{F}\big \Vert {y{+}\rho (F(u)-Gv)}\big \Vert +\rho l_{F}^{2}\right) \left\| {{\hat{u}}-u} \right\| {+}l_{F}\left( \left\| {{\hat{y}}-y} \right\| +\rho \big \Vert {G} \big \Vert \left\| {{\hat{v}}-v} \right\| \right) , \end{aligned}\end{aligned}$$

Combining the results above, and noting that $C\subseteq B_{u}\times {\mathbb {R}}^{m}\times {\mathbb {R}}^{q}$, we obtain that, for every $\omega =(u,v,y), {\hat{\omega }}=({\hat{u}},{\hat{v}},{\hat{y}})\in C$, we have

$$\begin{aligned} \begin{aligned} \left\| {\nabla _{u}\varphi ({\hat{\omega }})-\nabla _{u}\varphi (\omega )} \right\|&\le \left( L_{g}+L_{F}\left\| {y+\rho (F(u)-Gv)} \right\| +\rho l_{F}^{2}\right) \\&\left\| {{\hat{u}}-u} \right\| +l_{F}\left( \left\| {{\hat{y}}-y} \right\| +\rho \left\| {G} \right\| \left\| {{\hat{v}}-v} \right\| \right) \\&\le L_{\varphi }(u,v,y)\left\| {{\hat{\omega }} -\omega } \right\| , \end{aligned} \end{aligned}$$

with $L_{\varphi }(u,v,y){:=}\left( L_{g}+L_{F}\left\| {y+\rho (F(u)-Gv)} \right\| + l_{F}(1 + \rho l_{F} + \rho \left\| {G} \right\| ) \right) \left\| {{\hat{\omega }} -\omega } \right\| .$ We complete the proof for $\nabla _{u}\varphi$ and obtain inequality (C.2) by setting $L_{C,u}=\sup _{(u,v,y)\in C}L_{\varphi }(u,v,y)$ and noting that $L_{C,u}<\infty$ as $L_{\varphi }$ is continuous and C is compact.

Next, we examine $\nabla _{v}\varphi$ (cf. (4.2)). For every $\omega , {\hat{\omega }}\in C$, we have

$$\begin{aligned} \begin{aligned} \left\| {\nabla _{v}\varphi ({\hat{\omega }}) - \nabla _{v}\varphi ({\hat{\omega }})} \right\|&=\left\| {G^{T}\left( {\hat{y}}-y + \rho (F({\hat{u}})-F(u)) -\rho (G({\hat{v}}-v))\right) } \right\| \\&\le \left\| {G} \right\| \left( \left\| {{\hat{y}}-y} \right\| +\rho \left\| {F({\hat{u}})-F(u)} \right\| +\rho \left\| {G} \right\| \left\| {{\hat{v}}-v} \right\| \right) \\&\le \left\| {G} \right\| \left( \left\| {{\hat{y}}-y} \right\| +\rho l_{F}\left\| {{\hat{u}}-u} \right\| +\rho \left\| {G} \right\| \left\| {{\hat{v}}-v} \right\| \right) \\&\le L_{C,v}\left\| {{\hat{\omega }}-\omega } \right\| , \end{aligned} \end{aligned}$$

with $L_{C,v} = \left\| {G} \right\| \left( 1 + \rho (l_{F} + \left\| {G} \right\| )\right)$.

Finally, with $\nabla _{y}\varphi$ (cf. (4.3)), for every $\omega , {\hat{\omega }}\in C$, we have

$$\begin{aligned} \begin{aligned} \left\| {\nabla _{y}\varphi ({\hat{\omega }}) - \nabla _{y}\varphi (\omega )} \right\|&=\left\| {F({\hat{u}})-F(u)-G({\hat{v}}-v)} \right\| \\&\le \left\| {F({\hat{u}})-F(u)} \right\| + \left\| {G({\hat{v}}-v)} \right\| \\&\le l_{F}\left\| {{\hat{u}}-u} \right\| + \left\| {G} \right\| \left\| {{\hat{v}}-v} \right\| \\&\le L_{C,y}\left\| {{\hat{\omega }})-\omega } \right\| , \end{aligned} \end{aligned}$$

with $L_{C,y} = l_{F} + \left\| {G} \right\|$. $\square$