Complexity of Proximal Augmented Lagrangian for Nonconvex Optimization with Nonlinear Equality Constraints

Abstract

We analyze worst-case complexity of a Proximal augmented Lagrangian (Proximal AL) framework for nonconvex optimization with nonlinear equality constraints. When an approximate first-order (second-order) optimal point is obtained in the subproblem, an $\epsilon$ first-order (second-order) optimal point for the original problem can be guaranteed within ${\mathcal {O}}(1/ \epsilon ^{2 - \eta })$ outer iterations (where $\eta$ is a user-defined parameter with $\eta \in [0,2]$ for the first-order result and $\eta \in [1,2]$ for the second-order result) when the proximal term coefficient $\beta$ and penalty parameter $\rho$ satisfy $\beta = {\mathcal {O}}(\epsilon ^\eta )$ and $\rho = \varOmega (1/\epsilon ^\eta )$, respectively. We also investigate the total iteration complexity and operation complexity when a Newton-conjugate-gradient algorithm is used to solve the subproblems. Finally, we discuss an adaptive scheme for determining a value of the parameter $\rho$ that satisfies the requirements of the analysis.

Appendix: Proofs of Elementary Results

Proof of Theorem 1

Since $x^*$ is a local minimizer of (1), it is the unique global solution of

$$\begin{aligned} \min \, f(x) + \frac{1}{4} \Vert x - x^* \Vert ^4 \quad \hbox {subject to}\;\; c(x) = 0, \;\; \Vert x - x^* \Vert \le \delta , \end{aligned}$$

(68)

for $\delta > 0$ sufficiently small. For the same $\delta$, we define $x_k$ to be the global solution of

$$\begin{aligned} \min \, \quad f(x) + \frac{\rho _k}{2} \Vert c(x) \Vert ^2 + \frac{1}{4} \Vert x - x^* \Vert ^4 \quad \hbox {subject to}\;\; \Vert x - x^* \Vert \le \delta , \end{aligned}$$

(69)

for a given $\rho _k$, where $\{ \rho _k \}_{k \ge 1}$ is a positive sequence such that $\rho _k \rightarrow +\infty$. Note that $x_k$ is well defined because the feasible region is compact and the objective is continuous. Suppose that z is any accumulation point of $\{ x_k \}_{k \ge 1}$, that is, $x_k \rightarrow z$ for $k \in {\mathcal {K}}$, for some subsequence ${\mathcal {K}}$. Such a z exists because $\{ x_k \}_{k \ge 1}$ lies in a compact set, and moreover, $\Vert z - x^* \Vert \le \delta$. We want to show that $z = x^*$. By the definition of $x_k$, we have for any $k \ge 1$ that

$$\begin{aligned} f(x^*)&= f(x^*) + \frac{\rho _k}{2}\Vert c(x^*) \Vert ^2 + \frac{1}{4} \Vert x^* - x^* \Vert ^4 \nonumber \\&\ge f(x_k) + \frac{\rho _k}{2}\Vert c(x_k) \Vert ^2 + \frac{1}{4} \Vert x_k - x^* \Vert ^4 \ge f(x_k) + \frac{1}{4} \Vert x_k - x^* \Vert ^4. \end{aligned}$$

(70)

By taking the limit over ${\mathcal {K}}$, we have $f(x^*) \ge f(z) + \frac{1}{4} \Vert z - x^* \Vert ^4$. From (70), we have

$$\begin{aligned} \frac{\rho _k}{2}\Vert c(x_k) \Vert ^2 \le f(x^*) - f(x_k) \le f(x^*) - \inf _{k \ge 1} f(x_k) < +\infty . \end{aligned}$$

(71)

By taking limits over ${\mathcal {K}}$, we have that $c(z) = 0$. Therefore, z is the global solution of (68), so that $z = x^*$.

Without loss of generality, suppose that $x_k \rightarrow x^*$ and $\Vert x_k - x^* \Vert < \delta$. By first and second-order optimality conditions for (69), we have

$$\begin{aligned}&\nabla f(x_k) + \rho _k \nabla c(x_k) c(x_k) + \Vert x_k - x^* \Vert ^2 (x_k - x^*) = 0, \nonumber \\&\quad \nabla ^2 f(x_k) + \rho _k \sum _{i=1}^m c_i(x_k) \nabla ^2 c_i(x_k) + \rho _k \nabla c(x_k) [ \nabla c(x_k) ]^T \end{aligned}$$

(72)

$$\begin{aligned}&+ 2(x_k-x^*)(x_k-x^*)^T + \Vert x_k-x^* \Vert ^2 I \succeq 0. \end{aligned}$$

(73)

Define $\lambda _k \triangleq \rho _k c(x_k)$ and $\epsilon _k \triangleq \max \{ \Vert x_k - x^* \Vert ^3, 3 \Vert x_k - x^* \Vert ^2, \sqrt{ 2( f(x^*) - \inf _{k \ge 1} f(x_k) )/\rho _k} \}$. Then by (71), (72), (73) and Definition 2, $x_k$ is $\epsilon _k$-2o. Note that $x_k \rightarrow x^*$ and $\rho _k \rightarrow +\infty$, so $\epsilon _k \rightarrow 0^+$. $\square$

Proof of Lemma 1

We prove by contradiction. Otherwise for any $\alpha$ we could select sequence $\{ x_k \}_{k \ge 1} \subseteq S_{\alpha }^0$ such that $f(x_k) + \frac{\rho _0}{2} \Vert c(x_k) \Vert ^2 < - k$. Let $x^*$ be an accumulation point of $\{ x_k \}_{k \ge 1}$ (which exists by compactness of $S_\alpha ^0$). Then there exists index K such that $f(x^*) + \frac{\rho _0}{2} \Vert c(x^*) \Vert ^2 \ge -K + 1 > f(x_k) + \frac{\rho _0}{2} \Vert c(x_k) \Vert ^2 + 1$ for all $k \ge K$, which contradicts the continuity of $f(x) + \frac{\rho _0}{2} \Vert c(x) \Vert ^2$. $\square$