Globally convergent coderivative-based generalized Newton methods in nonsmooth optimization

Abstract

This paper proposes and justifies two globally convergent Newton-type methods to solve unconstrained and constrained problems of nonsmooth optimization by using tools of variational analysis and generalized differentiation. Both methods are coderivative-based and employ generalized Hessians (coderivatives of subgradient mappings) associated with objective functions, which are either of class ${{\mathcal {C}}}^{1,1}$, or are represented in the form of convex composite optimization, where one of the terms may be extended-real-valued. The proposed globally convergent algorithms are of two types. The first one extends the damped Newton method and requires positive-definiteness of the generalized Hessians for its well-posedness and efficient performance, while the other algorithm is of the regularized Newton-type being well-defined when the generalized Hessians are merely positive-semidefinite. The obtained convergence rates for both methods are at least linear, but become superlinear under the semismooth$^*$ property of subgradient mappings. Problems of convex composite optimization are investigated with and without the strong convexity assumption on smooth parts of objective functions by implementing the machinery of forward–backward envelopes. Numerical experiments are conducted for Lasso problems and for box constrained quadratic programs with providing performance comparisons of the new algorithms and some other first-order and second-order methods that are highly recognized in nonsmooth optimization.

Appendix: Some technical lemmas

This section contains four technical lemmas used in the text.

The first lemma is a local version of [44, Lemma 2.20]. The proof of this result is similar to the original one, and thus it is omitted.

Lemma 3

Let $\Omega \subset \mathrm{I\!R}^n$ be an open set, and let $\varphi :\Omega \rightarrow \mathrm{I\!R}$ be a continuously differentiable function such that $\nabla \varphi$ is Lipschitz continuous with modulus $L >0$. Then for any $\sigma >0$, $x \in \Omega$, and $d \in \mathrm{I\!R}^n$ satisfying $\langle \nabla \varphi (x),d\rangle <0$, the following inequality

$$\begin{aligned} \varphi (x+\tau d)\le \varphi (x) +\sigma \tau \langle \nabla \varphi (x),d\rangle \end{aligned}$$

(7.1)

holds whenever $\tau \in (0,\overline{\tau }]$ and $x+\tau d \in \Omega$, where

$$\begin{aligned} \overline{\tau }:= \frac{2(\sigma -1)\langle \nabla \varphi (x),d\rangle }{L\Vert d\Vert ^2}>0. \end{aligned}$$

The next lemma provides conditions for the R-linear and Q-linear convergence of sequences.

Lemma 4

(estimates for convergence rates) Let $\{\alpha _k\}, \{\beta _k \}$, and $\{\gamma _k \}$ be sequences of positive numbers. Assume that there exist numbers $c_i>0$, $i=1,2,3$, and $k_0\in \mathrm{I\!N}$ such that for all $k \ge k_0$ we have the estimates:

• (i) $\alpha _k - \alpha _{k+1} \ge c_1 \beta _k^2$.

• (ii) $\beta _k \ge c_2 \gamma _k$.

• (iii) $c_3 \gamma _k^2 \ge \alpha _k$.

Then the sequence $\{\alpha _k \}$ Q-linearly converges to zero, and the sequences $\{\beta _k \}$ and $\{\gamma _k \}$ R-linearly converge to zero as $k\rightarrow \infty$.

Proof

Combining (i), (ii), and (iii) yields the inequalities

$$\begin{aligned} \alpha _{k} - \alpha _{k+1} \ge c_1 \beta _k^2 \ge c_1 c_2^2 \gamma _k^2 \ge \frac{c_1c_2^2}{c_3}\alpha _{k} \quad \text {for all }\; k \ge k_0, \end{aligned}$$

which imply that $\alpha _{k+1}\le q \alpha _k$, where $q:= 1- (c_1c_2^2)/c_3\in (0,1)$. This verifies that the sequence $\{\alpha _{k}\}$ Q-linearly converges to zero. Using the latter and the assumed condition (i) ensures that

$$\begin{aligned} \beta _k^2 \le \frac{1}{c_1}(\alpha _k - \alpha _{k+1}) \le \frac{1}{c_1}\alpha _k \le \frac{1}{c_1}q\alpha _{k-1} \le \ldots \le \frac{1}{c_1} q^k \alpha _0 \quad \text {for all }\; k\ge k_0, \end{aligned}$$

which tells us that $\beta _k \le c\mu ^k$, where $c:= \sqrt{\alpha _0/c_1}$ and $\mu :=\sqrt{q}$. This justifies the R-linear convergence of the sequence $\{\beta _k \}$ to zero. Furthermore, it easily follows from (ii) that the sequence $\{\gamma _k \}$ R-linearly converge to zero, and thus we are done with the proof. $\square$

Now we obtain a useful result on the semismooth$^*$ property of compositions.

Lemma 5

(semismooth$^*$ property of composition mappings) Let $A \in \mathrm{I\!R}^{n\times n}$ be a symmetric nonsingular matrix, $b \in \mathrm{I\!R}^n$, ${\bar{x}}\in \mathrm{I\!R}^n$, and $f:\mathrm{I\!R}^n \rightarrow \mathrm{I\!R}^n$ be continuous and semismooth$^*$ at ${\bar{y}}:=A{\bar{x}}+b$. Then the mapping $g: \mathrm{I\!R}^n \rightarrow \mathrm{I\!R}^n$ defined by $g(x):= f(Ax+b)$ is semismooth$^*$ at ${\bar{x}}$.

Proof

Using the coderivative chain rule from [59, Theorem 1.66], we get

$$\begin{aligned} D^*g(x)(w)= A^* D^*f(Ax+b)(w) \quad \text {for all }\; w \in \mathrm{I\!R}^n. \end{aligned}$$

(7.2)

Denote $\mu := \sqrt{\textrm{max}\{1,\Vert A\Vert ^2 \}\cdot \textrm{max}\{1,\Vert A^{-1}\Vert ^2\}}>0$. Picking any $\varepsilon >0$ and employing the semismooth$^*$ property of f at ${\bar{y}}$, we find $\delta >0$ such that

$$\begin{aligned} |\langle x^*, y-{\bar{y}}\rangle - \langle y^*, f(y) - f({\bar{y}})\rangle |\le \frac{\varepsilon }{\mu } \big \Vert (y-{\bar{y}},f(y)-f({\bar{y}}))\big \Vert \cdot \Vert (x^*,y^*)\Vert \nonumber \\ \end{aligned}$$

(7.3)

for all $y \in {\mathbb {B}}_\delta ({\bar{y}})$ and all $(x^*,y^*) \in \textrm{gph}\,D^*f(y)$. Denoting $r:= \delta / \Vert A\Vert >0$ gives us $y:=Ax+b \in {\mathbb {B}}_\delta ({\bar{y}})$ whenever $x \in {\mathbb {B}}_r({\bar{x}})$. Picking now $x \in {\mathbb {B}}_r({\bar{x}})$ and $(z^*,w^*) \in \textrm{gph}\,D^*g(x)$, we get $(A^{-1}z^*,w^*)\in \textrm{gph}\,D^*f(Ax+b)$ due to (7.2). It follows from (7.3) that

$$\begin{aligned} |\langle z^*, x-{\bar{x}}\rangle - \langle w^*,g(x) - g({\bar{x}})\rangle |= & {} |\langle A^{-1}z^*, y-{\bar{y}}\rangle - \langle w^*, f(y) - f({\bar{y}})\rangle | \\\le & {} \frac{\varepsilon }{\mu }\big \Vert (y-{\bar{y}},f(y)-f({\bar{y}}))\Vert \cdot \big \Vert (A^{-1}z^*,w^*)\big \Vert \\= & {} \frac{\varepsilon }{\mu }\big \Vert (Ax-A{\bar{x}},g(x)-g({\bar{x}})) \big \Vert \cdot \big \Vert (A^{-1}z^*,w^*)\big \Vert \\\le & {} \varepsilon \Vert (x- {\bar{x}},g(x)-g({\bar{x}}))\Vert \cdot \Vert (z^*,w^*)\Vert , \end{aligned}$$

which verifies the semismooth$^*$ property of g at ${\bar{x}}$. $\square$

The final lemma establishes tilt stability of strongly convex functions at stationary points.

Lemma 6

(strong convexity and tilt-stability) Let $\varphi :\mathrm{I\!R}^n\rightarrow \overline{\mathrm{I\!R}}$ be an l.s.c. and strongly convex function with modulus $\kappa >0$, and let ${\bar{x}}\in \textrm{dom}\,\varphi$ such that $0\in \partial \varphi ({\bar{x}})$. Then ${\bar{x}}$ is a tilt-stable local minimizer of $\varphi$ with modulus $\kappa ^{-1}$.

Proof

By the second-order characterization of strongly convex functions [10, Theorem 5.1], we have

$$\begin{aligned}\langle z, w\rangle \ge \kappa \Vert w\Vert ^2 \quad \text {for all }\; z \in \partial ^2\varphi (x,y)(w),\; (x,y) \in \textrm{gph}\,\partial \varphi ,\;\text{ and } \;w \in \mathrm{I\!R}^n.\end{aligned}$$

This implies in turn that ${\bar{x}}$ is a tilt-stable local minimizer with modulus $\kappa ^{-1}$ by second-order characterization of tilt stability taken from [61, Theorem 3.5]. $\square$