Finding Second-Order Stationary Points in Constrained Minimization: A Feasible Direction Approach

Abstract

This paper introduces a method for computing points satisfying the second-order necessary optimality conditions for nonconvex minimization problems subject to a closed and convex constraint set. The method comprises two independent steps corresponding to the first- and second-order conditions. The first-order step is a generic closed map algorithm, which can be chosen from a variety of first-order algorithms, making it adjustable to the given problem. The second-order step can be viewed as a second-order feasible direction step for nonconvex minimization subject to a convex set. We prove that any limit point of the resulting scheme satisfies the second-order necessary optimality condition, and establish the scheme’s convergence rate and complexity, under standard and mild assumptions. Numerical tests illustrate the proposed scheme.

Appendices

Appendix A: Continuity of the Second-Order Optimality Measure

We start by recalling here some well-known definitions/properties on multi-valued maps that can be found for example in [37]. Assume hereafter that $C\subseteq \mathbb {R}^n$ is closed, convex, nonempty set.

Definition A.1

(Basic properties of point-to-set maps) Let $\Omega$ be a point-to-set map at $\bar{\mathbf{x}}\in C$. Then:

• 1. $\Omega$ is open at $\bar{\mathbf{x}}$, if $\{\mathbf{x}^k\}_{k\ge 0}\subseteq C$, $\mathbf{x}^k\rightarrow \bar{\mathbf{x}}$, and $\bar{\mathbf{y}}\in \Omega (\bar{\mathbf{x}})$ imply the existence of an integer K and a sequence $\{ \mathbf{y}^k \}_{k\ge 0}\subseteq C$ such that $\mathbf{y}^k\in \Omega (\mathbf{x}^k)$ for $k\ge K$ and $\mathbf{y}^k\rightarrow \bar{\mathbf{y}}$.

• 2. $\Omega$ is closed at $\bar{\mathbf{x}}$, if $\{\mathbf{x}^k\}_{k\ge 0}\subseteq C$, $\mathbf{x}^k\rightarrow \bar{\mathbf{x}}$, $\mathbf{y}^k\in \Omega (\mathbf{x}^k)$, and $\mathbf{y}^k\rightarrow \bar{\mathbf{y}}$ imply that $\bar{\mathbf{y}}\in \Omega (\bar{\mathbf{x}})$.

• 3. $\Omega$ is continuous at $\bar{\mathbf{x}}$ if it is both open and closed at $\bar{\mathbf{x}}$.

• 4. $\Omega$ is uniformly compact near $\bar{\mathbf{x}}$ if there is a neighborhood N of $\bar{\mathbf{x}}$ such that the closure of the set $\bigcup _{\mathbf{x}\in N}$ is compact.

The continuity property of Q follows from the next result (which we mildly adjusted to our setting).

Theorem A.1

(Continuity of problems [37, Theorem 7]) Let $\Omega$ be a point-to-set map, and let

$$\begin{aligned} \nu (\mathbf{x}) = \displaystyle \mathop {\mathrm{sup}}_{\mathbf{y}\in \Omega (\mathbf{x})} \phi (\mathbf{x},\mathbf{y}), \end{aligned}$$

(16)

where $\phi : C \times C \rightarrow ]-\infty ,\infty [$. If $\Omega$ is continuous at $\bar{\mathbf{x}}\in C$ and uniformly compact near $\bar{\mathbf{x}}$, and if $\phi$ is continuous on $\bar{\mathbf{x}}\times \Omega (\bar{\mathbf{x}})$, then $\nu$ is continuous at $\bar{\mathbf{x}}$.

To apply Theorem A.1 in the case of Q, we will restrict the discussion to the optimization problem (16) with the point-to-set mapping

$$\begin{aligned} \Omega (\mathbf{x}) = \{ \mathbf{y}\in C : \mathbf{a}(\mathbf{x})^T (\mathbf{y}-\mathbf{x}) \le 0, \Vert \mathbf{y}- \mathbf{x}\Vert _2 \le r \}, \qquad \forall \mathbf{x}\in C, \end{aligned}$$

(17)

with $\mathbf{a}:\mathbb {R}^n\rightarrow \mathbb {R}^n$ being a continuous vector mapping, while assuming that

$$\begin{aligned} \phi : C \times C \rightarrow ]-\infty ,\infty [ \end{aligned}$$

is a continuous function. Clearly, the optimality measure $Q(\cdot )$ can be defined in the form of (16) by setting $\mathbf{y}= \mathbf{x}+ \mathbf{d}$, $\phi (\mathbf{x},\mathbf{y}) = -f''(\mathbf{x}; \mathbf{y}-\mathbf{x})$, and $\mathbf{a}(\mathbf{x}) = \nabla f(\mathbf{x})$.

We will prove that $\Omega$ given by (17) is uniformly compact and continuous, where the continuity of $\Omega$ will be established using the following two results.

Theorem A.2

[37, Theorem 10] Let $\Omega$ be a point-to-set map given by

$$\begin{aligned} \Omega (\mathbf{x}) = \{ \mathbf{y}\in C : g(\mathbf{x},\mathbf{y}) \le 0\}, \end{aligned}$$

where each component of $g:C\times C \rightarrow [-\infty ,\infty ]^m$ is lower semicontinuous in $\{\bar{\mathbf{x}}\}\times C$. Then, $\Omega$ is closed at $\bar{\mathbf{x}}$.

Noting that C is convex, we have the following (adjusted) result (credited to Geoffrion).

Theorem A.3

[37, Theorem 12] Let $\Omega$ be a point-to-set map given by

$$\begin{aligned} \Omega (\mathbf{x}) = \{ \mathbf{y}\in C : g(\mathbf{x},\mathbf{y}) \le 0\}, \end{aligned}$$

where each component of $g:C\times C \rightarrow [-\infty ,\infty ]^m$ is continuous on $\{\bar{\mathbf{x}}\}\times \Omega (\bar{\mathbf{x}})$, and convex in $\mathbf{y}$ for each fixed $\mathbf{x}\in C$. If there exists a $\bar{\mathbf{y}}$ such that $g(\bar{\mathbf{x}},\bar{\mathbf{y}})<0$, then $\Omega$ is open at $\bar{\mathbf{x}}$.

We can now prove the continuity and uniform compactness of $\Omega$.

Lemma A.1

Suppose that $\Omega$ is given by (17). Then for any $\bar{\mathbf{x}}\in C$, we have that $\Omega$ is uniformly compact and continuous at $\bar{\mathbf{x}}$.

Proof

The uniform compactness of $\Omega$ follows trivially from its definition as the intersection of a half-plane and a norm ball.

To prove that $\Omega$ is continuous, we will show that it is both closed and open. The closeness of $\Omega$ is implied by Theorem A.2 as for any $\bar{\mathbf{x}}\in C$, all functions defining the point-to-set mapping $\Omega$ are continuous.

We will now prove that $\Omega$ is open. Let $\bar{\mathbf{x}}\in C$, $\mathbf{x}^k\rightarrow \bar{\mathbf{x}}$, and $\bar{\mathbf{y}}\in \Omega (\bar{\mathbf{x}})$. We need to prove that there exists a sequence $\{\mathbf{y}^k\}_{k\ge 0}$ such that $\mathbf{y}^k\rightarrow \bar{\mathbf{y}}$, satisfying that for any $k\ge 0$ it holds that $\mathbf{y}^k\in \Omega (\mathbf{x}^k)$.

Suppose that there exists $\hat{\mathbf{y}}\in \Omega (\bar{\mathbf{x}})$ such that $\mathbf{a}(\bar{\mathbf{x}})^T (\hat{\mathbf{y}} - \bar{\mathbf{x}}) < 0$. If $\Vert \hat{\mathbf{y}} - \bar{\mathbf{x}} \Vert _2 = r$, then there exists $\mathbf{d}\in \mathbb {R}^n$ and a sufficient small $\alpha$ such that $\mathbf{a}(\bar{\mathbf{x}})^T (\hat{\mathbf{y}} + \alpha \mathbf{d}- \bar{\mathbf{x}}) < 0$ and $\Vert \hat{\mathbf{y}} + \alpha \mathbf{d}- \bar{\mathbf{x}} \Vert _2 < r$. Hence, there exists $\tilde{\mathbf{y}}\in \Omega (\bar{\mathbf{x}})$ such that $\mathbf{a}(\bar{\mathbf{x}})^T (\tilde{\mathbf{y}} - \bar{\mathbf{x}}) < 0$ and $\Vert \tilde{\mathbf{y}} - \bar{\mathbf{x}} \Vert _2 < r$. Subsequently, we can invoke Theorem A.3 to obtain the desired claim that $\Omega (\bar{\mathbf{x}})$ is open. Assume hereafter that $\mathbf{a}(\bar{\mathbf{x}})^T (\mathbf{y}- \bar{\mathbf{x}}) = 0$ for any $\mathbf{y}\in \Omega (\bar{\mathbf{x}})$.

Set $\bar{r} = \Vert \bar{\mathbf{y}} - \bar{\mathbf{x}} \Vert _2 \le r$, and define for any $k\ge 0$

$$\begin{aligned} \mathbf{y}^k&= {\left\{ \begin{array}{ll} \bar{\mathbf{y}}, &{} \mathbf{x}^k = \bar{\mathbf{y}},\\ \mathbf{x}^k + \frac{\bar{r}}{\Vert \mathbf{d}^k \Vert _2 } \mathbf{d}^k, &{} \mathbf{x}^k \ne \bar{\mathbf{y}}, \end{array}\right. } \text { and } \mathbf{d}^k&= {\left\{ \begin{array}{ll} \bar{\mathbf{y}} - \mathbf{x}^k, &{} \mathbf{a}(\mathbf{x}^k) = 0, \\ \bar{\mathbf{y}} - \mathbf{x}^k - \frac{\mathbf{a}(\mathbf{x}^k)^T (\bar{\mathbf{y}} - \mathbf{x}^k)}{\Vert \mathbf{a}(\mathbf{x}^k)\Vert _2^2} \mathbf{a}(\mathbf{x}^k), &{} \mathbf{a}(\mathbf{x}^k) \ne 0 . \end{array}\right. } \end{aligned}$$

To see that indeed $\mathbf{y}^k \rightarrow \bar{\mathbf{y}}$, note that by the continuity of all terms and the assumption that $\mathbf{a}(\bar{\mathbf{x}})^T (\bar{\mathbf{y}} - \bar{\mathbf{x}}) = 0$ we have that

$$\begin{aligned} \mathbf{d}^k \rightarrow \bar{\mathbf{y}} - \bar{\mathbf{x}}, \text { and } \mathbf{y}^k \rightarrow \bar{\mathbf{y}} . \end{aligned}$$

The claim $\mathbf{y}^k\in \Omega (\mathbf{x}^k)$ for any $k\ge 0$ follows from the fact that $\mathbf{y}^k = \bar{\mathbf{y}}\in \Omega (\bar{\mathbf{x}})$ whenever $\mathbf{x}^k = \bar{\mathbf{y}}$ or $\mathbf{a}(\mathbf{x}^k) = 0$, and otherwise

$$\begin{aligned} \mathbf{a}(\mathbf{x}^k)^T \mathbf{y}^k&= \mathbf{a}(\mathbf{x}^k)^T \mathbf{x}^k + \frac{\bar{r}}{\Vert \mathbf{d}^k \Vert _2} \mathbf{a}(\mathbf{x}^k)^T \left( \bar{\mathbf{y}} - \mathbf{x}^k - \frac{\mathbf{a}(\mathbf{x}^k)^T (\bar{\mathbf{y}} - \mathbf{x}^k)}{\Vert \mathbf{a}(\mathbf{x}^k)\Vert _2^2} \mathbf{a}(\mathbf{x}^k) \right) = \mathbf{a}(\mathbf{x}^k)^T \mathbf{x}^k,\\ \Vert \mathbf{y}^k - \mathbf{x}^k \Vert _2&=\bar{r} \le r. \end{aligned}$$

To summarize, for any $\bar{\mathbf{y}}\in \Omega (\bar{\mathbf{x}})$, there exists a sequence $\{\mathbf{y}^k\}_{k\ge 0}$ such that $\mathbf{y}^k\rightarrow \bar{\mathbf{y}}$ satisfying that for any $k\ge 0$ it holds that $\mathbf{y}^k\in \Omega (\mathbf{x}^k)$, meaning that $\Omega (\bar{\mathbf{x}})$ is open. $\square$

Proof of Lemma 4.3

(Continuity of the second-order optimality measure Q) Since $\Omega$ is continuous and uniformly compact (Lemma A.1) at every point $\mathbf{x}\in C$, and $f''(\mathbf{x};\mathbf{d})$ is continuous, we have by Theorem A.1 that Q is continuous. $\square$

Appendix B: The Projected Gradient Descent Method with Backtracking

The Projected Gradient Descent (PGD) method is a classical algorithm with abundant references describing it in details; see for instance the survey of [41] and the extensive list references therein. The complexity of the PGD to find an $\varepsilon$ first order stationary point (measured through the usual norm of the gradient map) is of the order of $(O(\varepsilon ^{-2}))$; see, e.g., [30, Theorem 9.15]. Below, we describe the PGD used in our numerical tests with step size determined by backtracking policy below (cf. [30, Section 9.4]).