A fast gradient and function sampling method for finite-max functions

Abstract

This paper proposes an algorithm for the unconstrained minimization of a class of nonsmooth and nonconvex functions that can be written as finite-max functions. A gradient and function-based sampling method is proposed which, under special circumstances, either moves superlinearly to a minimizer of the problem of interest or improves the optimality certificate. Global and local convergence analysis are presented, as well as examples that illustrate the obtained theoretical results.

Appendix

The aim of this appendix is to show that the assumption $(\lambda _{k,{\overline{l}}_k})_i = 0$, whenever $i \notin \mathcal I(x_*)$ and $k\in \mathcal K$, with $\mathcal K$ defined in Corollary 2, is not necessary. For this goal, we will show that even without such an assumption, the results from the local convergence subsection remain the same.

We divide our reasoning in two cases and remind the reader that we have assumed $\mathcal I(x_*) = \{1,\ldots ,r+1\}$:

• (A1) The cardinality of $\mathcal I(x_*)$ is $n+1$;

• (A2) The cardinality of $\mathcal I(x_*)$ is $r+1$ with $r < n$.

Suppose first that A1 holds and let us consider an iterate $x_k$ sufficiently close to $x_*$. Moreover, assume that $k\in \mathcal K$, where $\mathcal K$ is the index set defined in Corollary 2. Then, looking at the optimization problem in (22), we see that any additional active constraint will generate an additional active constraint to (22) in a way that it will be a linear combination of the first $n+1$ active constraints (by Remark 2 and because the rank of ${\tilde{J}}_{k}$ remains constant in a close neighborhood of $x_*$). Hence, the solution obtained with, or without, this additional constraint is the same, which yields that the results presented at the local convergence subsection do not change for this special case.

So, let us consider the more intricate case A2. Moreover, let us assume that there is only one additional constraint, i.e., the number of active constraints is $r+2$ (we will see that the occurrence of more than one additional constraint will be a straightforward generalization of this simpler case). In other words, we are saying that solving (5) is equivalent to minimize

$$\begin{aligned} \begin{aligned} \min _{\left( d,z\right) \in {\mathbb {R}}^{n+1}~~}&z + \frac{1}{2}d^TH_kd \\ \text {s.t. }&f\left( x_{k,i}^{{\overline{l}}_k}\right) + \nabla f\left( x_{k,i}^{{\overline{l}}_k}\right) ^T\left( x_k + d - x_{k,i}^{{\overline{l}}_k}\right) = z_\text{, }~~1\le i\le r+2\text{, } \end{aligned} \end{aligned}$$

where here we assume that rearrangements were done in order to have the additional constraint as the $(r+2)$-th constraint and that it has the associated sampled point $x_{k,r+2}^{{\overline{l}}_k}$. Therefore, for an iterate $x_k$ sufficiently close to the solution and a sufficiently small sampling radius, we have, by the continuity of the functions $\phi _i$, that only the functions $\phi _1,\ldots ,\phi _{r+1}$ can assume the maximum at any sampled point. So, there exists $j\in \{1,\ldots ,r+1\}$ such that $f(x_{k,r+2}^{{\overline{l}}_k}) = \phi _j(x_{k,r+2}^{{\overline{l}}_k})$. Consequently, recalling that $k\in \mathcal K$, the dual problem of the above minimization problem can be seen as

$$\begin{aligned} \begin{aligned}&\max _{\lambda \in {\mathbb {R}}^{r+2}~~} \sum _{i=1}^{r+1} \lambda _i\left[ \phi _i\left( x_{k,i}^{{\overline{l}}_k}\right) + \nabla \phi _i\left( x_{k,i}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,i}^{{\overline{l}}_k}\right) \right] \\&\quad + \lambda _{r+2}\left[ \phi _j\left( x_{k,r+2}^{{\overline{l}}_k}\right) + \nabla \phi _j\left( x_{k,r+2}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,r+2}^{{\overline{l}}_k}\right) \right] \\&\quad - \frac{1}{2}\left\| \sum _{i=1}^{r+1} \lambda _i \nabla \phi _i(x_{k,i}^{{\overline{l}}_k}) + \lambda _{r+2} \nabla \phi _j(x_{k,r+2}^{{\overline{l}}_k}) \right\| _{H_k^{-1}}^2\\&\text {s.t. } e^T\lambda = 1\text{. } \end{aligned} \end{aligned}$$

(36)

Therefore, we can turn this last constrained maximization problem into an unconstrained one by making the following substitution $\lambda _{r+2} = 1 - \sum _{i=1}^{r+1} \lambda _i$. So, we have

$$\begin{aligned} \begin{aligned}&\max _{\lambda \in {\mathbb {R}}^{r+1}~~} \sum _{i=1}^{r+1} \lambda _i\left[ \phi _i\left( x_{k,i}^{{\overline{l}}_k}\right) + \nabla \phi _i\left( x_{k,i}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,i}^{{\overline{l}}_k}\right) - \phi _j\left( x_{k,r+2}^{{\overline{l}}_k}\right) \right. \\&\left. \quad - \nabla \phi _j\left( x_{k,r+2}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,r+2}^{{\overline{l}}_k}\right) \right] + \phi _j\left( x_{k,r+2}^{{\overline{l}}_k}\right) \\&\quad + \nabla \phi _j\left( x_{k,r+2}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,r+2}^{{\overline{l}}_k}\right) \\&\quad - \frac{1}{2}\left\| \sum _{i=1}^{r+1} \lambda _i \left[ \nabla \phi _i(x_{k,i}^{{\overline{l}}_k}) - \nabla \phi _j(x_{k,r+2}^{{\overline{l}}_k}) \right] + \nabla \phi _j(x_{k,r+2}^{{\overline{l}}_k}) \right\| _{H_k^{-1}}^2\text{. }\\ \end{aligned} \end{aligned}$$

Since the above problem is concave and smooth, its solution ${\overline{\lambda }}\in {\mathbb {R}}^{r+1}$ can be obtained by equaling the derivative of the objective function to the null vector. Consequently, assuming without loss of generality that the function $\phi _j$ involved in the additional constraint is $\phi _{r+1}$, we have

$$\begin{aligned} \begin{aligned}&\left( \begin{array}{c} \nabla \phi _1\left( x_{k,1}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+2}^{{\overline{l}}_k}\right) ^T\\ \vdots \\ \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+2}^{{\overline{l}}_k}\right) ^T \end{array}\right) H_k^{-1} \left( \begin{array}{c} \nabla \phi _1\left( x_{k,1}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+2}^{{\overline{l}}_k}\right) ^T\\ \vdots \\ \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+2}^{{\overline{l}}_k}\right) ^T \end{array}\right) ^T{\overline{\lambda }} \\&\quad = \left( \begin{array}{c} \phi _1\left( x_{k,1}^{{\overline{l}}_k}\right) + \nabla \phi _1\left( x_{k,1}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,1}^{{\overline{l}}_k}\right) \\ \vdots \\ \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) + \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,r+1}^{{\overline{l}}_k}\right) \end{array}\right) \\&\qquad - \left( \begin{array}{c} \phi _{r+1}\left( x_{k,r+2}^{{\overline{l}}_k}\right) + \nabla \phi _{r+1}\left( x_{k,r+2}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,r+2}^{{\overline{l}}_k}\right) \\ \vdots \\ \phi _{r+1}\left( x_{k,r+2}^{{\overline{l}}_k}\right) + \nabla \phi _{r+1}\left( x_{k,r+2}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,r+2}^{{\overline{l}}_k}\right) \end{array}\right) \\&\qquad -\left( \begin{array}{c} \nabla \phi _1\left( x_{k,1}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+2}^{{\overline{l}}_k}\right) ^T\\ \vdots \\ \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+2}^{{\overline{l}}_k}\right) ^T \end{array}\right) H_k^{-1}\nabla \phi _{r+1}\left( x_{k,r+2}^{{\overline{l}}_k}\right) \text{. } \end{aligned} \end{aligned}$$

Now, changing the points $x_{k,r+2}^{{\overline{l}}_k}$ for $x_{k,r+1}^{{\overline{l}}_k}$ and redefining

$$\begin{aligned} \tau _{k,{\overline{l}}_k} := \max _{1\le i\le r+2}\left\| x_{k,i}^{{\overline{l}}_k} - x_k\right\| \text{, } \end{aligned}$$

we get

$$\begin{aligned} \begin{aligned}&\left( \begin{array}{c} \nabla \phi _1\left( x_{k,1}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\\ \vdots \\ \nabla \phi _{r}\left( x_{k,r}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\\ 0^T \end{array}\right) H_k^{-1} \left( \begin{array}{c} \nabla \phi _1\left( x_{k,1}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\\ \vdots \\ \nabla \phi _{r}\left( x_{k,r}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\\ 0^T \end{array}\right) ^T{\overline{\lambda }} \\&\quad = \left( \begin{array}{c} \phi _1\left( x_{k,1}^{{\overline{l}}_k}\right) + \nabla \phi _1\left( x_{k,1}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,1}^{{\overline{l}}_k}\right) - \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,r+1}^{{\overline{l}}_k}\right) \\ \vdots \\ \phi _{r}\left( x_{k,r}^{{\overline{l}}_k}\right) + \nabla \phi _{r}\left( x_{k,r}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,r}^{{\overline{l}}_k}\right) - \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,r+1}^{{\overline{l}}_k}\right) \\ 0^T \end{array}\right) \\&\qquad -\left( \begin{array}{c} \nabla \phi _1\left( x_{k,1}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\\ \vdots \\ \nabla \phi _{r}\left( x_{k,r}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\\ 0^T \end{array}\right) H_k^{-1}\nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) + O\left( \tau _{k,{\overline{l}}_k}\right) \text{. } \end{aligned} \end{aligned}$$

This last linear system yields

$$\begin{aligned} \begin{aligned}&\left( \begin{array}{c} \nabla \phi _1\left( x_{k,1}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\\ \vdots \\ \nabla \phi _{r}\left( x_{k,r}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T \end{array}\right) H_k^{-1} \left( \begin{array}{c} \nabla \phi _1\left( x_{k,1}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\\ \vdots \\ \nabla \phi _{r}\left( x_{k,r}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T \end{array}\right) ^T \left( \begin{array}{c} {{\overline{\lambda }}}_1\\ \vdots \\ {{\overline{\lambda }}}_r \end{array}\right) \\&\quad = \left( \begin{array}{c} \phi _1\left( x_{k,1}^{{\overline{l}}_k}\right) + \nabla \phi _1\left( x_{k,1}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,1}^{{\overline{l}}_k}\right) - \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,r+1}^{{\overline{l}}_k}\right) \\ \vdots \\ \phi _{r}\left( x_{k,r}^{{\overline{l}}_k}\right) + \nabla \phi _{r}\left( x_{k,r}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,r}^{{\overline{l}}_k}\right) - \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\left( x_k - x_{k,r+1}^{{\overline{l}}_k}\right) \end{array}\right) \\&\qquad -\left( \begin{array}{c} \nabla \phi _1\left( x_{k,1}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T\\ \vdots \\ \nabla \phi _{r}\left( x_{k,r}^{{\overline{l}}_k} \right) ^T - \nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) ^T \end{array}\right) H_k^{-1}\nabla \phi _{r+1}\left( x_{k,r+1}^{{\overline{l}}_k}\right) + O\left( \tau _{k,{\overline{l}}_k}\right) \text{. } \end{aligned} \end{aligned}$$

Therefore, following the same reasoning used by us to get here, it is possible to see that the first r components of the dual variable ${{\hat{\lambda }}} \in {\mathbb {R}}^{r+1}$ linked to the problem (21) must satisfy the last linear system obtained above (not considering the remaining error vector) and, moreover,

$$\begin{aligned} {\hat{\lambda }}_{r+1} = 1 - \sum _{i=1}^r {\hat{\lambda }}_i\text{. } \end{aligned}$$

(37)

Therefore, considering $\lambda ^*\in {\mathbb {R}}^{r+2}$ the solution of (36) and using equation (37), we must have

$$\begin{aligned} \lambda ^* = \left( \begin{array}{c} {\hat{\lambda }}_1\\ \vdots \\ {\hat{\lambda }}_r\\ \lambda ^*_{r+1}\\ 1-\sum _{i=1}^r {\hat{\lambda }}_i - \lambda ^*_{r+1} \end{array}\right) + O\left( \tau _{k,{\overline{l}}_k}\right) = \left( \begin{array}{c} {\hat{\lambda }}_1\\ \vdots \\ {\hat{\lambda }}_r\\ \lambda ^*_{r+1}\\ {\hat{\lambda }}_{r+1} - \lambda ^*_{r+1} \end{array}\right) + O\left( \tau _{k,{\overline{l}}_k}\right) \text{. } \end{aligned}$$

So, to complete our reasoning, we write the following relation between the primal-dual variables

$$\begin{aligned} \begin{aligned} d_{k,{\overline{l}}_k}&= -H_k^{-1}\left[ \sum _{i=1}^{r+1}\lambda _i^*\nabla \phi _i\Big (x_{k,i}^{{\overline{l}}_k}\Big ) + \lambda ^*_{r+2}\nabla \phi _{r+1}\Big (x_{k,r+2}^{{\overline{l}}_k}\Big ) \right] \\&= -H_k^{-1}\left[ \sum _{i=1}^{r}\lambda ^*_i\nabla \phi _i\Big (x_{k,i}^{{\overline{l}}_k}\Big ) + \left( \lambda ^*_{r+1} + \lambda ^*_{r+2}\right) \nabla \phi _{r+1}\Big (x_{k,r+1}^{{\overline{l}}_k}\Big ) \right] + O\left( \tau _{k,{\overline{l}}_k}\right) \\&= -H_k^{-1}\sum _{i=1}^{r+1}{\hat{\lambda }_i\nabla } \phi _i\Big (x_{k,i}^{{\overline{l}}_k}\Big ) + O\left( \tau _{k,{\overline{l}}_k}\right) \text{. } \end{aligned} \end{aligned}$$

Hence, $d_{k,{\overline{l}}_k}$ is exactly the search direction obtained in (21) with an additional error vector. Therefore, the term $O\left( \tau _{k,{\overline{l}}_k}\right)$ is absorbed by the other error vectors in Theorem 3 and the result is still valid.

Finally, remember that we have considered just one additional active constraint to the others $r+1$ active constraints. However, it is straightforward to see that exactly the same reasoning can be used to prove the result for any other number of additional constraints.