Stochastic inexact augmented Lagrangian method for nonconvex expectation constrained optimization

Abstract

Many real-world problems not only have complicated nonconvex functional constraints but also use a large number of data points. This motivates the design of efficient stochastic methods on finite-sum or expectation constrained problems. In this paper, we design and analyze stochastic inexact augmented Lagrangian methods (Stoc-iALM) to solve problems involving a nonconvex composite (i.e. smooth + nonsmooth) objective and nonconvex smooth functional constraints. We adopt the standard iALM framework and design a subroutine by using the momentum-based variance-reduced proximal stochastic gradient method (PStorm) and a postprocessing step. Under certain regularity conditions (assumed also in existing works), to reach an $\varepsilon$-KKT point in expectation, we establish an oracle complexity result of $O(\varepsilon ^{-5})$, which is better than the best-known $O(\varepsilon ^{-6})$ result. Numerical experiments on the fairness constrained problem and the Neyman–Pearson classification problem with real data demonstrate that our proposed method outperforms an existing method with the previously best-known complexity result.

A regularity condition for ball-constrained fairness and Neyman–Pearson problems

In this section, we show that the regularity condition in (7) holds for the nonconvex fairness problem in (46) and the Neyman-Pearson classification problem in (48), when a ball constraint is imposed and the input data satisfies certain conditions. We consider the two problems together in the following form:

$$\begin{aligned} \min _{{\textbf{x}}, s} f_0({\textbf{x}}), \text{ s.t. } \hat{f}_1({\textbf{x}},s):=f_1({\textbf{x}}) + s = 0, s \ge 0, {\textbf{x}}\in {\mathcal {X}}:=\{{\textbf{x}}\in \mathbb {R}^d: \Vert {\textbf{x}}\Vert \le \lambda \}, \end{aligned}$$

(49)

where $\lambda > 0$, and $f_0$ and $f_1$ are the functions defined in (46) and (48) respectively for the fairness problem and the Neyman-Pearson classification problem. The slack variable s is used to reformulate (46) and (48) into equality-constrained problems. We did not include the constraint ${\textbf{x}}\in {\mathcal {X}}$ in our experiments but the generated iterate sequence remained bounded.

Let ${\mathcal {N}}_{\mathcal {X}}({\textbf{x}})$ be the normal cone of ${\mathcal {X}}$ at ${\textbf{x}}\in {\mathcal {X}}$ and ${\mathcal {N}}_+(s)$ the normal cone of $\mathbb {R}_+$ at $s \ge 0$. Then the regularity condition in (7) for the problem (49) becomes: there exists $\nu >0$ such that

$$\begin{aligned} \nu ^2 (f_1({\textbf{x}}) + s)^2 \le \textrm{dist}\left( -(f_1({\textbf{x}}) + s) \left[ \begin{array}{c}\nabla f_1({\textbf{x}})\\ 1 \end{array}\right] ,\ {\mathcal {N}}_{\mathcal {X}}({\textbf{x}}) \otimes {\mathcal {N}}_+(s)\right) ^2, \forall \, {\textbf{x}}\in {\mathcal {X}}, s\ge 0. \end{aligned}$$

(50)

Notice that ${\mathcal {N}}_{\mathcal {X}}({\textbf{x}}) = \{\textbf{0}\}$ if $\Vert {\textbf{x}}\Vert < \lambda$ and ${\mathcal {N}}_{\mathcal {X}}({\textbf{x}}) = \{\alpha {\textbf{x}}: \alpha \ge 0\}$ if $\Vert {\textbf{x}}\Vert =\lambda$. Also, ${\mathcal {N}}_+(s) = \{0\}$ if $s> 0$ and ${\mathcal {N}}_+(s) = \mathbb {R}_-$ if $s=0$. Hence, for any $({\textbf{x}}, s) \in {\mathcal {X}}\otimes \mathbb {R}_+$,

$$\begin{aligned} \begin{aligned}&\textrm{dist}\left( -(f_1({\textbf{x}}) + s) \left[ \begin{array}{c}\nabla f_1({\textbf{x}})\\ 1 \end{array}\right] ,\ {\mathcal {N}}_{\mathcal {X}}({\textbf{x}}) \otimes {\mathcal {N}}_+(s)\right) ^2\\&\quad = \left\{ \begin{array}{ll} (f_1({\textbf{x}}) + s)^2 + \textrm{dist}\big (-(f_1({\textbf{x}}) + s)\nabla f_1({\textbf{x}}),\, {\mathcal {N}}_{\mathcal {X}}({\textbf{x}}) \big )^2, &{} \text { if } s >0\\[0.1cm] \min \big (f_1({\textbf{x}}), 0\big )^2 + |f_1({\textbf{x}})|^2\Vert \nabla f_1({\textbf{x}})\Vert ^2, &{} \text { if } s =0, \Vert {\textbf{x}}\Vert < \lambda \\[0.1cm] \min \big (f_1({\textbf{x}}), 0\big )^2 + \min _{\alpha \ge 0} \Vert f_1({\textbf{x}}) \nabla f_1({\textbf{x}}) + \alpha {\textbf{x}}\Vert ^2, &{} \text { if } s =0, \Vert {\textbf{x}}\Vert = \lambda . \end{array} \right. \end{aligned} \end{aligned}$$

(51)

From (51), we can easily have that when $s>0$ or when $s=0$ and $f_1({\textbf{x}}) \le 0$, it holds

$$\begin{aligned} (f_1({\textbf{x}}) + s)^2 \le \textrm{dist}\left( -(f_1({\textbf{x}}) + s) \left[ \begin{array}{c}\nabla f_1({\textbf{x}})\\ 1 \end{array}\right] ,\ {\mathcal {N}}_{\mathcal {X}}({\textbf{x}}) \otimes {\mathcal {N}}_+(s)\right) ^2. \end{aligned}$$

Thus we only need to show the regularity condition at $({\textbf{x}},0)$ with ${\textbf{x}}\in {\mathcal {X}}$ such that $f_1({\textbf{x}}) > 0$. We make the following assumption about the data involved in (46) and (48).

Assumption 6

The feature vectors in (46) and (48) satisfy:

• (i) In (46), $\Vert {\textbf{a}}\Vert = q, \forall \, {\textbf{a}}\in S$ for some $q>0$ and $\langle {\textbf{a}}_1, {\textbf{a}}_2 \rangle \ge 0$ for any ${\textbf{a}}_1, {\textbf{a}}_2 \in S$. In addition,

  $$\begin{aligned} \frac{e^{\lambda q} }{(1+e^{\lambda q})^2} \sqrt{\sum _{{\textbf{a}}_1, {\textbf{a}}_2 \in S\backslash S_{\min }} {\textbf{a}}_1^\top {\textbf{a}}_2}> \frac{1-c}{4c} \sqrt{\sum _{{\textbf{a}}_1, {\textbf{a}}_2 \in S_{\min }} {\textbf{a}}_1^\top {\textbf{a}}_2}. \end{aligned}$$

  (52)

• (ii) In (48), $\Vert {\textbf{a}}_i^-\Vert = q, \forall \, i$ for some $q>0$, and $\langle {\textbf{a}}_i^-, {\textbf{a}}_j^- \rangle \ge 0$ for any i, j.

The above assumption will hold if each data point is first normalized and then appended by 1 at the end, which is equivalent to having an intercept term in the model, and in addition, for (46) the minority group $S_{\min }$ is only a small fraction of S.

Claim A1

Under Assumption 6, let $f_1$ be given in (46) or (48). Then $\nu _1:= \min _{\Vert {\textbf{x}}\Vert \le \lambda } \Vert \nabla f_1({\textbf{x}})\Vert >0$.

Proof

We first prove the claim for (48), for which case,

$$\begin{aligned} \nabla f_1({\textbf{x}}) = -\frac{1}{n^-}\sum _{i=1}^{n^-} \phi '(-{\textbf{x}}^\top {\textbf{a}}_i^-) {\textbf{a}}_i^-. \end{aligned}$$

(53)

Suppose $\nu _1 = \Vert \nabla f_1(\tilde{{\textbf{x}}})\Vert$, i.e., the minimum is reached at $\tilde{{\textbf{x}}}$. Notice $\phi '(u) = - \frac{e^u}{(1+e^u)^2} < 0$. Thus

$$\begin{aligned} \nu _1^2 = \frac{1}{(n^-)^2} \sum _{i, j =1}^{n^-} \phi '(-\tilde{{\textbf{x}}}^\top {\textbf{a}}_i^-) \phi '(-\tilde{{\textbf{x}}}^\top {\textbf{a}}_j^-) \langle {\textbf{a}}_i^-, {\textbf{a}}_j^-\rangle \ge \frac{1}{(n^-)^2} \sum _{i=1}^{n^-} \big [\phi '(-\tilde{{\textbf{x}}}^\top {\textbf{a}}_i^-)\big ]^2 \Vert {\textbf{a}}_i^- \Vert ^2, \end{aligned}$$

where the inequality follows from $\langle {\textbf{a}}_i^-, {\textbf{a}}_j^-\rangle \ge 0$ for all i, j. Hence, $\nu _1>0$ must hold by Assumption 6(ii).

Next we prove the claim for (46). When $f_1$ is the function in (46), it holds

$$\begin{aligned} \nabla f_1({\textbf{x}}) =c \sum _{{\textbf{a}}\in S\backslash S_{\min }} \sigma '({\textbf{a}}^\top {\textbf{x}}) {\textbf{a}}- (1-c) \sum _{{\textbf{a}}\in S_{\min }} \sigma '({\textbf{a}}^\top {\textbf{x}}) {\textbf{a}}. \end{aligned}$$

Again, suppose $\nu _1 = \Vert \nabla f_1(\tilde{{\textbf{x}}})\Vert$. Notice $\sigma '(u) = \frac{e^u}{(1+e^u)^2}$ is decreasing on $[0, +\infty )$ and increasing on $(-\infty , 0]$. Also, by Assumption 6(i) and $\Vert \tilde{{\textbf{x}}}\Vert \le \lambda$, we have $|{\textbf{a}}^\top \tilde{{\textbf{x}}}| \le q\lambda$. Hence, $\frac{e^{q\lambda }}{(1+e^{q\lambda })^2} \le \sigma '({\textbf{a}}^\top \tilde{{\textbf{x}}}) \le \frac{1}{4}$. Thus

$$\begin{aligned} \left\| c \sum _{{\textbf{a}}\in S\backslash S_{\min }} \sigma '({\textbf{a}}^\top \tilde{{\textbf{x}}}) {\textbf{a}}\right\| ^2= & {} c^2 \sum _{{\textbf{a}}_1, {\textbf{a}}_2 \in S\backslash S_{\min }} \sigma '({\textbf{a}}_1^\top \tilde{{\textbf{x}}}) \sigma '({\textbf{a}}_2^\top \tilde{{\textbf{x}}}) {\textbf{a}}_1^\top {\textbf{a}}_2 \\\ge & {} \frac{c^2 e^{2q\lambda }}{(1+e^{q\lambda })^4} \sum _{{\textbf{a}}_1, {\textbf{a}}_2 \in S\backslash S_{\min }} {\textbf{a}}_1^\top {\textbf{a}}_2, \end{aligned}$$

and

$$\begin{aligned} \left\| (1-c) \sum _{{\textbf{a}}\in S_{\min }} \sigma '({\textbf{a}}^\top {\textbf{x}}) {\textbf{a}}\right\| ^2 \le \frac{(1-c)^2}{16} \sum _{{\textbf{a}}_1, {\textbf{a}}_2 \in S_{\min }} {\textbf{a}}_1^\top {\textbf{a}}_2. \end{aligned}$$

By the triangle inequality, it holds that

$$\begin{aligned} \Vert \nabla f_1({\textbf{x}})\Vert \ge c \left\| \sum _{{\textbf{a}}\in S\backslash S_{\min }} \sigma '({\textbf{a}}^\top {\textbf{x}}) {\textbf{a}}\right\| - (1-c) \left\| \sum _{{\textbf{a}}\in S_{\min }} \sigma '({\textbf{a}}^\top {\textbf{x}}) {\textbf{a}}\right\| . \end{aligned}$$

Therefore, from (52), we obtain $\nu _1 > 0$ and complete the proof. $\square$

Claim A2

Suppose Assumption 6(ii) holds and in addition, the origin is a feasible point of (48), i.e., $f_1(\textbf{0}) \le 0$. Then it holds

$$\begin{aligned} \nu _2 := \min _{\alpha \ge 0, {\textbf{x}}}\big \{ \Vert f_1({\textbf{x}})\nabla f_1({\textbf{x}}) + \alpha {\textbf{x}}\Vert : \Vert {\textbf{x}}\Vert =\lambda , f_1({\textbf{x}}) \ge 0 \big \} > 0. \end{aligned}$$

(54)

Proof

Suppose that the minimum in (54) is reached at $\tilde{{\textbf{x}}}$, i.e., $\Vert \tilde{{\textbf{x}}}\Vert = \lambda$, $f_1(\tilde{{\textbf{x}}}) \ge 0$, and

$$\begin{aligned} \nu _2 = \min _{\alpha \ge 0} \Vert f_1(\tilde{{\textbf{x}}})\nabla f_1(\tilde{{\textbf{x}}}) + \alpha \tilde{{\textbf{x}}}\Vert . \end{aligned}$$

If $\nu _2 = 0$, then we must have $\tilde{\textbf{x}}= -\lambda \frac{\nabla f_1(\tilde{{\textbf{x}}})}{\Vert \nabla f_1(\tilde{{\textbf{x}}})\Vert }$ and the optimal $\alpha = \frac{ \Vert \nabla f_1(\tilde{{\textbf{x}}})\Vert }{\lambda }$. By (53), we have

$$\begin{aligned} -\tilde{{\textbf{x}}}^\top {\textbf{a}}_j^- = -\frac{\lambda }{\Vert \nabla f_1(\tilde{{\textbf{x}}})\Vert } \frac{1}{n^-} \sum _{i=1}^{n^-} \phi '(-{\textbf{x}}^\top {\textbf{a}}_i^-) \langle {\textbf{a}}_i^-, {\textbf{a}}_j^-\rangle > 0, \end{aligned}$$

where the inequality follows from $\phi '(u) < 0, \forall \, u$ and Assumption 6(ii). Now notice that $\phi (u)$ is an decreasing function. We have $f_1(\tilde{{\textbf{x}}}) < f_1(\textbf{0}) \le 0$, which contradicts to $f_1(\tilde{{\textbf{x}}}) \ge 0$. Therefore, we must have $\nu _2>0$ and thus complete the proof. $\square$

By Claims A1 and A2, we immediately obtain the theorem below.

Theorem 4

Suppose Assumption 6 holds and in addition, the origin is feasible in (48). Then there must exist a constant $\nu >0$ such that if $f_1$ is given in (46),

$$\begin{aligned} \nu |{\hat{f}}_1({\textbf{x}}, s)| \le \textrm{dist}\left( -{\hat{f}}_1({\textbf{x}}, s) \nabla {\hat{f}}_1({\textbf{x}}, s),\ {\mathcal {N}}_{\mathcal {X}}({\textbf{x}}) \otimes {\mathcal {N}}_+(s)\right) , \forall \, {\textbf{x}}\in \textrm{int}({\mathcal {X}}), s\ge 0, \end{aligned}$$

(55)

and if $f_1$ is given in (48),

$$\begin{aligned} \nu |{\hat{f}}_1({\textbf{x}}, s)| \le \textrm{dist}\left( -{\hat{f}}_1({\textbf{x}}, s) \nabla {\hat{f}}_1({\textbf{x}}, s),\ {\mathcal {N}}_{\mathcal {X}}({\textbf{x}}) \otimes {\mathcal {N}}_+(s)\right) , \forall \, {\textbf{x}}\in {\mathcal {X}}, s\ge 0, \end{aligned}$$

(56)

where ${\hat{f}}_1$ and ${\mathcal {X}}$ are defined in (49).

Proof

From (50) and (51), we obtain (55) with $\nu = \min \{1, \nu _1\}$, where $\nu _1$ is defined in Claim A1, and we obtain (56) with $\nu = \min \{1, \nu _1, \nu _2\}$, where $\nu _1$ and $\nu _2$ are defined in Claims A1 and A2. $\square$

Remark 9

From Theorem 4, we see that the regularity condition will hold for the ball constrained version of (48) under a certain data preprocessing and an origin feasibility condition. It can almost hold for a ball constrained version of (46), except for the points on the sphere of the constraint ball. For the tested instances of (46) and (48) without a ball constraint, we checked the regularity condition at the iterates (because we actually only need the condition at the generated iterates). We found that for (46), the introduced slack variable s would always be positive during the iterations, and for (48), the iterates became feasible after just a few iterations; see Fig. 2. Hence, for the instances tested in our experiments, the regularity condition in (7) holds with $\nu =1$ for ${\textbf{x}}$ being a generated iterate.