A J-symmetric quasi-newton method for minimax problems

Abstract

Minimax problems have gained tremendous attentions across the optimization and machine learning community recently. In this paper, we introduce a new quasi-Newton method for the minimax problems, which we call J-symmetric quasi-Newton method. The method is obtained by exploiting the J-symmetric structure of the second-order derivative of the objective function in minimax problem. We show that the Hessian estimation (as well as its inverse) can be updated by a rank-2 operation, and it turns out that the update rule is a natural generalization of the classic Powell symmetric Broyden method from minimization problems to minimax problems. In theory, we show that our proposed quasi-Newton algorithm enjoys local Q-superlinear convergence to a desirable solution under standard regularity conditions. Furthermore, we introduce a trust-region variant of the algorithm that enjoys global R-superlinear convergence. Finally, we present numerical experiments that verify our theory and show the effectiveness of our proposed algorithms compared to Broyden’s method and the extragradient method on three classes of minimax problems.

Existing definitions and results used in the proofs

Lemma A.1

(Sherman–Woodbury Formula) ( [37, page 19]) Suppose $A\in {\mathbb {R}}^{n\times n}$ is an invertible matrix and vectors $u,v\in {\mathbb {R}}^{n}$. Then $A+uv^T$ is invertible if and only is $1+v^TA^{-1}u\ne 0$. In this case,

$$\begin{aligned} (A+uv^T)^{-1}=A^{-1}-\frac{A^{-1}uv^TA^{-1}}{1+v^TA^{-1}u} . \end{aligned}$$

Lemma A.2

(Banach Perturbation Lemma) ( [54, page 45]) Consider square matrices $A,B \in {\mathbb {R}}^{d\times d}$. Suppose that A is invertible with $\Vert A^{-1}\Vert \le a$. If $\Vert A-B\Vert \le b$ and $ab < 1$, then B is also invertible and

$$\begin{aligned} \Vert B^{-1}\Vert \le \frac{a}{1-ab } . \end{aligned}$$

Lemma A.3

([23, Eq. (1.2)]) Consider square matrices $A,B \in {\mathbb {R}}^{d\times d}$. Then

$$\begin{aligned} \Vert AB \Vert _F \le \min \{ ~\Vert A \Vert _F\Vert B \Vert ~,~ \Vert A \Vert \Vert B \Vert _F~ \}. \end{aligned}$$

Definition A.4

(R-superlinear and Q-superlinear convergence rates [48]) We say the sequence $\{z_k\}$ is converging to $z^*$ R-superlinearly, if

$$\begin{aligned} \lim _{k \rightarrow \infty } \Vert z_k -z^*\Vert ^{1/k} =0, \end{aligned}$$

and $\{z_k\}$ is converging to $z^*$ Q-superlinearly, if there exists a sequence $\{q_k\}$ converging to zero such that

$$\begin{aligned} \lim _{k \rightarrow \infty } \frac{\Vert z_{k+1} -z^*\Vert }{\Vert z_k -z^*\Vert } \le q_k . \end{aligned}$$

Theorem A.5

(Dennis-Moré Q-superlinear Characterization Identity) ([22, Theorem 2.2]) Let the mapping F be differentiable in the open convex set $\mathbb {D}$ and assume that for some $z^* \in \mathbb {D}$, $\nabla F$ is continuous at $z^*$ and $\nabla F(z^*)$ is invertible. Let $\{B_k \}$ be a sequence of invertible matrices and suppose $\{z_k\}$, with $z_{k+1} = z_k -B_k^{-1}F(z_k)$, remains in $\mathbb {D}$ and converges to $z^*$. Then $\{z_k\}$ converges Q-superlinearly to $z^*$ and $F(z^*) = 0$ iff

$$\begin{aligned} \lim _{k\rightarrow \infty } \dfrac{\Big \Vert \big (B_k - \nabla F(z^*)\big )(z_{k+1}-z_k) \Big \Vert }{\Vert z_{k+1}-z_k\Vert } = 0 . \end{aligned}$$

Definition A.6

(Uniform linear independence) ( [48, Definition 5.1.]) A sequence of unit vectors $\{u_j\}$ in ${\mathbb {R}}^{n+m}$ is uniformly linearly independent if there is $\beta >0$, $k_0\ge 0$ and $t\ge n+m$, such that for $k\ge k_0$ and $\Vert x\Vert =1$, we have:

Theorem A.7

([48, Theorem 5.3.]) Let $\{ u_k\}$ be a sequence of unit vectors in ${\mathbb {R}}^{n+m}$. Then the following options are equivalent.

• The sequence $\{ u_k\}$ is uniformly linearly independent.

• For any ${\hat{\beta }} \in [0,1)$ there is a constant $\theta \in (0,1)$ such that if $|\beta _j-1| \le {\hat{\beta }}$ then:

  $$\begin{aligned} \left\| \prod _{j=k+1}^{k+t} \big ( I -\beta _j u_ju_j^T \big )\right\| \le \theta , ~~\textrm{for}~~k\ge k_0 \mathrm {~~and~~} t\ge n+m. \end{aligned}$$

Lemma A.8

( [48, Lemma 5.5.]) Let $\{ \phi _k\}$ and $\{ \delta _k\}$ be sequences of nonnegative numbers such that $\phi _{k+t} \le \theta \phi _k + \delta _k$ for some fixed integer $t \ge 1$ and $\theta \in (0,1)$. If $\{ \delta _k\}$ is bounded then $\{ \phi _k\}$ is also bounded, and if in addition, $\{ \delta _k\}$ converges to zero, then $\{ \phi _k\}$ converges to zero.