A unified stochastic approximation framework for learning in games

Abstract

We develop a flexible stochastic approximation framework for analyzing the long-run behavior of learning in games (both continuous and finite). The proposed analysis template incorporates a wide array of popular learning algorithms, including gradient-based methods, the exponential/multiplicative weights algorithm for learning in finite games, optimistic and bandit variants of the above, etc. In addition to providing an integrated view of these algorithms, our framework further allows us to obtain several new convergence results, both asymptotic and in finite time, in both continuous and finite games. Specifically, we provide a range of criteria for identifying classes of Nash equilibria and sets of action profiles that are attracting with high probability, and we also introduce the notion of coherence, a game-theoretic property that includes strict and sharp equilibria, and which leads to convergence in finite time. Importantly, our analysis applies to both oracle-based and bandit, payoff-based methods—that is, when players only observe their realized payoffs.

Appendices

A Regularizers and mirror maps

In this appendix we present some basic properties of the mirror map $Q$. To state them, recall first that the subdifferential of a $h$ at $x\in {\mathcal {X}}$ is defined as $\partial h( x) {:}{=} \{ y\in {\mathcal {Y}}: h(x') \ge h( x) + \left\langle y \right. ,\left. x'- x\right\rangle \; \hbox { for all}\ x'\in {\mathcal {V}} \},$ the domain of subdifferentiability of $h$ is ${{\,\textrm{dom}\,}}\partial h{:}{=} \{ x\in {{\,\textrm{dom}\,}}h: \partial h\ne \varnothing \}$, and the convex conjugate of $h$ is defined as $h^{*}( y) = \max _{ x\in {\mathcal {X}}} \{ \left\langle y \right. ,\left. x\right\rangle - h( x) \}$ for all $y\in {\mathcal {Y}}$. We then have the following basic results.

Lemma A.1

Let $h$ be a regularizer on ${\mathcal {X}}$, and let $Q:{\mathcal {Y}}\rightarrow {\mathcal {X}}$ be its induced mirror map. Then:

• 1. $Q$ is single-valued on ${\mathcal {Y}}$: in particular, for all $x\in {\mathcal {X}}$, $y\in {\mathcal {Y}}$, we have $x = Q( y) \iff y \in \partial h( x)$.

• 2. The prox-domain ${\mathcal {X}}_{h}{:}{=} {{\,\textrm{im}\,}}Q$ of $h$ satisfies ${\mathcal {X}}_{h}= {{\,\textrm{dom}\,}}\partial h$ and, hence, ${{\,\textrm{ri}\,}}{\mathcal {X}}\subseteq {\mathcal {X}}_{h}\subseteq {\mathcal {X}}$.

• 3. $Q$ is $(1/K)$-Lipschitz continuous and $Q= \nabla h^{*}$.

• 4. For all $x\in {{\,\textrm{ri}\,}}{\mathcal {X}}$, we have $y,y'\in \partial h( x)$ if and only if $\left\langle y'- y \right. ,\left. x'- x\right\rangle = 0$ for all $x'\in {\mathcal {X}}$.

Our second basic result concerns the Fenchel coupling

$$\begin{aligned} F(p, y) = h(p) + h^{*}( y) - \left\langle y \right. ,\left. p\right\rangle \quad \hbox {for} p\in {\mathcal {X}}\hbox {,} y\in {\mathcal {Y}}. \end{aligned}$$

(A.1)

For our purposes, the most relevant properties of $F$ are as follows:

Lemma A.2

For all $p\in {\mathcal {X}}$ and all $y,y'\in {\mathcal {Y}}$, we have:

$$\begin{aligned} \quad a)&\quad F(p, y) \ge 0 \quad \hbox { with equality if and only if}\ p= Q( y).&\end{aligned}$$

(A.2a)

$$\begin{aligned} \quad b)&\quad F(p, y) \ge \tfrac{1}{2} K\, \Vert Q( y) - p\Vert ^{2}.&\end{aligned}$$

(A.2b)

$$\begin{aligned} \quad c)&\quad F(p,y') \le F(p, y) + \left\langle y'- y \right. ,\left. Q( y) - p\right\rangle + \tfrac{1}{2K} \Vert y'- y\Vert _{*}^{2}. \end{aligned}$$

(A.2c)

In particular, if $h(0) = 0$, we have

$$\begin{aligned} (K/2) \Vert Q( y)\Vert ^{2}\le & {} h^{*}( y) \le -\min h+ \left\langle y \right. ,\left. Q( y)\right\rangle + (2/K) \Vert y\Vert _{*}^{2} \nonumber \\{} & {} \hbox {for all}\quad y\in {\mathcal {Y}}\end{aligned}$$

(A.3)

Variants of Lemmas A.1 and A.2 already exist in the literature (see e.g., [44] and references therein), so we do not provide a proof. Instead, we proceed below to show how the above extends to the setwise Fenchel coupling

$$\begin{aligned} F_{{\mathcal {S}}}( y) {:}{=} h^{*}( y) - h^{*}_{{\mathcal {S}}}( y) = \min _{p\in {\mathcal {S}}} \{ h(p) + h^{*}( y) - \left\langle y \right. ,\left. p\right\rangle \} = \min _{p\in {\mathcal {S}}} F(p, y)\qquad \end{aligned}$$

(A.4)

where ${\mathcal {S}}$ is a nonempty compact convex subset of ${\mathcal {X}}$ and $h^{*}_{{\mathcal {S}}}( y) = \max _{ x\in {\mathcal {S}}} \{\left\langle y \right. ,\left. x\right\rangle - h( y)\}$ denotes the convex conjugate of $h$ relative to ${\mathcal {S}}$. The most important properties of $F_{{\mathcal {S}}}$ are encoded in the following lemma.

Lemma A.3

With notation as above, we have:

• 1. $F_{{\mathcal {S}}}( y) \ge 0$ with equality if and only if $Q( y) \in {\mathcal {S}}$. Moreover, under the reciprocity condition (R), we have $F_{{\mathcal {S}}}( y) \rightarrow 0$ if and only if $Q( y) \rightarrow {\mathcal {S}}$.

• 2. $F_{{\mathcal {S}}}$ is differentiable and $\nabla F_{{\mathcal {S}}}( y) = Q( y) - Q_{{\mathcal {S}}}( y)$, where $Q_{{\mathcal {S}}}( y) = {{\,\mathrm{arg\,max}\,}}_{ x\in {\mathcal {S}}} \{ \left\langle y \right. ,\left. x\right\rangle - h( x) \}$.

• 3. For all $y,y'\in {\mathcal {Y}}$ we have $\Vert \nabla F_{{\mathcal {S}}}(y') - \nabla F_{{\mathcal {S}}}( y)\Vert \le (2/K) \Vert y'- y\Vert _{*}$.

Proof

Since ${\mathcal {S}}\subseteq {\mathcal {X}}$, we have $h^{*}_{{\mathcal {S}}} \le h^{*}$ by definition, and hence $F_{{\mathcal {S}}} \ge 0$. Moreover, since the minimum in (A.4) must be attained in ${\mathcal {S}}$, we get $F_{{\mathcal {S}}}( y) = 0$ if and only if $h(p) - h^{*}( y) - \left\langle y \right. ,\left. p\right\rangle = 0$ for some $p\in {\mathcal {S}}$; by Lemma A.2, this occurs if and only if $Q( y) = p\in {\mathcal {S}}$, so our first claim follows.

Moving forward, to show that $F_{{\mathcal {S}}}( y)\rightarrow 0$ if and only if $Q( y) \rightarrow {\mathcal {S}}$, let $y_{n}$ be a sequence in ${\mathcal {Y}}$, and let $x_{n} = Q( y_{n})$. For the “if” part, since ${\mathcal {S}}$ is compact, we may assume without loss of generality (and by descending to a subsequence if necessary) that $x_{n}$ converges to some $x\in {\mathcal {S}}$. Observe now that (a) $0 \le F_{{\mathcal {S}}}( y_{n}) \le F( x, y_{n})$ by the minimum (A.4); and (b) $F( x, y_{n})\rightarrow 0$ by (R). Thus, by sandwiching, we conclude that $F_{{\mathcal {S}}}( y_{n})\rightarrow 0$. Conversely, if $F_{{\mathcal {S}}}( y_{n}) \rightarrow 0$, we may again assume by compactness (and by descending to a subsequence if necessary) that $x_{n} = Q( y_{n})$ converges to some ${\hat{ x\in {\mathcal {X}}}}$. If ${\hat{ x\not \in {\mathcal {S}}}}$, then, by (R), we have $\lim _{n\rightarrow \infty } F( x, y_{n}) > 0$ for all $x\in {\mathcal {S}}$. Since ${\mathcal {S}}$ is compact and $F( x, y)$ is continuous in x, we conclude that $\liminf _{n\rightarrow \infty } F_{{\mathcal {S}}}( y_{n}) > 0$, a contradiction which establishes our claim.

Our last two claims follow by applying Lemma A.1 to $h$ and $h+\delta _{{\mathcal {S}}}$ where $\delta _{{\mathcal {S}}}$ denotes the convex indicator of ${\mathcal {S}}$. $\square$

The next properties we discuss concern the way that different regions of ${\mathcal {Y}}$ are mapped to ${\mathcal {X}}$ under $Q$.

Lemma A.4

[43, Prop. A.1]. Let $h$ be a regularizer on the simplex ${{\,\mathrm{{{\,\mathrm{\Delta }\,}}}\,}}(\mathcal {A}) \subseteq \mathbb {R}^{\mathcal {A}}$. If $y_{\alpha } - y_{\beta } \rightarrow -\infty$, then $Q_{\alpha }( y) \rightarrow 0$.

Lemma A.5

Let $h$ be a regularizer on ${\mathcal {X}}$, let $y_{n}$, $n=1,2,\dotsc$ be a sequence in ${\mathcal {Y}}$, and fix some $x\in {\mathcal {X}}$. If $\left\langle y_{n} \right. ,\left. z\right\rangle \rightarrow -\infty$ for every nonzero $z\in {{\,\textrm{TC}\,}}( x)$, we have $Q( y_{n}) \rightarrow x$.

Proof

Assume that $\limsup _{n} \Vert x_{n} - x\Vert > 0$. Then, given that $y_{n} \in \partial h( x_{n})$, we get $h( x) \ge h( x_{n}) + \left\langle y_{n} \right. ,\left. x - x_{n}\right\rangle \ge h( x_{n}) - \left\langle y_{n} \right. ,\left. z_{n}\right\rangle \Vert x_{n} - x\Vert ,$ where we set $z_{n} = ( x_{n} - x) / \Vert x_{n} - x\Vert$. If we further assume (by descending to a subsequence if needed) that $z_{n}$ converges in the unit sphere of $\Vert \cdot \Vert$, there exists some $z\in {{\,\textrm{TC}\,}}( x)$ with $\Vert z\Vert = 1$ and such that $\left\langle y_{n} \right. ,\left. z_{n}\right\rangle \le (1+\varepsilon ) \left\langle y_{n} \right. ,\left. z\right\rangle$ for some $\varepsilon >0$. Thus, taking the $\limsup$ of the above estimate gives $h( x) \ge \infty$, a contradiction which proves our claim. $\square$

Lemma A.6

Let $h$ be a regularizer on a convex polytope ${\mathcal {P}}$ of ${\mathcal {V}}$, let ${\mathcal {S}}$ be a face of ${\mathcal {P}}$, and let ${\mathcal {Z}}= \{ z_{1},\dotsc ,z_{m} \}$ be a set of unit vectors of ${\mathcal {V}}$ such that every point $x\in {\mathcal {P}}\setminus {\mathcal {S}}$ can be written as $x = p+ \lambda z$ for some $p\in {\mathcal {S}}$, $z\in {\mathcal {Z}}$ and $\lambda >0$. If $\max _{z\in {\mathcal {Z}}} \left\langle y \right. ,\left. z\right\rangle \rightarrow -\infty$, then $Q( y) \rightarrow {\mathcal {S}}$.

Proof

By the compactness of ${\mathcal {P}}$ (and descending to a subsequence if necessary), we may assume that $x_{n} = Q( y_{n})$ converges to some $x\in {\mathcal {P}}$. If $x\notin {\mathcal {S}}$, there exist $p\in {\mathcal {S}}$, $z\in {\mathcal {Z}}$ and $\lambda >0$ such that $x = p+ \lambda z$. In turn, this gives $h(p) \ge h( x_{n}) + \left\langle y_{n} \right. ,\left. p- x_{n}\right\rangle = h( x_{n}) - \left\langle y_{n} \right. ,\left. z_{n}\right\rangle \Vert x_{n} - p\Vert$ where we set $z_{n} = ( x_{n} - p)/\Vert x_{n} - p\Vert$. Since $z_{n} \rightarrow z$, taking $n\rightarrow \infty$ yields $h(p) \ge \infty$, a contradiction which shows that $x = \lim x_{n} \in {\mathcal {S}}$, as claimed. $\square$

We conclude this appendix with the dynamical properties of the Fenchel coupling under (MD). The heavy lifting will be provided by the following simple lemma:

Lemma A.7

Let $x_{}(t) = Q(y_{}(t))$ be an orbit of (MD). Then, for every nonempty closed convex subset ${\mathcal {S}}$ of ${\mathcal {X}}$, we have

$$\begin{aligned} {\dot{F}}_{{\mathcal {S}}}( y) = \left\langle v( x) \right. ,\left. x - x_{{\mathcal {S}}}\right\rangle \end{aligned}$$

(A.5)

where $x_{{\mathcal {S}}} = Q_{{\mathcal {S}}}( y)$ denotes the mirror image of y on ${\mathcal {S}}$. In particular, if ${\mathcal {S}}= \{p\}$, we have ${\dot{F}}(p, y) = \left\langle v( x) \right. ,\left. x - p\right\rangle$.

Proof

Simply note that ${\dot{ y}} = -v( x)$ and apply Lemma A.3. $\square$

B Omitted proofs and calculations

1.1 B.1 Error bounds for specific algorithms

Our aim in this appendix is to prove the bounds on the bias and magnitude of ${\hat{v}}_{n}$ reported in Proposition 3 and Table 1.

Proof of Proposition 3

We proceed in a method-by-method basis starting with the oracle-based methods of Sect. 3.2, that is, Algorithms 1–4. For this, we will make free use of the fact that we can take $M_{n}^{q} = 3^{q-1} (G^{q} + B_{n}^{q} + \sigma _{n}^{q})$ in (8), cf.  the discussion after (6).

$\blacktriangleright$  Algorithm 1: Stochastic gradient ascent  For (SGA), we have $U_{n} = {{\,\textrm{err}\,}}(X_{n};\theta _{n})$ and $b_{n} = 0$, so our claim follows immediately from the stated assumptions for (SFO).

$\blacktriangleright$  Algorithm 2: Extra-gradient  For (EG), we have ${\hat{v}}_{n} = V(X_{n+1/2};\theta _{n+1/2})$ so . We thus get

(B.1)

and, analogously

(B.2)

so under (13), as claimed.

$\blacktriangleright$  Algorithm 3: Optimistic gradient  For (OG), we have again , so the same series of arguments as above gives

(B.3)

under (13) with $q=\infty$. The noise term $U_{n}$ can be bounded in exactly the same way, so we omit the calculations.

$\blacktriangleright$  Algorithm 4: Exponential/multiplicative weights  We consider two cases, based on the information available to the players. For the full information oracle (14a), we have ${\hat{v}}_{n} = v(X_{n})$ so $b_{n} = U_{n} = 0$ by definition (i.e.,  the oracle is perfect). Otherwise, under the realization-based oracle (14b), we have because $\alpha _{n}$ is sampled according to $X_{n}$. We thus get $b_{n} = 0$ and $U_{n} = {{\,\mathrm{{\mathcal {O}}}\,}}(1)$, which proves our assertion.

We now proceed with the payoff-based methods of Sect. 3.2, namely Algorithms 5–7.

$\blacktriangleright$  Algorithm 5: Single-point stochastic approximation  Since $u_{i}$ is assumed bounded in the context of (SPSA), the bound for $M_{n}$ follows trivially. As for the bias of (SPSA), it will be convenient to set $V_{i}^{\delta }( x;w) = (d_{i}/\delta ) \, u_{i}( x+\delta w) \, w_{i}$ so, in obvious notation, ${\hat{v}}_{i,n} = V_{i}^{\delta _{n}}(X_{n};W_{n})$. Thus, if we fix a pivot point $x\in {\mathcal {X}}$ and a query point $\hat{x}= x+ \delta w$ for some $w\in {\mathcal {E}}= \prod _{i} {\mathcal {E}}_{i}$, a first-order Taylor expansion of $u_{i}$ with integral remainder gives

$$\begin{aligned} V_{i}^{\delta }( x;w)&= \frac{d_{i}}{\delta } u_{i}(\hat{x}) \cdot w_{i} = \frac{d_{i}}{\delta } u_{i}( x) \cdot w_{i} + \frac{d_{i}}{\delta } \left\langle \nabla u_{i}( x) \right. ,\left. z\right\rangle \cdot w_{i} \end{aligned}$$

(B.4a)

$$\begin{aligned}&+ \int _{0}^{1} \left\langle \nabla u_{i}( x+\tau z) - \nabla u_{i}( x) \right. ,\left. z\right\rangle \,d\tau \cdot w_{i} \end{aligned}$$

(B.4b)

where we set $z= \hat{x}- x= \delta w$. Hence, if $w$ is drawn uniformly at random from ${\mathcal {E}}$, taking expectations yields

(B.5)

where we used the fact that for all $i\in \mathcal {N}$ and that $w_{i}$ and $w_{j}$ are independent for all $i,j\in \mathcal {N}$, $i\ne j$. As for the second term, Assumption 1 readily yields

(B.6)

Thus, by combining (B.5) and (B.6), we conclude that , which immediately yields the desired bound $B_{n} = {{\,\mathrm{{\mathcal {O}}}\,}}(\delta _{n}) = {{\,\mathrm{{\mathcal {O}}}\,}}(1/n^{\ell _{\delta }})$ for (SPSA).

$\blacktriangleright$  Algorithm 6: Dampened gradient approximation  Recall that ${\hat{v}}_{i,n} = n\cdot \log ( 1 + (u_{i}(X_{n+1/2}) - u_{i}(X_{n})) W_{i,n} )$. Since $u_{i}(X_{n+1/2}) - u_{i}(X_{n}) = (1/n) v_{i}(X_{n}) W_{i,n} + {{\,\mathrm{{\mathcal {O}}}\,}}(1/n^{2})$ by the definition of $X_{n+1/2}$, expanding the logairthm readily yiels $B_{n} = {{\,\mathrm{{\mathcal {O}}}\,}}(1/n)$ and $M_{n} = {{\,\mathrm{{\mathcal {O}}}\,}}(1)$. Our claim then follows as above.

$\blacktriangleright$  Algorithm 7: Exponential weights for exploration and exploitation  Since $\hat{\alpha }_{n}$ is sampled according to $\hat{X}_{n}$, we readily get , so $B_{n} = {{\,\mathrm{{\mathcal {O}}}\,}}(\Vert \hat{X}_{n} - X_{n}\Vert ) = {{\,\mathrm{{\mathcal {O}}}\,}}(\delta _{n}) = {{\,\mathrm{{\mathcal {O}}}\,}}(1/n^{\ell _{\delta }})$. Moreover, since $\hat{X}_{i\alpha _{i},n} \ge \delta _{n}/A_{i}$, it follows that $\Vert {\hat{v}}_{n}\Vert _{*} = {{\,\mathrm{{\mathcal {O}}}\,}}(1/\delta _{n}) = {{\,\mathrm{{\mathcal {O}}}\,}}(n^{\ell _{\delta }})$, and our proof is complete. $\square$

1.2 B.2 Energy function derivations

Our aim in this last appendix is to prove the energy properties of the Fenchel coupling as stated in Lemmas 1 and 2. For concision, we will prove both as a special case of the following general result:

Proposition B.12

Let ${\mathcal {S}}$ be a nonempty compact convex subset of ${\mathcal {X}}$, and assume that there exists a neighborhood ${\mathcal {U}}$ of ${\mathcal {S}}$ such that

$$\begin{aligned} \left\langle v( x) \right. ,\left. x - p\right\rangle \le 0 \quad \hbox {for all}\quad x\in {\mathcal {U}},\quad p\in {\mathcal {S}}, \end{aligned}$$

(B.7)

with equality if and only if $x\in {\mathcal {S}}$. If (R) holds and $\varphi$ is defined as in (31), the function $E:{\mathcal {Y}}\rightarrow \mathbb {R}$ given by

$$\begin{aligned} E( y) = \varphi (F_{{\mathcal {S}}}( y)) \quad \hbox {for all}\quad y\in {\mathcal {Y}}\end{aligned}$$

(B.8)

is a local energy function for ${\mathcal {S}}$ under (MD). In addition, if ${\mathcal {U}}= {\mathcal {X}}$, $E$ is a global energy function for ${\mathcal {S}}$.

Proof

We will verify the requirements of Definition 5 in order.

• 1. For the first (Lipschitz continuity and smoothness), note that $\nabla E( y) = \varphi '(F_{{\mathcal {S}}}( y)) \nabla F_{{\mathcal {S}}}( y)$, so, lettting $x = Q( y)$ and $x_{{\mathcal {S}}} = Q_{{\mathcal {S}}}( y)$, Lemma A.3 yields

  $$\begin{aligned} \nabla E( y) = ( x - x_{{\mathcal {S}}}) \cdot {\left\{ \begin{array}{ll} 1 &{}\text {if}\quad F_{{\mathcal {S}}}( y) \le 1,\\ 1/\sqrt{F_{{\mathcal {S}}}( y)} &{}\text {otherwise.} \end{array}\right. } \end{aligned}$$

  (B.9)

  Furthermore, by Lemma A.2, we also have $F(p, y) \ge (K/2) \Vert x - p\Vert ^{2}$ for all $p\in {\mathcal {S}}$, so, by minimizing over $p\in {\mathcal {S}}$, we get

  $$\begin{aligned} F_{{\mathcal {S}}}( y) \ge (K/2) {{\,\textrm{dist}\,}}( x,{\mathcal {S}})^{2} = (K/2) \Vert x - {{\,\textrm{pr}\,}}_{{\mathcal {S}}}( x)\Vert ^{2} \end{aligned}$$

  (B.10)

  where ${{\,\textrm{pr}\,}}_{{\mathcal {S}}}( x) {:}{=} {{\,\mathrm{arg\,min}\,}}_{p\in {\mathcal {S}}} \Vert x - p\Vert$. In turn, this gives $\Vert x - {{\,\textrm{pr}\,}}_{{\mathcal {S}}}( x)\Vert \le \sqrt{2/K}$ whenever $F_{{\mathcal {S}}}( y) \le 1$, so we get

  $$\begin{aligned} \Vert \nabla E( y)\Vert= & {} \Vert x - x_{{\mathcal {S}}}\Vert \le \Vert x - {{\,\textrm{pr}\,}}_{{\mathcal {S}}}( x)\Vert + \Vert {{\,\textrm{pr}\,}}_{{\mathcal {S}}}( x) - x_{{\mathcal {S}}}\Vert \nonumber \\\le & {} \sqrt{2/K} + {{\,\textrm{diam}\,}}({\mathcal {S}}) \end{aligned}$$

  (B.11)

  whenever $F_{{\mathcal {S}}}( y) \le 1$. On the other hand, if $F_{{\mathcal {S}}}( y) \ge 1$, we have

  

  (B.12)

  By the reciprocity condition (R), it follows that the set $\{ x = Q( y): F_{{\mathcal {S}}}( y) \ge 1 \}$ is well-separated from ${\mathcal {S}}$, so $\Vert x - {{\,\textrm{pr}\,}}_{{\mathcal {S}}}( x)\Vert$ is bounded away from zero if $F_{{\mathcal {S}}}( y) \ge 1$. Thus, by combining Eqs. (B.11) and (B.12), we conclude that $\Vert \nabla E( y)\Vert$ is bounded. Finally, again by Lemma A.2, $F( y)$ is $(1/K)$-Lipschitz smooth, so Eq. 35—which is equivalent to the Lipschitz smoothness of $E$—follows immediately from the concavity of $\varphi$.

• 2. For the positive-definiteness requirement of Definition 5, note that Lemma A.3 and the reciprocity condition (R) yield $Q( y) \rightarrow {\mathcal {S}}$ if and only if $F_{{\mathcal {S}}}( y) \rightarrow 0$. Thus, given that $\varphi (z) = z$ for small z, the same will hold for $E= \varphi \circ F_{{\mathcal {S}}}$, and our claim follows.

• 3. Finally, for the Lyapunov properties of $E$ under (MD), recall that Lemma A.7 gives ${\dot{F}}_{{\mathcal {S}}}( y) = \left\langle v( x) \right. ,\left. x - x_{{\mathcal {S}}}\right\rangle$, so

  $$\begin{aligned} {\dot{E}}( y)= & {} \left\langle {\dot{ y}} \right. ,\left. \nabla E( y)\right\rangle = \varphi '(F( y)) \left\langle v( x) \right. ,\left. x - x_{{\mathcal {S}}}\right\rangle < 0 \nonumber \\{} & {} \hbox {whenever}\quad x \in {\mathcal {U}}\setminus {\mathcal {S}}\end{aligned}$$

  (B.13)

  where we used the defining property (B.7) of ${\mathcal {S}}$ (recall that $x_{{\mathcal {S}}} \in {\mathcal {S}}$ by construction). Moving forward, by Lemma A.3, there exists some $E_{+}>0$ such that the sublevel set ${\mathcal {D}}= \{ y\in {\mathcal {Y}}: F_{{\mathcal {S}}}( y) \le E_{+} \}$ is mapped to ${\mathcal {U}}$ under $Q$, i.e.,  $Q( y) \in {\mathcal {U}}$ whenever $F_{{\mathcal {S}}}( y) \le E_{+}$. Thus, putting everythrpf ing together, we conclude that ${\dot{E}}( y)\rightarrow 0$ if and only if $F_{{\mathcal {S}}}( y)\rightarrow 0$, which implies that $\sup \{ {\dot{E}}( y): E_{-}< E( y)< E_{+} \} < 0$ for all $E_{-}\in (0,E_{+})$, and our proof is complete.

$\square$