On Gradient Like Properties of Population Games, Learning Models and Self Reinforced Processes

Abstract

We consider ordinary differential equations on the unit simplex of $\mathbb{R}^{n}$ that naturally occur in population games, models of learning and self reinforced random processes. Generalizing and relying on an idea introduced in Dupuis and Fisher (On the construction of Lyapunov functions for nonlinear Markov processes via relative entropy, 2011), we provide conditions ensuring that these dynamics are gradient like and satisfy a suitable “angle condition”. This is used to prove that omega limit sets and chain transitive sets (under certain smoothness assumptions) consist of equilibria; and that, in the real analytic case, every trajectory converges toward an equilibrium. In the reversible case, the dynamics are shown to be C ¹ close to a gradient vector field. Properties of equilibria -with a special emphasis on potential games—and structural stability questions are also considered.

Appendix

Let L be an irreducible rate matrix and $\pi \in \dot{\varDelta }$ denote the invariant probability of L. That is the unique solution (in Δ) of π L = 0. For all $f,g \in \mathbb{R}^{n}$ we let

$$\displaystyle{\langle \,f,g\rangle =\sum _{i}f_{i}g_{i},\,\langle \,f,g\rangle _{\pi } =\sum _{i}f_{i}g_{i}\pi _{i}\mbox{ and }\langle \,f,g\rangle _{1/\pi } =\sum _{i}f_{i}g_{i}\frac{1} {\pi _{i}}.}$$

The Dirichlet form of L is the map $\mathcal{E}: \mathbb{R}^{n}\mapsto \mathbb{R}_{+}$ defined as

$$\displaystyle{\mathcal{E}(\,f) = -\langle \,f,Lf\rangle _{\pi } = \frac{1} {2}\sum _{i,j}(\,f_{i} - f_{j})^{2}L_{ ij}\pi _{i}.}$$

By irreducibility, $\mathcal{E}(\,f)> 0$ unless f is constant, and the spectral gap

$$\displaystyle{\lambda =\sup \{ \mathcal{E}(\,f):\:\langle \, f,1\rangle _{\pi } = 0,\langle \,f,f\rangle _{\pi } = 1\}}$$

is positive. We let L ^∗ be the irreducible rate matrix defined by

$$\displaystyle{L_{ij}^{{\ast}} = \frac{\pi _{j}L_{ji}} {\pi _{i}}.}$$

Note that L ^∗ admits π as invariant probability and that L ^∗ is the adjoint of L for $\langle,\rangle _{\pi }.$

We let $L^{T}: T\varDelta \mapsto T\varDelta$ be defined by

$$\displaystyle{L^{T}h = hL.}$$

Finally recall that for all $f \in \mathbb{R}^{n}$ $\frac{f} {\pi }$ stands for the vector defined by $(\frac{f} {\pi } )_{i} = \frac{f_{i}} {\pi _{i}},i = 1\ldots n.$

Lemma 8

For all u,v ∈ TΔ

$$\displaystyle{\langle L^{T}u,v\rangle _{ 1/\pi } =\langle L^{{\ast}}(\frac{u} {\pi } ), \frac{v} {\pi } )\rangle _{\pi }}$$

In particular L ^T is invertible and L ^T is a definite negative operator for $\langle,\rangle _{\frac{1} {\pi } }$ whenever L is reversible with respect to π.

Proof

The first assertion follows from elementary algebra. For the second, note that $\langle L^{T}u,u\rangle _{1/\pi } = -\mathcal{E}(\frac{u} {\pi } ).$ Thus, by irreducibility,

$$\displaystyle{\langle L^{T}u,u\rangle _{ 1/\pi } <0}$$

unless u = 0.   ■ 

1.1 Proof of Lemma 4

Given $f \in \mathbb{R}^{n}$ we write f ≥ 0 if f _i  ≥ 0 for all i. We let $1 \in \mathbb{R}^{n}$ denote the vector which components are all equal to 1. For all t ≥ 0 we let $P_{t} = e^{tL}.$ Since L is a rate matrix, (P _t ) is a Markov semigroup meaning that P _t f ≥ 0 for all $f \in \mathbb{R}^{n}$ with f ≥ 0 and P _t 1 = 1. 

Lemma 9

Let $I \subset \mathbb{R}$ be an open interval and $S: I\mapsto \mathbb{R}$ aC ² function such that $S^{{\prime\prime}}(t) \geq \alpha> 0.$ Let $f \in \mathbb{R}^{n}$ be such that f _i ∈ I for all i. Then

$$\displaystyle{ \frac{d} {dt}\langle S(P_{t}\,f),1\rangle _{\pi }\vert _{t=0} \leq -\alpha \mathcal{E}(\,f).}$$

Proof

For all u, v ∈ I $S(v) - S(u) - S^{{\prime}}(u)(v - u) \geq \alpha /2(v - u)^{2}.$ Hence for all i, j

$$\displaystyle{S(\,f_{j}) - S((P_{t}\,f)_{i}) - S^{{\prime}}((P_{ t}\,f)_{i})(\,f_{j} - (P_{t}\,f)_{i}) \geq \alpha /2(\,f_{j} - (P_{t}\,f)_{i})^{2}.}$$

Applying P _t to this inequality gives

$$\displaystyle{P_{t}(Sf)_{i} - S((P_{t}\,f)_{i}) \geq \alpha /2P_{t}(\,f_{i} - (P_{t}\,f)_{i})^{2}) =\alpha /2(P_{ t}\,f_{i}^{2} - (P_{ t}\,f)_{i}^{2})}$$

Hence

$$\displaystyle{P_{t}(Sf) - S((P_{t}\,f)) \geq \alpha /2P_{t}(\,f - (P_{t}\,f))^{2}) =\alpha /2(P_{ t}\,f^{2} - (P_{ t}\,f)^{2}).}$$

Therefore, using the fact that $\langle P_{t}g,1\rangle _{\pi } =\langle g,1\rangle _{\pi }$ leads to

$$\displaystyle{\langle Sf - S(P_{t}\,f),1\rangle _{\pi } \geq \alpha \langle \, f^{2} - (P_{ t}\,f)^{2},1\rangle _{\pi }.}$$

Dividing by t and letting $t \rightarrow 0$ leads to the desired inequality.  ■ 

Let $S:]0,\infty [\mapsto \mathbb{R}$ be a C ² function with positive second derivative. Let $H_{\pi }^{S}:\varDelta \mapsto \mathbb{R}$ be the map defined by

$$\displaystyle{H_{\pi }^{S}(x) =\sum _{ i}\pi _{i}S(\frac{x_{i}} {\pi _{i}} ).}$$

Corollary 4

For all x ∈Δ

$$\displaystyle{\langle \nabla H_{\pi }^{S}(x),xL\rangle \leq -\alpha \lambda V ar_{\pi }(\,f)}$$

where $f_{i} = \frac{x_{i}} {\pi _{i}}$

Proof

For x ∈ Δ let x(t) = xe ^tL, $f_{i} = \frac{x_{i}} {\pi _{i}},\,f_{i}(t) = \frac{x_{i}(t)} {\pi _{i}}$ and $P_{t}^{{\ast}}g = e^{tL^{{\ast}} }g.$ Note that $P_{t}^{{\ast}}$) is the adjoint of P _t with respect to $\langle,\rangle _{\pi }.$

For all $g \in \mathbb{R}^{n},\langle x(t),g\rangle =\langle x,P_{t}g\rangle =\langle \, f,P_{t}g\rangle _{\pi } =\langle P_{t}^{{\ast}}f,g\rangle _{\pi }$ so that $f(t) = P_{t}^{{\ast}}f.$ Hence by the preceding lemma applied to L ^∗ it follows that

$$\displaystyle{\langle \nabla H_{\pi }^{S}(x),xL\rangle = \frac{d} {dt}\langle S(P_{t}^{{\ast}}f),1\rangle _{\pi }\vert _{ t=0} \leq -\alpha \mathcal{E}(\,f) \leq -\alpha \lambda V ar_{\pi }(\,f)}$$

where $\alpha =\min _{i}S^{{\prime\prime}}(\frac{x_{i}} {\pi _{i}} )> 0.$  ■ 

We now prove the Lemma. Set $S(t) =\int _{ 1}^{t}s(u)du.$ Then for all u ∈ T Δ

$$\displaystyle{\langle \nabla H_{\pi }^{S}(x),u\rangle =\sum _{ i}u_{i}s(\frac{x_{i}} {\pi _{i}} )}$$

and the results follows from Corollary 4.