Convergence results on stochastic adaptive learning

Abstract

We investigate an adaptive learning model which nests several existing learning models such as payoff assessment learning, valuation learning, stochastic fictitious play learning, experience-weighted attraction learning and delta learning with foregone payoff information in normal form games. In particular, we consider adaptive players each of whom assigns payoff assessments to his own actions, chooses the action which has the highest assessment with some perturbations and updates the assessments using observed payoffs, which may include payoffs from unchosen actions. Then, we provide conditions under which the learning process converges to a quantal response equilibrium in normal form games.

Appendices

Appendices

1.1 The Proof of Proposition 4

The following argument is an extension of the proof of Proposition 5 in Cominetti et al. (2010) when players observe foregone payoff information and there exists a discount factor for the foregone payoff information. In the argument, we provide a condition under which F is a contraction mapping.

Now for Q, $Q'$, $x=(C_{i,s_{i}}(Q_{i}))_{i,s_{i}}$ and $x'=(C_{i,s_{i}}(Q'_{i}))_{i,s_{i}}$, let i and $s_{i}$ be such that

$$\begin{aligned} ||F(Q) - F(Q')||_{\infty } = |\overline{\pi }_{i}(s_{i}, x_{-i}) - \overline{\pi }_{i}(s_{i}, x'_{-i}) |. \end{aligned}$$

(A.1)

Consider a telescopic series $y_{0},\ldots ,y_{N}$ such that $y_{N}= x$, $y_{0}=x'$ and

$$\begin{aligned} y_{n}= & {} (x_{1},\ldots ,x_{n},x'_{n+1},\ldots ,x'_{N}) \end{aligned}$$

for $n \notin \{0,N\}$. Then, equation (A.1) is expressed as follows:

$$\begin{aligned} |\overline{\pi }_{i}(s_{i}, x_{-i}) - \overline{\pi }_{i}(s_{i}, x'_{-i})| = |\sum ^{N}_{l = 1}(\overline{\pi }_{i}(s_{i}, y_{l,-i}) - \overline{\pi }_{i}(s_{i}, y_{l-1,-i}))|. \end{aligned}$$

Now, the summand for $l = i$ is expressed as follows:

$$\begin{aligned} | \overline{\pi }_{i}(s_{i}, y_{l,-i}) - \overline{\pi }_{i}(s_{i}, y_{l-1,-i}) | = | (1-\delta )\pi _{i}(s_{i},y_{l,-i})(x_{i,s_{i}}-x'_{i,s_{i}})|, \end{aligned}$$

whereas the summand for $l \ne i$ is expressed as follows:

$$\begin{aligned} |\overline{\pi }_{i}(s_{i}, y_{l,-i}) - \overline{\pi }_{i}(s_{i}, y_{l-1,-i}) | \le&| \big ( y_{l,i,s_{i}} + (1-y_{l,i,s_{i}}) \delta \big )| |\pi _{i}(s_{i}, y_{l,-i}) - \pi _{i}(s_{i}, y_{l-1,-i}) | \\ \le&|\pi _{i}(s_{i}, y_{l,-i}) - \pi _{i}(s_{i}, y_{l-1,-i}) | \\ =&|\sum _{s_{l} \in S_{l}} \pi _{i}(s_{i}, s_{l}, y_{l, -(i,l)}) (x_{l,s_{l}} - x'_{l, s_{l}}) | \\ =&|\sum _{s_{l} \in S_{l}} \big ( \pi _{i}(s_{i}, s_{l}, y_{l, -(i,l)}) - \pi _{i}(s_{i}, \overline{s}_{l}, y_{l,-(i,l)})\big ) (x_{l,s_{l}} - x'_{l, s_{l}}) | \end{aligned}$$

where $\overline{s}_{l}$ is fixed. Note that

$$\begin{aligned} |x_{l, s_{l}} - x'_{l,s_{l}}| \le&\sum _{s \in S_{l}}| \frac{\partial }{\partial Q_{s}}C_{l,s_{l}}(Q^{*}_{l})(Q_{l,s} - Q'_{l,s})| \\ \le&|| Q - Q'||_{\infty } \sum _{s \in S_{l}}| \frac{\partial }{\partial Q_{s}}C_{l,s_{l}}(Q^{*}_{l})| \\ =&|| Q - Q'||_{\infty } 2 \frac{1}{\tau _{l}}C_{l,s_{l}} (Q^{*}_{l})(1-C_{l,s_{l}} (Q^{*}_{l})) \end{aligned}$$

for some $Q^{*}_{l}$. For the last equality, we use the result of Lemma 1, which we show later. Therefore, for the summand of $l =i$, we have

$$\begin{aligned} | \overline{\pi }_{i}(s_{i}, y_{l,-i}) - \overline{\pi }_{i}(s_{i}, y_{l-1,-i}) |= & {} |(1-\delta )\pi _{i}(s_{i},y_{l,-i}) (x_{i,s_{i}}-x'_{i,s_{i}})| \\\le & {} 2K || Q - Q'||_{\infty } \frac{1}{\tau _{i}} \end{aligned}$$

where $K := \frac{(1-\delta )}{4}\max _{i,s_{i},s_{-i}} |\pi _{i}(s_{i},s_{-i})|$, whereas for the summand of $l \ne i$, we have

$$\begin{aligned} |\overline{\pi }_{i}(s_{i}, y_{l,-i}) - \overline{\pi }_{i}(s_{i}, y_{l-1,-i}) | \le&|\sum _{s_{l} \in S_{l}} \big ( \pi _{i}(s_{i}, s_{l}, y_{l,-(i,l)}) - \pi _{i}(s_{i}, \overline{s}_{l}, y_{l, -(i,l)}) \big ) (x_{l,s_{l}} - x'_{l, s_{l}}) | \\ \le&\sum _{s_{l} \in S_{l}} | \big ( \pi _{i}(s_{i}, s_{l}, y_{l,-(i,l)}) - \pi _{i}(s_{i}, \overline{s}_{l}, y_{l,-(i,l)}) \big ) (x_{l,s_{l}} - x'_{l, s_{l}}) | \\ \le&\sum _{s_{l} \in S_{l}} \theta | x_{l,s_{l}} - x'_{l, s_{l}} | \\ \le&\sum _{s_{l} \in S_{l}} \theta || Q - Q'||_{\infty } 2 \frac{1}{\tau _{l}}C_{l,s_{l}} (Q^{*}_{l}) \\ =\,&2 \theta || Q - Q'||_{\infty } \frac{1}{\tau _{l}} \end{aligned}$$

where $\theta := \max _{i,s_{i},s_{j},s'_{j}} | \pi _{i}(s_{i},s_{j},s_{-(i,j)}) - \pi _{i}(s_{i},s'_{j},s_{-(i,j)}) |$. Therefore,

$$\begin{aligned} ||F(Q) - F(Q')||_{\infty } \le&\sum _{l \ne i} 2 \theta || Q - Q'||_{\infty } \frac{1}{\tau _{l}} + 2K || Q - Q'||_{\infty } \frac{1}{\tau _{i}}\\ \le&2 \theta ' \beta || Q - Q'||_{\infty } \end{aligned}$$

where $\theta ':=\max \{ \theta , K \}$ and $\beta := \sum _{l \in {\mathcal {N}}} \frac{1}{\tau _{l}}$. Hence, if $(\tau _{l})$ is large enough, that is, $2 \theta ' \beta <1$, F is a contraction mapping. $\square$

Lemma 1

For the logit choice rule, we have

$$\begin{aligned} \frac{\partial C_{i,s_{i}}}{\partial Q_{i,t_{i}}}(Q_{i}) = \frac{1}{\tau _{i}} C_{i,s_{i}}(Q_{i}) \big ( {\mathbb {1}}\{ t_{i}=s_{i}\} - C_{i,t_{i}}(Q_{i}) \big ) \end{aligned}$$

and

$$\begin{aligned} \sum _{t \in S_{i}}\left| \frac{\partial C_{i,s_{i}}}{\partial Q_{i,t_{i}}}(Q_{i})\right| = 2 \frac{1}{\tau _{i}} C_{i,s_{i}}(Q_{i}) (1- C_{i,s_{i}}(Q_{i}) ) \end{aligned}$$

for each i, $Q_{i}$, $s_{i}$ and $t_{i}$.

Proof

$$\begin{aligned} \frac{\partial C_{i,s_{i}}}{\partial Q_{i,t_{i}}}(Q_{i})=&\frac{\partial }{\partial Q_{i,t_{i}}} \frac{\exp \left( \frac{1}{\tau _{i}}Q_{i,s_{i}} \right) }{\sum _{s'_{i} } \exp \left( \frac{1}{\tau _{i}}Q_{i,s'_{i}}\right) } \\ =&\frac{{\mathbb {1}}\{ t_{i}=s_{i}\}\frac{1}{\tau _{i}}\exp \left( \frac{1}{\tau _{i}}Q_{s_{i}}\right) \sum _{s'_{i} } \exp \left( \frac{1}{\tau _{i}}Q_{s'_{i}}\right) - \frac{1}{\tau _{i}}\exp \left( \frac{1}{\tau _{i}}Q_{s_{i}}\right) \exp \left( \frac{1}{\tau _{i}}Q_{t_{i}}\right) }{\left( \sum _{s'_{i} } \exp \left( \frac{1}{\tau _{i}}Q_{s'_{i}}\right) \right) ^{2}} \\ =&{\mathbb {1}}\{ t_{i}=s_{i}\} \frac{1}{\tau _{i}} C_{i,s_{i}}(Q_{i}) - \frac{1}{\tau _{i}} C_{i,s_{i}}(Q_{i}) C_{i,t_{i}}(Q_{i}) \\ =&\frac{1}{\tau _{i}} C_{i,s_{i}}(Q_{i}) \left( {\mathbb {1}}\{ t_{i}=s_{i}\} - C_{i,t_{i}}(Q_{i}) \right) \end{aligned}$$

and

$$\begin{aligned} \sum _{t_{i} \in S_{i}}\left| \frac{\partial C_{i,s_{i}}}{\partial Q_{i,t_{i}}}(Q_{i})\right| =&\sum _{t_{i} \in S_{i}} \left| \frac{1}{\tau _{i}} C_{i,s_{i}}(Q_{i}) \big ( {\mathbb {1}}\{ t_{i}=s_{i}\} - C_{i,t_{i}}(Q_{i}) \big )\right| \\ =&\frac{1}{\tau _{i}} C_{i,s_{i}}(Q_{i}) \sum _{t_{i} \in S_{i}} | {\mathbb {1}}\{ t_{i}=s_{i}\} - C_{i,t_{i}}(Q_{i}) | \\ =&2 \frac{1}{\tau _{i}} C_{i,s_{i}}(Q_{i}) (1- C_{i,s_{i}}(Q_{i}) ). \end{aligned}$$

$\square$

1.2 The Proof of Proposition 6

As we focus on a symmetric $2 \times 2$ game, for actions, the set of actions and the payoff function, we omit the subscript referring to players. Since only two actions are available and the weighting parameters of the actions are equivalent for each player, it is enough for the following convergence analysis to focus on the assessment difference. In the following argument, without confusion, let $Q_{n,i}$ denote the difference of player i’s assessments in period n: for $s, t \in S$ and $s \ne t$,

$$\begin{aligned} Q_{n,i}:=Q_{n, i,t} -Q_{n,i,s}. \end{aligned}$$

Let $Q_{n}=(Q_{n,i}, Q_{n,-i})$ be the assessment difference profile. Note that each player’s choice rule can be expressed as a function of the difference: let $C_{i}: {\mathbb {R}} \rightarrow {\mathbb {R}}$ be the choice rule of player i such that

$$\begin{aligned} C_{i}(Q_{n,i}) := \frac{1}{1+ \exp \left( \frac{1}{\tau _{i}}Q_{n,i}\right) } \end{aligned}$$

and $x_{n,i,s}=C_{i}(Q_{n,i})$.

Now, we express the updating rule of the assessment differences in the following manner: for each n and i,

$$\begin{aligned} Q_{n+1,i} = Q_{n,i} + \alpha _{n,i}\big ( G_{i}(Q_{n}) -Q_{n,i} + M'_{n,i} \big ) \end{aligned}$$

where

• 1. $M'_{n,i} := M_{n, i, t} - M_{n, i, s}$, which is still a martingale difference noise;

• 2. $G=(G_{i}, G_{-i}): {\mathbb {R}}^2 \rightarrow {\mathbb {R}}^{2}$ is defined in the following manner: for each i and $Q=(Q_{i},Q_{-i}) \in {\mathbb {R}}^{2}$,

  $$\begin{aligned} G_{i}(Q) :=\,&\big ( \pi (t, x_{-i}) - \pi (s, x_{-i}) \big ) \\ =\,&bx_{-i, s}+ c \end{aligned}$$

  where

  • (a) $x_{i, s}:=C_{i}(Q_{i})$, $x_{i,t}:= 1-x_{i,s}$ and $x_{i}:=(x_{i,s}, x_{i,t})$ for each i;

  • (b) $b:=( \pi (t, s)- \pi (s, s)) - (\pi (t, t) - \pi (s, t))$;

  • (c) $c:= \pi (t, t)- \pi (s, t)$.

By applying the asynchronous stochastic approximation method of Tsitsiklis (1994) to the assessment difference process, we show the convergence to a quantal response equilibrium, at which the following equations hold: for each i,

$$\begin{aligned} x^{e}_{i, s}= & {} \frac{1}{1+ \exp \left( \frac{1}{\tau _{i}} Q^{e}_{i}\right) } \quad \text {and} \\ Q^{e}_{i}= & {} bx^{e}_{-i, s}+c. \end{aligned}$$

Without loss of generality, let s be the strictly dominant action of the game. Then, $Q^{e}_{i}<0$ and $C_{i}(Q^{e}_{i})> \frac{1}{2}$, as $Q^{e}_{i} = (\pi (t, s) -\pi (s, s) )x^{e}_{-i, s} +(\pi (t, t) -\pi (s, t) )(1-x^{e}_{-i, s}) <0$. Note also that $x^{e}_{i,s} \rightarrow 1$ as $\tau _{i} \rightarrow 0$, which means that the quantal response equilibrium approaches the dominant strategy equilibrium of the game.

Now consider

$$\begin{aligned} | G_{i}(Q_{n}) - Q^{e}_{i} | =&|b| |x_{n,-i,s} -x^{e}_{-i, s}| \\ =&k_{n,-i} |Q_{n,-i}-Q^{e}_{-i}| \end{aligned}$$

where

$$\begin{aligned} k_{n,i} := {\left\{ \begin{array}{ll} |b|\frac{ |C_{i}(Q_{n,i})-C_{i}(Q^{e}_{i}) |}{|Q_{n,i}-Q^{e}_{i}| } &{} \text {if } Q_{n,i} \ne Q^{e}_{i}\\ 0 &{} \text {otherwise}. \end{array}\right. } \end{aligned}$$

Since $C_{i}$ is decreasing, concave on the negative domain and convex on the positive domain for each i, we know that

$$\begin{aligned} \max _{Q_{i}} \frac{|C_{i}(Q_{i}) - C_{i}(Q^{e}_{i})|}{|Q_{i}-Q^{e}_{i}|} \le \frac{|C_{i}(Q^{e}_{i})|}{|Q^{e}_{i}|} \end{aligned}$$

for $Q^{e}_{i}<0$ and $C_{i}(Q^{e}_{i})>0$. Also, note that for each i,

$$\begin{aligned} |\pi (t, s) -\pi (s, s)| \vee |\pi (t, t)-\pi (s, t)| \ge |Q^{e}_{i} |, \end{aligned}$$

and thus

$$\begin{aligned} k_{n,i} \le \frac{|b|}{|\pi (t,s) -\pi (s, s)| \vee |\pi (t, t)-\pi (s, t)|}=:b'. \end{aligned}$$

Therefore, if $b'<1$, we know that $x_{n}$ converges to the quantal response equilibrium by the asynchronous stochastic approximation method.

Note that for symmetric $2 \times 2$ games with a strictly dominant action,

$$\begin{aligned} |( \pi (t, s)- \pi (s, s)) - (\pi (t, t) - \pi (s, t) ) | < |\pi (t,s) -\pi (s, s)| \vee |\pi (t, t)-\pi (s, t)|, \end{aligned}$$

and thus we obtain the convergence for the game. $\square$