# State-Policy Dynamics in Evolutionary Games

- 201 Downloads
- 1 Citations

## Abstract

Standard evolutionary game theory framework is a useful tool to study large interacting systems and to understand the strategic behavior of individuals in such complex systems. Adding an individual state to model local feature of each player in this context allows one to study a wider range of problems in various application areas as networking, biology, etc. In this paper, we introduce such an extension of evolutionary game framework and particularly, we focus on the dynamical aspects of this system. Precisely, we study the coupled dynamics of the policies and the individual states inside a population of interacting individuals. We first define a general model by coupling replicator dynamics and continuous-time Markov decision processes, and we then consider a particular case of a two policies and two states evolutionary game. We first obtain a system of combined dynamics, and we show that the rest points of this system are equilibria profiles of our evolutionary game with individual state dynamics. Second, by assuming two different timescales between states and policies dynamics, we can compute explicitly the equilibria. Then, by transforming our evolutionary game with individual states into a standard evolutionary game, we obtain an equilibrium profile which is equivalent, in terms of occupation measures and expected fitness to the previous one. All our results are illustrated with numerical analysis.

## Keywords

Evolutionary game theory Dynamic processes Replicator dynamics Singular perturbation## List of symbols

- \(\mathcal {A}\)
Finite set of pure actions, with \(|\mathcal {A}|=K\)

- \(\varDelta :=\{\mathbf {p}\in \mathbb {R}^K_+|\sum _{i\in \mathcal {A}}p_i=1\}\)
Set of strategies

- \(\mathcal {S}\)
Set of states, with \(|\mathcal {S}|=N\)

- \(\mathcal {U}\)
Set of Markov policies

- \(\mathcal {U}_S\)
Set of stationary policies

- \(\mathcal {U}_D\)
Set of deterministic policies

- \(\mathcal {R}_s(s',a)\)
Transition rate from state \(s'\in \mathcal {S}\) to state \(s\in \mathcal {S}\) given action \(a\in \mathcal {A}\)

- \(\pi _s(u_i)\)
Time ratio spent in state

*s*under policy \(u_i\)- \(w_s\)
Fraction of individual in the population in state

*s*; \(\mathbf {w}=(w_1,\ldots ,w_N)\): distribution over all states in the population- \(q_i\)
Fraction of individuals choosing deterministic policy \(u_i\)

- \(\xi =(w_1,q_x)\)
Population profile

- \(J(s,a;s',a')\)
Immediate fitness that a player get when in state

*s*plays action*a*in an interaction with an individual in state \(s'\) playing \(a'\)- \(F(\cdot ,\cdot )\)
Fitness of an individual

- \(\bar{F}(\cdot )\)
Average expected fitness of a population

- \(F_i(\cdot )\)
Expected fitness of an individual choosing deterministic policy \(u_i\in \mathcal {U}_D\)

- \(J_i(\cdot )\)
Expected fitness of an individual playing action \(i\in \mathcal {A}\)

## References

- 1.Altman E, Hayel Y (2008) Stochastic evolutionary game approach to energy management in a distributed aloha network. In: IEEE INFOCOMGoogle Scholar
- 2.Altman E, Hayel Y (2010) Markov decision evolutionary games. IEEE Trans Autom Control 55:1560–1569MathSciNetCrossRefzbMATHGoogle Scholar
- 3.Beggs A (2005) On the convergence of reinforcement learning. J Econ Theory 122:1–36MathSciNetCrossRefzbMATHGoogle Scholar
- 4.Borgers T, Sarin R (1997) Learning through reinforcement and replicator dynamics. J Econ Theory 77:1–14MathSciNetCrossRefzbMATHGoogle Scholar
- 5.Flesch J, Parthasarathy T, Thuijsman F, Uyttendaele P (2013) Evolutionary stochastic games. Dyn Games Appl 3:207–219MathSciNetCrossRefzbMATHGoogle Scholar
- 6.Fudenberg D, Levine D (1998) Theory of learning in games. MIT press, CambridgezbMATHGoogle Scholar
- 7.Gros PMS, Buccieri D, Bonvin D (2005) Two-time-scale control scheme is illustrated via the simulation of a flying robotic structure. In: IFAC conference on nonlinear model predictive controlGoogle Scholar
- 8.Guo X, Hernandez-Lerma O (2009) Continuous time Markov decision process. Springer, BerlinCrossRefzbMATHGoogle Scholar
- 9.Hofbauer J, Schuster P, Sigmund K (1979) A note on evolutionarily stable strategies and game dynamics. J Theor Biol 81:609–612CrossRefGoogle Scholar
- 10.Hofbauer J, Sigmund K (1998) Evolutionary games and population dynamics. Cambridge University Press, CambridgeCrossRefzbMATHGoogle Scholar
- 11.Hofbauer J, Sigmund K (2003) Evolutionary game dynamics. Am Math Soc 40:479–519MathSciNetCrossRefzbMATHGoogle Scholar
- 12.Jackson M (2005) A survey of network formation models: stability and efficiency. Group formation in economics: networks, clubs, and coalitions book. Cambridge University Press, CambridgeGoogle Scholar
- 13.Jovanovic B, Rosenthal R (1988) Anonymous sequential games. J Math Econ 17:77–87MathSciNetCrossRefzbMATHGoogle Scholar
- 14.Kokotovic P, Khalil H, O’reilly J (1986) Singular perturbation methods in control. SIAM, PhiladelphiazbMATHGoogle Scholar
- 15.Mannor S, Shamma J (2007) Multi-agent learning for engineers. Artif Intell 171:417–422MathSciNetCrossRefzbMATHGoogle Scholar
- 16.Poveda J, Quijano N (2013) Extremum seeking for multi-population games. In: 52th IEEE conference on decision and controlGoogle Scholar
- 17.Sandholm WH (2010) Population games and evolutionary dynamics. MIT Press, CambridgezbMATHGoogle Scholar
- 18.Smith JM, Price GR (1973) The logic of animal conflict. Nature 246:15–18CrossRefzbMATHGoogle Scholar
- 19.Taylor P, Jonker LB (1978) Evolutionary stable strategies and game dynamics. Math Biosci 40:145–156MathSciNetCrossRefzbMATHGoogle Scholar
- 20.Weibull J (1995) Evolutionary game theory. MIT Press, CambridgezbMATHGoogle Scholar
- 21.Wiecek P, Altman E (2015) Stationary anonymous sequential games with undiscounted rewards. J Optim Theory Appl 166(2):686–710MathSciNetCrossRefzbMATHGoogle Scholar