Abstract
Standard evolutionary game theory framework is a useful tool to study large interacting systems and to understand the strategic behavior of individuals in such complex systems. Adding an individual state to model local feature of each player in this context allows one to study a wider range of problems in various application areas as networking, biology, etc. In this paper, we introduce such an extension of evolutionary game framework and particularly, we focus on the dynamical aspects of this system. Precisely, we study the coupled dynamics of the policies and the individual states inside a population of interacting individuals. We first define a general model by coupling replicator dynamics and continuous-time Markov decision processes, and we then consider a particular case of a two policies and two states evolutionary game. We first obtain a system of combined dynamics, and we show that the rest points of this system are equilibria profiles of our evolutionary game with individual state dynamics. Second, by assuming two different timescales between states and policies dynamics, we can compute explicitly the equilibria. Then, by transforming our evolutionary game with individual states into a standard evolutionary game, we obtain an equilibrium profile which is equivalent, in terms of occupation measures and expected fitness to the previous one. All our results are illustrated with numerical analysis.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Abbreviations
- \(\mathcal {A}\) :
-
Finite set of pure actions, with \(|\mathcal {A}|=K\)
- \(\varDelta :=\{\mathbf {p}\in \mathbb {R}^K_+|\sum _{i\in \mathcal {A}}p_i=1\}\) :
-
Set of strategies
- \(\mathcal {S}\) :
-
Set of states, with \(|\mathcal {S}|=N\)
- \(\mathcal {U}\) :
-
Set of Markov policies
- \(\mathcal {U}_S\) :
-
Set of stationary policies
- \(\mathcal {U}_D\) :
-
Set of deterministic policies
- \(\mathcal {R}_s(s',a)\) :
-
Transition rate from state \(s'\in \mathcal {S}\) to state \(s\in \mathcal {S}\) given action \(a\in \mathcal {A}\)
- \(\pi _s(u_i)\) :
-
Time ratio spent in state s under policy \(u_i\)
- \(w_s\) :
-
Fraction of individual in the population in state s; \(\mathbf {w}=(w_1,\ldots ,w_N)\): distribution over all states in the population
- \(q_i\) :
-
Fraction of individuals choosing deterministic policy \(u_i\)
- \(\xi =(w_1,q_x)\) :
-
Population profile
- \(J(s,a;s',a')\) :
-
Immediate fitness that a player get when in state s plays action a in an interaction with an individual in state \(s'\) playing \(a'\)
- \(F(\cdot ,\cdot )\) :
-
Fitness of an individual
- \(\bar{F}(\cdot )\) :
-
Average expected fitness of a population
- \(F_i(\cdot )\) :
-
Expected fitness of an individual choosing deterministic policy \(u_i\in \mathcal {U}_D\)
- \(J_i(\cdot )\) :
-
Expected fitness of an individual playing action \(i\in \mathcal {A}\)
References
Altman E, Hayel Y (2008) Stochastic evolutionary game approach to energy management in a distributed aloha network. In: IEEE INFOCOM
Altman E, Hayel Y (2010) Markov decision evolutionary games. IEEE Trans Autom Control 55:1560–1569
Beggs A (2005) On the convergence of reinforcement learning. J Econ Theory 122:1–36
Borgers T, Sarin R (1997) Learning through reinforcement and replicator dynamics. J Econ Theory 77:1–14
Flesch J, Parthasarathy T, Thuijsman F, Uyttendaele P (2013) Evolutionary stochastic games. Dyn Games Appl 3:207–219
Fudenberg D, Levine D (1998) Theory of learning in games. MIT press, Cambridge
Gros PMS, Buccieri D, Bonvin D (2005) Two-time-scale control scheme is illustrated via the simulation of a flying robotic structure. In: IFAC conference on nonlinear model predictive control
Guo X, Hernandez-Lerma O (2009) Continuous time Markov decision process. Springer, Berlin
Hofbauer J, Schuster P, Sigmund K (1979) A note on evolutionarily stable strategies and game dynamics. J Theor Biol 81:609–612
Hofbauer J, Sigmund K (1998) Evolutionary games and population dynamics. Cambridge University Press, Cambridge
Hofbauer J, Sigmund K (2003) Evolutionary game dynamics. Am Math Soc 40:479–519
Jackson M (2005) A survey of network formation models: stability and efficiency. Group formation in economics: networks, clubs, and coalitions book. Cambridge University Press, Cambridge
Jovanovic B, Rosenthal R (1988) Anonymous sequential games. J Math Econ 17:77–87
Kokotovic P, Khalil H, O’reilly J (1986) Singular perturbation methods in control. SIAM, Philadelphia
Mannor S, Shamma J (2007) Multi-agent learning for engineers. Artif Intell 171:417–422
Poveda J, Quijano N (2013) Extremum seeking for multi-population games. In: 52th IEEE conference on decision and control
Sandholm WH (2010) Population games and evolutionary dynamics. MIT Press, Cambridge
Smith JM, Price GR (1973) The logic of animal conflict. Nature 246:15–18
Taylor P, Jonker LB (1978) Evolutionary stable strategies and game dynamics. Math Biosci 40:145–156
Weibull J (1995) Evolutionary game theory. MIT Press, Cambridge
Wiecek P, Altman E (2015) Stationary anonymous sequential games with undiscounted rewards. J Optim Theory Appl 166(2):686–710
Author information
Authors and Affiliations
Corresponding author
Additional information
This work has been partially supported by the European Commission within the framework of the CONGAS Project FP7-ICT-2011-8-317672, see http://www.congas-project.eu.
Rights and permissions
About this article
Cite this article
Brunetti, I., Hayel, Y. & Altman, E. State-Policy Dynamics in Evolutionary Games. Dyn Games Appl 8, 93–116 (2018). https://doi.org/10.1007/s13235-016-0208-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13235-016-0208-0