Dynamic Games and Applications

, Volume 8, Issue 1, pp 93–116 | Cite as

State-Policy Dynamics in Evolutionary Games

  • Ilaria Brunetti
  • Yezekael Hayel
  • Eitan AltmanEmail author


Standard evolutionary game theory framework is a useful tool to study large interacting systems and to understand the strategic behavior of individuals in such complex systems. Adding an individual state to model local feature of each player in this context allows one to study a wider range of problems in various application areas as networking, biology, etc. In this paper, we introduce such an extension of evolutionary game framework and particularly, we focus on the dynamical aspects of this system. Precisely, we study the coupled dynamics of the policies and the individual states inside a population of interacting individuals. We first define a general model by coupling replicator dynamics and continuous-time Markov decision processes, and we then consider a particular case of a two policies and two states evolutionary game. We first obtain a system of combined dynamics, and we show that the rest points of this system are equilibria profiles of our evolutionary game with individual state dynamics. Second, by assuming two different timescales between states and policies dynamics, we can compute explicitly the equilibria. Then, by transforming our evolutionary game with individual states into a standard evolutionary game, we obtain an equilibrium profile which is equivalent, in terms of occupation measures and expected fitness to the previous one. All our results are illustrated with numerical analysis.


Evolutionary game theory Dynamic processes Replicator dynamics Singular perturbation 

List of symbols

\(\mathcal {A}\)

Finite set of pure actions, with \(|\mathcal {A}|=K\)

\(\varDelta :=\{\mathbf {p}\in \mathbb {R}^K_+|\sum _{i\in \mathcal {A}}p_i=1\}\)

Set of strategies

\(\mathcal {S}\)

Set of states, with \(|\mathcal {S}|=N\)

\(\mathcal {U}\)

Set of Markov policies

\(\mathcal {U}_S\)

Set of stationary policies

\(\mathcal {U}_D\)

Set of deterministic policies

\(\mathcal {R}_s(s',a)\)

Transition rate from state \(s'\in \mathcal {S}\) to state \(s\in \mathcal {S}\) given action \(a\in \mathcal {A}\)

\(\pi _s(u_i)\)

Time ratio spent in state s under policy \(u_i\)


Fraction of individual in the population in state s; \(\mathbf {w}=(w_1,\ldots ,w_N)\): distribution over all states in the population


Fraction of individuals choosing deterministic policy \(u_i\)

\(\xi =(w_1,q_x)\)

Population profile


Immediate fitness that a player get when in state s plays action a in an interaction with an individual in state \(s'\) playing \(a'\)

\(F(\cdot ,\cdot )\)

Fitness of an individual

\(\bar{F}(\cdot )\)

Average expected fitness of a population

\(F_i(\cdot )\)

Expected fitness of an individual choosing deterministic policy \(u_i\in \mathcal {U}_D\)

\(J_i(\cdot )\)

Expected fitness of an individual playing action \(i\in \mathcal {A}\)


  1. 1.
    Altman E, Hayel Y (2008) Stochastic evolutionary game approach to energy management in a distributed aloha network. In: IEEE INFOCOMGoogle Scholar
  2. 2.
    Altman E, Hayel Y (2010) Markov decision evolutionary games. IEEE Trans Autom Control 55:1560–1569MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Beggs A (2005) On the convergence of reinforcement learning. J Econ Theory 122:1–36MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Borgers T, Sarin R (1997) Learning through reinforcement and replicator dynamics. J Econ Theory 77:1–14MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Flesch J, Parthasarathy T, Thuijsman F, Uyttendaele P (2013) Evolutionary stochastic games. Dyn Games Appl 3:207–219MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Fudenberg D, Levine D (1998) Theory of learning in games. MIT press, CambridgezbMATHGoogle Scholar
  7. 7.
    Gros PMS, Buccieri D, Bonvin D (2005) Two-time-scale control scheme is illustrated via the simulation of a flying robotic structure. In: IFAC conference on nonlinear model predictive controlGoogle Scholar
  8. 8.
    Guo X, Hernandez-Lerma O (2009) Continuous time Markov decision process. Springer, BerlinCrossRefzbMATHGoogle Scholar
  9. 9.
    Hofbauer J, Schuster P, Sigmund K (1979) A note on evolutionarily stable strategies and game dynamics. J Theor Biol 81:609–612CrossRefGoogle Scholar
  10. 10.
    Hofbauer J, Sigmund K (1998) Evolutionary games and population dynamics. Cambridge University Press, CambridgeCrossRefzbMATHGoogle Scholar
  11. 11.
    Hofbauer J, Sigmund K (2003) Evolutionary game dynamics. Am Math Soc 40:479–519MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Jackson M (2005) A survey of network formation models: stability and efficiency. Group formation in economics: networks, clubs, and coalitions book. Cambridge University Press, CambridgeGoogle Scholar
  13. 13.
    Jovanovic B, Rosenthal R (1988) Anonymous sequential games. J Math Econ 17:77–87MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Kokotovic P, Khalil H, O’reilly J (1986) Singular perturbation methods in control. SIAM, PhiladelphiazbMATHGoogle Scholar
  15. 15.
    Mannor S, Shamma J (2007) Multi-agent learning for engineers. Artif Intell 171:417–422MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Poveda J, Quijano N (2013) Extremum seeking for multi-population games. In: 52th IEEE conference on decision and controlGoogle Scholar
  17. 17.
    Sandholm WH (2010) Population games and evolutionary dynamics. MIT Press, CambridgezbMATHGoogle Scholar
  18. 18.
    Smith JM, Price GR (1973) The logic of animal conflict. Nature 246:15–18CrossRefzbMATHGoogle Scholar
  19. 19.
    Taylor P, Jonker LB (1978) Evolutionary stable strategies and game dynamics. Math Biosci 40:145–156MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Weibull J (1995) Evolutionary game theory. MIT Press, CambridgezbMATHGoogle Scholar
  21. 21.
    Wiecek P, Altman E (2015) Stationary anonymous sequential games with undiscounted rewards. J Optim Theory Appl 166(2):686–710MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.INRIA Sophia Antipolis and LIA/CERIUniversity of AvignonAvignonFrance
  2. 2.LIA/CERIUniversity of AvignonAvignonFrance
  3. 3.INRIA Sophia Antipolis and LINCS - Laboratory of Information, Network and Communication SciencesValbonneFrance

Personalised recommendations