Extended Replicator Dynamics as a Key to Reinforcement Learning in Multi-agent Systems

  • Karl Tuyls
  • Dries Heytens
  • Ann Nowe
  • Bernard Manderick
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2837)


Modeling learning agents in the context of Multi-agent Systems requires an adequate understanding of their dynamic behaviour. Evolutionary Game Theory provides a dynamics which describes how strategies evolve over time. Börgers et al. [1] and Tuyls et al. [11] have shown how classical Reinforcement Learning (RL) techniques such as Cross-learning and Q-learning relate to the Replicator Dynamics (RD). This provides a better understanding of the learning process. In this paper, we introduce an extension of the Replicator Dynamics from Evolutionary Game Theory. Based on this new dynamics, a Reinforcement Learning algorithm is developed that attains a stable Nash equilibrium for all types of games. Such an algorithm is lacking for the moment. This kind of dynamics opens an interesting perspective for introducing new Reinforcement Learning algorithms in multi-state games and Multi-Agent Systems.


Nash Equilibrium Reinforcement Learn Replicator Dynamics Evolutionary Game Theory Learn Automaton 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Börgers, T., Sarin, R.: Learning Through Reinforcement and Replicator Dynamics. Journal of Economic Theory 77(1) (November 1997)Google Scholar
  2. 2.
    Hofbauer, J., Sigmund, K.: Evolutionary Games and Population Dynamics. Cambridge University Press, Cambridge (1998)zbMATHGoogle Scholar
  3. 3.
    Gintis, C.M.: Game Theory Evolving, 2000. Princeton University Press, Princeton (2000)zbMATHGoogle Scholar
  4. 4.
    Narendra, K., Thathachar, M.: Learning Automata: An Introduction. Prentice-Hall, Englewood Cliffs (1989)Google Scholar
  5. 5.
    Redondo, F.V.: Game Theory and Economics. Cambridge University Press, Cambridge (2001)Google Scholar
  6. 6.
    Schneider, T.D.: Evolution of biological information. Journal of NAR 28, 2794–2799 (2000)CrossRefGoogle Scholar
  7. 7.
    Stauffer, D.: Life, Love and Death: Models of Biological Reproduction and Aging. Institute for Theoretical physics, Köln, Euroland (1999)Google Scholar
  8. 8.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An introduction. MIT Press, Cambridge (1998)Google Scholar
  9. 9.
    Samuelson, L.: Evolutionary Games and Equilibrium Selection. MIT Press, Cambridge (1997)zbMATHGoogle Scholar
  10. 10.
    Tuyls, K., Lenaerts, T., Verbeeck, K., Maes, S., Manderick, B.: Towards a Relation Between Learning Agents and Evolutionary Dynamics. In: Proceedings of BNAIC 2002, KU Leuven, Belgium (2002)Google Scholar
  11. 11.
    Tuyls, K., Verbeeck, K., Lenaerts, T.: A Selection-Mutation model for Qlearning in MAS. Accepted at AAMAS 2003, Melbourne, Australia (2003)Google Scholar
  12. 12.
    Tuyls, K., Verbeeck, K., Maes, S.: On a Dynamical Analysis of Reinforcement Learning in Games: Emergence of Occam’s Razor. Accepted at CEEMAS 2003 (2003)Google Scholar
  13. 13.
    Weibull, J.W.: Evolutionary Game Theory. MIT Press, Cambridge (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Karl Tuyls
    • 1
  • Dries Heytens
    • 1
  • Ann Nowe
    • 1
  • Bernard Manderick
    • 1
  1. 1.Computational Modeling Lab, Department of Computer ScienceVrije Universiteit BrusselBelgium

Personalised recommendations