Bulletin of Mathematical Biology

, Volume 71, Issue 8, pp 1818–1850

A Theoretical Analysis of Temporal Difference Learning in the Iterated Prisoner’s Dilemma Game

Original Article

DOI: 10.1007/s11538-009-9424-8

Cite this article as:
Masuda, N. & Ohtsuki, H. Bull. Math. Biol. (2009) 71: 1818. doi:10.1007/s11538-009-9424-8


Direct reciprocity is a chief mechanism of mutual cooperation in social dilemma. Agents cooperate if future interactions with the same opponents are highly likely. Direct reciprocity has been explored mostly by evolutionary game theory based on natural selection. Our daily experience tells, however, that real social agents including humans learn to cooperate based on experience. In this paper, we analyze a reinforcement learning model called temporal difference learning and study its performance in the iterated Prisoner’s Dilemma game. Temporal difference learning is unique among a variety of learning models in that it inherently aims at increasing future payoffs, not immediate ones. It also has a neural basis. We analytically and numerically show that learners with only two internal states properly learn to cooperate with retaliatory players and to defect against unconditional cooperators and defectors. Four-state learners are more capable of achieving a high payoff against various opponents. Moreover, we numerically show that four-state learners can learn to establish mutual cooperation for sufficiently small learning rates.


Cooperation Direct reciprocity Prisoner’s dilemma Reinforcement learning 

Copyright information

© Society for Mathematical Biology 2009

Authors and Affiliations

  1. 1.Graduate School of Information Science and TechnologyThe University of TokyoBunkyoJapan
  2. 2.Department of Value and Decision ScienceTokyo Institute of TechnologyTokyoJapan

Personalised recommendations