Abstract
This paper examines the performance of simple reinforcement learningalgorithms in a stationary environment and in a repeated game where theenvironment evolves endogenously based on the actions of other agents. Sometypes of reinforcement learning rules can be extremely sensitive to smallchanges in the initial conditions, consequently, events early in a simulationcan affect the performance of the rule over a relatively long time horizon.However, when multiple adaptive agents interact, algorithms that performedpoorly in a stationary environment often converge rapidly to a stableaggregate behaviors despite the slow and erratic behavior of individuallearners. Algorithms that are robust in stationary environments can exhibitslow convergence in an evolving environment.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Arthur, W.B. (1994). Inductive reasoning and bounded rationality: The El Farol problem. American Economic Association Papers and Proceedings, 84, 406–411.
Bell, A.M., Sethares, W.A. and Bucklew, J.A. (1999). Coordination failure as a source of congestion in information networks. Working paper. NASA Ames Research Center, http://ic.arc.nasa.gov/ic/people/bell/bell.html.
Bush, R. and Mosteller, F. (1955). Stochastic Models for Learning. Wiley, New York.
Erev, I. and Rapoport, A. (1997). Coordination, ‘magic’, and reinforcement learning in a market entry game. Working paper. Hong Kong University of Science and Technology Department of Marketing MKTG 97.103.
Er'ev, I. and Roth, A.E. (1998). Predicting how people play games: Reinforcement learning in experimental games with unique, mixed strategy equilibria. American Economic Review, 8(4), 848–881.
Fogel, D.B., Chellapilla, K. and Angeline, P.J. (1999). Inductive reasoning and bounded rationality reconsidered. IEEE Transactions on Evolutionary Computation, 3(2), 142–146.
Fudenberg, D. and Levine, D.K. (1998). The Theory of Learning in Games. The MIT Press, Cambridge, MA.
Gary-Bobo, R. (1990). On the existence of equilibrium points in a class of asymmetric market entry games. Games and Economic Behavior,2, 239–246.
Kephart, J.O., Hogg, T. and Huberman, B.A. (1989). Dynamics of computational. Physical Review A, 40(1), 404–421.
Rapoport, A., Seale, D. and Winter, E. (1998). An experimental study of coordination and learning in iterated two-market entry games. Working paper, Hong Kong University of Science and Technology Department of Marketing MKTG 98.116.
Roth, A.E. and Er'ev, I. (1995). Learning in extensive form games: Experimental data and simple dynamic models in the intermediate term. Games and Economic Behavior,8, 164–212.
Selten, R. and Guth, W. (1982). Equilibrium point selection in a class of market entry games. In Games, Economic Dynamics, and Time Series Analysis, 101–116. Physica-Verlag, Vienna.
Sethares, W.A. and Bell, A.M. (1998). An adaptive solution to the El Farol problem. In Proceedings of the Thirty-Sixth Annual Allerton Conference on Communication, Control, and Computing. Allerton IL.
Sundali, J., Rapoport, A. and Seale, D. (1995). Coordination in market entry games with symmetric players. Organizational Behavior and Human Decision Processes, 64(2), 203–218.
Sutton, R.S. and Barto, A.G. (1998). Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Bell, A.M. Reinforcement Learning Rules in a Repeated Game. Computational Economics 18, 89–110 (2001). https://doi.org/10.1023/A:1013818611576
Issue Date:
DOI: https://doi.org/10.1023/A:1013818611576