From Markov Chains to Stochastic Games

  • Abraham Neyman
Conference paper
Part of the NATO Science Series book series (ASIC, volume 570)

Abstract

Markov chains1 and Markov decision processes (MDPs) are special cases of stochastic games. Markov chains describe the dynamics of the states of a stochastic game where each player has a single action in each state. Similarly, the dynamics of the states of a stochastic game form a Markov chain whenever the players’ strategies are stationary. Markov decision processes are stochastic games with a single player. In addition, the decision problem faced by a player in a stochastic game when all other players choose a fixed profile of stationary strategies is equivalent to an MDP.

Keywords

Markov Chain Discount Factor Mixed Strategy Pure Strategy Markov Decision Process 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bewley, T. and Kohlberg, E. (1976) The asymptotic theory of stochastic games, Mathematics of Operations Research 1, 197–208.MathSciNetMATHCrossRefGoogle Scholar
  2. 2.
    Bewley, T. and Kohlberg, E. (1976) The asymptotic solution of a recursion equation occurring in stochastic games, Mathematics of Operations Research 1, 321–336.MathSciNetMATHCrossRefGoogle Scholar
  3. 3.
    Blackwell, D. (1962) Discrete dynamic programming, Annals of Mathematical Statistics 33, 719–726.MathSciNetMATHCrossRefGoogle Scholar
  4. 4.
    Denardo, E. V. (1982) Dynamic Programming, Prentice-Hall, Englewood Cliffs, NJ.Google Scholar
  5. 5.
    Derman, C. (1970) Finite State Markov Decision Processes, Academic Press, New York.Google Scholar
  6. 6.
    Filar, J.A. and Vrieze, O.J. (1996) Competitive Markov Decision Processes, Springer-Verlag, Berlin.CrossRefGoogle Scholar
  7. 7.
    Hardy, G. H. and Littlewood, J. E. (1931) Notes on the theory of series (xiv): Two Tauberian theorems, Journal of London Mathematical Society 6, 281–286.MathSciNetCrossRefGoogle Scholar
  8. 8.
    Kemeny, J. G. and Snell, J. L. (1960) Finite Markov Chains, Van Nostrand Reinhold, New York.MATHGoogle Scholar
  9. 9.
    Neyman, A. (2003) Real algebraic tools in stochastic games, in A. Neyman and S. Sorin (eds.), Stochastic Games and Applications, NATO Science Series C, Mathematical and Physical Sciences, Vol. 570, Kluwer Academic Publishers, Dordrecht, Chapter 6, pp. 57–75.Google Scholar
  10. 10.
    Neyman, A. (2003) Stochastic games: Existence of the minmax, in A. Neyman and S. Sorin (eds.), Stochastic Games and Applications, NATO Science Series C, Mathematical and Physical Sciences, Vol. 570, Kluwer Academic Publishers, Dordrecht, Chapter 11, pp. 173–193.Google Scholar
  11. 11.
    Neyman, A. (2003) Stochastic games and nonexpansive maps, in A. Neyman and S. Sorin (eds.), Stochastic Games and Applications, NATO Science Series C, Mathematical and Physical Sciences, Vol. 570, Kluwer Academic Publishers, Dordrecht, Chapter 26, pp. 397–415.Google Scholar
  12. 12.
    Puterman, M. L. (1994) Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley, New York.MATHGoogle Scholar
  13. 13.
    Raghavan, T.E.S. (2003) Finite-step algorithms for single-controller and perfect information stochastic games, in A. Neyman and S. Sorin (eds.), Stochastic Games and Applications, NATO Science Series C, Mathematical and Physical Sciences, Vol. 570, Kluwer Academic Publishers, Dordrecht, Chapter 15, pp. 227–251.CrossRefGoogle Scholar
  14. 14.
    Shapley, L.S. (1953) Stochastic games, Proceedings of the National Academy of Sciences of the U.S.A. 39, 1095–1100 (Chapter 1 in this volume).MathSciNetGoogle Scholar
  15. 15.
    Solan, E. (2003) Perturbations of Markov chains with applications to stochastic games, in A. Neyman and S. Sorin (eds.), Stochastic Games and Applications, NATO Science Series C, Mathematical and Physical Sciences, Vol. 570, Kluwer Academic Publishers, Dordrecht, Chapter 17, pp. 265–280.CrossRefGoogle Scholar
  16. 16.
    Solan, E. and Vieille, N. (2002), Correlated equilibrium in stochastic games, Games and Economic Behavior 38, 362–399.MathSciNetMATHCrossRefGoogle Scholar
  17. 17.
    Sorin, S. (2003) Discounted stochastic games: The finite case, in A. Neyman and S. Sorin (eds.), Stochastic Games and Applications, NATO Science Series C, Mathematical and Physical Sciences, Vol. 570, Kluwer Academic Publishers, Dordrecht, Chapter 5, pp. 51–55.Google Scholar
  18. 18.
    Vieille, N. (2000) Small perturbations and stochastic games, Israel Journal of Mathematics 119, 127–142.MathSciNetMATHCrossRefGoogle Scholar
  19. 19.
    Vrieze, O.J. (2003) Stochastic games and stationary strategies, in A. Neyman and S. Sorin (eds.), Stochastic Games and Applications, NATO Science Series C, Mathematical and Physical Sciences, Vol. 570, Kluwer Academic Publishers, Dordrecht, Chapter 4, pp. 37–50.Google Scholar
  20. 20.
    Vrieze, O.J. (2003) Stochastic games, practical motivation and the orderfield property for special classes, in A. Neyman and S. Sorin (eds.), Stochastic Games and Applications, NATO Science Series C, Mathematical and Physical Sciences, Vol. 570, Kluwer Academic Publishers, Dordrecht, Chapter 14, pp. 215–225.Google Scholar
  21. 21.
    White, D. J. (1993) Markov Decision Processes, John Wiley, Chichester.Google Scholar

Copyright information

© Springer Science+Business Media New York 2003

Authors and Affiliations

  • Abraham Neyman
    • 1
  1. 1.Hebrew University of JerusalemJerusalemIsrael

Personalised recommendations