A Pumping Algorithm for Ergodic Stochastic Mean Payoff Games with Perfect Information

• Endre Boros
• Khaled Elbassioni
• Kazuhisa Makino
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6080)

Abstract

In this paper, we consider two-person zero-sum stochastic mean payoff games with perfect information, or BWR-games, given by a digraph G = (V = V B  ∪ V W  ∪ V R , E), with local rewards $$r: E \to {\mathbb R}$$, and three types of vertices: black V B , white V W , and random V R . The game is played by two players, White and Black: When the play is at a white (black) vertex v, White (Black) selects an outgoing arc (v,u). When the play is at a random vertex v, a vertex u is picked with the given probability p(v,u). In all cases, Black pays White the value r(v,u). The play continues forever, and White aims to maximize (Black aims to minimize) the limiting mean (that is, average) payoff. It was recently shown in  that BWR-games are polynomially equivalent with the classical Gillette games, which include many well-known subclasses, such as cyclic games, simple stochastic games (SSG′s), stochastic parity games, and Markov decision processes. In this paper, we give a new algorithm for solving BWR-games in the ergodic case, that is when the optimal values do not depend on the initial position. Our algorithm solves a BWR-game by reducing it, using a potential transformation, to a canonical form in which the optimal strategies of both players and the value for every initial position are obvious, since a locally optimal move in it is optimal in the whole game. We show that this algorithm is pseudo-polynomial when the number of random nodes is constant. We also provide an almost matching lower bound on its running time, and show that this bound holds for a wider class of algorithms. Let us add that the general (non-ergodic) case is at least as hard as SSG′s, for which no pseudo-polynomial algorithm is known.

Keywords

mean payoff games local reward Gillette model perfect information potential stochastic games

References

1. 1.
Andersson, D., Miltersen, P.B.: The complexity of solving stochastic games on graphs. In: Dong, Y., Du, D.-Z., Ibarra, O. (eds.) ISAAC 2009. LNCS, vol. 5878, pp. 112–121. Springer, Heidelberg (2009)
2. 2.
Beffara, E., Vorobyov, S.: Adapting Gurvich-Karzanov-Khachiyan’s algorithm for parity games: Implementation and experimentation. Technical Report 2001-020, Department of Information Technology, Uppsala University (2001), https://www.it.uu.se/research/reports/#2001
3. 3.
Beffara, E., Vorobyov, S.: Is randomized Gurvich-Karzanov-Khachiyan’s algorithm for parity games polynomial? Technical Report 2001-025, Department of Information Technology, Uppsala University (2001), https://www.it.uu.se/research/reports/#2001
4. 4.
Björklund, H., Sandberg, S., Vorobyov, S.: A combinatorial strongly sub-exponential strategy improvement algorithm for mean payoff games. DIMACS Technical Report 2004-05, DIMACS, Rutgers University (2004)Google Scholar
5. 5.
Björklund, H., Vorobyov, S.: Combinatorial structure and randomized subexponential algorithms for infinite games. Theoretical Computer Science 349(3), 347–360 (2005)
6. 6.
Björklund, H., Vorobyov, S.: A combinatorial strongly sub-exponential strategy improvement algorithm for mean payoff games. Discrete Applied Mathematics 155(2), 210–229 (2007)
7. 7.
Boros, E., Elbassioni, K., Gurvich, V., Makino, K.: Every stochastic game with perfect information admits a canonical form. RRR-09-2009, RUTCOR. Rutgers University (2009)Google Scholar
8. 8.
Boros, E., Elbassioni, K., Gurvich, V., Makino, K.: A pumping algorithm for ergodic stochastic mean payoff games with perfect information. RRR-19-2009, RUTCOR. Rutgers University (2009)Google Scholar
9. 9.
Boros, E., Gurvich, V.: Why chess and back gammon can be solved in pure positional uniformly optimal strategies? RRR-21-2009, RUTCOR. Rutgers University (2009)Google Scholar
10. 10.
Chatterjee, K., Henzinger, T.A.: Reduction of stochastic parity to stochastic mean-payoff games. Inf. Process. Lett. 106(1), 1–7 (2008)
11. 11.
Chatterjee, K., Jurdziński, M., Henzinger, T.A.: Quantitative stochastic parity games. In: SODA ’04: Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms, pp. 121–130. Society for Industrial and Applied Mathematics, Philadelphia (2004)Google Scholar
12. 12.
Condon, A.: The complexity of stochastic games. Information and Computation 96, 203–224 (1992)
13. 13.
Condon, A.: An algorithm for simple stochastic games. In: Advances in computational complexity theory. DIMACS series in discrete mathematics and theoretical computer science, vol. 13 (1993)Google Scholar
14. 14.
Dhingra, V., Gaubert, S.: How to solve large scale deterministic games with mean payoff by policy iteration. In: Valuetools ’06: Proceedings of the 1st international conference on Performance evaluation methodolgies and tools, vol. 12. ACM, New York (2006)Google Scholar
15. 15.
Eherenfeucht, A., Mycielski, J.: Positional strategies for mean payoff games. International Journal of Game Theory 8, 109–113 (1979)
16. 16.
Friedmann, O.: An exponential lower bound for the parity game strategy improvement algorithm as we know it. In: Symposium on Logic in Computer Science, pp. 145–156 (2009)Google Scholar
17. 17.
Gillette, D.: Stochastic games with zero stop probabilities. In: Dresher, M., Tucker, A.W., Wolfe, P. (eds.) Contribution to the Theory of Games III. Annals of Mathematics Studies, vol. 39, pp. 179–187. Princeton University Press, Princeton (1957)Google Scholar
18. 18.
Gimbert, H., Horn, F.: Simple stochastic games with few random vertices are easy to solve. In: Amadio, R.M. (ed.) FOSSACS 2008. LNCS, vol. 4962, pp. 5–19. Springer, Heidelberg (2008)
19. 19.
Gurvich, V., Karzanov, A., Khachiyan, L.: Cyclic games and an algorithm to find minimax cycle means in directed graphs. USSR Computational Mathematics and Mathematical Physics 28, 85–91 (1988)
20. 20.
Halman, N.: Simple stochastic games, parity games, mean payoff games and discounted payoff games are all LP-type problems. Algorithmica 49(1), 37–50 (2007)
21. 21.
Hoffman, A.J., Karp, R.M.: On nonterminating stochastic games. Management Science, Series A 12(5), 359–370 (1966)
22. 22.
Jurdziński, M.: Deciding the winner in parity games is in UP ∩ co-UP. Inf. Process. Lett. 68(3), 119–124 (1998)
23. 23.
Jurdziński, M.: Games for Verification: Algorithmic Issues. PhD thesis, Faculty of Science, University of Aarhus, USA (2000)Google Scholar
24. 24.
Jurdziński, M., Paterson, M., Zwick, U.: A deterministic subexponential algorithm for solving parity games. In: SODA ’06: Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm, pp. 117–123. ACM, New York (2006)
25. 25.
Karp, R.M.: A characterization of the minimum cycle mean in a digraph. Discrete Math. 23, 309–311 (1978)
26. 26.
Karzanov, A.V., Lebedev, V.N.: Cyclical games with prohibition. Mathematical Programming 60, 277–293 (1993)
27. 27.
Kratsch, D., McConnell, R.M., Mehlhorn, K., Spinrad, J.P.: Certifying algorithms for recognizing interval graphs and permutation graphs. In: SODA ’03: Proceedings of the fourteenth annual ACM-SIAM symposium on Discrete algorithms, pp. 158–167. Society for Industrial and Applied Mathematics, Philadelphia (2003)Google Scholar
28. 28.
Liggett, T.M., Lippman, S.A.: Stochastic games with perfect information and time-average payoff. SIAM Review 4, 604–607 (1969)
29. 29.
Littman, M.L.: Algorithm for sequential decision making, CS-96-09. PhD thesis, Dept. of Computer Science, Brown Univ., USA (1996)Google Scholar
30. 30.
Mine, H., Osaki, S.: Markovian decision process. American Elsevier Publishing Co., New York (1970)Google Scholar
31. 31.
Moulin, H.: Extension of two person zero sum games. Journal of Mathematical Analysis and Application 5(2), 490–507 (1976)
32. 32.
Moulin, H.: Prolongement des jeux à deux joueurs de somme nulle. Bull. Soc. Math. France, Memoire 45 (1976)Google Scholar
33. 33.
Pisaruk, N.N.: Mean cost cyclical games. Mathematics of Operations Research 24(4), 817–828 (1999)
34. 34.
Vöge, J., Jurdzinski, M.: A discrete strategy improvement algorithm for solving parity games. In: Emerson, E.A., Sistla, A.P. (eds.) CAV 2000. LNCS, vol. 1855, pp. 202–215. Springer, Heidelberg (2000)
35. 35.
Vorobyov, S.: Cyclic games and linear programming. Discrete Applied Mathematics 156(11), 2195–2231 (2008)
36. 36.
Zwick, U., Paterson, M.: The complexity of mean payoff games on graphs. Theoretical Computer Science 158(1-2), 343–359 (1996)

Authors and Affiliations

• Endre Boros
• 1
• Khaled Elbassioni
• 2