Dynamic Games and Applications

, Volume 6, Issue 3, pp 263–276 | Cite as

A Dijkstra-Type Algorithm for Dynamic Games



We study zero-sum dynamic games with deterministic transitions and alternating moves of the players. Player 1 aims at reaching a terminal set and minimizing a possibly discounted running and final cost. We propose and analyze an algorithm that computes the value function of these games extending Dijkstra’s algorithm for shortest paths on graphs. We also show the connection of these games with numerical schemes for differential games of pursuit-evasion type, if the grid is adapted to the dynamical system. Under suitable conditions, we prove the convergence of the value of the discrete game to the value of the differential game as the step of approximation tends to zero.


Zero sum dynamic games Dijkstra algorithm Pursuit evasion games Discrete time games 

Mathematics Subject Classification

91A25 65Kxx 49N70 91A23 91A24 49N75 


  1. 1.
    Alfaro T, Henzinger T, Kupferman O (2007) Concurrent reachability games. Theor Comput Sci 386:188–217MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Andrews J, Vladimirsky A (2014) Deterministic control of randomly-terminated processes. Interfaces Free Bound 16:1–40MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Bardi M, Capuzzo-Dolcetta I (1997) Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations. Birkhäuser Boston Inc, BostonCrossRefMATHGoogle Scholar
  4. 4.
    Bardi M, Bottacin S, Falcone M (1995) Convergence of discrete schemes for discontinuous value functions of pursuit-evasion games. In: New trends in dynamic games and applications. Ann. Internat. Soc. Dynam. Games, 3, Birkhäuser Boston, pp 273–304Google Scholar
  5. 5.
    Bardi M, Falcone M and Soravia P (1994) Fully discrete schemes for the value function of pursuit-evasion games. In: Advances in dynamic games and applications. Ann. Internat. Soc. Dynam. Games, 1, Birkhäuser Boston, pp 89–105Google Scholar
  6. 6.
    Bardi M, Falcone M, Soravia P (1999) Numerical methods for pursuit-evasion games via viscosity solutions. In: Bardi M, Parthasarathy T, Raghavan e TES (eds) Stochastic and differential games: theory and numerical methods. Ann. Internat. Soc. Dynam. Games, 4. Birkhäuser, Boston, pp 105–175Google Scholar
  7. 7.
    Bardi M, Soravia P (1991) Approximation of differential games of pursuit-evasion by discrete-time games. In: Differential games-developments in modelling and computation, Lecture Notes in Control and Inform. Sci., 156, Springer, pp 131–143Google Scholar
  8. 8.
    Bardi M, Soravia P (1994) A comparison result for Hamilton-Jacobi equations and applications to some differential games lacking controllability. Funkcial Ekvac 37:19–43MathSciNetMATHGoogle Scholar
  9. 9.
    Bardi M, Terrone G (2013) On the homogenization of some non-coercive Hamilton-Jacobi-Isaacs equations. Commun Pure Appl Anal 12:207–236MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Bertsekas D (2001) Dynamic programming and optimal control, 2nd edn, vol II. Athena Scientific, BostonGoogle Scholar
  11. 11.
    Bertsekas D (2015) Robust shortest path planning and semicontractive dynamic programming, Report LIDS-2915Google Scholar
  12. 12.
    Cacace S, Cristiani E, Falcone M (2014) Can local single-pass methods solve any stationary Hamilton-Jacobi equation? SIAM J Sci Comput 36:A570–A587MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Cacace S, Cristiani E, Falcone M, Picarelli A (2012) A patchy dynamic programming scheme for a class of Hamilton–Jacobi–Bellman equations. SIAM J Sci Comput 34:A2625–A2649MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Cardaliaguet P, Quincampoix M, Saint-Pierre P (1999) Set-valued numerical analysis for optimal control and differential games. In: Bardi M, Parthasarathy T, Raghavan e TES (eds) Stochastic and differential games: theory and numerical methods. Ann. Internat. Soc. Dynam. Games. Birkhäuser, Boston, pp 177–247Google Scholar
  15. 15.
    Cristiani E (2009) A fast marching method for Hamilton–Jacobi equations modeling monotone front propagations. J Sci Comput 39:189–205MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Cristiani E, Falcone M (2006) A fast marching method for pursuit-evasion games. In: Communications to SIMAI congress 1. doi:10.1685/CSC06059
  17. 17.
    Cristiani E, Falcone M (2009) Fully-discrete schemes for the value function of pursuit-evasion games with state constraints. In: Advances in dynamic games and their applications, Ann. Internat. Soc. Dynam. Games, 10. Birkhäuser Boston, pp 177–206Google Scholar
  18. 18.
    Dijkstra EW (1959) A note on two problems in connection with graphs. Numer Math 1:269–271MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Falcone M, Ferretti R (2014) Semi-Lagrangian approximations schemes for linear and Hamilton-Jacobi equations. SIAM, PhiladelphiaMATHGoogle Scholar
  20. 20.
    Filar JA, Raghavan TES (1991) Algorithms for stochastic games—A survey. Z Oper Res 35:437–472MathSciNetMATHGoogle Scholar
  21. 21.
    Filar JA, Vrieze OJ (1996) Competitive Markov decision processes. Springer, BerlinCrossRefMATHGoogle Scholar
  22. 22.
    Fredman ML, Tarjan RE (1987) Fibonacci heaps and their uses in improved network optimization algorithms. J Assoc Comput Mach 34:596–615MathSciNetCrossRefGoogle Scholar
  23. 23.
    Grüne L, Junge O (2008) Global optimal control of perturbed systems. J Optim Theory Appl 136:411–429MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Kushner HJ (2004) The Gauss-Seidel numerical procedure for Markov stochastic games. IEEE Trans Autom Control 49:1779–1784MathSciNetCrossRefGoogle Scholar
  25. 25.
    LaValle SM (2006) Planning algorithms. Cambridge University Press, CambridgeCrossRefMATHGoogle Scholar
  26. 26.
    McEneaney WM (2006) Max-plus methods for nonlinear control and estimation. Birkhäuser Boston Inc, BostonMATHGoogle Scholar
  27. 27.
    Parthasarathy T, Raghavan TES (1981) An orderfield property for stochastic games when one player controls transition probabilities. J Optim Theory Appl 33:375–392MathSciNetCrossRefMATHGoogle Scholar
  28. 28.
    Raghavan TES, Syed Z (2003) A policy-improvement type algorithm for solving zero-sum two-person stochastic games of perfect information. Math Program Ser A 95:513–532MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Sethian JA (1996) A fast marching level set method for monotonically advancing fronts. Proc Nat Acad Sci USA 93:1591–1595MathSciNetCrossRefMATHGoogle Scholar
  30. 30.
    Sethian JA (1999) Level set methods and fast marching methods. Evolving interfaces in computational geometry, fluid mechanics, computer vision, and materials science, Second edn. Cambridge University Press, CambridgeMATHGoogle Scholar
  31. 31.
    Sethian JA, Vladimirsky A (2003) Ordered upwind methods for static Hamilton–Jacobi equations: theory and algorithms. SIAM J Numer Anal 41:325–363MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Shapley LS (1953) Stochastic games. Proc Nat Acad Sci USA 39:1095–1100MathSciNetCrossRefMATHGoogle Scholar
  33. 33.
    Sorin S (2002) A first course on zero-sum repeated games, Mathématiques & Applications, vol 37. Springer, BerlinGoogle Scholar
  34. 34.
    Tsitsiklis JN (1995) Efficient algorithms for globally optimal trajectories. IEEE Trans Autom Control 40:1528–1538MathSciNetCrossRefMATHGoogle Scholar
  35. 35.
    Vladimirsky A (2008) Label-setting methods for multimode stochastic shortest path problems on graphs. Math Oper Res 33:821–838MathSciNetCrossRefMATHGoogle Scholar
  36. 36.
    von Lossow M (2007) A min-max version of Dijkstra’s algorithm with application to perturbed optimal control problems. In: Proceedings of the 6th international congress on industrial and applied mathematics (ICIAM07), Proceedings of the applied mathematics and mechanics, vol 7, pp 4130027–4130028Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Dipartimento di MatematicaUniversità degli Studi di PadovaPaduaItaly
  2. 2.UPMC Univ Paris 06, CNRS, UMR 7586, IMJ-PRGSorbonne UniversitésParisFrance
  3. 3.Sorbonne Paris Cité, UMR 7586Univ Paris-DiderotParisFrance

Personalised recommendations