Skip to main content
SpringerLink
Log in

Turnpike Theorems for Markov Games

  • Published:
Dynamic Games and Applications Aims and scope Submit manuscript

Abstract

This paper has a two-fold purpose. First, we attempt to outline the development of the turnpike theorems in the last several decades. Second, we study turnpike theorems in finite-horizon two-person zero-sum Markov games on a general Borel state space. Utilising the Bellman (or Shapley) operator defined for this game, we prove stochastic versions of the early turnpike theorem on the set of optimal strategies and the middle turnpike theorem on the distribution of the state space.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Akian M, Gaubert S, Kolokoltsov V (2009) The optimal assignment problem for a countable state space. Contemp Math 495:39–60

    Article  MathSciNet  Google Scholar 

  2. Alvarez O, Bardi M (2007) Ergodic problems in differential games. In: Jorgensen S, Quincampoix M, Vincent TL (eds) Advances in dynamic game theory. Ann Int Soc Dyn Games, vol 9. Birkhäuser, Boston, pp 131–152

    Chapter  Google Scholar 

  3. Alvarez O, Bardi M (2010) Ergodic ergodicity, stabilization, and singular perturbations of Bellman-Isaacs equations. Memoirs American math soc, 77 pages

    Google Scholar 

  4. Araujo A, Scheinkman JA (1977) Smoothness, comparative dynamics, and the turnpike property. Econometrica 45(3):601–620

    Article  MathSciNet  MATH  Google Scholar 

  5. Arkin VI, Evstigneev IV (1987) Stochastic models of control and economic dynamics (economic theory, econometrics, and mathematical economics). Academic Press, San Diego

    Google Scholar 

  6. Atsumi H (1965) Neoclassical growth and the efficient program of capital accumulation. Rev Econ Stud 32(2):127–136

    Article  Google Scholar 

  7. Bardi M (2009) On differential games with long-time-average cost. Ann Int Soc Dyn Games 10(1):1–16

    Google Scholar 

  8. Bewley TF (1982) An integration of equilibrium theory and turnpike theory. J Math Econ 20:233–267

    Article  MathSciNet  Google Scholar 

  9. Bewley TF (2007) General equilibrium, overlapping generations models and optimal growth theory. Harvard University Press, Cambridge

    Google Scholar 

  10. Blot J, Crettez B (2007) On the smoothness of optimal paths II: some local turnpike results. Dec Econ Financ 30:137–150

    Article  MathSciNet  MATH  Google Scholar 

  11. Brock WA, Mirman LJ (1972) Optimal economic growth and uncertainty: the discounted case. J Econ Theory 4:479–513

    Article  MathSciNet  Google Scholar 

  12. Brock WA, Mirman LJ (1973) Optimal economic growth and uncertainty: the no discounting case. Int Econ Rev 14(3):560–573

    Article  MathSciNet  MATH  Google Scholar 

  13. Brock WA, Majumdar M (1978) Global asymptotic stability results for multisector models of optimal growth under uncertainty when future utilities are discounted. J Econ Theory 18:225–243

    Article  MathSciNet  MATH  Google Scholar 

  14. Carlson DA (1990) The existence of catching-up optimal solutions for a class of infinite horizon optimal control problems with time delay. SIAM J Control Optim 28:402–422

    Article  MathSciNet  MATH  Google Scholar 

  15. Carlson DA, Haurie A, Jabrane A (1987) Existence of overtaking solutions to infinite dimensional control problems on unbounded time intervals. SIAM J Control Optim 25:1517–1541

    Article  MathSciNet  MATH  Google Scholar 

  16. Carlson DA, Haurie AB (1996) A turnpike theory for infinite horizon open-loop differential games with decoupled controls. SIAM J Control Optim 34(4):1405–1419

    Article  MathSciNet  MATH  Google Scholar 

  17. Cass D (1966) Optimum growth in an aggregative model of capital accumulation: a Turnpike theorem. Econometrica 34(4):833–850

    Article  Google Scholar 

  18. Chang FR (1982) A note on the stochastic value loss assumption. J Econ Theory 26:164–170

    Article  MATH  Google Scholar 

  19. Coles JL (1985) Equilibrium turnpike theory with constant returns to scale and possibly heterogeneous discount factors. Int Econ Rev 26(3):671–679

    Article  MathSciNet  MATH  Google Scholar 

  20. Cox JC, Huang C-F (1992) A continuous-time portfolio turnpike theorem. J Econ Dyn Control 16:491–507

    Article  MathSciNet  MATH  Google Scholar 

  21. Dana RA (1974) Evaluation of development programs in a stationary stochastic economy with bounded primary resources. In: Proceedings of the Warsaw symposium on mathematical methods in economics. North-Holland, Amsterdam, pp 179–205

    Google Scholar 

  22. Dasgupta S, Mckenzie KW (1985) A note on comparative dynamics of stationary states. Econ Lett 18:333–338

    Article  MathSciNet  Google Scholar 

  23. Denardo EV, Rothblum UG (2006) A turnpike theorem for a risk-sensitive Markov decision process with stopping. SIAM J Control Optim 45(2):414–431

    Article  MathSciNet  Google Scholar 

  24. Dorfman R, Samuelson P, Solow R (1958) Linear programming and economic analysis. McGraw-Hill, New York

    MATH  Google Scholar 

  25. Dybvig PH, Rogers LCG, Back K (1999) Portfolio Turnpikes. Rev Financ Stud 12(1):165–195

    Article  Google Scholar 

  26. Escobedo-Trujillo B, López-Barrientos D, Hernández-Lerma O (2012) Bias and overtaking equilibria for zero-sum stochastic differential games. J Optim Theory Appl 153:662–687

    Article  MathSciNet  MATH  Google Scholar 

  27. Evstigneev IV (1974) Optimal stochastic programs and their stimulating prices. In: Los J, Los MW (eds) Mathematical models in economics. North-Holland, Amsterdam, pp 219–252

    Google Scholar 

  28. Evstigneev IV (1976) Turnpike theorems in probabilistic models of economic dynamics. Mat Zametki 19(2):165–171 (translated from Mat. Z. 19(2), 279–290 (1976))

    MathSciNet  MATH  Google Scholar 

  29. Fershtman C, Kamien MI (1990) Turnpike properties in a finite horizon differential game: dynamic duopoly with sticky prices. Int Econ Rev 31(1):49–60

    Article  MathSciNet  MATH  Google Scholar 

  30. Feinstein CD, Oren SS (1985) A “funnel” turnpike theorem for optimal growth problems with discounting. J Econ Dyn Control 9:25–39

    Article  MathSciNet  Google Scholar 

  31. Fleming W, Sethi SP, Soner HM (1987) An optimal stochastic production planning problem with randomly fluctuating demand. SIAM J Control Optim 25:1494–1502

    Article  MathSciNet  MATH  Google Scholar 

  32. Föllmer H, Majumdar M (1978) On the asymptotic behaviour of stochastic economic processes. J Math Econ 5:275–287

    Article  MATH  Google Scholar 

  33. Gale D (1967) On optimal development in a multi-sector economy. Rev Econ Stud 34:1–18

    Article  MathSciNet  Google Scholar 

  34. Guo X, Hernández-Lerma O (2005) Zero-sum continuous-time Markov games with unbounded transition and discounted payoff rates. Bernoulli 11:1009–1029

    Article  MathSciNet  MATH  Google Scholar 

  35. Hankansson N (1974) Convergence to isoelastic utility and policy in multiperiod portfolio choice. J Financ Econ 1:201–224

    Google Scholar 

  36. Haurie A (1976) Optimal control on an infinite time horizon: the turnpike approach. J Math Econ 3(1):81–102

    Article  MathSciNet  MATH  Google Scholar 

  37. Haurie A, Delft CV (1991) Turnpike properties for a class of piecewise deterministic systems arising in manufacturing flow control. Ann Oper Res 29:351–374

    Article  MathSciNet  MATH  Google Scholar 

  38. Hernandez-Lerma O, Lasserre JB (2001) Zero-sum stochastic games in Borel spaces: average Payoff criteria. SIAM J Control Optim 39:1520–1539

    Article  MathSciNet  MATH  Google Scholar 

  39. Holzbaur UD (1986) Fixed point theorems for discounted finite Markov decision processes. J Math Anal Appl 116(2):594–597

    Article  MathSciNet  MATH  Google Scholar 

  40. Huang MY, Malhame RP, Caines PE (2006) Large population stochastic dynamic games: closed-loop Mckean–Vlasov systems and the Nash certainty equivalence principle. Commun Inf Syst 6(3):221–251

    MathSciNet  MATH  Google Scholar 

  41. Huang CFu Zariphopoulou T (1999) Turnpike behaviour of long-term investments. Finance Stoch 3:15–34

    Article  MathSciNet  MATH  Google Scholar 

  42. Huberman G, Ross S (1983) Portfolio turnpike theorems, risk aversion and regularly varying functions. Econometrica 51:1345–1361

    Article  MathSciNet  MATH  Google Scholar 

  43. Jaśkiewicz A (2002) Zero-sum semi-Markov games. SIAM J Control Optim 41:723–739

    Article  MathSciNet  MATH  Google Scholar 

  44. Jaśkiewicz A (2009) Zero-sum ergodic semi-Markov games with weakly continuous transition probabilities. J Optim Theory Appl 141:321–347

    Article  MathSciNet  MATH  Google Scholar 

  45. Jaśkiewicz A, Nowak AS (2006) Zero-sum ergodic stochastic games with feller transition probabilities. SIAM J Control Optim 45:773–789

    Article  MathSciNet  MATH  Google Scholar 

  46. Joshi S (1997) Turnpike theorems in nonconvex nonstationary environments. Int Econ Rev 38(1):225–248

    Article  MATH  Google Scholar 

  47. Joshi S (2003) The stochastic turnpike property without uniformity in convex aggregate growth models. J Econ Dyn Control 27:1289–1315

    Article  MATH  Google Scholar 

  48. Khan MA, Piazza A (2011) An overview of turnpike theory: towards the discounted deterministic case. Adv Math Econ 14:39–67

    Article  Google Scholar 

  49. Kolokoltsov VN (1989) Turnpikes and infinite extremals in Markov decision processes. Mat Zametki 46(4):118–120 (in Russian)

    MathSciNet  Google Scholar 

  50. Kolokoltsov VN (1992) On linear, additive, and homogeneous operators. In: Maslov VP, Samborski SN (eds) Idempotent analysis. Advances in soviet mathematics, vol 13, pp 87–101

    Google Scholar 

  51. Kolokoltsov VN, Malafeyev OA (2010) Understanding game theory: introduction to the analysis of many agent systems of competition and cooperation. World Scientific, Singapore

    Book  MATH  Google Scholar 

  52. Kolokoltsov VN (2010) Nonlinear Markov processes and kinetic equations. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  53. Kolokoltsov VN (2012) Nonlinear Markov games on a finite state space (mean-field and binary interactions). Int J Stat Probab 1(1):77–91. http://www.ccsenet.org/journal/index.php/ijsp/article/view/16682

    Google Scholar 

  54. Kolokoltsov VN, Li J, Yang W (2012) Mean field games and nonlinear Markov processes. arXiv:1112.3744v2

  55. Küenle HU (2007) On Markov games with average reward criterion and weakly continuous transition probabilities. SIAM J Control Optim 45:2156–2168

    Article  MathSciNet  MATH  Google Scholar 

  56. Leland H (1972) On turnpike portfolios. In: Szego G, Shell K (eds) Mathematical methods in investment and finance. North-Holland, Amsterdam

    Google Scholar 

  57. Guerrero-Luchtenberg, LC (2000) A uniform neighborhood turnpike theorem and applications. J Nath Econ 34(3):329–357

    Article  MathSciNet  MATH  Google Scholar 

  58. Le Van C, Morhaim L (2006) On optimal growth models when the discount factor is near 1 or equal to 1. Int J Econ Theory 2(1):55–76

    Article  Google Scholar 

  59. Maitra A, Sudderth W (1993) Borel stochastic games with limsup payoffs. Ann Probab 21:861–885

    Article  MathSciNet  MATH  Google Scholar 

  60. Majumdar M, Mitra T (1982) Intertemporal allocation with a non-convex technology: the aggregative framework. J Econ Theory 27:101–136

    Article  MathSciNet  MATH  Google Scholar 

  61. Majumdar M, Nermuth (1982) Dynamic optimisation in non-convex models with irreversible investment: monotonicity and Turnpike theorems. J Econ 42:339–362

    Article  MathSciNet  MATH  Google Scholar 

  62. Majumdar M, Zilcha I (1987) Optimal growth in a stochastic environment: some sensitivity and turnpike results. J Econ Theory 43:116–133

    Article  MathSciNet  MATH  Google Scholar 

  63. Makarov VL, Rubinov AM (1973) Mathematical theory of economic dynamics and equilibria. Nauka, Moscow (english transl.: Springer, New York)

    Google Scholar 

  64. Mamedov MA (2009) Asymptotical stability of optimal paths in nonconvex problems. In: Pearce C, Hunt E (eds) Optimization: structure and applications. Optimization and its applications, vol 32. Springer, Berlin, pp 95–134

    Google Scholar 

  65. Mamedov MA (2003) A turnpike theorem for continuous-time control systems when the optimal stationary point is not unique. Abstr Appl Anal 11:631–650

    Article  Google Scholar 

  66. Mamedov MA, Pehlivan S (2001) Statistical cluster points and Turnpike theorem in nonconvex problems. J Math Anal Appl 256:686–693

    Article  MathSciNet  MATH  Google Scholar 

  67. Marena M, Montrucchio L (1999) Neighborhood turnpike theorem for continuous time optimization models. J Optim Theory Appl 101:651–676

    Article  MathSciNet  MATH  Google Scholar 

  68. McKenzie LW (1963) The turnpike theorem of Morishima. Rev Econ Stud 30:169–176

    Article  Google Scholar 

  69. McKenzie LW (1976) Turnpike theory. Econometrica 44(5):841–865

    Article  MathSciNet  MATH  Google Scholar 

  70. McKenzie LW (1977) A new route to the turnpike. In: Henn R, Moeachlin O (eds) Mathematical economics and games theory. Springer, New York

    Google Scholar 

  71. Mckenzie LW (1979) Optimal economic growth and turnpike theorems. Discussion paper 79-1, University of Rochester

  72. McKenzie LW (1986) Optimal economic growth, and turnpike theorems and comparative dynamics. In: Handbook of mathematical economics, vol 3, pp 1281–1355

    Google Scholar 

  73. McKenzie LW (1998) Turnpikes. Am Econ Rev 88(2):1–14. Papers and proceedings of the hundred and tenth annual meeting of the American economic association

    Google Scholar 

  74. Mirman LJ, Zilcha I (1975) On optimal growth under uncertainty. J Econ Theory 11:329–339

    Article  MathSciNet  MATH  Google Scholar 

  75. Mirman LJ, Zilcha I (1977) Characterizing optimal policies in a one-sector model of economic growth under uncertainty. J Econ Theory 14:389–401

    Article  MathSciNet  MATH  Google Scholar 

  76. Montrucchio L (1995) A turnpike theorem for continuous-time optimal-control models. J Econ Dyn Control 19:599–619

    Article  MathSciNet  MATH  Google Scholar 

  77. Montrucchio L (1995) A new turnpike theorem for discounted programs. Econ Theory 5:371–382

    Article  MathSciNet  MATH  Google Scholar 

  78. Morishima M (1961) Proof of a turnpike theorem: the no joint production case. Rev Econ Stud 28:89–97

    Article  Google Scholar 

  79. Mossin J (1968) Optimal multiperiod policies. J Bus 31:215–229

    Article  Google Scholar 

  80. Nowak AS (1999) Sensitivity equilibrium for ergodic stochastic games with countable state space. Math Methods Oper Res 50:65–76

    MathSciNet  MATH  Google Scholar 

  81. Nowak AS (2008) Equilibrium in a dynamic game of capital accumulation with the overtaking criterion. Econ Lett 99:233–237

    Article  Google Scholar 

  82. Park H (2000) Global asymptotic stability of a competitive equilibrium with recursive preferences. Econ Theory 15(3):565–584

    Article  MATH  Google Scholar 

  83. Pehlivan S, Mamedov MA (2000) Statistical cluster points and turnpike. Optimization 48(1):91–106

    Article  MathSciNet  Google Scholar 

  84. Radner R (1961) Paths of economic growth that are optimal with regard only to final states. Rev Econ Stud 28:98–104

    Article  Google Scholar 

  85. Rapaport A, Cartigny P (2004) Turnpike theorems by a value function approach. In: ESAIM: control, optimisation and calculus of variations, vol 10, pp 123–141

    Google Scholar 

  86. Samuelson PA, Solow RM (1956) A complete capital model involving heterogeneous capital goods. Q J Econ 27:537–562

    Article  Google Scholar 

  87. Scheinkman J (1976) On optimal steady states of n-sector growth models when utility is discounted. J Econ Theory 12:11–30

    Article  MathSciNet  MATH  Google Scholar 

  88. Shapiro JF (1968) Turnpike planning horizon for a Markov decision model. Manag Sci 14:292–300

    Article  MATH  Google Scholar 

  89. Sorin S (2005) New approaches and recent advances in two-person zero-sum repeated games. In: Nowak AS, Szajowski K (eds) Advances in dynamic games, pp 67–94

    Chapter  Google Scholar 

  90. Takashi K, Roy S (2006) Dynamic optimization with a non smooth, nonconvex technology: the case of a linear objective function. Econ Theory 29:325–340

    Article  MATH  Google Scholar 

  91. Vega-Amaya O (2003) Zero-sum semi-Markov games. Fixed-point solutions of the Shapley equation. SIAM J Control Optim 42:1876–1894

    Article  MathSciNet  MATH  Google Scholar 

  92. Veinott AF (1966) On finding optimal policies in discrete dynamic programming with no discounting. Ann Math Stat 37:1284–1294

    Article  MathSciNet  MATH  Google Scholar 

  93. Yakovenko SYu Kontorer LA (1992) Nonlinear semigroups and infinite horizon optimization. In: Maslov VP, Samborski SN (eds) Idempotent analysis. Advances in Soviet Mathematics, vol 13, pp 167–210

    Google Scholar 

  94. Yano M (1984) Competitive equilibria on turnpikes in a McKenzie economy, I: A neighborhood turnpike theorem. Int Econ Rev 25(3):695–717

    Article  MATH  Google Scholar 

  95. Zaslavski AJ (1995) Optimal programs on infinite horizon 1. SIAM J Control Optim 33:1643–1660

    Article  MathSciNet  MATH  Google Scholar 

  96. Zaslavski AJ (1995) Optimal programs on infinite horizon 2. SIAM J Control Optim 33:1661–1686

    Article  MathSciNet  MATH  Google Scholar 

  97. Zaslavski AJ (1999) The turnpike property for dynamic discrete time zero-sum games. Abstr Appl Anal 4(1):21–48

    Article  MathSciNet  MATH  Google Scholar 

  98. Zaslavski AJ (1998) Turnpike theorem for convex infinite dimensional. J Convex Anal 5(2):237–248

    MathSciNet  MATH  Google Scholar 

  99. Zaslavski AJ (2004) Turnpike theorem for a class of discrete time optimal control problem. Fixed Point Theory Appl 5:175–182

    Google Scholar 

  100. Zaslavski AJ (2006) Turnpike properties in the calculus of variations and optimal control. Springer, New York

    MATH  Google Scholar 

  101. Zaslavski AJ (2009) Two turnpike results for a discrete-time optimal control system. Nonlinear Anal 71:902–909

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

Supported by the AFOSR grant FA9550-09-1-0664 ‘Nonlinear Markov control processes and games’. The authors would like to thank the anonymous referee for helpful comments and suggestions.

Author information

Authors and Affiliations

  1. Department of Statistics, University of Warwick, Coventry, CV4 7AL, UK

    Vassili Kolokoltsov & Wei Yang

Authors
  1. Vassili Kolokoltsov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vassili Kolokoltsov.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kolokoltsov, V., Yang, W. Turnpike Theorems for Markov Games. Dyn Games Appl 2, 294–312 (2012). https://doi.org/10.1007/s13235-012-0047-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13235-012-0047-6

Keywords

Search

Navigation