Zeitschrift für Operations Research

, Volume 28, Issue 1, pp 17–27 | Cite as

On Markovian decision models with a finite skeleton

  • A. Wrobel


We present a class of countable state space Markovian decision models that can be investigated by means of an associated finite-state, finite-action reduced model which we call the skeleton. In particular, we obtain a turnpike theorem for the original model (Theorem 2 in Section 5) from a known turnpike theorem for the reduced finite model. For illustration, we present in detail an application of this approach to an inventory model (re-establishing a known turnpike result) and sketch analogous results for a cash-balance model and a growth model.


State Space Growth Model Original Model Decision Model Analogous Result 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Wir führen eine Klasse von Markovschen Entscheidungsmodellen mit abzählbarem Zustandsraum ein, die mittels eines verbundenen, reduzierten Modells mit endlichem Zustands- und Aktionsraum, welches wir das Skelett nennen, untersucht werden können. Insbesondere erhalten wir ein Turnpike Theorem für das ursprüngliche Modell (Theorem 2 im Abschnitt 5) von einem bekannten Turnpike Theorem für das reduzierte endliche Modell. Zur Erläuterung stellen wir im einzelnen eine Anwendung dieses Ansatzes für ein Lagerhaltungsmodell (Wiederherleitung eines bekannten Turnpike Ergebnisses) dar, und wir skizzieren analoge Ergebnisse für ein Kassenhaltungsmodell und ein Wachstumsmodell.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Anthonisse, J.M., andH. Tijms: Exponential convergence of products of stochastic matrices. J. Math. Anal. Appl.59, 1977, 360–364.Google Scholar
  2. Constantinides, G.M., andS.F. Richard: Existence of optimal simple policies for discounted-cost inventory and cash management in continuous time. Opns. Res.26, 1978, 620–636.Google Scholar
  3. Hinderer, K., andG. Hübner: An improvement of J.F. Shapiro's turnpike theorem for the horizon of finite stage discrete dynamic programs. Transaction of the Seventh Prague Conference on Math. Statistics, Stochastic Processes and Information Science, vol. A, Prague 1974, 245–257.Google Scholar
  4. Hordijk, A., andH. Tijms: Convergence results and approximations for optimal (s, S) policies. Man. Sci.20, 1974, 1432–1438.Google Scholar
  5. —: On a conjecture of Iglehart. Man. Sci.21, 1975, 1342–1345.Google Scholar
  6. Isaacson, D.L., andR.W. Madsen: Markov Chains. New York 1976.Google Scholar
  7. Kadelka, D.: On existence of optimal control limit policies in a model of optimal growth. Methods of Opns. Res.44, 1981, 341–346.Google Scholar
  8. Kaplan, R.S.: Stochastic growth models. Man. Sci.18, 1972, 249–264.Google Scholar
  9. Neave, E.H.: The stochastic cash balance problem with fixed costs for increases and decreases. Man. Sci.16, 1970, 472–490.Google Scholar
  10. Scarf, H.: The optimality of (S, s) policies in the dynamic inventory model. Mathematical Models in the Social Sciences. Ed. by K.J. Arrow, S. Karlin, and P. Suppes. Stanford 1960.Google Scholar
  11. Schäl, M.: On the optimality of (s, S)-policies in dynamic inventory model with finite horizon. SIAM J. Appl. Math.30, 1976, 528–537.Google Scholar
  12. Shapiro, J.F.: Turnpike planning horizons for a Markovian decision model. Man. Sci.14, 1968, 292–300.Google Scholar
  13. Seneta, E.: Coefficients of ergodicity: structure and applications. Adv. Appl. Prob.5, 1979, 576–590.Google Scholar
  14. Veinott, A.F., Jr.: On the optimality of (s, S) inventory policies: new conditions and a new proof. SIAM J. Appl. Math.14, 1966, 1067–1083.Google Scholar

Copyright information

© Physica-Verlag 1984

Authors and Affiliations

  • A. Wrobel
    • 1
  1. 1.Institute of Computer SciencePolish Academy of SciencesWarsaw PKiNPoland

Personalised recommendations