Annals of Operations Research

, Volume 29, Issue 1, pp 417–426 | Cite as

On an extremal property of Markov chains and sufficiency of Markov strategies in Markov decision processes with the Dubins-Savage criterion

  • I. M. Sonin
Borel State Space


An inequality regarding the minimum ofP(lim inf(X n εD n )) is proved for a class of random sequences. This result is related to the problem of sufficiency of Markov strategies for Markov decision processes with the Dubins-Savage criterion, the asymptotical behaviour of nonhomogeneous Markov chains, and some other problems.


Markov decision process Markov strategy Dubins-Savage criterion nonhomogeneous Markov chain 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    D. Blackwell, Finite nonhomogeneous Markov chains, Ann. Math. 46 (1945) 594–599.Google Scholar
  2. [2]
    H. Cohn, Finite nonhomogeneous Markov chains: asymptotic behaviour, Adv. Appl. Prob. 8 (1976) 502–516.Google Scholar
  3. [3]
    H. Cohn, Products of stochastic matrices and applications, Int. J. Math. Sci. 12 (1989) 209–233.Google Scholar
  4. [4]
    E. Dynkin and A. Yushkevich,Controlled Markov processes (Springer, 1979).Google Scholar
  5. [5]
    E.A. Feinberg, Nonrandomized Markov and semi-Markov strategies in dynamic programming, Theor. Prob. App. 27 (1982) 116–126.Google Scholar
  6. [6]
    T. Hill, On the existence of good Markov strategies, Trans. Am. Math. Soc. 247 (1979) 157–176.Google Scholar
  7. [7]
    T. Hill and V. Pestien, The existence of good Markov strategies for decision processes with general payoffs, Stoch. Proc. Appl. 23 (1986) 412.Google Scholar
  8. [8]
    W. Hopp, J. Bean and R. Smith, A new optimality criterion for nonhomogeneous Markov decision processes, Oper. Res. 35 (1987) 875–883.Google Scholar
  9. [9]
    A.N. Kolmogoroff, Zur Theorie der Markoffschen Ketten, Math. Ann. Bd. 112 (1936).Google Scholar
  10. [10]
    I.M. Sonin, A theorem on separation of jets and some properties of random sequences, Stochastics 21 (1987) 231–250.Google Scholar
  11. [11]
    I.M. Sonin, The separation of jets and some asymptotic properties of random sequences,Discrete Event Systems: Models and Applications, Lecture Notes in Contr. Inf., vol. 103 (Springer, 1988) pp. 275–282.Google Scholar
  12. [12]
    I.M. Sonin, The separation of jets and generalization into nonhomogeneous case of the theorem on decomposition of finite Markov chain into ergodic components, Dokl. Acad. Sci. 304 (1989) 36–40 (in Russian).Google Scholar
  13. [13]
    H. Thorisson, Backward limits, Ann. Prob. 16 (1988) 914–924.Google Scholar

Copyright information

© J.C. Baltzer A.G. Scientific Publishing Company 1991

Authors and Affiliations

  • I. M. Sonin
    • 1
  1. 1.Central Economic Mathematics InstituteUSSR Academy of SciencesMoscowUSSR

Personalised recommendations