Skip to main content

Advertisement

SpringerLink
Log in
Menu
Find a journal Publish with us
Search
Cart
  1. Home
  2. Probability Theory and Related Fields
  3. Article
Stationary policies and Markov policies in Borel dynamic programming
Download PDF
Download PDF
  • Published: March 1987

Stationary policies and Markov policies in Borel dynamic programming

  • Manfred Schäl1 &
  • William Sudderth2 

Probability Theory and Related Fields volume 74, pages 91–111 (1987)Cite this article

  • 124 Accesses

  • 8 Citations

  • Metrics details

Summary

The question of the existence of good Markov [good stationary] policies is studied for a general class of Borel [stationary] dynamic programming models. It is shown, for example, that Markov [stationary] policies are uniformly adequate if every transition law is absolutely continuous with respect to a fixed measure [and the reward function is positive or the model satisfies certain compactness and continuity conditions].

Download to read the full article text

Working on a manuscript?

Avoid the common mistakes

References

  • Barbosa-Dantas, C.A.: The existence of stationary optimal plans. Dissertation. Univ. of California, Berkeley (1966)

  • Bertsekas, D.P., Shreve, S.E.: Stochastic optimal control. New York: Academic Press 1978

    Google Scholar 

  • Blackwell, D.: Positive dynamic programming. Proc. Fifth Berkeley Sympos. Math. Stat. Probab.1, 415–418. University of California (1965)

    Google Scholar 

  • Blackwell, D.: The stochastic processes of Borel gambling and dynamic programming. Ann. Stat.4, 370–374 (1976)

    Google Scholar 

  • Blackwell, D., Freedman, D., Orkin, M.: The optimal reward operator in dynamic programming. Ann. Probab.2, 926–941 (1974)

    Google Scholar 

  • Bodewig, H.-H.: Über dynamische Optimierung mit endlich additiven Maßen. Master thesis, University of Bonn (1979)

  • Brown, L.D., Purves, R.: Measurable selections of extrema. Ann. Stat.1, 902–912 (1973)

    Google Scholar 

  • Dawen, van R.: Stationäre Politiken in stochastischen Entscheidungsproblemen. Dissertation, Univ. of Bonn (1983). One part will appear in Math. of Op. Res. under the title: Pointwise and uniformly good stationary strategies for dynamic programming models

  • Dawen, van R.: On stationary strategies in positive stochastic 1 and 2 person games with general state space. ZAMM64, T327–328 (1984)

    Google Scholar 

  • Dawen, van R., Schäl, M.: On the existence of stationary optimal policies in Markov decision models. ZAMM63, T403–404 (1983a)

    Google Scholar 

  • Dubins, L.E., Savage, L.J.: How to gamble if you must. New York: McGraw-Hill 1965

    Google Scholar 

  • Dubins, L.E., Sudderth, W.: Persistently ε-optimal strategies. Math. Op. Res.2, 125–134 (1977a)

    Google Scholar 

  • Dubins, L.E., Sudderth, W.: Countably additive gambling and optimal stopping. Z. Wahrscheinlichkeitstheor. Verw. Geb.41, 59–72 (1977b)

    Google Scholar 

  • Dubins, L.E., Sudderth, W.: On stationary strategies for absolutely continuous houses. Ann. Probab.7, 461–467 (1979)

    Google Scholar 

  • Engelbert, A., Engelbert, H.J.: Optimal stopping and almost sure convergence of random sequences. Z. f. Wahrscheinlichkeitstheor. Verw. Geb.48, 309–325 (1979)

    Google Scholar 

  • Fainberg, E.A., Sonin, I.M.: Stationary and Markov policies in countable state dynamic programming. Lect. Notes Math.1021, 111–129. Berlin Heidelberg New York: Springer 1983

    Google Scholar 

  • Fainberg, E.A., Sonin, I.M.: Persistently nearly optimal strategies in stochastic dynamic programming. Statistics and control of stochastic processes, Proc. Steklov Semin., Moscow 1984, Transl. Ser. Math. Engl. (1985)

  • Federgruen, A., Hordijk, A., Tijms, H.C.: Denumerable state semi-Markov decision processes with unbounded costs. Average cost criterion. Stochastic Processes Appl.9, 223–235 (1979)

    Google Scholar 

  • Frid, E.B.: On a problem of D. Blackwell from the theory of dynamic programming, Theor. Probab. Appl.15, 719–722 (1976)

    Google Scholar 

  • Hinderer, K.: Foundations of non-stationary dynamic programming with discrete time-parameter. Lect. Notes Operat. Res. Math. Systems33. Berlin Heidelberg New York: Springer 1970

    Google Scholar 

  • Kertz, R.P.: Renewal plans and persistent optimality in countable additive gambling. Math. Op. Res.7, 361–382 (1982)

    Google Scholar 

  • Ornstein, D.: On the existence of stationary optimal strategies. Proc. Am. Math. Soc.20, 563–569 (1969)

    Google Scholar 

  • Schäl, M.: Conditions for optimality in dynamic programming and for the limit ofn-stage optimal policies to be optimal. Z. Wahrscheinlichkeitstheor. Verw. Geb.32, 179–196 (1975)

    Google Scholar 

  • Schäl, M.: Stationary policies in dynamic programming models under compactness assumptions. Math. Op. Res.8, 366–372 (1983)

    Google Scholar 

  • Strauch, R.E.: Negative dynamic programming. Ann. Math. Stat.37, 871–890 (1966)

    Google Scholar 

  • Sudderth, W.D.: On the existence of good stationary strategies. Trans. Am. Math. Soc.135, 399–414 (1965)

    Google Scholar 

  • Sudderth, W.D.: A ‘Fatou equation’ for randomly stopped variables. Ann. Math. Stat.42, 2143–2146 (1971)

    Google Scholar 

  • Wal, van der J.: On stationary strategies in countable state total reward Markov decision processes. Math. Op. Res.9, 290–300 (1984)

    Google Scholar 

  • Wal, van der J.: On uniformly nearly-optimal Markov strategies, pp. 461–467. Operations Research Proceedings 1982 (1983)

Download references

Author information

Authors and Affiliations

  1. Institut für Angewandte Mathematik, Universität Bonn, Wegeler Str. 6, D-5300, Bonn, Federal Republic of Germany

    Manfred Schäl

  2. School of Statistics, University of Minnesota, 270 Vincent Hall, 55455, Minneapolis, MN, USA

    William Sudderth

Authors
  1. Manfred Schäl
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. William Sudderth
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Research supported by “Deutsche Forschungsgemeinschaft, Sonderforschungsbereich 72”

Research supported by National Science Foundation Grant MCS 8100789

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Schäl, M., Sudderth, W. Stationary policies and Markov policies in Borel dynamic programming. Probab. Th. Rel. Fields 74, 91–111 (1987). https://doi.org/10.1007/BF01845641

Download citation

  • Received: 06 June 1984

  • Revised: 20 April 1986

  • Issue Date: March 1987

  • DOI: https://doi.org/10.1007/BF01845641

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Keywords

  • Stochastic Process
  • Probability Theory
  • Dynamic Programming
  • Programming Model
  • Stationary Policy
Download PDF

Working on a manuscript?

Avoid the common mistakes

Advertisement

Search

Navigation

  • Find a journal
  • Publish with us

Discover content

  • Journals A-Z
  • Books A-Z

Publish with us

  • Publish your research
  • Open access publishing

Products and services

  • Our products
  • Librarians
  • Societies
  • Partners and advertisers

Our imprints

  • Springer
  • Nature Portfolio
  • BMC
  • Palgrave Macmillan
  • Apress
  • Your US state privacy rights
  • Accessibility statement
  • Terms and conditions
  • Privacy policy
  • Help and support

167.114.118.210

Not affiliated

Springer Nature

© 2023 Springer Nature