Skip to main content

Advertisement

SpringerLink
Log in
Menu
Find a journal Publish with us
Search
Cart
  1. Home
  2. Probability Theory and Related Fields
  3. Article
A convex analytic approach to Markov decision processes
Download PDF
Download PDF
  • Published: August 1988

A convex analytic approach to Markov decision processes

  • Vivek S. Borkar1 nAff2 

Probability Theory and Related Fields volume 78, pages 583–602 (1988)Cite this article

  • 608 Accesses

  • 65 Citations

  • 1 Altmetric

  • Metrics details

Summary

This paper develops a new framework for the study of Markov decision processes in which the control problem is viewed as an optimization problem on the set of canonically induced measures on the trajectory space of the joint state and control process. This set is shown to be compact convex. One then associates with each of the usual cost criteria (infinite horizon discounted cost, finite horizon, control up to an exit time) a naturally defined occupation measure such that the cost is an integral of some function with respect to this measure. These measures are shown to form a compact convex set whose extreme points are characterized. Classical results about existence of optimal strategies are recovered from this and several applications to multicriteria and constrained optimization problems are briefly indicated.

Download to read the full article text

Working on a manuscript?

Avoid the common mistakes

References

  1. Billingsley, P.: Convergence of probability measures. New York; Wiley 1968

    Google Scholar 

  2. Bertsekas, D.P.: Dynamic Programming and stochastic control. New York: Academic 1976

    Google Scholar 

  3. Borkar, V.S.: On minimum cost per unit time control of Markov chains, SIAM J. Control Optimization 22, 965–978 (1984)

    Google Scholar 

  4. Borkar, V.S.: Control of Markov chains with long-run average cost criterion. In: Fleming, W., Lions, P.L. (eds.) Stochastic differential systems, stochastic control theory and applications, IMA vol. 10, pp. 57–77. Berlin Heidelberg New York: Springer 1988

    Google Scholar 

  5. Phelps, R.: Lectures on Choquet's theorem. New York: Nostrand 1966

    Google Scholar 

  6. Makowski, A. Schwartz, A.: Implementation issues for Markov decision processes. In: Fleming, W., Lions, P.L. (eds.). Stochastic differential systems, stochastic control theory and applications, IMA vol. 10, pp. 323–337. Berlin Heidelberg New York: Springer 1988

    Google Scholar 

  7. Ross, S.: Introduction to stochastic dynamic programming. New York: Academic 1984

    Google Scholar 

Download references

Author information

Author notes
  1. Vivek S. Borkar

    Present address: Bangalore Center, Tata Institute of Fundamental Research, I.I.Sc. Campus, P.O. Box 1234, 560012, Bangalore, India

Authors and Affiliations

  1. Systems Research Center, University of Maryland, 20742, College Park, MD, USA

    Vivek S. Borkar

Authors
  1. Vivek S. Borkar
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Research supported by NSF Grant CDR-85-00108

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Borkar, V.S. A convex analytic approach to Markov decision processes. Probab. Th. Rel. Fields 78, 583–602 (1988). https://doi.org/10.1007/BF00353877

Download citation

  • Received: 15 March 1987

  • Revised: 08 January 1988

  • Issue Date: August 1988

  • DOI: https://doi.org/10.1007/BF00353877

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Keywords

  • Control Problem
  • Optimal Strategy
  • Statistical Theory
  • Extreme Point
  • Classical Result
Download PDF

Working on a manuscript?

Avoid the common mistakes

Advertisement

Search

Navigation

  • Find a journal
  • Publish with us

Discover content

  • Journals A-Z
  • Books A-Z

Publish with us

  • Publish your research
  • Open access publishing

Products and services

  • Our products
  • Librarians
  • Societies
  • Partners and advertisers

Our imprints

  • Springer
  • Nature Portfolio
  • BMC
  • Palgrave Macmillan
  • Apress
  • Your US state privacy rights
  • Accessibility statement
  • Terms and conditions
  • Privacy policy
  • Help and support

167.114.118.210

Not affiliated

Springer Nature

© 2023 Springer Nature