Skip to main content
SpringerLink
Log in

Optimal Control of Markov Processes with Age-Dependent Transition Rates

  • Published:
Applied Mathematics & Optimization Aims and scope Submit manuscript

Abstract

We study optimal control of Markov processes with age-dependent transition rates. The control policy is chosen continuously over time based on the state of the process and its age. We study infinite horizon discounted cost and infinite horizon average cost problems. Our approach is via the construction of an equivalent semi-Markov decision process. We characterise the value function and optimal controls for both discounted and average cost cases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Davis, M.H.A.: Markov Models and Optimization. Chapman and Hall, London (1993)

    MATH  Google Scholar 

  2. Ethier, S.N., Kurtz, T.G.: Markov Processes: Characterization and Convergence. Wiley, New York (1986)

    MATH  Google Scholar 

  3. Finberg, E.A., Yushkevich, A.A.: Homogeneous controllable Markov models with continuous time and with a finite or countable state space. Teor. Veroâtn. Ee Primen. 24, 155–160 (1979)

    Google Scholar 

  4. Federgruen, A., Hordijk, A., Tijms, H.C.: Denumerable state semi-Markov decision processes with unbounded costs, average cost criteria. Stoch. Process. Appl. 9, 223–235 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  5. Federgruen, A., Hordijk, A., Tijms, H.C.: Denumerable undiscounted semi-Markov decision processes with unbounded rewards. Math. Oper. Res. 8, 298–313 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  6. Ghosh, M.K., Goswami, A.: Risk minimising option pricing in a semi-Markov modulated market. SIAM J. Control Optim. 48, 1519–1541 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  7. Guo, X., Hernández-Lerma, O.: Continuous-time controlled Markov chains. Ann. Appl. Probab. 13, 363–388 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  8. Guo, X., Hernández-Lerma, O.: Continuous-Time Markov Decision Processes. Theory and Applications. Springer, Berlin (2009)

    Book  MATH  Google Scholar 

  9. Guo, X., Hernández-Lerma, O., Prieto-Rumeau, T.: A survey of recent results on continuous-time Markov decision processes. Top 14, 177–261 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  10. Hordijk, A., Van Der Duyn Schouten, F.A.: Average optimal policies in Markov decision drift processes with applications to a queueing and a replacement model. Adv. Appl. Probab. 15, 274–303 (1983)

    Article  MATH  Google Scholar 

  11. Hordijk, A., Van Der Duyn Schouten, F.A.: Discretization and weak convergence in Markov decision drift processes. Math. Oper. Res. 9, 112–141 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  12. Hordijk, A., Van Der Duyn Schouten, F.A.: Markov decision drift processes; conditions for optimality obtained by discretization. Math. Oper. Res. 10, 160–173 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  13. Ikeda, N., Watanabe, S.: Stochastic Differential Equations and Diffusion Processes. North-Holland, Amsterdam (1989)

    MATH  Google Scholar 

  14. Pliska, S.R.: Controlled jump processes. Stoch. Process. Appl. 3, 259–282 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  15. Ross, S.M.: Applied Probability Models with Optimization Applications. Dover, New York (1992)

    MATH  Google Scholar 

  16. Wakuta, K.: Arbitrary state semi-Markov decision processes with unbounded rewards. Optimization 18, 447–454 (1987)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, Indian Institute of Science, Bangalore-12, India

    Mrinal K. Ghosh & Subhamay Saha

Authors
  1. Mrinal K. Ghosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Subhamay Saha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mrinal K. Ghosh.

Additional information

This work is supported in part by SPM fellowship of CSIR and in part by UGC Centre for Advanced Study.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ghosh, M.K., Saha, S. Optimal Control of Markov Processes with Age-Dependent Transition Rates. Appl Math Optim 66, 257–271 (2012). https://doi.org/10.1007/s00245-012-9171-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00245-012-9171-3

Keywords

Search

Navigation