Applied Mathematics & Optimization

, Volume 66, Issue 2, pp 257–271 | Cite as

Optimal Control of Markov Processes with Age-Dependent Transition Rates

  • Mrinal K. GhoshEmail author
  • Subhamay Saha


We study optimal control of Markov processes with age-dependent transition rates. The control policy is chosen continuously over time based on the state of the process and its age. We study infinite horizon discounted cost and infinite horizon average cost problems. Our approach is via the construction of an equivalent semi-Markov decision process. We characterise the value function and optimal controls for both discounted and average cost cases.


Age-dependent transition rates Semi-Markov decision process Infinite horizon discounted cost Infinite horizon average cost 


  1. 1.
    Davis, M.H.A.: Markov Models and Optimization. Chapman and Hall, London (1993) zbMATHGoogle Scholar
  2. 2.
    Ethier, S.N., Kurtz, T.G.: Markov Processes: Characterization and Convergence. Wiley, New York (1986) zbMATHGoogle Scholar
  3. 3.
    Finberg, E.A., Yushkevich, A.A.: Homogeneous controllable Markov models with continuous time and with a finite or countable state space. Teor. Veroâtn. Ee Primen. 24, 155–160 (1979) Google Scholar
  4. 4.
    Federgruen, A., Hordijk, A., Tijms, H.C.: Denumerable state semi-Markov decision processes with unbounded costs, average cost criteria. Stoch. Process. Appl. 9, 223–235 (1979) MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Federgruen, A., Hordijk, A., Tijms, H.C.: Denumerable undiscounted semi-Markov decision processes with unbounded rewards. Math. Oper. Res. 8, 298–313 (1983) MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Ghosh, M.K., Goswami, A.: Risk minimising option pricing in a semi-Markov modulated market. SIAM J. Control Optim. 48, 1519–1541 (2009) MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Guo, X., Hernández-Lerma, O.: Continuous-time controlled Markov chains. Ann. Appl. Probab. 13, 363–388 (2003) MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Guo, X., Hernández-Lerma, O.: Continuous-Time Markov Decision Processes. Theory and Applications. Springer, Berlin (2009) zbMATHCrossRefGoogle Scholar
  9. 9.
    Guo, X., Hernández-Lerma, O., Prieto-Rumeau, T.: A survey of recent results on continuous-time Markov decision processes. Top 14, 177–261 (2006) MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Hordijk, A., Van Der Duyn Schouten, F.A.: Average optimal policies in Markov decision drift processes with applications to a queueing and a replacement model. Adv. Appl. Probab. 15, 274–303 (1983) zbMATHCrossRefGoogle Scholar
  11. 11.
    Hordijk, A., Van Der Duyn Schouten, F.A.: Discretization and weak convergence in Markov decision drift processes. Math. Oper. Res. 9, 112–141 (1984) MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Hordijk, A., Van Der Duyn Schouten, F.A.: Markov decision drift processes; conditions for optimality obtained by discretization. Math. Oper. Res. 10, 160–173 (1985) MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Ikeda, N., Watanabe, S.: Stochastic Differential Equations and Diffusion Processes. North-Holland, Amsterdam (1989) zbMATHGoogle Scholar
  14. 14.
    Pliska, S.R.: Controlled jump processes. Stoch. Process. Appl. 3, 259–282 (1975) MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    Ross, S.M.: Applied Probability Models with Optimization Applications. Dover, New York (1992) zbMATHGoogle Scholar
  16. 16.
    Wakuta, K.: Arbitrary state semi-Markov decision processes with unbounded rewards. Optimization 18, 447–454 (1987) MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  1. 1.Department of MathematicsIndian Institute of ScienceBangalore-12India

Personalised recommendations