Skip to main content
SpringerLink
Log in

Semi-Markov Control Models with Partially Known Holding Times Distribution: Discounted and Average Criteria

  • Published:
Acta Applicandae Mathematicae Aims and scope Submit manuscript

Abstract

The paper deals with a class of semi-Markov control models with Borel state and control spaces and possibly unbounded costs, where the holding times distribution F depends on an unknown and possibly non-observable parameter which may change from stage to stage. The system is modeled as a game against nature, which is a particular case of a minimax control system. The objective is to show the existence of minimax strategies under the discounted and average cost criteria.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Altman, E., Hordijk, A.: Zero-sum Markov games and worst-case optimal control of queueing systems. Queueing Syst. Theory Appl. 21, 415–447 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  2. Coraluppi, S.P., Marcus, S.I.: Mixed risk-neutral/minimax control of discrete-time finite state Markov decision process. IEEE Trans. Autom. Control 45, 528–532 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  3. Dynkin, E.B., Yushkevich, A.A.: Controlled Markov Processes. Springer, New York (1979)

    Google Scholar 

  4. Federgruen, A., Tijms, H.C.: The optimality equation in average cost denumerable state semi-Markov decision problems. Recurrence conditions and algorithms. J. Appl. Probab. 15, 356–373 (1978)

    Article  MathSciNet  MATH  Google Scholar 

  5. Federgruen, A., Schweitzer, P.J., Tijms, H.C.: Denumerable undiscounted semi-Markov decision processes with unbounded rewards. Math. Oper. Res. 8, 298–313 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  6. González-Trejo, T.J., Hernández-Lerma, O., Hoyos-Reyes, L.F.: Minimax control of discrete-time stochastic systems. SIAM J. Control Optim. 41, 1626–1659 (2003)

    Article  MATH  Google Scholar 

  7. Hernández-Lerma, O., Lasserre, J.B.: Discrete-Time Markov Control Processes: Basic Optimality Criteria. Springer, New York (1996)

    Google Scholar 

  8. Hordijk, A., Passchier, O., Spieksma, F.M.: Optimal control against worst case admission policies: A multichained stochastic game. Math. Methods Oper. Res. 45, 281–301 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  9. Jagannathan, R.: A minimax ordering policy for the infinite stage dynamic inventory problem. Manag. Sci. 24, 1138–1149 (1978)

    Article  MathSciNet  MATH  Google Scholar 

  10. Jaskiewicz, A.: An approximation approach to ergodic semi-Markov control processes. Math. Methods Oper. Res. 54, 1–19 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  11. Jaskiewicz, A.: A fixed point approach to solve the average cost optimality equation for semi-Markov decision processes with Feller transition probabilities. Commun. Stat., Theory Methods 36, 2559–2575 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  12. Kalyanasundaram, S., Chong, E.K.P., Shroff, N.B.: Markov decision processes with uncertain transition rates: sensitivity and max-min control. Asian J. Control 6(2), 253–269 (2004)

    Article  Google Scholar 

  13. Küenle, H.-U.: Stochastiche Spiele und Entscheidungsmodelle. B. G. Teubner, Leipzig (1986)

    Google Scholar 

  14. Küenle, H.-U.: On the optimality of (s,S)-strategies in a minimax inventory model with average cost criterion. Optimization 22, 123–138 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  15. Kurano, M.: Discrete-time Markovian decision processes with an unknown parameter-average return criterion. J. Oper. Res. Soc. Jpn. 15, 67–76 (1972)

    MathSciNet  MATH  Google Scholar 

  16. Kurano, M.: Minimax strategies for average cost stochastic games with an application to inventory models. J. Oper. Res. Soc. Jpn. 30, 232–247 (1987)

    MathSciNet  MATH  Google Scholar 

  17. Luque-Vásquez, F., Hernández-Lerma, O.: Semi-Markov models with average costs. Appl. Math. 26, 315–331 (1999)

    MathSciNet  MATH  Google Scholar 

  18. Luque-Vásquez, F., Minjárez-Sosa, J.A.: Semi-Markov control processes with unknown holding times distribution under a discounted criterion. Math. Methods Oper. Res. 61, 455–468 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  19. Luque-Vásquez, F., Minjárez-Sosa, J.A., Rosas-Rosas, L.C.: Semi-Markov control processes with unknown holding times distribution under an average cost criterion. Appl. Math. Optim. 61, 217–336 (2010)

    Article  Google Scholar 

  20. Mandl, P.: Estimation and control in Markov chains. Adv. Appl. Probab. 6, 40–60 (1974)

    Article  MathSciNet  MATH  Google Scholar 

  21. Milliken, P., Marsh, C., Van Brunt, B.: Minimax controller design for a class of uncertain nonlinear systems. Automatica 35, 583–590 (1999)

    Article  MATH  Google Scholar 

  22. Puterman, M.L.: Markov Decision Processes. Discrete Stochastic Dynamic Programming. Wiley, New York (1994)

    Book  MATH  Google Scholar 

  23. Rieder, U.: Measurable selection theorems for optimization problems. Manuscripta Math. 115–131 (1978)

  24. Savkin, A.V., Peterson, I.R.: Minimax optimal control of uncertain systems with structured uncertainty. Int. J. Robust Nonlinear Control 5, 119–137 (1995)

    Article  MATH  Google Scholar 

  25. Schäl, M.: Conditions for optimality and for the limit on n-stage optimal policies to be optimal. Z. Wahrscheinlichkeitstheor. Verw. Geb. 32, 179–196 (1975)

    Article  MATH  Google Scholar 

  26. Schweitzer, P.J.: Iterative solution of the functional equations of undiscounted Markov renewal programming. J. Math. Anal. Appl. 34, 495–501 (1971)

    Article  MathSciNet  MATH  Google Scholar 

  27. Vega-Amaya, O.: The average cost optimality equation: a fixed point approach. Bol. Soc. Mat. Mexicana 9, 185–195 (2003)

    MathSciNet  MATH  Google Scholar 

  28. Yu, W., Guo, X.: Minimax controller design for discrete-time time-varying stochastic systems. In: Proceedings of the 41st IEEE CDC, Las Vegas, Nevada, USA, pp. 598–603 (2002)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Departamento de Matemáticas, Universidad de Sonora, Rosales s/n, Col. Centro, 83000, Hermosillo, Sonora, Mexico

    Fernando Luque-Vásquez, J. Adolfo Minjárez-Sosa & Luz del Carmen Rosas-Rosas

Authors
  1. Fernando Luque-Vásquez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. J. Adolfo Minjárez-Sosa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luz del Carmen Rosas-Rosas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to J. Adolfo Minjárez-Sosa.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Luque-Vásquez, F., Minjárez-Sosa, J.A. & Rosas-Rosas, L.d.C. Semi-Markov Control Models with Partially Known Holding Times Distribution: Discounted and Average Criteria. Acta Appl Math 114, 135–156 (2011). https://doi.org/10.1007/s10440-011-9605-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10440-011-9605-y

Keywords

Mathematics Subject Classification (2000)

Search

Navigation