Zeitschrift für Operations Research

, Volume 39, Issue 2, pp 131–155 | Cite as

Controlled Markov processes on the infinite planning horizon: Weighted and overtaking cost criteria

  • Emmanuel Fernández-Gaucherand
  • Mrinal K. Ghosh
  • Steven I. Marcus


Stochastic control problems for controlled Markov processes models with an infinite planning horizon are considered, under some non-standard cost criteria. The classical discounted and average cost criteria can be viewed as complementary, in the sense that the former captures the short-time and the latter the long-time performance of the system. Thus, we study a cost criterion obtained as weighted combinations of these criteria, extending to a general state and control space framework several recent results by Feinberg and Shwartz, and by Krass et al. In addition, a functional characterization is given for overtaking optimal policies, for problems with countable state spaces and compact control spaces; our approach is based on qualitative properties of the optimality equation for problems with an average cost criterion.

Key words

Controlled Markov Processes Infinite Planning Horizon Weighted and Overtaking Cost Criteria 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Arapostathis A, Borkar VS, Fernández-Gaucherand E, Ghosh MK, Marcus SI (1993) Discrete-time controlled Markov processes with an average cost criterion: A survey. SIAM J Control Optim 31:282–344Google Scholar
  2. Bertsekas DP (1987) Dynamic programming: Deterministic and stochastic models. Prentice-Hall, Englewood CliffsGoogle Scholar
  3. Bertsekas DP, Shreve SE (1978) Stochastic optimal control: The discrete time case. Academic Press, New YorkGoogle Scholar
  4. Borkar VS (1983) Controlled Markov chains and stochastic networks. SIAM J Control Optim 21:652–666Google Scholar
  5. Borkar VS (1991) Topics in controlled Markov chains. Pitman Research Notes in Mathematics Series 240 Longman Scientific & Technical, UKGoogle Scholar
  6. Borkar VS, Ghosh MK (1991) Ergodic and adaptive control of nearest neighbor motions. Math Control Signals and Systems 4:81–98Google Scholar
  7. Cavazos-Cadena R, Hernández-Lerma O (1992) Equivalence of Lyapunov stability criteria in a class of Markov decision processes. Appl Math Optim 26:113–137Google Scholar
  8. Derman C (1970) Finite state markovian decision processes. Academic Press, New YorkGoogle Scholar
  9. Derman C, Strauch RE (1966) A note on memoryless rules for controlling sequential control processes. Ann Math Stat 37:276–278Google Scholar
  10. Dynkin EB, Yushkevich AA (1979) Controlled Markov processes. Springer-Verlag, New YorkGoogle Scholar
  11. Fernández-Gaucherand E, Arapostathis A, Marcus SI (1990) Remarks on the existence of solutions to the average cost optimality equation in Markov decision processes. Systems Control Lett 5:425–432Google Scholar
  12. Feinberg EA (1982) Controlled Markov processes with arbitrary numerical criteria. Theory Prob Applications 27:486–503Google Scholar
  13. Feinberg EA, Shwartz A (1992) Markov decision models with weighted discounted criteria. Preprint (to appear in Math. Operat. Res.)Google Scholar
  14. Filar JA, Vrieze OJ (1992) Weighted reward criteria in competitive Markov decision processes. ZOR — Methods and Models of Operations Research 36:343–358Google Scholar
  15. Gale D (1967) On optimal development in a multisector economy. Rev Econom Stud 34:1–19Google Scholar
  16. Ghosh MK (1990) Markov decision process with multiple costs. Operat Res Lett 9:257–260Google Scholar
  17. Ghosh MK, Marcus SI (1991) Infinite horizon controlled diffusion problems with some nonstandard criteria. J Mathematical Systems and Control 1:45–70Google Scholar
  18. Hinderer K (1970) Foundations of non-stationary dynamic programming with discrete time parameters. Lect Notes Oper Res Math Syst 33, Springer-Verlag BerlinGoogle Scholar
  19. Hernández-Lerma O (1989) Adaptive Markov control processes. Springer-Verlag New YorkGoogle Scholar
  20. Hernández-Lerma O, Lasserre JB (1990) Average cost optimal policies for Markov control processes with Borel state space and unbounded costs. Systems Control Lett 15:349–356Google Scholar
  21. Krass D, Filar JA, Sinha SS (1992) A weighted Markov decision process. Operat Res 40:1180–1187Google Scholar
  22. Leizarowitz A (1987) Infinite horizon optimization for finite state Markov chain. SIAM J Control Optim 25:1601–1618Google Scholar
  23. Leizarowitz A (1988) Controlled diffusion processes on infinite horizon with the overtaking criterion. Appl Math Optim 17:61–78Google Scholar
  24. Rieder U (1978) Measurable selection theorems for optimization problems. Manuscripta Mathematica 24:115–131Google Scholar
  25. Ritt RK, Sennott LI (1992) Optimal stationary policies in general state space Markov decision chains with finite action sets. Math Operat Res 17:901–909Google Scholar
  26. Ross SM (1983) Introduction to stochastic dynamic programming. Academic Press New YorkGoogle Scholar
  27. Royden HL (1968) Real analysis, 2nd ed Macmillan New YorkGoogle Scholar
  28. SchÄl M (1993) Average optimality in dynamic programming with general state space. Math Operat Res 18:163–172Google Scholar
  29. Stidham S (1988) Scheduling, routing, and flow control in stochastic networks. In: Stochastic Differential Systems, Stochastic Control Theory and Applications, Fleming W, Lions PL (eds) The IMA Volumes in Mathematics and Its Applications; Springer-Verlag Berlin 10:529–561Google Scholar
  30. Tijms HC (1986) Stochastic modelling and analysis: A computational approach. John Wiley ChichesterGoogle Scholar
  31. White DJ (1985) Real applications of Markov decision processes. Interfaces 15:73–83Google Scholar
  32. White DJ (1987) A selective survey of hypothetical applicatons of Markov decision processes. Dept. of Systems Engineering Report, University of Virginia, Charlottesville, VAGoogle Scholar
  33. White DJ (1988) Further real applications of Markov decision processes. Interfaces 18:55–61Google Scholar
  34. Von WeizsÄcker CC (1965) Existence of optimal programs of accumulation for an infinite time horizon. Rev Econom Stud 32:85–164Google Scholar
  35. Yakowitz S (1982) Dynamic programming applications in water resources. Water Resour Res 18:673–696Google Scholar
  36. Yushkevich AA (1973) On a class of strategies in general Markov decision models. Theory Prob Applications 18:777–779Google Scholar

Copyright information

© Physica-Verlag 1994

Authors and Affiliations

  • Emmanuel Fernández-Gaucherand
    • 1
  • Mrinal K. Ghosh
    • 2
  • Steven I. Marcus
    • 3
  1. 1.Systems & Industrial Engineering DepartmentThe University of ArizonaTucsonUSA
  2. 2.Department of MathematicsIndian Institute of ScienceBangaloreIndia
  3. 3.Institute for Systems Research and Electrical Engineering DepartmentThe University of MarylandCollege ParkUSA

Personalised recommendations