Journal of Optimization Theory and Applications

, Volume 35, Issue 4, pp 581–610 | Cite as

Optimal switching among a finite number of Markov processes

  • B. Doshi
Contributed Papers


This paper investigates the problem of the optimal switching among a finite number of Markov processes, generalizing some of the author's earlier results for controlled one-dimensional diffusion. Under rather general conditions, it is shown that the optimal discounted cost function is the unique solution of a functional equation. Under more restrictive assumptions, this function is shown to be the unique solution of some quasi-variational inequalities. These assumptions are verified for a large class of control problems. For controlled Markov chains and controlled one-dimensional diffusion, the existence of a stationary optimal policy is established. Finally, a policy iteration method is developed to calculate an optimal stationary policy, if one exists.

Key Words

Markov decision processes optimal switching discounted costs quasi-variational inequalities stationary optimal policy policy iteration 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Robin, M.,Some Optimal Control Problems for Queueing Systems, Mathematical Programming Studies, Vol. 6, pp. 154–169, 1975.Google Scholar
  2. 2.
    Doshi, B. T.,Optimal Control of a Diffusion Process with Reflecting Boundaries and Both Continuous and Lump Costs, Dynamic Programming and Its Applications, Edited by M. Puterman, Academic Press, New York, New York, 1978.Google Scholar
  3. 3.
    Sheng, D.,Some Problems in the Optimal Control of Diffusions, Stanford University, Department of Operations Research, Technical Report No. 50, 1978.Google Scholar
  4. 4.
    Chernoff, H., andPetkau, J.,Optimal Control of a Brownian Motion, University of British Columbia, Technical Report, 1977.Google Scholar
  5. 5.
    Zabczyk, J.,Optimal Control by Means of Switching, Studia Mathematica, Vol. 65, pp. 161–171, 1973.Google Scholar
  6. 6.
    Zabczyk, J.,Introduction to the Theory of Optimal Stopping, Proceedings of Conference on Stochastic Control, Bonn, Germany, 1979.Google Scholar
  7. 7.
    Robin, M.,Controle Impulsionnel des Processus de Markov, IRIA, These de Doctorat, 1978.Google Scholar
  8. 8.
    Lepeltier, J. P., andMarchal, B.,Techniques Probabilistes dans le Control Impulsionnel, Stochastics, Vol. 2, pp. 243–287, 1979.Google Scholar
  9. 9.
    Bismut, J. M., Contsole de Processus Alternants et Applications, Wahrschenlickhetstheorie verw Gebiete, Vol. 47, pp. 241–288, 1979.Google Scholar
  10. 10.
    Blumenthal, R. M., andGetoor, R. K.,Markov Processes and Potential Theory, Academic Press, New York, New York, 1968.Google Scholar
  11. 11.
    Dynkin, E. B.,Markov Processes, Vols. 1 and 2, Academic Press, New York, New York, 1965.Google Scholar
  12. 12.
    Dynkin, E. G.,Optimal Choice of a Stopping Time for a Markov Process, Doklady Akademiya Nauk, SSSR, Vol. 150, pp. 238–240, 1963.Google Scholar
  13. 13.
    Friedman, A., andRobin, M.,The Free Boundary for Variational Inequality with Nonlocal Operators, SIAM Journal on Control, Vol. 16, pp. 347–372, 1978.Google Scholar
  14. 14.
    Bensoussan, A.,Optimal Impulse Control Theory, Proceedings of the Workshop on Stochastic Control, Bonn, Germany, 1979.Google Scholar
  15. 15.
    Bensoussan, A., andLions, J. L.,Nouvelles Methodes in Controle Impulsionnel, Applied Mathematics and Optimization, Vol. 1, pp. 289–312, 1979.Google Scholar
  16. 16.
    DeLeve, G., Federgruen, A., andTijms, H.,A General Markov Decision Model, I, Model and Techniques, Advances in Applied Probability, Vol. 10, 1979.Google Scholar
  17. 17.
    Doshi, B. T.,Controlled One-Dimensional Diffusions with Switching Costs—Average Cost Criterion, Stochastic Processes and Their Applications, Vol. 8, pp. 211–227, 1978.Google Scholar

Copyright information

© Plenum Publishing Corporation 1981

Authors and Affiliations

  • B. Doshi
    • 1
  1. 1.Bell LaboratoriesHolmdel

Personalised recommendations