Abstract
This paper investigates the problem of the optimal switching among a finite number of Markov processes, generalizing some of the author's earlier results for controlled one-dimensional diffusion. Under rather general conditions, it is shown that the optimal discounted cost function is the unique solution of a functional equation. Under more restrictive assumptions, this function is shown to be the unique solution of some quasi-variational inequalities. These assumptions are verified for a large class of control problems. For controlled Markov chains and controlled one-dimensional diffusion, the existence of a stationary optimal policy is established. Finally, a policy iteration method is developed to calculate an optimal stationary policy, if one exists.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Robin, M.,Some Optimal Control Problems for Queueing Systems, Mathematical Programming Studies, Vol. 6, pp. 154–169, 1975.
Doshi, B. T.,Optimal Control of a Diffusion Process with Reflecting Boundaries and Both Continuous and Lump Costs, Dynamic Programming and Its Applications, Edited by M. Puterman, Academic Press, New York, New York, 1978.
Sheng, D.,Some Problems in the Optimal Control of Diffusions, Stanford University, Department of Operations Research, Technical Report No. 50, 1978.
Chernoff, H., andPetkau, J.,Optimal Control of a Brownian Motion, University of British Columbia, Technical Report, 1977.
Zabczyk, J.,Optimal Control by Means of Switching, Studia Mathematica, Vol. 65, pp. 161–171, 1973.
Zabczyk, J.,Introduction to the Theory of Optimal Stopping, Proceedings of Conference on Stochastic Control, Bonn, Germany, 1979.
Robin, M.,Controle Impulsionnel des Processus de Markov, IRIA, These de Doctorat, 1978.
Lepeltier, J. P., andMarchal, B.,Techniques Probabilistes dans le Control Impulsionnel, Stochastics, Vol. 2, pp. 243–287, 1979.
Bismut, J. M., Contsole de Processus Alternants et Applications, Wahrschenlickhetstheorie verw Gebiete, Vol. 47, pp. 241–288, 1979.
Blumenthal, R. M., andGetoor, R. K.,Markov Processes and Potential Theory, Academic Press, New York, New York, 1968.
Dynkin, E. B.,Markov Processes, Vols. 1 and 2, Academic Press, New York, New York, 1965.
Dynkin, E. G.,Optimal Choice of a Stopping Time for a Markov Process, Doklady Akademiya Nauk, SSSR, Vol. 150, pp. 238–240, 1963.
Friedman, A., andRobin, M.,The Free Boundary for Variational Inequality with Nonlocal Operators, SIAM Journal on Control, Vol. 16, pp. 347–372, 1978.
Bensoussan, A.,Optimal Impulse Control Theory, Proceedings of the Workshop on Stochastic Control, Bonn, Germany, 1979.
Bensoussan, A., andLions, J. L.,Nouvelles Methodes in Controle Impulsionnel, Applied Mathematics and Optimization, Vol. 1, pp. 289–312, 1979.
DeLeve, G., Federgruen, A., andTijms, H.,A General Markov Decision Model, I, Model and Techniques, Advances in Applied Probability, Vol. 10, 1979.
Doshi, B. T.,Controlled One-Dimensional Diffusions with Switching Costs—Average Cost Criterion, Stochastic Processes and Their Applications, Vol. 8, pp. 211–227, 1978.
Author information
Authors and Affiliations
Additional information
Communicated by P. Varaiya
This research was sponsored by the Air Force Office of Scientific Research (AFSC), United States Air Force, under Contract No. F-49620-79-C-0165.
The author would like to thank the referee for bringing Refs. 7, 8, and 9 to his attention.
Rights and permissions
About this article
Cite this article
Doshi, B. Optimal switching among a finite number of Markov processes. J Optim Theory Appl 35, 581–610 (1981). https://doi.org/10.1007/BF00934933
Issue Date:
DOI: https://doi.org/10.1007/BF00934933