Skip to main content
SpringerLink
Log in

Optimal control problem regularization for the Markov process with finite number of states and constraints

  • Stochastic Systems, Queueing Systems
  • Published:
Automation and Remote Control Aims and scope Submit manuscript

Abstract

The optimal control problem is considered for a system given by the Markov chain with integral constraints. It is shown that the solution to the optimal control problem on the set of all predictable controls satisfies Markov property. This optimal Markov control can be obtained as a solution of the corresponding dual problem (in case if the regularity condition holds) or (in other case) by means of proposed regularization method. The problems arising due to the system nonregularity along with the way to cope with those problems are illustrated by an example of optimal control problem for a single channel queueing system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Low, S.H., Paganini, F., and Doyle, J.C., Internet Congestion Control, IEEE Control Syst. Mag., 2002, vol. 22, no. 1, pp. 28–43.

    Article  MATH  Google Scholar 

  2. Vasenin, V.A. and Simonova, G.I., Mathematical Models of Traffic Control in Internet. New Approaches Based of TCP/AQM Schemes, Autom. Remote Control, 2005, vol. 66, no. 8, pp. 1274–1286.

    Article  MathSciNet  MATH  Google Scholar 

  3. Kelly, F.P., Maulloo, A., and Tan, D., Rate Control in Communication Networks. Shadow Prices, Proportional Fairness and Stability, J. Oper. Res. Soc., 1998, vol. 49, pp. 237–252.

    Article  MATH  Google Scholar 

  4. Hordijk, A. and Spieksma, F., Constrained Admission Control to a Queueing System, Adv. Appl. Probab., 1989, vol. 21, no. 2, pp. 409–431.

    Article  MathSciNet  MATH  Google Scholar 

  5. Kitaev, M.Y. and Rykov, V.V., Controlled Queueing Systems, Boca Raton: CRC, 1995.

    MATH  Google Scholar 

  6. Hordijk, A. and van der Duyn Schouten, F.A., Average Optimal Policies in Markov Decision Drift Processes with Application to a Queueing and a Replacement Model, Adv. Appl. Probab., 1983, vol. 15, pp. 274–303.

    Article  MATH  Google Scholar 

  7. Yeh, L., A Repair Replacement Model, Adv. Appl. Probab., 1990, vol. 22, no. 2, pp. 494–497.

    Article  MathSciNet  MATH  Google Scholar 

  8. Piunovskiy, A.B., Bicriteria Optimization of a Queue with a Controlled Input Stream, Queueing Syst., 2004, vol. 48, pp. 159–184.

    Article  MathSciNet  MATH  Google Scholar 

  9. Abramov, V., Optimal Control of Large Dam, J. Appl. Probab., 2007, vol. 44, pp. 249–258.

    Article  MathSciNet  MATH  Google Scholar 

  10. Nandalal, K.D.W. and Bogardi, J.J., Dynamic Programming Based Operation of Reservoirs. Applicability and Limits. International Hydrology Series, Cambridge: Cambridge Univ. Press, 2007.

    Google Scholar 

  11. Miller, A., Miller, B., Popov, A., and Stepanyan, K., Towards the Development of Numerical Procedure for Control of Connected Markov Chains, in Proc. 5th Australian Control Conf. (AUCC), Gold Coast, 2015, 5–6 November, pp. 336–341.

    Google Scholar 

  12. Altman, E., Constrained Markov Decision Processes, Boca Raton: Chapman, 1999.

    MATH  Google Scholar 

  13. Bertsekas, D.P., and Sreve, S.E., Stochastic Optimal Control, New York: Academic, 1978. Translated under the title Stokhasticheskoe optimal’noe upravlenie, Moscow: Nauka, 1984.

    Google Scholar 

  14. Fainberg, E.A., Nonrandomized Markov and Semi-Markov Strategies in Dynamic Programming, Theory Probab. Appl., 1982, vol. 27, no. 1, pp. 116–126.

    Article  MathSciNet  Google Scholar 

  15. Serfozo, R., Optimal Control of Random Walks, Birth and Death Processes, and Queues, Adv. Appl. Probab., 1981, vol. 13, no. 1, pp. 61–83.

    Article  MathSciNet  MATH  Google Scholar 

  16. Howard, R., Dynamic Programming and Markov Processes, New York: Wiley, 1960.

    MATH  Google Scholar 

  17. Miller, B.L., Finite State Continuous Time Markov Decision Process with a Finite Planning Horizon, SIAM J. Control, 1968, vol. 6, no. 2, pp. 266–279.

    Article  MathSciNet  MATH  Google Scholar 

  18. Pliska, S., Controlled Jump Processes, Stoch. Process Appl., 1975, vol. 3, pp. 259–282.

    Article  MathSciNet  MATH  Google Scholar 

  19. Yushkevich, A.A., Controlled Markov Models with Countable State Space and Continuous Time, Theory Probab. Appl., 1977, vol. 22, no. 2, pp. 215–235.

    Article  MathSciNet  MATH  Google Scholar 

  20. Elliott, R.J., Aggoun, L., and Moore, J.B., Hidden Markov Models. Estimation and Control, New York: Springer, 1995.

    MATH  Google Scholar 

  21. Liptser, R.Sh. and Shiryaev, A.N., Statistics of Random Processes, New York: Springer, 1979.

    MATH  Google Scholar 

  22. Davis, M.H.A. and Elliott, R.J., Optimal Control of Jump Process, Z. Wahrscheinlichkeitstheorie verw. Gebiete, 1977, vol. 40, pp. 183–202.

    Article  MathSciNet  MATH  Google Scholar 

  23. Wan, C.B. and Davis, M.H.A., Existence of Optimal Control for Stochastic Jump Processes, SIAM J. Control Optim., 1979, vol. 17, no. 4, pp. 511–524.

    Article  MathSciNet  MATH  Google Scholar 

  24. Tidball, M.M., Lombardi, A., Pourtaller, O., et al., Continuity of Optimal Values and Solutions for Control of Markov Chains with Constraints, SIAM J. Control Optim., 2000, vol. 38, no. 4, pp. 1204–1222.

    Article  MathSciNet  MATH  Google Scholar 

  25. Piunovskii, A.B., Controllable Jump Model with Discounting under Constraints, Theory Probab. Appl., 1997, vol. 42, no. 1, pp. 51–72.

    Article  MathSciNet  Google Scholar 

  26. Piunovskiy, A.B., Optimal Control of Random Sequences in Problems with Constraints, Dordrecht: Kluwer, 1997.

    Book  MATH  Google Scholar 

  27. Mendiondo, M. and Stockbridge, R., Approximation of Infinite-dimensional Linear Programming Problems Which Arise in Stochastic Control, SIAM J. Control Optim., 1998, vol. 36, no. 4, pp. 1448–1472.

    Article  MathSciNet  MATH  Google Scholar 

  28. Miller, B.M., Miller, G.B., and Semenikhin, K.V., Methods to Design Optimal Control of Markov Process with Finite State Set in the Presence of Constraints, Autom. Remote Control, 2011, vol. 72, no. 2, pp. 323–341.

    Article  MathSciNet  MATH  Google Scholar 

  29. Gikhman, I.I. and Skorokhod, A.V., Controllable Random Processes, Kiev: Naukova Dumka, 1977.

    MATH  Google Scholar 

  30. Davis, M.H.A., Markov Models and Optimization, London: Chapman, 1993.

    Book  MATH  Google Scholar 

  31. Elliott, R.J., A Partially Observed Control Problem for Markov Chains, Appl. Math. Optim., 1992, vol. 25, pp. 151–169.

    Article  MathSciNet  MATH  Google Scholar 

  32. Fleming, W.H. and Rishel, R.W., Deterministic and Stochastic Optimal Control, New York: Springer, 1975. Translated under the title Optimal’noe upravlenie determinirovannymi i stokhasticheskimi sistemami, Moscow: Mir, 1978.

    Book  MATH  Google Scholar 

  33. Lee, E.B. and Marcus, L., Foundation of Optimal Control Theory, New York: Wiley, 1975.

    Google Scholar 

  34. Bremaud, P., Optimal Thinning of a Point Process, SIAM J. Control Optim., 1979, vol. 17, no. 2, pp. 222–230.

    Article  MathSciNet  MATH  Google Scholar 

  35. Miller, B., Optimization of Stochastic Networks via Stochastic Controls, in Proc. Int. Conf. “System Identification and Control Problems” (SICPRO’08), Moscow, 2008, January 28–31.

    Google Scholar 

  36. Miller, B., Access and Service Rate Control in Queueing Systems, in Proc. IFAC World Congr., Seoul, 2008, July 6–11.

    Google Scholar 

  37. Gonzalez-Hernandez, J. and Hernandez-Lerma, O., Extreme Points of Sets of Randomized Strategies in Constrained Optimization and Control Problems, SIAM J. Optim., 2005, vol. 15, no. 4, pp. 1085–1104.

    Article  MathSciNet  MATH  Google Scholar 

  38. Ioffe, A.D. and Tikhomirov, V.M., Teoriya ekstremal’nykh zadach (Theory of Extremal Problems), Moscow: Nauka, 1974.

    Google Scholar 

  39. Polyak, B.T., Vvedenie v optimizatsiyu, Moscow: Nauka, 1983. Translated into English under the title Introduction to Optimization, New York: Optimization Software, 1987.

    MATH  Google Scholar 

  40. Miller, A.B., Using Methods of Stochastic Control to Prevent Overloads in Data Transmission Networks, Autom. Remote Control, 2010, vol. 71, no. 9, pp. 1804–1815.

    Article  MathSciNet  MATH  Google Scholar 

  41. Miller, B.M., Myasnikov, D.V., and Semenikhin, K.V., Designing the Optimal Stochastic Control of a Queuing System with Regard for Constraints, in Proc. 37th School-Conf. of Young Researchers and Experts “Information Technologies and Systems,” Kaliningrad, 2013, September 1–6.

    Google Scholar 

  42. Myasnikov, D.V. and Semenikhin, K.V., Control of M|M|1|N Queue Parameters under Constraints, J. Comput. Syst. Sci. Int., 2016, vol. 55, no. 1, pp. 59–78.

    Article  Google Scholar 

  43. Rockafellar, R.T., Convex Analysis, Princeton: Princeton Univ. Press, 1970. Translated under the title Vypuklyi analiz, Moscow: Mir, 1973.

    MATH  Google Scholar 

  44. Zangwill, W.I., Nonlinear Programming. A Unified Approach, Englewood Cliffs: Prentice-Hall, 1969. Translated under the title Nelineinoe programmirovanie. Edinyi podkhod, Moscow: Sovetskoe Radio, 1973.

    MATH  Google Scholar 

  45. Berkovitz, L., Convexity and Optimization in Rn, New York: Wiley, 2002.

    Book  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Kharkevich Institute for Information Transmission Problems, Russian Academy of Sciences, Moscow, Russia

    B. M. Miller

  2. Institute of Informatics Problems, Federal Research Center “Computer Science and Control” of Russian Academy of Sciences, Moscow, Russia

    G. B. Miller

  3. Moscow State Aviation Institute, Moscow, Russia

    K. V. Semenikhin

Authors
  1. B. M. Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. G. B. Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. K. V. Semenikhin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to B. M. Miller.

Additional information

Original Russian Text © B.M. Miller, G.B. Miller, K.V. Semenikhin, 2016, published in Avtomatika i Telemekhanika, 2016, No. 9, pp. 96–123.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Miller, B.M., Miller, G.B. & Semenikhin, K.V. Optimal control problem regularization for the Markov process with finite number of states and constraints. Autom Remote Control 77, 1589–1611 (2016). https://doi.org/10.1134/S0005117916090071

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0005117916090071

Search

Navigation