Skip to main content

Flow Control as a Stochastic Optimal Control Problem with Incomplete Information

Problems of Information Transmission volume 41, pages 150–170 (2005)Cite this article

Abstract

A nonlinear stochastic control problem related to ow control is considered. It is assumed that the state of a link is described by a controlled hidden Markov process with a finite state set, while the loss ow is described by a counting process with intensity depending on a current transmission rate and an unobserved link state. The control is the transmission rate, and it has to be chosen as a nonanticipating process depending on the observation of the loss process. The aim of the control is to achieve the maximum of some utility function that takes into account losses of the transmitted information. Originally, the problem belongs to the class of stochastic control problems with incomplete information; however, optimal filtering equations that provide estimation of the current link state based on observations of the loss process allow one to reduce the problem to a standard stochastic control problem with full observations. Then a necessary optimality condition is derived in the form of a stochastic maximum principle, which allows us to obtain explicit analytic expressions for the optimal control in some particular cases. Optimal and suboptimal controls are investigated and compared with the ow control schemes used in TCP/IP (Transmission Control Protocols/Internet Protocols) networks. In particular, the optimal control demonstrates a much smoother behavior than the TCP/IP congestion control currently used.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

REFERENCES

  1. Jacobson, V., Congestion Avoidance and Control, in Proc. ACM SIGCOMM’88, New York: ACM, 1988, pp. 314–329.

    Google Scholar 

  2. Allman, M., Paxson, V., and Stevens, W., TCP Congestion Control, RFC 2581, April, 1999. Available at http: //www.ietf.org/rfc/rfc2581.txt.

  3. Ramakrishnan, K., Floyd, S., and Black, D., The Addition of Explicit Congestion Notification (ECN) to IP, RFC 3168, September, 2001. Available at http://www.ietf.org/rfc/rfc3168.txt.

  4. Ramakrishnan, K. and Jain, R., A Binary Feedback Scheme for Congestion Avoidance in Computer Networks with Connectionless Network Layer, ACM Trans. Comput. Syst., 1990, vol. 8, no.2, pp. 158–181.

    Google Scholar 

  5. Altman, E., Avrachenkov, K., and Barakat, C., A Stochastic Model of TCP/IP with Stationary Random Losses, Comput. Commun. Rev., 2000, vol. 30, no.4, pp. 231–242.

    Google Scholar 

  6. Altman, E., Avrachenkov, K., and Barakat, C., TCP in Presence of Bursty Losses, Perform. Eval., 2000, vol. 42, no.2–3, pp. 129–147.

    Google Scholar 

  7. Barakat, C., TCP/IP Modeling and Validation, IEEE Network, 2001, vol. 15, no.3, pp. 38–47.

    Google Scholar 

  8. Altman, E., Avrachenkov, K.E., Barakat, C., and Dube, P., TCP over a Multi-state Markovian Path, in Performance and QoS of Next Generation Networking, New York: Springer, 2000, pp. 103–122.

    Google Scholar 

  9. Davis, M.H.A., Markov Models and Optimization, London: Chapman & Hall, 1993.

    Google Scholar 

  10. Elliott, R.J., Aggoun, L., and Moore J.B., Hidden Markov Models: Estimation and Control, NewYork: Springer, 1995.

    Google Scholar 

  11. Fleming, W.H. and Rishel, R.W., Deterministic and Stochastic Optimal Control, Berlin: Springer, 1975.

    Google Scholar 

  12. Gilbert, E.N., Capacity of a Burst-Noise Channel, Bell Syst. Tech. J., 1960, vol. 39, no.5, pp. 1253–1265.

    Google Scholar 

  13. Athuraliya, S. and Low, S., Optimization Flow Control with Newton-like Algorithm, Telecommun. Syst., 2000, vol. 15, no.3/4, pp. 345–358.

    Google Scholar 

  14. Kelly, F.P., Mathematical Modelling of the Internet, Mathematics Unlimited: 2001 and Beyond, Engquist, B. and Schmid, W., Eds., Berlin: Springer, 2001, pp. 685–702.

    Google Scholar 

  15. Kelly, F.P., Maulloo, A.K., and Tan, D.K.H., Rate Control in Communication Networks: Shadow Prices, Proportional Fairness and Stability, J. Oper. Res. Soc., 1998, vol. 49, no.3, pp. 237–252.

    Google Scholar 

  16. Kunniyur, S. and Srikant, R., End-to-End Congestion Control: Utility Functions, Random Losses and ECN Marks, IEEE/ACM Trans. Networking, 2003, vol. 11, no.5, pp. 689–702.

    Google Scholar 

  17. Low, S.H. and Lapsley, D.E., Optimization Flow Control, I: Basic Algorithm and Convergence, IEEE/ACM Trans. Networking, 1999, vol. 7, no.6, pp. 861–874.

    Google Scholar 

  18. Massoulie, L. and Roberts, J., Bandwidth Sharing: Objectives and Algorithms, IEEE/ACM Trans. Networking, 2002, vol. 10, no.3, pp. 320–328.

    Google Scholar 

  19. Dumas, V., Guillemin, F., and Robert, P., A Markovian Analysis of Additive-Increase Multiplicative-Decrease (AIMD) Algorithms, Adv. in Appl. Probab., 2002, vol. 34, no.1, pp. 85–111.

    Google Scholar 

  20. Gal’chuk, L.I., A Generalization of Girsanov’s Theorem on Substitution of Measures to the Case of Semi-Martingales with Jumps, Teor. Veroyatn. Primen., 1977, vol. 22, no.2, pp. 279–294 [Theory Probab. Appl. (Engl. Transl.), 1977, vol. 22, no. 2, pp. 271–285].

    Google Scholar 

  21. Bremaud, P., Point Processes and Queues, Martingale Dynamics, New York: Springer, 1981.

    Google Scholar 

  22. Liptser, R.Sh. and Shiryaev, A.N., Statistics of Random Processes, New York: Springer, 1979.

    Google Scholar 

  23. Gikhman, I.I. and Skorokhod, A.V., Upravlyaemye sluchainye protsessy, Kiev: Nauk. Dumka, 1977. Translated under the title Controlled Stochastic Processes, New York: Springer, 1979.

    Google Scholar 

  24. Yushkevich, A.A., Controlled Markov Models with Countable State Space and Continuous Time, Teor. Veroyatn. Primen., 1977, vol. 22, no.2, pp. 222–241 [Theory Probab. Appl. (Engl. Transl.), 1977, vol. 22, no. 2, pp. 215–235].

    Google Scholar 

  25. Low, S.H., Paganini, F., and Doyle, J.C., Internet Congestion Control, IEEE Control Syst. Mag., 2002, vol. 22, no.1, pp. 28–43.

    Google Scholar 

  26. Miller, B.M. and Runggaldier, W.J., Kalman Filtering for Linear Systems with Coefficients Driven by a Hidden Markov Jump Process, Syst. Control Lett., 1997, vol. 31, no.2, pp. 93–102.

    Google Scholar 

  27. Liptser, R.Sh. and Shiryaev, A.N., Teoriya Martingalov, Nauka, 1986. Translated under the title Theory of Martingales, Dordrecht: Kluwer, 1989.

  28. Wang, E. and Hajek, B., Stochastic Processes in Engineering Systems, New York: Springer, 1985.

    Google Scholar 

  29. Jacod, J. and Shiryaev, A.N., Limit Theorems for Stochastic Processes, Berlin: Springer, 1987. Translated under the title Predel’nye teoremy dlya sluchainykh protsessov, Moscow: Fizmatlit, 1994.

    Google Scholar 

  30. Kabanov, Yu.M., On the Pontriagin Maximum Principle for SDEs with Poisson-type Driving Noise, Statistics and Control of Stochastic Processes. The Liptser Festschrift, Kabanov, Yu.M., Rozovskii, B.L., and Shiryaev, A.N., Eds., River Egde: World Scientific, 1997, pp. 173–190.

    Google Scholar 

  31. Tang, S. and Li, X., Necessary Conditions for Optimal Control of Stochastic Systems with Random Jumps, SIAM J. Control Optim., 1994, vol. 32, no.5, pp. 1447–1475.

    Google Scholar 

  32. El Karoui, N. and Huang, S.-J., A General Result of Existence and Uniqueness of Backward Stochastic Differential Equations, in Backward stochastic differential equations (Paris, 1995–1996), Pitman Res. Notes Math. Ser., vol. 364, Harlow: Longman, 1997, pp. 27–36.

    Google Scholar 

  33. Mathis, M., Semke, J., Mahdavi, J., and Ott, T., The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm, Comput. Commun. Rev., 1997, vol. 27, no.3, pp. 67–82.

    Google Scholar 

  34. Padhye, J., Firoiu, V., Towsley, D., and Kurose, J., Modeling TCP Reno Performance: A Simple Model and Its Empirical Validation, IEEE/ACM Trans. Networking, 2000, vol. 8, no.2, pp. 133–145.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute for Information Transmission Problems, RAS, Moscow, Russia

    B. M. Miller & K. V. Stepanyan

  2. INRIA, Sophia-Antipolis, France

    K. E. Avrachenkov

  3. Moscow Aviation Institute, State University of Aerospace Technologies, Russia

    G. B. Miller

Authors
  1. B. M. Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. K. E. Avrachenkov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. K. V. Stepanyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. G. B. Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

__________

Translated from Problemy Peredachi Informatsii, No. 2, 2005, pp. 89–110.

Original Russian Text Copyright © 2005 by B. Miller, Avrachenkov, Stepanyan, G. Miller.

Supported in part by Program 7.2 of Branch of Informatics, Computer Equipment, and Automation of RAS “New Physical and Structural Solutions in Infotelecommunication,” project no. 4.6d; Russian Foundation for Basic Research, project no. 05-01-00508; INRIA, project ARC TCP; and INTAS, grant YSF 04-83-3623.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Miller, B.M., Avrachenkov, K.E., Stepanyan, K.V. et al. Flow Control as a Stochastic Optimal Control Problem with Incomplete Information. Probl Inf Transm 41, 150–170 (2005). https://doi.org/10.1007/s11122-005-0020-8

Download citation

  • Received:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11122-005-0020-8

Keywords

  • Optimal Control Problem
  • Congestion Control
  • Loss Process
  • Link State
  • Stochastic Optimal Control