Skip to main content
SpringerLink
Log in

Learning Intelligent Controls in High Speed Networks: Synergies of Computational Intelligence with Control and Q-Learning Theories

  • Chapter
  • First Online:
Innovative Issues in Intelligent Systems

Part of the book series: Studies in Computational Intelligence ((SCI,volume 623))

Abstract

The phenomena of congestion and packet-drops in high-speed communications and computer networks do affect the quality-of-service and overall performance more than ever existing time-delays with uncertain variations. Their control and, possibly, prevention are subject to extensive research ever since the internet is available. Because of the uncertainties and time-varying phenomena, obtaining the accurate and complete information on the network traffic patterns, especially for the multi-bottleneck case, is rather difficult hence learning intelligent controls are needed. One such control is a multi-agent flow controller (MAFC) based on Q-learning algorithm in conjunction with the theory of Nash equilibrium of opponents’ strategies. The other is a model-independent Q-learning control (MIQL) scheme having focus on the flow with higher priority, which also does not need prior-knowledge on communication traffic and congestion. The competition of communication flows with different priorities is considered as a two-player non-cooperative game. The Nash Q-learning algorithm control obtains the Nash Q-values through trial-and-error and interaction with the network environment so as to improve its behaviour policy. The MAFC can learn to take the best actions in order to regulate source flows that guarantee high throughput and low packet-loss ratio. The MIQL control, through a specific learning processing, does achieve the optimum sending rate for the sources with lower priority while observing the sources with higher priority. Designed intelligent controls achieve superior performances in controlling the flows in high-speed networks in comparison to the standard ones and avoid communications congestion.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aström, K.J.: Introduction: complex systems and control. In: Astöm, KJ., Albertos, P., Blanke, M., Isidori, A., Sanz, R. (eds.) Control of Complex Systems, Chapter 1, pp. 1–20. Springer, London (2001)

    Google Scholar 

  2. Siljak, D.D.: Decentralized Control of Complex Systems. Academic Press, Boston (1991)

    Google Scholar 

  3. Wiener, N.: Cybernetics or Control and Communication in the Animal and the Machine. MIT Press, Coambridge, MA (1948).

    Google Scholar 

  4. Tsien, H.-S.: Engineering Cybermetics. McGraw-Hill, New York, NY (1954).

    Google Scholar 

  5. Dimirovski, G.M., Gough, N.E., Barnett, S.: Categories in systems and control theory. Int. J. Control 8(9), 1081–1090 (1977)

    MATH  Google Scholar 

  6. Gitt, W.: Information: the third fundamental quantity. Siemens Rev. 56, 36–41 (1989)

    Google Scholar 

  7. Kalman, R.E.: On the general theory of control: In: Proceedings of the First International Congress on Automatic Control. Butterworth Scientific Institute, London, UK, vol. 1, pp. 481–506. Also in Russian: Trudyi I Kongressa IFAK, Izdatelystvo AN SSSR, Moskva, SSSR, vol. 2, pp. 521–547 (1961)

    Google Scholar 

  8. Pontryagin, L.S., Boltyanskiy, V.G., Gamkrelidze, R.V., Mishchenko, E.F.: Matematicheskaya teoriya optimalynih processov, Izdatelystvo fizichko-matematicheskoy literaturi, Moskva, SSSR. Also in English (1962): The Mathematical Theory of Optimal Processes. J. Wiley InterScience, New York (1961)

    Google Scholar 

  9. Savkin, A.V., Matveev, A.S.: Cyclic linear differential automata: a simple class of hybrid dynamical systems. Automatica 36(5), 727–734 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  10. Imer, O.C., Basar, T.: Control of congestion in high-speed networks. Eur. J. Control 7(2–3), 132–144 (2001)

    Article  MATH  Google Scholar 

  11. Jacobson, V.: Congestion avoidance and control. ACM Comput. Commun. Rev. 18, 314–329 (1988)

    Article  Google Scholar 

  12. Alpcan, T., Basar, T.: A game theoretical analysis of intrusion detection in access control systems. In: Proceedings of the 43rd IEEE Conference on Decision and Control, Paradise Island, Bahamas, 14–17 Dec. The IEEE Press, Piscataway, NJ, USA, pp. 1568–1573 (2004)

    Google Scholar 

  13. Basar, T., Olsder, G.J.: Dynamic Non-cooperative Game Theory, 2nd edn. The SIAM, Philadelphia, PA (1999)

    Google Scholar 

  14. Dimirovski, G.M: Network Q-Learning control prevent cyber intrusion risks: synergies of control theory and computational intelligence. In: Duman, E., Atiya, A. (eds.) Use of Risk Analysis in Computer-Aided Persuasion, vol. 88. NATO Science Sub-Series E, Chapter 20–Part III. The IOS Press, Amsterdam, Berlin Tokyo Washington DC, pp. 281–303 (2011)

    Google Scholar 

  15. Gevros, P., Crowcoft, J., Kirstein, P., Bhatti, S.: Congestion control mechanisms and the best effort service model. IEEE Netw. 15(3), 16–26 (2001)

    Article  Google Scholar 

  16. Imer, O.C., Compans, S., Basar, T., Srikant, R.: ABR congestion control in ATM networks. IEEE Control Syst. Mag. 21(1), 38–56 (2001)

    Article  Google Scholar 

  17. Jing, Y.-W., Li, X., Dimirovski, G.M., Zheng, Y., Zhang, S.-Y.: Nash Q-learning multi-agent flow control for high-speed networks. In: Proceedings of the 28th American Control Conference, St. Louis, MO, USA, 10–12 June, pp. 3304–3309. The IEEE Press, Piscataway, NJ, USA (2009)

    Google Scholar 

  18. Lestas, M., Pitsillides, A., Ioannou, P., Hadjipollas, G.: Adaptive congestion protocol: A congestion control protocol with learning capability. Comput. Netw. Int. J. Comput. Telecommun. Netw. 51(13), 3773–3798 (2007)

    MATH  Google Scholar 

  19. Li, X., Dimirovski, G.M., Jing, Y.-W., Zhang, S.-Y.: A Q-learning model-independent flow controller for high-speed networks. In: Proceedings of the 28th American Control Conference, St. Louis, MO, USA, 10–12 June, pp. 1544–1548. The IEEE Press, Piscataway, NJ, USA (2009)

    Google Scholar 

  20. Li, X., Zhou, Y.C., Dimirovski, G.M., Jing, Y.-W.: Simulated annealing Q-learning algorithm for ABR traffic control of ATM networks. In: Proceedings of the 27th American Control Conference, Seattle, WA, USA, 11–13 June, pp. 4462–4467. The IEEE Press, Piscataway, NJ, USA (2008)

    Google Scholar 

  21. Li, X., Shen, X.J., Jing, Y.-W., Zhang, S.-Y.: Simulated annealing-reinforcement learning algorithm for ABR traffic control of ATM networks. In: Proceedings of the 46th IEEE Conference on Decision and Control, New Orleans, LA. USA, 12–14 Dec, pp. 5716-5721. The IEEE Press, Piscataway, NJ, USA (2007)

    Google Scholar 

  22. Ren, T., Zhu, Z., Yu, H., Dimirovski, G.M.: Integral sliding-mode controller for congestion problem in ATM networks. Int. J. Control 86(3), 529–539 (2013)

    Google Scholar 

  23. Ren, T., Wang, C., Luo, X., Jing, Y., Dimirovski, G.M.: Robust controller design for ABR traffic control of ATM networks with time-varying multiple time delays. Int. J. Innovative Comput. Inf. Control 5(8), 68–74 (2009)

    Google Scholar 

  24. Ren, T., Gao, Z., Kong, W., Jing, Y., Yang, M., Dimirovski, G.M.: Performance and robustness analysis of a fuzzy-immune flow controller in ATM networks with time-varying multiple time-delays. J. Control Theory Appl. 6(3), 253–258 (2008)

    Article  MathSciNet  Google Scholar 

  25. Aizerman, M.A., Breverman, E.M., Rosonoer, L.I.: Theoretical foundation of the potential function method in pattern recognition learning (in Russian). Avtomatika i Telemechanika 25, 821–837 (1964)

    Google Scholar 

  26. Aizerman, M.A., Breverman, E.M., Rosonoer, L.I.: The probability problem of pattern recognition learning and the potential function method (in Russian). Avtomatika i Telemechanika 25, 821–837 (1964)

    Google Scholar 

  27. Chatovich, A., Okug, S., Dundar, G.: Hierarchical neuro-fuzzy call admission controller for ATM networks. Comput. Commun. 24(11), 1031–1044 (2001)

    Article  Google Scholar 

  28. Cheng, R.G., Chang, C.J., Lin, L.F.: a QoS-provisioning neural fuzzy connection admission controller for multimedia high-speed networks. IEEE ACM Trans. Netw. 7(1), 111–121 (1999)

    Article  Google Scholar 

  29. Harmer, P.K., Williams, P.D., Gunsh, G.H., Lamont, G.B.: An artificial immune system architecture for computer security applications. IEEE Trans. Evol. Comput. 6(2), 252–280 (2002)

    Article  Google Scholar 

  30. Hsiao, M.C., Tan, S.W., Hwang, K.S., Wu, C.S.: a reinforcement learning approach to congestion control of high-speed multimedia networks. Cybern. Syst. 36(2), 181–202 (2005)

    Article  Google Scholar 

  31. Hwang, K.S., Tan, S.W., Hsiao, M.C., Wu, C.S.: Cooperative multi-agent congestion control for high-speed networks. IEEE Trans. Syst. Man Cybern. Part B: Cybern. 35(2), 255–268 (2005)

    Article  Google Scholar 

  32. The MathWorks Laboratory: Matlab: Control Toolbox, Fuzzy Toolbox. LMI Toolbox, The MathWorks Inc, Natick (1991)

    Google Scholar 

  33. The MathWorks Laboratory: Matlab: Simulink – Dynamic System Simulation Software. The MathWorks Inc, Natick (1996)

    Google Scholar 

  34. Dimirovski, G.M.: Lyapunov stability in control synthesis design using fuzzy-logic and neural-networks. In: Proceedings of the 17th IMACS World Congress, Paris, 11–15 July. The IMACS and Ecole Centrale de Lille, Villeneuve d’Ascq, FR, Paper T5-I-01-0907, pp. 1–8 (2005)

    Google Scholar 

  35. Marinos, P.N.: Fuzzy logic and its applications to switching systems. IEEE Trans. Comput. 18(4), 343–348 (1969)

    Article  MATH  Google Scholar 

  36. Palm, R., Driankov, D.: Fuzzy switched hybrid systems – Modelling and identification. In: Proceedings of the 1998 IEEE International Conference on Fuzzy Systems, Gaithersburg, MD, pp. 130–135. The IEEE Press, Piscataway, NJ, USA (1998)

    Google Scholar 

  37. Tanaka, K., Masaaki, I., Wang, H.O.: Stability and smoothness conditions for switching fuzzy systems. Proceedings of the 19th American Control Conference, pp. 2474–2478. The IEEE Press, Piscataway, NJ, USA (2000)

    Google Scholar 

  38. Wang, H.O., Tanaka, K., Griffin, M.: An approach to fuzzy control of nonlinear systems: Stability and design issues. IEEE Trans. Fuzzy Syst. 4(1), 14–23 (1996)

    Article  Google Scholar 

  39. Ye, H., Michael, A.N.: Stability theory for hybrid dynamical systems. IEEE Trans. Autom. Control 43(4), 464–474 (1998)

    Google Scholar 

  40. Zhao, J., Dimirovski, G.M.: Quadratic stability for a class of switched nonlinear systems. IEEE Trans. Autom. Control 49(4), 574–578 (2004)

    Article  MathSciNet  Google Scholar 

  41. Yang, H., Dimirovski, G.M., Zhao, J.: Switched fuzzy systems: representation modelling, stability, and control design. In: Chountas, P., Petrounias, I., Kacprzyk, J. (eds.) Studies in Computational Intelligence Volume 109 – Intelligent Techniques and Tool for Novel Systems Architectures, Chapter 9, pp. 169–184. Springer, Berlin Heidelberg (2008)

    Google Scholar 

  42. Yang, M., Jing, Y., Dimirovski, G.M., Zhang, N.: Stability and performance analysis of a congestion control algorithm for networks. In: Proceedings of the 46th IEEE Conference on Decision and Control, New Orleans, LA. USA, 12–14 Dec, pp. 4453–4458. The IEEE Press, Piscataway, NJ, USA (2007)

    Google Scholar 

  43. Jing, Y. W.: A Study on Advanced Fuzzy and Learning Controls for Communication Netwroks and Effects of Their Applications. Private Communication Report CCN-2007-2008-Shenyang-Skopje, College of Informations Science and Engineering, Northeastern University, Shenyang, PR China (2009).

    Google Scholar 

  44. Watkins, C.J.C.H., Dayan, P.: Q-learning. Mach. Learn. 8(3), 279–292 (1992)

    MATH  Google Scholar 

  45. Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: a survey. J. Artif. Intell. Res. 4(1), 237–285 (1996)

    Google Scholar 

  46. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA, USA (1998)

    Google Scholar 

  47. Littman, M.L.: Value-function reinforcement learning in Markov games. J Cogn. Syst. Res. 2(1), 55–66 (2001)

    Article  Google Scholar 

  48. Szepesvari, C., Littman, M.L.: a unified analysis of value-function-based reinforcement-learning algorithms. Neural Comput. 11(8), 2017–2060 (1999)

    Article  Google Scholar 

  49. Hu, J., Wellman, M.P.: Nash Q-learning for general-sum stochastic games. J. Mach. Learn. Res. 4, 1039–1069 (2003)

    MathSciNet  Google Scholar 

  50. Jing, Y.W., Dimirovski, G.M.: Multimedia transfer over the internet: a control based improvement of the AIMD algorithm. In: Proceedings of the 3rd IEEE International Conference on Information and Communication Technologies: From Theory to Applications, Damascus, Syria, 24–28 April. The IEEE Press, Piscataway, NJ, USA and ENST Brest, Bretagne, FR, Paper SEN06-1/1-6 (2006)

    Google Scholar 

  51. Dimirovski, G.M.: Complexity versus integrity solution in adaptive fuzzy-neural inference models. Int. J. Intell. Syst. 23(5), 556–573 (2008)

    Article  MATH  Google Scholar 

  52. Nash, J.F.: Non-cooperative games. Ann. Math. 54(2), 286–295 (1951)

    Article  MathSciNet  MATH  Google Scholar 

  53. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, London, UK (2006)

    MATH  Google Scholar 

  54. Chiu, D.M., Jain, R.: Analysis of the increase decrease algorithms for congestion avoidance in computer networks. Comput. Netw. ISDN Syst. 17, 1–14 (1989)

    Article  MATH  Google Scholar 

  55. Bradtke, S.J., Ydstie, BE., Barto, A.G.: Adaptive linear quadratic control using policy iteration. In: Proceedings of the 13th American Control Conference, Chicago, IL, 13–15 June, pp. 3475–3479. The IEEE Press, Piscataway, NJ (1994)

    Google Scholar 

  56. Haykin, S.: Neural Networks and Learning Machines, 3rd edn. Pearson Prentice Hall, Upper Saddle River (2009)

    Google Scholar 

  57. Ljung, L.: System Identification—Theory for the User, 2nd edn. Prentice Hall PTR, Upper Saddle River (1999)

    Google Scholar 

Download references

Acknowledgments

The author has the honour to acknowledge and point out that this research was accomplished due to his collaboration with the young Dr. Xin Li and Dr. Yan Zheng, their mentor Prof. Yuan-Wei Jing from the Northeastern University, Shenyang, P.R. of China. Furthermore, special thanks are due to Academician Si-Ying Zhang, our common advisor and teacher, whose guidance has been instrumental. The author acknowledges their crucial merits, respectively, for fruitful carrying out this joint research endeavour.

Author information

Authors and Affiliations

  1. School of Electrical Engineering & Information Technologies, St Cyril and St Methodius University, MK-1000, Skopje, Karpos 2 BB, Republic of Macedonia

    Georgi M. Dimirovski

  2. School of Engineering, Acibadem, Dogus University of Istanbul, Zeamet Sk 21, TR-34722, Istanbul, Republic of Turkey

    Georgi M. Dimirovski

Authors
  1. Georgi M. Dimirovski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Georgi M. Dimirovski .

Editor information

Editors and Affiliations

  1. Inst of Info & Commu Tech, Bulgarian Academy of Sciences, Sofia, Bulgaria

    Vassil Sgurev

  2. Iona College, Machine Intelligence Institute, New Rochelle, New York, USA

    Ronald R. Yager

  3. Sys Res Intit of Polish Acad of Science, Intelligent Systems Laboratory, Warsaw, Poland

    Janusz Kacprzyk

  4. and Information Technologies, University of Library Studies, Sofia, Bulgaria

    Vladimir Jotsov

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Dimirovski, G.M. (2016). Learning Intelligent Controls in High Speed Networks: Synergies of Computational Intelligence with Control and Q-Learning Theories. In: Sgurev, V., Yager, R., Kacprzyk, J., Jotsov, V. (eds) Innovative Issues in Intelligent Systems. Studies in Computational Intelligence, vol 623. Springer, Cham. https://doi.org/10.1007/978-3-319-27267-2_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-27267-2_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-27266-5

  • Online ISBN: 978-3-319-27267-2

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation