Evaluation of Optimal Resource Management Policies for WiMAX Networks with AMC: A Reinforcement Learning Approach

  • Adam Flizikowski
  • Mateusz Majewski
  • Marcin Przybyszewski
Part of the Advances in Intelligent and Soft Computing book series (AINSC, volume 102)


Call admission control in access network has become an interesting topic for the research community due to its potential applicability in broadband wireless systems. Admission control problem can be formulated as Markov decision process (MDP) and has proven to deliver optimal policies for blocking and dropping probabilities in wireless networks. This however typically requires the model to know the system dynamics in advance. One approach to solving MDPs considers letting CAC agent interact with the environment and learn by ”trial and error” to choose optimal actions - thus Reinforcement Learning algorithms are applied. Abstraction and generalization techniques can be used with RL algorithms to solve MDPs with large state space. In this paper authors decribe and evaluate a MDP formulated problem to find optimal Call Admission Control policies for WiMAX networks with adaptive modulation and coding. We consider two classes of service (BE and UGS-priority) and a variable capacity channel with constant error bit rate. Hierarchical Reinforcement Learning (HRL) techniques are applied to find optimal policies for multi-task CAC agent. In addition this article validates several neural network training algorithms to deliver a training algorithm suitable for the CAC agent problem.


Optimal Policy Admission Control Markov Decision Process Adaptive Modulation Call Admission Control 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Putterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley and Sons Inc. Publication, Hoboken (2005)Google Scholar
  2. 2.
    Nasser, N., Hassanein, H.: An optimal and fair call admission control policy for seamless handoff in multimedia wireless networks with QoS guarantees. In: Global Telecommunications Conference, vol. 6, pp. 3926–3930 (2004)Google Scholar
  3. 3.
    Yang, X., Feng, G.: Cost Minimization for Admission Control in Bandwidth Asymmetry Wireless Networks. In: IEEE International Conference on Communications, pp. 5484–5489 (2007)Google Scholar
  4. 4.
    Wenlong, N., Wei, L., Alam, M.: Determination of optimal call admission control policy in wireless networks. IEEE Transactions on Wireless Communications 8, 1038–1044 (2009)CrossRefGoogle Scholar
  5. 5.
    Nordstrom, E., Carlstrom, J.: Call admission control and routing for integrated CBR/VBR and ABR services: a Markov decision approach. In: IEEE Proceedings, pp. 71–76 (1999)Google Scholar
  6. 6.
    Senouci, S., Beylot, A., Pujolle, G.: Call Admission Control in Cellular Networks: A Reinforcement Learning Solution. International Journal of Network Management 13, 89–103 (2004)CrossRefGoogle Scholar
  7. 7.
    Mitchell, T.M.: Machine Learning. McGraw-Hill Science-Engineering-Math, New York (1997)zbMATHGoogle Scholar
  8. 8.
    Pietrabissa, A., Delli, P.F.: MDP call control in variable capacity communication networks. In: 18th Mediterranean Conference on Control and Automation, pp. 483–488 (2010)Google Scholar
  9. 9.
    Ren-Hung, H., Kurose, J.F., Towsley, D.: MDP routing in ATM networks using virtual path concept. In: 13th Proceedings IEEE Networking for Global Communications, pp. 1509–1517 (1994)Google Scholar
  10. 10.
    IEEE 802.16. Part 16: Air Interface for Broadband Wireless Access Systems (2009)Google Scholar
  11. 11.
    Yu, K., Wang, X., Sun, S., Zhang, L., Wu, X.: A Statistical Connection Admission Control Mechanism for Multiservice IEEE 802.16 Network. In: Vehicular Technology Conference, pp. 1–5 (2009)Google Scholar
  12. 12.
    Pietrabissa, A.: Reinforcement learning call control in variable capacity links. In: 18th Mediterranean Conference on Control and Automation, pp. 933–938 (2010)Google Scholar
  13. 13.
    Vu, X.-T., Nguyen, D.-T., Vu, T.A.: An finite-state Markov channel model for ACM scheme in WiMAX. In: Region 10 Conference, pp. 1–6 (2009)Google Scholar
  14. 14.
    Flizikowski, A., Hołubowicz, W., Przybyszewski, M., Grzegorzewski, S.: Admission control and system capacity assessment of WiMAX with ACM and nb-LDPC codes - simulation study with ViMACCS ns2 patch. In: The International Conference on Advanced Information Networking and Applications, Perth (2009)Google Scholar
  15. 15.
    Flizikowski, A., Przybyszewski, M., Majewski, M., Kozik, R.: Evaluation of guard channel admission control schemes for IEEE 802.16 with integrated nb-LDPC codes. In: International Conference on Ultra Modern Telecommunications, St.-PetersburgGoogle Scholar
  16. 16.
    Frontiers of Mobile and Wireless Communication. In: Proceedings of the IEEE 6th Circuits and Systems Symposium, vol. 2, pp. 549–552 (2004)Google Scholar
  17. 17.
    Barto, A., Mahadevan, S.: Recent Advances in Hierarchical Reinforcement Learning. Discrete Event Dynamic Systems: Theory and Applications 13, 343–379 (1999)Google Scholar
  18. 18.
    Sutton, R., Precup, D., Singh, S.: Between MDPs and Semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112, 181–211 (1999)MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Adam Flizikowski
    • 1
  • Mateusz Majewski
    • 2
  • Marcin Przybyszewski
    • 2
  1. 1.University of Technology and Life ScienceBydgoszczPoland
  2. 2.ITTIPoznańPoland

Personalised recommendations