Skip to main content
SpringerLink
Log in

Efficient QoS Provisioning for Adaptive Multimedia in Mobile Communication Networks by Reinforcement Learning

  • Published:
Mobile Networks and Applications Aims and scope Submit manuscript

Abstract

The scarcity and large fluctuations of link bandwidth in wireless networks have motivated the development of adaptive multimedia services in mobile communication networks, where it is possible to increase or decrease the bandwidth of individual ongoing flows. This paper studies the issues of quality of service (QoS) provisioning in such systems. In particular, call admission control and bandwidth adaptation are formulated as a constrained Markov decision problem. The rapid growth in the number of states and the difficulty in estimating state transition probabilities in practical systems make it very difficult to employ classical methods to find the optimal policy. We present a novel approach that uses a form of discounted reward reinforcement learning known as Q-learning to solve QoS provisioning for wireless adaptive multimedia. Q-learning does not require the explicit state transition model to solve the Markov decision problem; therefore more general and realistic assumptions can be applied to the underlying system model for this approach than in previous schemes. Moreover, the proposed scheme can efficiently handle the large state space and action set of the wireless adaptive multimedia QoS provisioning problem. Handoff dropping probability and average allocated bandwidth are considered as QoS constraints in our model and can be guaranteed simultaneously. Simulation results demonstrate the effectiveness of the proposed scheme in adaptive multimedia mobile communication networks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. E. Altman, Constrained Markov Decision Process (Chapman and Hall, London, 1999).

    Google Scholar 

  2. N. Argiriou and L. Georgiadis, Channel sharing by rate-adaptive streaming applications, in: Proc. IEEE Infocom'02 (June 2002).

  3. D.P. Bertsekas and J.N. Tsitsiklis, Neuro-Dynamic Programming (Athena Scientific, 1996).

  4. F.J. Beutler and K.W. Ross, Optimal policies for controlled Markov chains with a constraint, J. Math. Anal. Appl. 112 (1985) 236–252.

    Article  MathSciNet  Google Scholar 

  5. F.J. Beutler and K.W. Ross, Time-average optimal constrained semi-Markov decision processes, Adv. Appl. Prob. 18 (1986) 341–359.

    MathSciNet  Google Scholar 

  6. A. Bhattacharya and S.K. Das, LeZi-update: An information-theoretic framework for personal mobility tracking in PCS networks, Wireless Networks 8(2/3) (2002) 121–135.

    Google Scholar 

  7. J.A. Boyan and M.L. Littman, Packet routing in dynamically changing networks: A reinforcement learning approach, in: Advances in NIPS 6, J.D. Cowan et al. (eds.) (1994) pp. 671–678.

  8. C. Chou and K.G. Shin, Analysis of combined adaptive bandwidth allocation and admission control in wireless networks, in: Proc. IEEE Infocom'02 (June 2002).

  9. 3GPP, RRC protocol specification, 3G TS25.331 version 3.20.0 (Sept. 2004).

  10. Z. Gabor, Z. Kalmar and C. Szepesvari, Multi-criteria reinforcement learning, in: Proc. Int'l Conf. Machine Learning, Madison, WI (July 1998).

  11. D. Hong and S.S. Rappaport, Traffic model and performance analysis for cellular mobile radio telephone systems with prioritised and non-prioritised handoff procedures, IEEE Trans. Veh. Technol. VT-35 (1986) 77–92.

    Google Scholar 

  12. ISO/IEC 144962-2, Information Technology Coding of Audio-Visual Objects: Visual (Committee draft, Oct. 1997).

  13. ITU-T H. 263, Video Coding for Low Bitrate Communication (Jan. 1998).

  14. T. Kwon, J. Choi, Y. Choi and S.K. Das, Near optimal bandwidth adaptation algorithm for adaptive multimedia services in: Wireless/Mobile Networks, in: Proc. IEEE VTC'99-Fall, vol. 2, Amsterdam, The Netherland (Sept. 1999) pp. 874–878.

  15. T. Kwon, Y. Choi, C. Bisdikian and M. Naghshineh, QoS provisioning in wireless/mobile multimedia networks using an adaptive framework, Wireless Networks 9 (2003) 51–59.

    Article  Google Scholar 

  16. P. Marbach, O. Mihatsch and J.N. Tsitsiklis, Call admission control and routing in integrated services networks using neuro-dynamic programming, IEEE J. Select. Areas Commun. 18(2) (2000) 197–208.

    Google Scholar 

  17. J. Nie and S. Haykin, A Q-learning based dynamic channel assignment technique for mobile communication systems, IEEE Trans. Veh. Technol. 48(5) (1999) 1676–1687.

    Google Scholar 

  18. M.L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming (Wiley, New York, 1994).

    Google Scholar 

  19. S.P. Singh and D.P. Bertsekas, Reinforcement learning for dynamic channel allocation in cellular telephone systems, in: Advances in NIPS Vol. 9, M. Mozer et al. (eds.) (1997) pp. 974–980.

    Google Scholar 

  20. A.K. Talukdar, B.R. Badrinath and A. Acharya, Rate adaptation schemes in networks with mobile hosts, in: Proc. ACM/IEEE MobiCom'98 (Oct. 1998).

  21. H. Tong and T.X. Brown, Adaptive call admission control under quality of service constraints: a reinforcement learning solution, IEEE J. Select. Areas Commun. 18(2) (2000) 209–221.

    Google Scholar 

  22. D. Taubman and A. Zakhor, A common framework for rate and distortion based scaling of highly scalable compressed video, IEEE Trans. Circuits Syst. Video Technol. 6(4) (1996) 329–354.

    Google Scholar 

  23. C.J.C.H. Watkins and P. Dayan, Q-learning, Machine Learning 8 (1992) 279–292.

    Google Scholar 

  24. D. Wu, Y.T. Hou and Y.Q. Zhang, Scalable video coding and transport over broadband wireless networks, Proc. IEEE 89(1) (2001) 6–20.

    Google Scholar 

  25. S. Wu et al., A dynamic call admission policy with precision QoS guarantee using stochastic control for mobile wireless networks, IEEE/ACM Trans. Networking 10(2) (2002) 257–271.

    Google Scholar 

  26. F. Yu, V.W.S. Wong and V.C.M. Leung, Reinforcement learning for call admission control and bandwidth adaptation in mobile multimedia networks, in: Proc. of ICICS-PCM'3, Singapore (Dec. 2003).

  27. F. Yu, V.W.S. Wong and V.C.M. Leung, A new QoS provisioning method for adaptive multimedia in cellular wireless networks, in: Proc. IEEE Infocom'04, HongKong, China, (Apr. 2004).

  28. G.V. Zaruba, I. Chlamtac and S.K. Das, A prioritized real-time wireless call degradation framework for optimal call mix selection, Mobile Networks and Applications 7 (2002) 143–151.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, The University of British Columbia, 2356 Main Mall, Vancouver, BC, Canada, V6T 1Z4

    Yu Fei, Vincent W. S. Wong & Victor C. M. Leung

Authors
  1. Yu Fei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vincent W. S. Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Victor C. M. Leung
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yu Fei.

Additional information

This work is based in part on a paper presented at BroadNet's 04, San Jose, CA, Oct. 2004.

Fei Yu received the M.S. degree in Computer Engineering from Beijing University of Posts and Telecommunications, P.R. China, in 1998, and the Ph.D. degree in Electrical Engineering from the University of British Columbia (UBC), Canada, in 2003. From 1998 to 1999, Dr. Yu was a system engineer at China Telecom, P.R. China, working on the planning, design and performance analysis of national SS7 and GSM networks. From 2002 to 2004, He was a research and development engineer at Ericsson Mobile Platforms, Sweden, where he worked on dual-mode UMTS/GPRS handsets. He is currently a postdoctoral research fellow at UBC. His research interests are quality of service, cross-layer design and mobility management in wireless networks.

Vincent W.S. Wong (S'94-M'00) received the B.Sc. (with distinction) degree from the University of Manitoba, Winnipeg, MB, Canada, in 1994, the M.A.Sc. degree from the University of Waterloo, Waterloo, ON, Canada, in 1996, and the Ph.D. degree from the University of British Columbia (UBC), Vancouver, BC, Canada, in 2000, all in electrical engineering. From 2000 to 2001, he was a Systems Engineer at PMC-Sierra, Inc., Burnaby, BC. Since 2002, he has been with the Department of Electrical and Computer Engineering, UBC, where he is currently an Assistant Professor. His research interests are in wireless communications and networking. Dr. Wong received the Natural Science and Engineering Research Council (NSERC) postgraduate scholarship and the Fessenden Postgraduate Scholarship from Communications Research Centre, Industry Canada, during his graduate studies.

Victor C.M. Leung received the B.A.Sc. (Hons.) degree in electrical engineering from the University of British Columbia (U.B.C.) in 1977, and was awarded the APEBC Gold Medal as the head of the graduating class in the Faculty of Applied Science. He attended graduate school at U.B.C. on a Natural Sciences and Engineering Research Council Postgraduate Scholarship and obtained the Ph.D. degree in electrical engineering in 1981.

From 1981 to 1987, Dr. Leung was a Senior Member of Technical Staff at Microtel Pacific Research Ltd. (later renamed MPR Teltech Ltd.), specializing in the planning, design and analysis of satellite communication systems. He also held a part-time position as Visiting Assistant Professor at Simon Fraser University in 1986 and 1987. In 1988, he was a Lecturer in the Department of Electronics at the Chinese University of Hong Kong. He joined the Department of Electrical Engineering at U.B.C. in 1989, where he is a Professor, Associate Head of Graduate Affairs, holder of the TELUS Mobility Industrial Research Chair in Advanced Telecommunications Engineering, and a member of the Institute for Computing, Information and Cognitive Systems. His research interests are in the areas of architectural and protocol design and performance analysis for computer and telecommunication networks, with applications in satellite, mobile, personal communications and high speed networks.

Dr. Leung is a Fellow of IEEE and a voting member of ACM. He is an editor of the IEEE Transactions on Wireless Communications, and an associate editor of the IEEE Transactions on Vehicular Technology. He has served on the technical program committees of numerous conferences, and is serving as the Technical Program Vice-Chair of IEEE WCNC 2005.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fei, Y., Wong, V.W.S. & Leung, V.C.M. Efficient QoS Provisioning for Adaptive Multimedia in Mobile Communication Networks by Reinforcement Learning. Mobile Netw Appl 11, 101–110 (2006). https://doi.org/10.1007/s11036-005-4464-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11036-005-4464-2

Keywords

Search

Navigation