Skip to main content
SpringerLink
Log in

Policy-Based and QoE-Aware Content Delivery Using Q-Learning Method

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

With the increasing popularity of various video services, video content is becoming a dominant traffic type in mobile networks. This poses a serious challenge to mobile network operators as well as service providers when it comes to delivering video content in a controllable and resource-efficient way to multiple users. Meeting various quality of experience and quality of service requirements is a difficult task especially in a wireless environment where several different priority based user classes can be included. This paper proposes an intelligent and context-aware application level fair scheduler, which is based on reinforcement learning and which can dynamically adjust relevant scheduling parameters in reaction to specific events or context information. The implemented Q-learning method is analyzed with reference to the delivery of progressive video streaming services employed by the likes of YouTube, Daily Motion, etc. In this regard, we study the performance observed by the end users in a scenario where the backhaul link in a mobile network infrastructure may become congested. Using the application level scheduler to intelligently orchestrate between multiple concurrent flows will minimize the number of buffer starvation events and thus enable smooth playback in cases where a pure TCP based delivery would fail. We also demonstrate the effectiveness of the Q-learning based scheduler to provide service separation between the user classes and fairness within a user class.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

References

  1. Alcock, S., & Nelson, R. (2011). Application flow control in YouTube video streams. SIGCOMM Computer Communications Review, 41(2), 24–30.

    Article  Google Scholar 

  2. Bertsekas, D. (2007). Dynamic programming and optimal control, vol. 1&2. Belmont, MA: Athena Scientific.

    Google Scholar 

  3. Changuel, N., Sayadi, B., & Kieffer, M. (2012). Online learning for QoE-based video streaming to mobile receivers. In Proceedings of the 2012 IEEE globecom workshops (GC Wkshps), pp. 1319–1324.

  4. Cisco Visual Networking Index: Global mobile data traffic forecast Update, 2011–2016 (2013). http://www.puremobile.com/media/infortis/documents/cisco_mobile_forecast.pdf

  5. Ghobadi, M., Cheng, Y., Jain, A., & Mathis, M. (2012). Trickle: Rate limiting youtube video streaming. In Proceedings of the 2012 USENIX conference on annual technical conference, USENIX ATC’12 (pp. 17–17). USENIX Association, Berkeley, CA, USA. http://dl.acm.org/citation.cfm?id=2342821.2342838

  6. Habachi, O., Hu, Y., Van der Schaar, M., Hayel, Y., & Wu, F. (2012). QoE-aware congestion control algorithm for conversational services. In Proceedings of the 2012 IEEE international conference on communications (ICC), pp. 5233–5537.

  7. Hoen, P. J., Tuyls, K., Panait, L., Luke, S., & La Poutré, J. A. (2006). An overview of cooperative and competitive multiagent learning. In K. Tuyls, P. J. Hoen, K. Verbeeck, S. Sen (Eds.) Proceedings of the first international on learning and adaption in multi-agent systems (LAMAS 2005), Lecture Notes in Computer Science, Vol. 3898, pp. 1–46. Springer.

  8. INET Framework for OMNeT++: (2013). http://inet.omnetpp.org

  9. Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237–285.

    Google Scholar 

  10. Lederer, S., Mueller, C., & Timmerer, C. (2012). Dynamic adaptive streaming over http dataset. In Proceedings of the ACM multimedia systems conference, Chapel Hill, NC, USA.

  11. Maier, G., Feldmann, A., Paxson, V., & Allman, M. (2009). On dominant characteristics of residential broadband internet traffic. In Proceedings of the 9th ACM SIGCOMM conference on internet measurement conference, IMC ’09, pp. 90–102.

  12. Matos, R., Coutinho, N., Marques, C., Sargento, S., Chakareski, J., & Kassler, A. (2012). Quality of experience-based routing in multi-service wireless mesh networks. In Proceedings of the 2012 IEEE international conference on communications (ICC), pp. 7060–7065.

  13. Navarro-Ortiz, J., Ameigeiras, P., Lopez-Soler, J., Lorca-Hernando, J., Perez-Tarrero, Q., & Garcia-Perez, R. (2013). A QoE-aware scheduler for http progressive video in OFDMA systems. IEEE Communications Letters, 17(4), 677–680.

    Article  Google Scholar 

  14. Satoda, K., Yoshida, H., Ito, H., & Ozawa, K. (2012). Adaptive video pacing method based on the prediction of stochastic tcp throughput. In Global communications conference (GLOBECOM), 2012 IEEE, pp. 1944–1950.

  15. Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction. Cambridge: MIT Press.

    Google Scholar 

  16. Tran, H. A., Mellouk, A., Hoceini, S., & Augustin, B. (2012). Global state-dependent qoe based routing. In Proceedings of the 2012 IEEE international conference on communications (ICC), pp. 131–135.

  17. Tran, H. A., Mellouk, A., Hoceini, S., & Souihi, S. (2012). User QoE-based adaptive routing system for future internet cdn. In Proceedings of the 2012 computing, communications and applications conference (ComComAp), pp. 90–95.

  18. Watkins, C. (1989). Learning from delayed eewards. Ph.D. thesis, Cambridge University, Cambridge.

  19. Yousaf, F. Z., Liebsch, M., Maeder, A., & Schmid, S. (2013). Mobile CDN enhancements for QoE-improved content delivery in mobile operator networks. IEEE Network Magazine (Special Issue on Video Over Mobile Networks), 27(2), 14–21.

    Google Scholar 

  20. Yousaf, F. Z., Mämmelä, O., & Mannersalo, P. (2014). Reinforcement learning method for QoE-aware optimization of content delivery. In IEEE wireless communications and networking conference (WCNC), pp. 1–6.

Download references

Acknowledgments

This research has been performed within the UniverSelf Project (www.univerself-project.eu) and received funding from the European Community’s Seventh Framework Programme (FP7/ 2007-2013) under Grant Agreement No. 257513. The work by Mannersalo and Mämmelä was also supported by the Finnish Funding Agency for Technology and Innovation (Tekes) in the framework of the EUREKA/Celtic COMMUNE.

Author information

Authors and Affiliations

  1. VTT Technical Research Centre of Finland, P.O. Box 1100, 90571, Oulu, Finland

    Olli Mämmelä & Petteri Mannersalo

  2. NEC Laboratories Europe, Kurfuersten-Anlage 36, 69115, Heidelberg, Germany

    Faqir Zarrar Yousaf & Johannes Lessmann

Authors
  1. Olli Mämmelä
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Faqir Zarrar Yousaf
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Petteri Mannersalo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Johannes Lessmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Olli Mämmelä.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mämmelä, O., Yousaf, F.Z., Mannersalo, P. et al. Policy-Based and QoE-Aware Content Delivery Using Q-Learning Method. Wireless Pers Commun 83, 315–342 (2015). https://doi.org/10.1007/s11277-015-2395-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-015-2395-1

Keywords

Search

Navigation