Skip to main content
SpringerLink
Log in

Deep reinforcement learning based QoE-aware actor-learner architectures for video streaming in IoT environments

  • Special Issue Article
  • Published:
Computing Aims and scope Submit manuscript

Abstract

The number of connected smart devices enabling multimedia applications has expanded tremendously in Internet-of-Things (IoT) environments. Specifically, the requirement for a high quality of experience (QoE) for video streaming services is a crucial prerequisite for a range of use cases, including smart surveillance, smart healthcare, smart agriculture and many more. However, providing a high QoE for video streaming is challenging due to underlying dynamic network conditions. To address this issue, several adaptive bit rate (ABR) algorithms based on predetermined rules have been developed. However, they do not generalize well to a wide variety of network conditions. ABR algorithms, based on reinforcement learning (RL), have been proven to be more effective at generalizing to varying network conditions but they still have limitations, specifically, constrained exploration and high variance in value estimates. In this paper, we propose asynchronous advantage actor-critic (A3C) based actor-learner architectures for generating the adaptive bit rates for video streaming in IoT environments. To address the existing issues, we propose integrating two advanced A3C algorithms: Follow then Forage Exploration (FFE) and Averaged A3C. We demonstrate their efficacy in improving the QoE over vanilla A3C. Additionally, we also demonstrate the benefits of the proposed architecture for video streaming under different network conditions and for different variants of the QoE metric. We show that advanced A3C methods provide up to 30.70% improvement in QoE over vanilla A3C and a considerably higher QoE over other fixed-rule-based ABR algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Mumtaz S, Al-Dulaimi A, Frascolla V, Hassan SA, Dobre OA (2019) Guest editorial special issue on 5G and beyond-mobile technologies and applications for IoT. IEEE Internet Things J 6(1):203–206

    Article  Google Scholar 

  2. Alvi SA, Afzal B, Shah GA, Atzori L, Mahmood W (2015) Internet of multimedia things: vision and challenges. Ad Hoc Netw 33:87–111

    Article  Google Scholar 

  3. Barakabitze AA, Barman N, Ahmad A, Zadtootaghaj S, Sun L, Martini MG et al (2019) QoE management of multimedia streaming services in future networks: a tutorial and survey. IEEE Commun Surv Tutor 22(1):526–565

    Article  Google Scholar 

  4. Floris A, Atzori L (2015) Quality of experience in the multimedia internet of things: definition and practical use-cases. In: 2015 IEEE international conference on communication workshop (ICCW). IEEE, 2015. pp 1747–1752

  5. Floris A, Atzori L (2016) Managing the quality of experience in the multimedia internet of things: a layered-based approach. Sensors 16(12):2057

    Article  Google Scholar 

  6. Karaadi A, Sun L, Mkwawa IH (2017) Multimedia communications in internet of things QoT or QoE? In: 2017 IEEE international conference on internet of things (iThings) and IEEE green computing and communications (GreenCom) and IEEE cyber, physical and social computing (CPSCom) and IEEE smart data (SmartData). IEEE, 2017. pp 23–29

  7. Fizza K, Banerjee A, Mitra K, Jayaraman PP, Ranjan R, Patel P et al (2021) QoE in IoT: a vision, survey and future directions. Discov Internet Things 1(1):1–14

    Article  Google Scholar 

  8. Rajavel R, Ravichandran S, Harimoorthy K, Nagappan P, Kanagachidambaresan G (2021) IoT-based smart healthcare video surveillance system using edge computing. J Ambient Intell Humaniz Comput 03:1–13

    Google Scholar 

  9. Plageras A, Psannis K, Ishibashi Y, Kim BG (2016) IoT-based surveillance system for ubiquitous healthcare

  10. Islam M, Rahaman A, Islam R (2020) Development of smart healthcare monitoring system in IoT environment. SN Comput Sci 05(1):185

    Article  Google Scholar 

  11. Khan J, Li JP, Ahamad B, Parveen S, Ul Haq A, Khan GA et al (2020) SMSH: secure surveillance mechanism on smart healthcare IoT system with probabilistic image encryption. IEEE Access 8:15747–15767

    Article  Google Scholar 

  12. Haque A, Milstein A, Fei-Fei L (2020) Illuminating the dark spaces of healthcare with ambient intelligence. Nature 09(585):193–202

    Article  Google Scholar 

  13. Nayyar A, Puri V (2016) Smart farming: IoT based smart sensors agriculture stick for live temperature and moisture monitoring using Arduino, cloud computing and solar technology, pp 673–680

  14. Farooq MS, Riaz S, Abid A, Umer T, Zikria YB (2020) Role of IoT technology in agriculture: a systematic literature review. Electronics 9(2). Available from: https://www.mdpi.com/2079-9292/9/2/319

  15. Boobalan J, Jacintha V, Nagarajan J, Thangayogesh K, Tamilarasu S (2018) An IOT based agriculture monitoring system. In: 2018 international conference on communication and signal processing (ICCSP), pp 0594–0598

  16. Ayaz M, Ammad-Uddin M, Sharif Z, Mansour A, Aggoune EHM (2019) Internet-of-Things (IoT)-based smart agriculture: toward making the fields talk. IEEE Access 7:129551–129583

    Article  Google Scholar 

  17. Hussain T, Muhammad K, Khan S, Ullah A, Lee M, Baik S (2019) Intelligent baby behavior monitoring using embedded vision in IoT for smart healthcare centers. J Artif Intell Syst 01(1):110–124

    Article  Google Scholar 

  18. Sharma S, Rajan udeja R, Gagangeet ujla S, Rasmeet et al (2020) DeTrAs: deep learning-based healthcare framework for IoT-based assistance of Alzheimer patients. Neural Comput Appl pp 1-13

  19. Vasisht D, Kapetanovic Z, Won J, Jin X, Chandra R, Sinha S, et al (2017) FarmBeats: An IoT platform for data-driven agriculture. In: 14th USENIX symposium on networked systems design and implementation (NSDI 17). Boston, MA: USENIX Association, pp 515–529. Available from: https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/vasisht

  20. Datta SK, Dugelay JL, Bonnet C (2018) IoT Based UAV platform for emergency services. In: 2018 international conference on information and communication technology convergence (ICTC), pp 144–147

  21. Sandvine Global Interrnet Phenomena Report (2020) Available from: https://www.sandvine.com/phenomena

  22. Dash.js Available from: https://github.com/Dash-Industry-Forum/dash.js/

  23. Shahzadi S, Iqbal M, Dagiuklas T, Qayyum ZU (2017) Multi-access edge computing: open issues, challenges and future perspectives. J Cloud Comput 6(1):1–13

    Article  Google Scholar 

  24. Jiang X, Yu FR, Song T, Leung VC (2021) A survey on multi-access edge computing applied to video streaming: some research issues and challenges. IEEE Commun Surv Tutor 23(2):871–903

    Article  Google Scholar 

  25. Li Y, Frangoudis PA, Hadjadj-Aoul Y, Bertin P (2016) A mobile edge computing-based architecture for improved adaptive HTTP video delivery. In: 2016 IEEE conference on standards for communications and networking (CSCN). IEEE, pp 1–6

  26. Bentaleb A, Taani B, Begen AC, Timmerer C, Zimmermann R (2018) A survey on bitrate adaptation schemes for streaming media over HTTP. IEEE Commun Surv Tutor 21(1):562–585

    Article  Google Scholar 

  27. Spiteri K, Urgaonkar R, Sitaraman RK (2020) BOLA: near-optimal bitrate adaptation for online videos. IEEE/ACM Trans Netw 28(4):1698–1711

    Article  Google Scholar 

  28. Sutton RS, Barto AG (2011) Reinforcement learning: an introduction

  29. Mao H, Netravali R, Alizadeh M (2017) Neural adaptive video streaming with pensieve. In: Proceedings of the conference of the ACM special interest group on data communication, pp 197–210

  30. Saxena P, Naresh M, Gupta M, Achanta A, Kota S, Gupta S (2020) NANCY: neural adaptive network coding methodology for video distribution over wireless networks. arXiv preprint arXiv:2008.09559

  31. Mnih V, Badia AP, Mirza M, Graves A, Harley T, Lillicrap TP, et al (2016) Asynchronous methods for deep reinforcement learning. In: Proceedings of the 33rd international conference on international conference on machine learning - Volume 48. ICML’16. JMLR.org, pp 1928–1937

  32. Babaeizadeh M, Frosio I, Tyree S, Clemons J, Kautz J (2016) GA3C: GPU-based A3C for deep reinforcement learning. CoRR abs/161106256

  33. Espeholt L, Soyer H, Munos R, Simonyan K, Mnih V, Ward T, et al (2018) Impala: scalable distributed deep-rl with importance weighted actor-learner architectures. In: International conference on machine learning. PMLR, pp 1407–1416

  34. Chen S, Zhang XF, Wu JJ, Liu D (2018) Averaged-A3C for asynchronous deep reinforcement learning. International conference on neural information processing. Springer, Berlin, pp 277–288

    Chapter  Google Scholar 

  35. Holliday JB, Le TN (2020) follow then forage exploration: improving asynchronous advantage actor critic. In: International conference on soft computing, artificial intelligence and applications (SAI). pp 107–118

  36. Huang TY, Johari R, McKeown N, Trunnell M, Watson M (2014) A buffer-based approach to rate adaptation: evidence from a large video streaming service. In: Proceedings of the 2014 ACM Conference on SIGCOMM. SIGCOMM ’14. pp 187–198. Association for Computing Machinery, New York

  37. Sun Y, Yin X, Jiang J, Sekar V, Lin F, Wang N, et al (2016) CS2P: Improving video bitrate selection and adaptation with data-driven throughput prediction. In: Proceedings of the 2016 ACM SIGCOMM Conference

  38. Jiang J, Sekar V, Zhang H (2012) Improving fairness, efficiency, and stability in HTTP-based adaptive video streaming with FESTIVE. CoNEXT ’12. pp 97–108. Association for Computing Machinery, New York

  39. Spiteri K, Urgaonkar R, Sitaraman RK (2016) BOLA: near-optimal bitrate adaptation for online videos. In: IEEE INFOCOM 2016 - The 35th Annual IEEE international conference on computer communications, pp 1–9

  40. Federal Communications Commission (2016) Raw Data - Measuring Broadband America, Available from: https://www.fcc.gov/reports-research/reports/ measuring- broadband- america/raw- data- measuring- broadband- america- 2016

  41. Riiser H, Vigmostad P, Griwodz C, Halvorsen P (2013) Commute path bandwidth traces from 3G networks: analysis and applications. MMSys ’13. pp 114–118. Association for Computing Machinery, New York

  42. Akhtar Z (2018) Oboe: Auto-tuning Video ABR Algorithms to Network Conditions. Oboe: Auto-tuning Video ABR Algorithms to Network Conditions. August 20–25, Budapest, Hungary

  43. Yin X, Jindal A, Sekar V, Sinopoli B (2015) A control-theoretic approach for dynamic adaptive video streaming over HTTP. SIGCOMM Comput Commun Rev 45(4):325–338

    Article  Google Scholar 

  44. De Cicco L, Caldaralo V, Palmisano V, Mascolo S (2013) Elastic: a client-side controller for dynamic adaptive streaming over http (dash). In: 20th International packet video workshop. IEEE, pp 1–8

  45. Yousef H, Feuvre JL, Storelli A (2020) ABR prediction using supervised learning algorithms. In: 2020 IEEE 22nd international workshop on multimedia signal processing (MMSP), pp 1–6

  46. Sani Y, Raca D, Quinlan JJ, Sreenan CJ (2020) SMASH: A supervised machine learning approach to adaptive video streaming over HTTP. In: 2020 twelfth international conference on quality of multimedia experience (QoMEX), pp 1–6

  47. Huang T, Sun L (2020) Deepmpc: a mixture abr approach via deep learning And Mpc. In: 2020 IEEE International conference on image processing (ICIP), pp 1231–1235

  48. Amour L, Souihi S, Mellouk A, Mushtaq MS (2019) Q2ABR: QoE-aware adaptive video bit rate solution. Int J Commun Syst 11:33

    Google Scholar 

  49. Tian Z, Zhao L, Nie L, Chen P, Chen S (2019) Deeplive: QoE optimization for live video streaming through deep reinforcement learning. In: 2019 IEEE 25th international conference on parallel and distributed systems (ICPADS), pp 827–831

  50. Liu J, Tao X, Lu J (2019) QoE-oriented rate adaptation for DASH with enhanced deep Q-learning. IEEE Access 7:8454–8469

    Article  Google Scholar 

  51. Mao H, Netravali R, Alizadeh M (2017) Neural adaptive video streaming with pensieve. In: Proceedings of the conference of the ACM special interest group on data communication. SIGCOMM ’17. pp 197–210. Association for Computing Machinery, New York

  52. Akhtar Z, Nam YS, Govindan R, Rao S, Chen J, Katz-Bassett E, et al (2018) Oboe: auto-tuning video ABR algorithms to network conditions. In: Proceedings of the 2018 conference of the ACM special interest group on data communication. SIGCOMM ’18. pp 44–58. Association for Computing Machinery, New York

  53. Mnih V, Badia AP, Mirza M, Graves A, Lillicrap TP, Harley T et al (2016) Asynchronous Methods for Deep Reinforcement Learning. CoRR arXiv:1602.01783

  54. Yang SR, Tseng YJ, Huang CC, Lin WC (2018) Multi-access edge computing enhanced video streaming: proof-of-concept implementation and prediction/QoE models. IEEE Trans Veh Technol 68(2):1888–1902

    Article  Google Scholar 

  55. Schulman J, Moritz P, Levine S, Jordan M, Abbeel P (2018) High-dimensional continuous control using generalized advantage estimation

  56. Tuli S, Ilager S, Buyya R (2020) Dynamic scheduling for stochastic edge-cloud computing environments using A3C learning and residual recurrent neural networks

  57. Goudarzi M (2021) A distributed deep reinforcement learning technique for application placement in edge and fog computing environments

  58. Wang X, Wang C, Li X, Leung VC, Taleb T (2020) Federated deep reinforcement learning for Internet of Things with decentralized cooperative edge caching. IEEE Internet Things J 7(10):9441–9455

    Article  Google Scholar 

  59. Wu CL, Chiu TC, Wang CY, Pang AC (2020) Mobility-aware deep reinforcement learning with glimpse mobility prediction in edge computing. In: ICC 2020-2020 IEEE international conference on communications (ICC). IEEE, pp 1–7

  60. Mondal A, Palit B, Khandelia S, Pal N, Jayatheerthan J, Paul K et al (2020) Efficient EnDASH-A mobility adapted energy, video streaming ABR, for cellular networks. In: IFIP networking conference (Networking). IEEE, pp 127–135

  61. Chen S, Zhang XF, Wu JJ, Liu D (2018) Averaged-A3C for asynchronous deep reinforcement learning. In: Cheng L, Leung ACS, Ozawa S (eds) Neural information processing. Springer International Publishing, Cham, pp 277–288

    Chapter  Google Scholar 

  62. Netravali R, Sivaraman A, Das S, Goyal A, Winstein K, Mickens J, et al (2015) Mahimahi: accurate record-and-replay for HTTP. USENIX ATC ’15. pp 417–429. USENIX Association, USA

  63. Narayanan A, Ramadan E, Carpenter J, Liu Q, Liu Y, Qian F et al (2020) A first look at commercial 5G performance on smartphones. In: Proceedings of the web conference, pp 894–905

Download references

Acknowledgements

This work has been supported by TCS foundation, India under the TCS research scholar program, 2019-2023.

Author information

Authors and Affiliations

  1. Department of CSIS, BITS Pilani, Hyderabad, India

    Mandan Naresh, Vikramjeet Das, Paresh Saxena & Manik Gupta

Authors
  1. Mandan Naresh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vikramjeet Das
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paresh Saxena
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Manik Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paresh Saxena.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Naresh, M., Das, V., Saxena, P. et al. Deep reinforcement learning based QoE-aware actor-learner architectures for video streaming in IoT environments. Computing 104, 1527–1550 (2022). https://doi.org/10.1007/s00607-021-01046-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-021-01046-1

Keywords

Mathematics Subject Classification

Search

Navigation