Skip to main content

Advertisement

SpringerLink
Log in

Qoe-guaranteed distributed offloading decision via partially observable deep reinforcement learning for edge-enabled Internet of Things

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In edge-enabled Internet of Things (IoT), Quality of Experience (QoE)-guaranteed offloading decision is to decide which IoT tasks can be offloaded to edge servers with QoE guarantee. Centralized QoE-guaranteed offloading decision methods construct a global decision model for all IoT tasks with complete information. However, centralized offloading decision methods entail collecting global information from IoT devices, edge servers, and network environment, which may not be practical in large-scale distributed edge-enabled IoT environments, and it is unrealistic for privacy-critical and heterogeneous IoT tasks in many real-world edge-enabled IoT systems, where IoT devices may refuse to expose their private information and heterogeneous IoT tasks may have different QoE requirements, these issues make the application of centralized offloading decision method limited. To address these limitations, we propose a distributed offloading decision method which enables each IoT device to make decisions by partially observable global information in a decentralized manner. The distributed offloading decision process is modeled as a multi-agent partially observable Markov decision process, and a novel model-free deep reinforcement learning-based distributed algorithm named GRU Fictitious Self-Play Dueling Double Deep Recurrent Q Network(GFSP-D3RQN) is introduced to solve the problem. Furthermore, we measure the QoE of each IoT device based on a combination of latency and energy consumption, which are weighted differently according to the individual preferences of each IoT device, using a non-dimensionalized adjustment to accommodate the varying requirements of these IoT devices. Extensive simulation results show that our algorithm can achieve a higher average QoE and higher success ratio compared with baseline algorithms, which improved by at least 6.38\(\%\) and 5.91\(\%\), respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availibility

The data used in this paper can be obtained by numerical simulation through the experimental design in this paper.

References

  1. Hong S-T, Kim H (2019) QoE-aware computation offloading to capture energy-latency-pricing tradeoff in mobile clouds. IEEE Trans Mob Comput 18(9):2174–2189

    Article  Google Scholar 

  2. Meng Q, Wang K, Liu B, Miyazaki T, He X (2018) QoE-based big data analysis with deep learning in pervasive edge environment. In: Proceedings of the IEEE International Conference on Communications (ICC), Kansas City, MO, USA, Jul. , pp 1–6

  3. Yuyi Mao et al (2017) A survey on mobile edge computing: the communication perspective. IEEE Commun Surv Tutor 19(4):2322–2358

    Article  Google Scholar 

  4. Li X, Sun C, Wen J et al (2022) Multi-user QoE enhancement: federated multi-agent reinforcement learning for cooperative edge intelligence. IEEE Netw 36(5):144–151

    Article  Google Scholar 

  5. Jiang H, Dai X, Xiao Z, Iyengar AK (2022) Joint task offloading and resource allocation for energy-constrained mobile edge computing. IEEE Trans Mob Comput. https://doi.org/10.1109/TMC.2022.3150432

    Article  Google Scholar 

  6. Miao Hu et al (2020) Heterogeneous edge offloading with incomplete information: a minority game approach. IEEE Trans Parallel Distrib Syst 31(9):2139–2154

    Article  Google Scholar 

  7. Alqerm I, Pan J (2021) DeepEdge: a new QoE-based resource allocation framework using deep reinforcement learning for future heterogeneous edge-IoT applications. IEEE Trans Netw Serv Manage 18(4):3942–3954. https://doi.org/10.1109/TNSM.2021.3123959

    Article  Google Scholar 

  8. Jungyeon Baek, Kaddoum Georges (2021) Online partial offloading and task scheduling in SDN-Fog networks with deep recurrent reinforcement learning. IEEE Internet Things J. https://doi.org/10.1109/JIOT.2021.3130474

    Article  Google Scholar 

  9. Baek J, Kaddoum G (2021) Heterogeneous task offloading and resource allocations via deep recurrent reinforcement learning in partial observable multifog networks. IEEE Internet Things J 8(2):1041–1056. https://doi.org/10.1109/JIOT.2020.3009540

    Article  Google Scholar 

  10. Huo Y, Meng C, Li R, Jing T (2020) An overview of privacy preserving schemes for industrial Internet of Things. China Commun 17(10):1–18

    Article  Google Scholar 

  11. Zeng Y , Huang Y , Liu J et al (2020) Privacy-preserving distributed edge caching for mobile data offloading in 5G networks. In: IEEE 40th International conference on distributed computing systems. IEEE

  12. Jošilo S, Dán G (2019) Decentralized algorithm for randomized task allocation in fog computing systems. IEEE/ACM Trans Netw 27(1):85–97. https://doi.org/10.1109/TNET.2018.2880874

    Article  Google Scholar 

  13. Chen Y, Zhao J, Wu Y et al (2022) Qoe-aware decentralized task offloading and resource allocation for end-edge-cloud systems: a game-theoretical approach. IEEE Trans Mob Comput. https://doi.org/10.1109/TMC.2022.3223119

    Article  Google Scholar 

  14. Chen Y, Zhang S, Xiao M, Qian Z, Wu J, Lu S (2020) Multi-user edge-assisted video analytics task offloading game based on deep reinforcement learning. In: 2020 IEEE 26th international conference on parallel and distributed systems, ICPADS

  15. Chen Z, Wang X (2020) Decentralized computation offloading for multi-user mobile edge computing: a deep reinforcement learning approach. EURASIPJ Wirel Commun Network 2020(1):1–21

    Google Scholar 

  16. Wu Y-C, Dinh TQ, Fu Y, Lin C, Quek TQS (2021) A hybrid DQN and optimization approach for strategy and resource allocation in MEC networks. IEEE Trans Wirel Commun 20(7):4282–4295. https://doi.org/10.1109/TWC.2021.3057882

    Article  Google Scholar 

  17. Afrin M, Jin J, Rahman A, Rahman A, Wan J, Hossain E (2021) Resource allocation and service provisioning in multi-agent cloud robotics: a comprehensive survey. IEEE Commun Surv Tutor Secondquarter 23(2):842–870. https://doi.org/10.1109/COMST.2021.3061435

    Article  Google Scholar 

  18. Wang J (2022) Edge artificial intelligence-based affinity task offloading under resource adjustment in a 5G network. Appl Intell 52:8167–8188

    Article  Google Scholar 

  19. Papoudakis G, Christianos F, Rahman A Albrecht SV (2019) Dealing with non-stationarity in multi-agent deep reinforcement learning. arXiv:1906.04737 [cs, stat]

  20. Kumar RR, Varakantham P (2020) On solving cooperative MARL problems with a few good experiences. arXiv:2001.07993 [Cs]

  21. Heess N, Hunt JJ, Lillicrap TP Silver D (2015) Memory-based control with recurrent neural networks. arXiv preprint arXiv:1512.04455

  22. Guo C, Liang L, Li GY (2019) Resource allocation for high-reliability low-latency vehicular communications with packet retransmission. IEEE Trans Veh Technol 68(7):6219–6230

    Article  Google Scholar 

  23. Guo Hongzhi, Liu Jiajia (2018) Collaborative computation offloading for multiaccess edge computing over fiber-wireless networks. IEEE Trans Veh Technol 67(5):4514–4526

    Article  Google Scholar 

  24. Haodong Lu et al (2020) Edge QoE: computation offloading with deep reinforcement learning for Internet of Things. IEEE Internet Things J 7(10):9255–9265

    Article  Google Scholar 

  25. Alexey Rudenko et al (1998) Saving portable computer battery power through remote process execution. ACM SIGMOBILE Mob Comput Commun Rev 2(1):19–26

    Article  Google Scholar 

  26. Xu Jie, Chen Lixing, Ren Shaolei (2017) Online learning for offloading and autoscaling in energy harvesting mobile edge computing. IEEE Trans Cognit Commun Netw 3(3):361–373

    Article  Google Scholar 

  27. Pepper (2019) Robert: Cisco visual networking index (VNI) global mobile data traffic forecast update. In: Cisco, Tech. Rep., Feb. 2013. Accessed Jul 10

  28. Chen S, Tang B, Wang K (2022) Twin delayed deep deterministic policy gradient-based intelligent computation offloading for IoT. Digit Commun Netw. https://doi.org/10.1016/j.dcan.2022.06.008

    Article  Google Scholar 

  29. Wang P, Di B, Song L, Jennings NR (2022) Multi-layer computation offloading in distributed heterogeneous mobile edge computing networks. IEEE Trans Cognit Commun Netw 8(2):1301–1315. https://doi.org/10.1109/TCCN.2022.3161955

    Article  Google Scholar 

  30. Peng K, Huang H, Bilal M et al (2022) Distributed incentives for intelligent offloading and resource allocation in digital twin driven smart industry. IEEE Trans Ind Inform. https://doi.org/10.1109/TII.2022.3184070

    Article  Google Scholar 

  31. Pham XQ, Huynh-The T, Huh EN et al (2022) Partial computation offloading in parked vehicle-assisted multi-access edge computing: a game-theoretic approach. IEEE Trans Veh Technol 71(9):10220–10225

    Article  Google Scholar 

  32. Zhou J, Tian D, Sheng Z, Duan X, Shen X (2021) Distributed task offloading optimization with queueing dynamics in multiagent mobile-edge computing networks. IEEE Internet Things J 8(15):12311–12328. https://doi.org/10.1109/JIOT.2021.3063509

    Article  Google Scholar 

  33. Lu H et al (2020) Optimization of task offloading strategy for mobile edge computing based on multi-agent deep reinforcement learning. IEEE Access 8:202573–202584

    Article  Google Scholar 

  34. Cao Z, Zhou P, Li R, Huang S, Wu D (2020) Multiagent deep reinforcement learning for joint multichannel access and task offloading of mobile-edge computing in industry 4.0. IEEE Internet Things J 7(7):6201–6213. https://doi.org/10.1109/JIOT.2020.2968951

    Article  Google Scholar 

  35. Chen S, Chen J, Miao Y, Wang Q, Zhao C (2022) Deep reinforcement learning-based cloud-edge collaborative mobile computation offloading in industrial networks. IEEE Trans Signal Inf Process Netw 8:364–375. https://doi.org/10.1109/TSIPN.2022.3171336

    Article  MathSciNet  Google Scholar 

  36. Ming Tang, Wong Vincent WS (2020) Deep reinforcement learning for task offloading in mobile edge computing systems. IEEE Trans Mobile Comput. https://doi.org/10.1109/TMC.2020.3036871

    Article  Google Scholar 

  37. Zhan Y, Guo S, Li P, Zhang J (2020) A deep reinforcement learning based offloading game in edge computing. IEEE Trans Comput 69(6):883–893. https://doi.org/10.1109/TC.2020.2969148

    Article  MathSciNet  MATH  Google Scholar 

  38. Huang J, Wang M, Wu Y, Chen Y, Shen X (2022) Distributed offloading in overlapping areas of mobile edge computing for Internet of Things. IEEE Internet Things J. https://doi.org/10.1109/JIOT.2022.3143539

    Article  Google Scholar 

  39. Liu Jk, Mao Y, Zhang J, Letaief KB (2016) Delay-optimal computation task scheduling for mobile-edge computing systems. In: IEEE international symposium on information theory (ISIT), pp 1451–1455. https://doi.org/10.1109/ISIT.2016.7541539.

  40. Rappaport TS (1996) Wireless communications: principles and practice, vol 2. Prentice Hall PTR, New Jersey

    MATH  Google Scholar 

  41. Zou J, Hao T, Yu C, Jin H (2021) A3C-DO: a regional resource scheduling framework based on deep reinforcement learning in edge scenario. IEEE Trans Comput 70(2):228–239

    Article  MATH  Google Scholar 

  42. Nguyen DC et al (2021) Deep reinforcement learning for collaborative offloading in heterogeneous edge networks. In: IEEE/ACM 21st International symposium on cluster, cloud and internet computing (CCGrid). IEEE

  43. Miettinen Antti P, Nurminen Jukka K (2010) Energy efficiency of mobile clients in cloud computing. HotCloud 10(4–4):19

    Google Scholar 

  44. Heinrich J, Silver D (2016) Deep reinforcement learning from self-play in imperfect-information games. arXiv preprint arXiv:1603.01121

  45. Heinrich J, Lanctot M, Silver D (2015) Fictitious self-play in extensive-form games. In: Proceedings of the 32nd international conference on machine learning

  46. Kawamura K, Mizukami N, Tsuruoka Y (2018) Neural fictitious self-play in imperfect information games with many players. In: Computer games, communications in computer and information science

  47. Tampuu A et al (2017) Multiagent cooperation and competition with deep reinforcement learning. PloS One 12(4):e0172395

    Article  Google Scholar 

  48. Wang Z et al (2016) Dueling network architectures for deep reinforcement learning. In: International conference on machine learning. PMLR

  49. Van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI conference on artificial intelligence 30(1)

  50. Hausknecht M, Stone P (2015) Deep recurrent q-learning for partially observable mdps. In: 2015 AAAI fall symposium series

  51. Kapturowski S et al (2018) Recurrent experience replay in distributed reinforcement learning. In: International conference on learning representations

  52. Yang Y, Wang J (2020) An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:2011.00583

  53. Liu L, Li H, Gruteser M (2019) Edge assisted real-time object detection for mobile augmented reality. In: Proceedings of the 25th annual international conference on mobile computing networking, pp 1–16

  54. Wang W et al (2017) Gated self-matching networks for reading comprehension and question answering. In: Proceedings of the 55th annual meeting of the association for computational Linguistics, vol 1. Long Papers

  55. Zhang L, Jabbari B, Ansari N (2022) Deep reinforcement learning driven UAV-assisted edge computing. IEEE Internet Things J 9(24):25449–25459. https://doi.org/10.1109/JIOT.2022.3196842

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by National Key Research and Development Project under Grant 2019YFB1706101; in part by the Science-Technology Foundation of Chongqing, China under Grant cstc2019jscx-mbdxX0047; in part by the Fundamental Research Funds for the Central Universities under Grant 2020CDCGRJ50.

Author information

Authors and Affiliations

  1. The School of Big data and Software Engineering, Chongqing University, Chongqing, 400044, China

    Jiaxin Hou, Yingbo Wu, Junpeng Cai & Zhiwen Zhou

Authors
  1. Jiaxin Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yingbo Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junpeng Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhiwen Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yingbo Wu.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hou, J., Wu, Y., Cai, J. et al. Qoe-guaranteed distributed offloading decision via partially observable deep reinforcement learning for edge-enabled Internet of Things. Neural Comput & Applic 35, 21603–21619 (2023). https://doi.org/10.1007/s00521-023-08905-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08905-2

Keywords

Search

Navigation