Skip to main content

Advertisement

SpringerLink
Log in

Deep Reinforcement Learning-based Power Control and Bandwidth Allocation Policy for Weighted Cost Minimization in Wireless Networks

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Mobile edge computing (MEC) can dispatch its powerful servers close by to assist with the computation workloads that intelligent wireless terminals have offloaded. The MEC server’s physical location is closer to the intelligent wireless terminals, which can satisfy the low latency and high reliability demands. In this paper, we formulate an MEC framework with multiple vehicles and service devices that considers the priority and randomness of arriving workloads from roadside units (RSUs), cameras, laser radars (Lidar) and the time-varying channel state between the service device and MEC server (MEC-S). To minimize the long-term weighted average cost of the proposed MEC system, we transit this issue (cost minimization problem) into the Markov decision process (MDP). Furthermore, considering the difficulty realizing the state transition probability matrix, the dimensional complexity of the state space, and the continuity of the action space, we propose a deterministic policy gradient (MADDPG)-based bandwidth partition and power allocation optimization policy. The proposed MADDPG-based policy is a model-free deep reinforcement learning (DRL) method, which can effectively deal with continuous action space and further guide multi-agent to execute decision-making. The comprehensive results verify that the proposed MADDPG-based optimization scheme has fine convergence and performance that is better than that of the other four baseline algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Lakhan A, Ahmad M, Bilal M, Jolfaei A, Mehmood RM (2021) Mobility aware blockchain enabled offloading and scheduling in vehicular fog cloud computing. IEEE Trans Intell Transp Syst 22(7):4212–4223

    Article  Google Scholar 

  2. Zhang X, Wang Y (2022) Deepmecagent: multi-agent computing resource allocation for uav-assisted mobile edge computing in distributed iot system. Appl Intell 1–12

  3. Ding Y, Li K, Liu C, Li K (2021) A potential game theoretic approach to computation offloading strategy optimization in end-edge-cloud computing. IEEE Trans Parallel Distrib Syst 33(6):1503–1519

    Article  Google Scholar 

  4. Alhelaly S, Muthanna A, Elgendy IA (2022) Optimizing task offloading energy in multi-user multi-uav-enabled mobile edge-cloud computing systems. Appl Sci 12(13):6566

    Article  Google Scholar 

  5. Cozzolino V, Tonetto L, Mohan N, Ding AY, Ott J (2022) Nimbus: Towards latency-energy efficient task offloading for ar services. IEEE Trans Cloud Comput

  6. Jin X, Hua W,Wang Z, Chen Y (2022) A survey of research on computation offloading in mobile cloud computing. Wireless Networks 1–23

  7. Hu S, Xiao Y (2021) Design of cloud computing task offloading algorithm based on dynamic multi-objective evolution. Future Generation Computer Systems 122:144–148

    Article  Google Scholar 

  8. De D, Mukherjee A, Guha Roy D (2020) Power and delay efficient multilevel offloading strategies for mobile cloud computing. Wireless Personal Communications 112(4):2159–2186

    Article  Google Scholar 

  9. Plachy J, Becvar Z, Strinati EC, Pietro Nd (2021) Dynamic allocation of computing and communication resources in multi-access edge computing for mobile users. IEEE Trans Netw Serv Manag 18(2):2089–2106. https://doi.org/10.1109/TNSM.2021.3072433

    Article  Google Scholar 

  10. uz Zaman SK, Jehangiri AI, Maqsood T, Ahmad Z, Umar AI, Shuja J, Alanazi E, Alasmary W (2021) Mobility-aware computational offloading in mobile edge networks: a survey. Cluster Computing 1–22

  11. Chakraborty S, De D, Mazumdar K (2022) Dome: Dew computing based microservice execution in mobile edge using q-learning. Appl Intell 1–20

  12. Shuja J, Bilal K, Alasmary W, Sinky H, Alanazi E (2021) Applying machine learning techniques for caching in next-generation edge networks: A comprehensive survey. J Netw Comput App 103005

  13. Zhao F, Chen Y, Zhang Y, Liu Z, Chen X (2021) Dynamic offloading and resource scheduling for mobile-edge computing with energy harvesting devices. IEEE Trans Netw Serv Manag 18(2):2154–2165. https://doi.org/10.1109/TNSM.2021.3069993

    Article  Google Scholar 

  14. Tian K, Chai H, Liu Y, Liu B (2022) Edge intelligence empowered dynamic offloading and resource management of mec for smart city internet of things. Electronics 11(6):879

    Article  Google Scholar 

  15. Gao M, Shen R, Li J, Yan S, Li Y, Shi J, Han Z, Zhuo L (2020) Computation offloading with instantaneous load billing for mobile edge computing. IEEE Trans Serv Comput

  16. Alfakih T, Hassan MM, Gumaei A, Savaglio C, Fortino G (2020) Task offloading and resource allocation for mobile edge computing by deep reinforcement learning based on sarsa. IEEE Access 8:54074–54084

    Article  Google Scholar 

  17. Hadi M, Ghazizadeh R (2022) Joint resource allocation, user clustering and 3-d location optimization in multi-uav-enabled mobile edge computing. Computer Networks 109420

  18. Wang Z, Lv T, Chang Z (2022) Computation offloading and resource allocation based on distributed deep learning and software defined mobile edge computing. Computer Networks 205:108732

    Article  Google Scholar 

  19. Lu W, Mo Y, Feng Y, Gao Y, Zhao N, Wu Y, Nallanathan A (2022) Secure transmission for multi-uav-assisted mobile edge computing based on reinforcement learning. IEEE Trans Netw Sci Eng

  20. Jitani A, Mahajan A, Zhu Z, Abou-Zeid H, Fapi ET, Purmehdi H (2022) Structure-aware reinforcement learning for node-overload protection in mobile edge computing. IEEE Trans Cogn Commun Netw

  21. Sutton RS, Barto AG et al (1998) Introduction to Reinforcement Learning vol 135. MIT press Cambridge, ???

  22. Sutton RS, McAllester DA, Singh SP, Mansour Y (2000) Policy gradient methods for reinforcement learning with function approximation. In: Advances in Neural Information Processing Systems 1057–1063

  23. Lyu Y, Liu Z, Fan R, Zhan C, Hu H, An J (2022) Optimal computation offloading in collaborative leo-iot enabled mec: A multi-agent deep reinforcement learning approach. IEEE Trans Green Commun Netw

  24. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529

    Article  Google Scholar 

  25. Van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: Thirtieth AAAI Conference on Artificial Intelligence

  26. Wang Z, Schaul T, Hessel M, Hasselt H, Lanctot M, Freitas N (2016) Dueling network architectures for deep reinforcement learning. In: International Conference on Machine Learning 1995–2003

  27. Kakade SM (2001) A natural policy gradient. Advances in neural information processing systems 14

  28. Mnih V, Badia AP, Mirza M, Graves A, Lillicrap T, Harley T, Silver D, Kavukcuoglu K (2016) Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning 1928–1937

  29. Haarnoja T, Zhou A, Abbeel P, Levine S (2018) Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International Conference on Machine Learning 1861–1870 PMLR

  30. Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D (2016) Continuous control with deep reinforcement learning 1–14

  31. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. https://doi.org/https://arxiv.org/pdf/1707.06347.pdf

  32. Abouaomar A, Mlika Z, Filali A, Cherkaoui S, Kobbane A (2021) A deep reinforcement learning approach for service migration in mec-enabled vehicular networks. In: 2021 IEEE 46th Conference on Local Computer Networks (LCN) 273–280. IEEE

  33. Wang L, Wang K, Pan C, Xu W, Aslam N, Nallanathan A (2021) Deep reinforcement learning based dynamic trajectory control for uav-assisted mobile edge computing. IEEE Transactions on Mobile Computing

  34. Karimi E, Chen Y, Akbari B (2022) Task offloading in vehicular edge computing networks via deep reinforcement learning. Computer Communications 189:193–204

    Article  Google Scholar 

  35. Nduwayezu M, Yun JH (2022) Latency and energy aware rate maximization in mc-noma-based multi-access edge computing: A two-stage deep reinforcement learning approach. Computer Networks 207:108834

    Article  Google Scholar 

  36. Ngo HQ, Larsson EG, Marzetta TL (2013) Energy and spectral efficiency of very large multiuser mimo systems. IEEE Trans Commun 61(4):1436–1449

    Article  Google Scholar 

  37. Ke H, Wang J, Deng L, Ge Y, Wang H (2020) Deep reinforcement learning-based adaptive computation offloading for mec in heterogeneous vehicular networks. IEEE Trans Veh Technol 69(7):7916–7929

    Article  Google Scholar 

  38. Chen Z, Zhang L, Pei Y, Jiang C, Yin L (2021) Noma-based multi-user mobile edge computation offloading via cooperative multi-agent deep reinforcement learning. IEEE Trans Cogn Commun Netw 8(1):350–364

  39. Chen Z (2020) Wang X (2020) Decentralized computation offloading for multiuser mobile edge computing: A deep reinforcement learning approach. EURASIP Journal on Wireless Communications and Networking 1:1–21

    Google Scholar 

  40. Kuang Z, Shi Y, Guo S, Dan J, Xiao B (2019) Multi-user offloading game strategy in ofdma mobile cloud computing system. IEEE Trans Veh Technol 68(12):12190–12201

    Article  Google Scholar 

  41. Wu Y, Wang Y, Zhou F, Hu RQ (2019) Computation efficiency maximization in ofdma-based mobile edge computing networks. IEEE Commun Lett 24(1):159–163

    Article  Google Scholar 

  42. Chen X, Jiao L, Li W, Fu X (2015) Efficient multi-user computation offloading for mobile-edge cloud computing. IEEE/ACM transactions on networking 24(5):2795–2808

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Jilin Provincial Science and Technology Department Natural Science Foundation of China (20210101415JC) and the Jilin Provincial Science and Technology Department Free Exploration Research Project of China (YDZJ202201ZYTS556).

Author information

Authors and Affiliations

  1. School of Computer Technology and Engineering, Changchun Institute of Technology, Kuanping, Changchun, 130012, JiLin, China

    Hongchang Ke & Hongbin Sun

  2. National and Local Joint Engineering Research Center for Intelligent Distribution Network Measurement, Control and Safe Operation Technology of National Development and Reform Commission, Changchun Institute of Technology, Kuanping, Changchun, 130012, JiLin, China

    Hongchang Ke & Hongbin Sun

  3. College of Computer Science and Engineering, Changchun University of Technology, Yan’an, Changchun, 130012, JiLin, China

    Hui Wang

Authors
  1. Hongchang Ke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongbin Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hui Wang.

Ethics declarations

Conflicts of interest

The authors declare that there are no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Hongchang Ke, Hui Wang and Hongbin Sun These authors contributed equally to this work.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ke, H., Wang, H. & Sun, H. Deep Reinforcement Learning-based Power Control and Bandwidth Allocation Policy for Weighted Cost Minimization in Wireless Networks. Appl Intell 53, 26885–26906 (2023). https://doi.org/10.1007/s10489-023-04929-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04929-2

Keywords

Search

Navigation