Skip to main content

Advertisement

SpringerLink
Log in

Computation offloading optimization for UAV-assisted mobile edge computing: a deep deterministic policy gradient approach

  • Original Paper
  • Published:
Wireless Networks Aims and scope Submit manuscript

Abstract

Unmanned Aerial Vehicle (UAV) can play an important role in wireless systems as it can be deployed flexibly to help improve coverage and quality of communication. In this paper, we consider a UAV-assisted Mobile Edge Computing (MEC) system, in which a UAV equipped with computing resources can provide offloading services to nearby user equipments (UEs). The UE offloads a portion of the computing tasks to the UAV, while the remaining tasks are locally executed at this UE. Subject to constraints on discrete variables and energy consumption, we aim to minimize the maximum processing delay by jointly optimizing user scheduling, task offloading ratio, UAV flight angle and flight speed. Considering the non-convexity of this problem, the high-dimensional state space and the continuous action space, we propose a computation offloading algorithm based on Deep Deterministic Policy Gradient (DDPG) in Reinforcement Learning (RL). With this algorithm, we can obtain the optimal computation offloading policy in an uncontrollable dynamic environment. Extensive experiments have been conducted, and the results show that the proposed DDPG-based algorithm can quickly converge to the optimum. Meanwhile, our algorithm can achieve a significant improvement in processing delay as compared with baseline algorithms, e.g., Deep Q Network (DQN).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Dinh, H. T., Lee, C., Niyato, D., & Wang, P. (2013). A survey of mobile cloud computing: architecture, applications, and approaches. Wireless communications and mobile computing, 13(18), 1587–1611

    Article  Google Scholar 

  2. Mach, P., & Becvar, Z. (2017). Mobile edge computing: A survey on architecture and computation offloading. IEEE Communications Surveys & Tutorials, 19(3), 1628–1656

    Article  Google Scholar 

  3. Abbas, N., Zhang, Y., & Taherkordi, A. (2018). Mobile edge computing: A survey. IEEE Internet of Things Journal, 5(1), 450–465

    Article  Google Scholar 

  4. Mao, Y., You, C., Zhang, J., Huang, K., & Letaief, K. B. (2017). A survey on mobile edge computing: The communication perspective. IEEE Communications Surveys & Tutorials, 19(4), 2322–2358

    Article  Google Scholar 

  5. Tran, T. X., & Pompili, D. (2018). Joint task offloading and resource allocation for multi-server mobile-edge computing networks. IEEE Transactions on Vehicular Technology, 18(1), 856–868

    Article  Google Scholar 

  6. Zhao, J., Li, Q., Gong, Y., & Zhang, K. (2019). Computation offloading and resource allocation for cloud assisted mobile edge computing in vehicular networks. IEEE Transactions on Vehicular Technology, 18(8), 7944–7956

    Article  Google Scholar 

  7. Chen, Z., & Wang, X. (2020). Decentralized computation offloading for multi-user mobile edge computing: A deep reinforcement learning approach. EURASIP Journal on Wireless Communications and Networking, 2020(1), 1–21

    Article  Google Scholar 

  8. Feng, G., Wang, C., & Li, B. (2019). UAV-assisted wireless relay networks for mobile offloading and trajectory optimization. Peer-to-Peer Networking and Applications, 12(6), 1820–1834

    Article  Google Scholar 

  9. Hu, Q., Cai, Y., Yu, G., Qin, Z., Zhao, M., & Li, G. Y. (2019). Joint offloading and trajectory design for UAV-enabled mobile edge computing systems. IEEE Internet of Things Journal, 6(2), 879–1892

    Google Scholar 

  10. Diao, X., Zheng, J., Cai, Y., Wu, Y., & Anpalagan, A. (2019). Fair data allocation and trajectory optimization for UAV-Assisted mobile edge computing. IEEE Communications Letters, 23(12), 2357–2361

    Article  Google Scholar 

  11. Cheng, N., Lyu, F., & Quan, W. (2019). Space/aerial-assisted computing offloading for IoT applications: A learning-based approach. IEEE Journal on Selected Areas in Communications, 37(5), 1117–1129

    Article  Google Scholar 

  12. Li, J., Liu, Q., Wu, P., Shu, F., & Jin, S. (2018). Task offloading for uav-based mobile edge computing via deep reinforcement learning. IEEE/CIC International Conference on Communications in China (ICCC), 2018, 798–802

    Google Scholar 

  13. Xiong, J., Guo, H., & Liu, J. (2019). Task offloading in UAV-aided edge computing: Bit allocation and trajectory optimization. IEEE Communications Letters, 23(3), 538–541

    Article  Google Scholar 

  14. Selim, M.M., Rihan, M., & Yang, Y. (2020). Optimal task partitioning, Bit allocation and trajectory for D2D-assisted UAV-MEC systems. Peer-to-Peer Networking and Applications, 1–10.

  15. Ge, L., Dong, P., & Zhang, H. (2020). Joint beamforming and trajectory optimization for intelligent reflecting surfaces-assisted UAV communications. IEEE Access, 8, 78702–78712

    Article  Google Scholar 

  16. Lillicrap, T.P., Hunt, J.J., & Pritzel, A. (2015). Continuous control with deep reinforcement learning. arXiv:1509.02971.

  17. Dai, Y., Xu, D., & Maharjan, S. (2019). Artificial intelligence empowered edge computing and caching for internet of vehicles. IEEE Wireless Communications, 26(3), 12–18

    Article  Google Scholar 

  18. Fang, W., Ding, S., Li, Y., Zhou, W., & Xiong, N. (2019). OKRA: optimal task and resource allocation for energy minimization in mobile edge computing systems. Wireless Networks, 25(4), 2851–2867

    Article  Google Scholar 

  19. Coldrey, M. (2013). Non-Line-of-Sight small cell backhauling using microwave technology. IEEE Communications Magazine, 51(9), 78–84

    Article  Google Scholar 

  20. Saleem, U., Liu, Y., Jangsher, S., & Li, Y. (2018). Performance guaranteed partial offloading for mobile edge computing. In 2018 IEEE Global Communications Conference (GLOBECOM) (pp. 1–6). IEEE, New York.

  21. Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. Cambridge: MIT press.

    MATH  Google Scholar 

  22. Nie, J., & Haykin, S. (1999). A Q-learning-based dynamic channel assignment technique for mobile communication systems. IEEE Transactions on Vehicular Technology, 48(5), 1676–1687

    Article  Google Scholar 

  23. Mnih, V., Kavukcuoglu, K., & Silver, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533

    Article  Google Scholar 

  24. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014, January). Deterministic policy gradient algorithms. In International conference on machine learning (pp. 387–395). PMLR.

  25. Qiu, C., Hu, Y., & Chen, Y. (2019). Deep deterministic policy gradient (DDPG)-based energy harvesting wireless communications. IEEE Internet of Things Journal, 6(5), 8577–8588

    Article  Google Scholar 

  26. Jeong, S., Simeone, O., & Kang, J. (2017). Mobile edge computing via a UAV-mounted cloudlet: Optimization of bit allocation and path planning. IEEE Transactions on Vehicular Technology, 67(3), 2049–2063

    Article  Google Scholar 

  27. Galatolo, F. A., Cimino, M. G., & Vaglini, G. (2021). Solving the scalarization issues of Advantage-based Reinforcement Learning algorithms. Computers & Electrical Engineering, 92, 107117.

Download references

Acknowledgements

This work is supported by the Beijing Municipal Natural Science Foundation (Joint Fund for Frontier Research of Fengtai Rail-Transit) under Grant L191019, the Open Project of Key Laboratory of Industrial Internet of Things & Networked Control, Ministry of Education under Grant 2019FF03, the Open Project of Beijing Intelligent Logistics System Collaborative Innovation Center under Grant BILSCIC-2019KF-10, the Traffic Control Technology Innovation Fund under Grant 9907006515, the Research Base Project of Beijing Municipal Social Science Foundation under Grant 18JDGLB026 and the Science and Technique General Program of Beijing Municipal Commission of Education under Grant KM201910037003.

Author information

Authors and Affiliations

  1. School of Computer and Information Technology, Beijing Jiaotong University, Beijing, 100044, China

    Yunpeng Wang & Weiwei Fang

  2. Key Laboratory of Industrial Internet of Things & Networked Control, Ministry of Education, Chongqing, 400065, China

    Weiwei Fang

  3. School of Information, Beijing Wuzi University, Beijing, 101149, China

    Yi Ding

  4. College of Intelligence and Computing, Tianjin University, Tianjin, 300350, China

    Naixue Xiong

Authors
  1. Yunpeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weiwei Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Naixue Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weiwei Fang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Fang, W., Ding, Y. et al. Computation offloading optimization for UAV-assisted mobile edge computing: a deep deterministic policy gradient approach. Wireless Netw 27, 2991–3006 (2021). https://doi.org/10.1007/s11276-021-02632-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11276-021-02632-z

Keywords

Search

Navigation