Advertisement

Unmanned Aerial Vehicles Path Planning Based on Deep Reinforcement Learning

  • Guoqiu Wang
  • Xuanyu Zheng
  • Haitong Zhao
  • Qidong Zhao
  • Changsheng ZhangEmail author
  • Bin Zhang
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1074)

Abstract

Obstacle avoidance and path planning of unmanned aerial vehicles (UAVs) is an essential and challenging task, especially in the unknown environment with dynamic obstacles. To address this problem, a method of UAV path planning based on Deep Q-Learning is proposed. The experience replay mechanism is introduced in the deep reinforcement learning (DRL) process, and a value network is established to calculate the optimal value for the action of the UAV. The optimal flight policy of the UAV is determined through the \(\epsilon \)-greed algorithm. The experimental results show that the UAV with well-trained model can avoid the obstacles in motion perfectly, and the cruise time is reduced by half compared with the untrained UAV.

Keywords

UAV path planning Obstacle avoidance Deep reinforcement learning DQN 

References

  1. 1.
    Dorling, K., Heinrichs, J., Messier, G.G.: Vehicle routing problems for drone delivery. IEEE Trans. Syst. Man Cybern. 47(1), 70–85 (2017).  https://doi.org/10.1109/TSMC.2016.2582745CrossRefGoogle Scholar
  2. 2.
    Yin, C., Xiao, Z., Cao, X., et al.: Offline and online search: UAV multiobjective path planning under dynamic urban environment. IEEE Internet Things J. 5(2), 546–558 (2018).  https://doi.org/10.1109/JIOT.2017.2717078CrossRefGoogle Scholar
  3. 3.
    Bevacqua, G., Cacace, J., Finzi, A., Lippiello, V.: Mixed-initiative planning and execution for multiple drones in search and rescue missions. In: Twenty-Fifth International Conference on Automated Planning and Scheduling. AAAI Publications (2015)Google Scholar
  4. 4.
    Chen, Y.B., Luo, G.C., Mei, Y.S., et al.: UAV path planning using artificial potential field method updated by optimal control theory. Int. J. Syst. Sci. 47(6), 1407–1420 (2016).  https://doi.org/10.1080/00207721.2014.929191MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Bortoff, S.A.: Path Planning for UAV. In: Proceedings of the 2000 American Control Conference. ACC (IEEE Cat. No. 00CH36334) (2000)Google Scholar
  6. 6.
    Peng, X., Xu, D., Zhang, F.: UAV online path planning based on dynamic multiobjective evolutionary algorithm. In: Proceedings of the 30th Chinese Control Conference, pp. 5424–5429. IEEE (2011)Google Scholar
  7. 7.
    Ragi, S., Chong, E.K.P.: UAV path planning in a dynamic environment via partially observable Markov decision process. IEEE Trans. Aerosp. Electron. Syst. 49(4), 2397–2412 (2013).  https://doi.org/10.1109/TAES.2013.6621824CrossRefGoogle Scholar
  8. 8.
    Roberge, V., Tarbouchi, M., Labonte, G.: Comparison of parallel genetic algorithm and particle swarm optimization for real-time UAV path planning. IEEE Trans. Industr. Inf. 9(1), 132–141 (2013).  https://doi.org/10.1109/TII.2012.2198665CrossRefGoogle Scholar
  9. 9.
    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015).  https://doi.org/10.1038/nature14236CrossRefGoogle Scholar
  10. 10.
    Chen, Y.F., Liu, M., Everett, M., et al.: Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 285–292. IEEE (2017).  https://doi.org/10.1109/ICRA.2017.7989037
  11. 11.
    Chen, C., Liu, Y., Kreiss, S., et al.: Crowd-robot interaction: crowd-aware robot navigation with attention-based deep reinforcement learning. arXiv preprint arXiv:1809.08835 (2018)
  12. 12.
    Wan, Q., Liu, W., Xu, L., et al.: Research on optimal strategy planning of uncertain environment BDI agent based on Q-learning. Comput. Eng. Sci. 41(01), 166–172 (2019)Google Scholar
  13. 13.
    Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
  14. 14.
    Nair, A., McGrew, B., Andrychowicz, M., et al.: Overcoming exploration in reinforcement learning with demonstrations. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6292–6299. IEEE (2018)Google Scholar
  15. 15.
    Hu, Y., Da, Q., Zeng, A., et al.: Reinforcement learning to rank in e-commerce search engine: formalization, analysis, and application. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 368–377. ACM (2018)Google Scholar
  16. 16.
    Luo, W., Tang, Q., Fu, C., Eberhard, P.: Deep-Sarsa based multi-UAV path planning and obstacle avoidance in a dynamic environment. In: International Conference on Sensing and Imaging. Advances in Swarm Intelligence, pp. 102–111 (2018).  https://doi.org/10.1007/978-3-319-93818-9_10
  17. 17.
    van den Berg, J., Guy, S.J., Lin, M., Manocha, D.: Reciprocal \(n\)-body collision avoidance. In: Robotics Research. Springer Tracts in Advanced Robotics, vol. 70, pp. 3–19 (2011).  https://doi.org/10.1007/978-3-642-19457-3_1

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Guoqiu Wang
    • 1
  • Xuanyu Zheng
    • 1
  • Haitong Zhao
    • 1
  • Qidong Zhao
    • 1
  • Changsheng Zhang
    • 1
    Email author
  • Bin Zhang
    • 1
  1. 1.Northeastern UniversityShenyangPeople’s Republic of China

Personalised recommendations