Autonomous Maneuver Decision of UAV in Air Combat Based on Scenario-Transfer Deep Reinforcement Learning

Abstract

The traditional command operation depend on the ground station is difficult to adapt to the highly dynamic and uncertain UAV air combat environment. Previous researches on UAV autonomous maneuver decision have limitations due to oversimplified assumptions, large and complex calculations, and lack of flexibility. Aiming at the air combat scenario of Red and Blue UAV, a three-dimensional UAV air combat model based on Markov Decision Process (MDP) is established. We trained Red UAV with Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm to autonomously complete air combat missions. The performance is improved by scenario-transfer training and self-game training techniques. As the training scenario gradually transfers from simple to complex, the Red UAV continue to learn from previous experience steadily and improve the capabilities. The intelligence level is improved through self-game training. The simulation results present that the proposed maneuvering decision-making model and the training method can help the drone obtain effective decision-making strategies to get an advantage and defeat opponents in the confrontation.