Abstract
Recent years have witnessed the rapid development of aerospace science and technology, and the orbital game technology has shown great potential value in the field of failed satellite maintenance, debris removal, etc. In this case, orbital game is often characterized by nonlinear dynamic model, unknown state information, high randomness, but the existing approaches to deal with game problem are difficult to be applied. The analytical method based on game theory is only applicable to simple scenarios, and it is challenging to find the optimal strategy for such complex scenarios as satellite swarm game. It should be noted that deep reinforcement learning has some research basis in the cooperative decision-making and control of multi-agents. In view of its powerful perception and decision ability, this paper applies deep reinforcement learning to solve the orbital game problem of satellite swarm. Firstly, the game scenario is modeled, where typical constraints, e.g., minimum time, optimal fuel, and collision avoidance, are taken into consideration in the game process, and then the multi-agent reinforcement learning algorithm is developed to solve the optimal maneuver strategy. The algorithm is based on the Actor-Critic architecture and uses a centralized training and decentralized execution approach to solve the optimal joint maneuver strategy. For different task scenarios, the action space, state observation space, and reward space are designed to introduce more rewards that match the specific game tasks to make the algorithm converge quickly, so that the satellite swarm emerges and executes better intelligent strategies to complete the corresponding game task.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Zhao, L.R., Dang, Z.H., Zhang, Y.L.: Orbital game: concepts, principles and methods. J. Command Control 7(3), 215–224 (2021)
Zhou, J.: Manned space rendezvous and docking technology. Manned Space 17(2), 1–8 (2011)
Yuan, L.: Spacecraft intelligent autonomous control technology toward uncertain environment. J. Astronaut. 42(7), 839–849 (2021)
Sun, S., Zhang, Q., Loxton, R., et al.: Numerical solution of a pursuit-evasion differential game involving two spacecraft in low earth orbit. J. Ind. Manag. Optim. (JIMO) 11(4), 1127–1147 (2015)
Li, Z., Zhu, H., Yang, Z., et al.: Saddle point of orbital pursuit-evasion game under J2-perturbed dynamics. J. Guid. Control Dyn. 43(9), 1733–1739 (2020)
Pontani, M., Conway, B.A.: Numerical solution of the three-dimensional orbital pursuit-evasion game. J. Guid. Control Dyn. 2(32), 474–487 (2009)
Stupik, J., Pontani, M., Conway, B.: Optimal pursuit/evasion spacecraft trajectories in the hill reference frame. In: AIAA/AAS Astrodynamics Specialist Conference, p. 4882 (2012)
Hafer, W.T., Reed, H.L., Turner, J.D., et al.: Sensitivity methods applied to orbital pursuit evasion. J. Guid. Control Dyn. 38(6), 1118–1126 (2015)
Liu, Y., Ye, D., Hao, Y.: Distributed online mission planning for multi-player space pursuit and evasion. Chin. J. Aeronaut. 29(6), 1709–1720 (2016)
Harmon, M.E., Baird, L.C., Klopf, A.H.: Reinforcement learning applied to a differential game. Adapt. Behav. 4(1), 3–28 (1995)
Liu, B.Y., Ye, X.B., Gao, Y., et al.: Forward-looking imaginative planning framework combined with prioritized replay double DQN. In: International Conference Control, Automation and Robotics, pp. 336–341 (2019)
Xu, D., Hui, Z., Liu, Y.Q., et al.: Morphing control of a new bionic morphing UAV with deep reinforcement learning. Aerosp. Sci. Technol. 92, 232–243 (2019)
Zhu, Y., Mottaghi, R., Kolve, E., et al.: Target-driven visual navigation in indoor scenes using deep reinforcement learning. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 3357–3364. IEEE (2017)
Liu, B., Ye, X., Gao, Y., et al.: Strategy solution of non-cooperative target pursuit-evasion game based on branching deep reinforcement learning. Acta Aeronaut. Astronaut. Sinica 41(10), 348–358 (2020)
Wang, C., Ye, D., Sun, Z., et al.: Adaptive game strategy of spacecraft terminal interception. J. Astronaut. 41(3), 309–318 (2020)
Lowe, R., Wu, Y., Tamar, A., et al.: Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems, pp. 6379–6390 (2017)
Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Lillicrap, T., Hunt, J., Pritzel, A., et al.: Continuous control with deep reinforcement learning. Comput. Sci. (2015)
Zavoli, A., Federici, L.: Reinforcement learning for robust trajectory design of interplanetary missions. J. Guid. Control Dyn. 44(8), 1440–1453 (2021)
Hovell, K., Ulrich, S.: Deep reinforcement learning for spacecraft proximity operations guidance. J. Spacecr. Rocket. 58(2), 254–264 (2021)
Shirobokov, M., Trofimov, S., Ovchinnikov, M.: Survey of machine learning techniques in spacecraft control design. Acta Astronaut. 186, 87–97 (2021)
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grants U21B6001, 11972026 and U2013206, in part by Science and Technology on Space Intelligent Control Laboratory under Grant 2021-JCJQ-LB-010-07, and in part by Key Research and Development Program of Shaanxi under Grant 2023-YBGY-384.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Yu, W., Yue, X., Huang, P., Liu, C. (2024). Deep Reinforcement Learning-Based Intelligent Decision-Making for Orbital Game of Satellite Swarm. In: Li, S. (eds) Computational and Experimental Simulations in Engineering. ICCES 2023. Mechanisms and Machine Science, vol 145. Springer, Cham. https://doi.org/10.1007/978-3-031-42987-3_61
Download citation
DOI: https://doi.org/10.1007/978-3-031-42987-3_61
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-42986-6
Online ISBN: 978-3-031-42987-3
eBook Packages: EngineeringEngineering (R0)