Abstract
Future missions to the Moon and beyond are likely to involve low-thrust propulsion technologies due to their propellant efficiency. However, these still present a difficult trajectory design problem, owing to the near continuous thrust, lack of control authority and chaotic dynamics. Lyapunov control laws can generate sub-optimal trajectories for such missions with minimal computational cost and are suitable for feasibility studies and as initial guesses for optimisation methods. In this work a Reinforced Lyapunov Controller is used to design optimal low-thrust transfers from geostationary transfer orbit towards lunar polar orbit. Within the reinforcement learning (RL) framework, a dual-actor network setup is used, one in each of the Earth- and Moon-centred inertial frames respectively. A key contribution of this paper is the demonstration of a forwards propagated trajectory, removing the need to define a patch point a priori. This is enabled by an adaptive patch distance and extensive initial geometry exploration during the RL training. Results for both time- and fuel-optimal transfers are presented, along with a Monte Carlo analysis of the robustness to disturbances for such transfers. Phasing is introduced where necessary to aid rendezvous with the Moon. The results demonstrate the potential for such techniques to provide a basis for the design and guidance of low-thrust lunar transfers.
![](http://media.springernature.com/lw685/springer-static/image/art%3A10.1007%2Fs42064-024-0212-x/MediaObjects/42064_2024_212_Fig1_HTML.jpg)
Article PDF
Avoid common mistakes on your manuscript.
References
Peterson, P. Y., Herman, D. A., Kamhawi, H., Frieman, J. D., Huang, W., Verhey, T., Dinca, D., Boomer, K., Pinero, L., Criswell, K., et al. Overview of NASA’s solar electric propulsion project. In: Proceedings of the 36th International Electric Propulsion Conference, 2019: IEPC-2019-836.
Racca, G. D., Foing, B. H., Coradini, M. SMART-1: The first time of Europe to the Moon. In: Earth-Moon Relationships. Barbieri, C., Rampazzi, F., Eds. Springer Dordrecht, 2001: 379–390.
Thomas, V. C., Makowski, J. M., Brown, G. M., McCarthy, J. F., Bruno, D., Cardoso, J. C., Chiville, W. M., Meyer, T. F., Nelson, K. E., Pavri, B. E., et al. The Dawn spacecraft. In: The Dawn Mission to Minor Planets 4 Vesta and 1 Ceres. Russell, C., Raymond, C., Eds. Springer New York, 2011: 175–249.
Kawaguchi, J. I., Fujiwara, A., Uesugi, T. K. The ion engines cruise operation and the Earth swingby of ‘Hayabusa’ (MUSES-C). In: Proceedings of the 55th International Astronautical Congress of the International Astronautical Federation, the International Academy of Astronautics, and the International Institute of Space Law, 2005.
Tsuda, Y., Yoshikawa, M., Abe, M., Minamino, H., Nakazawa, S. System design of the Hayabusa 2—Asteroid sample return mission to 1999 JU3. Acta Astronautica, 2013, 91: 356–362.
Benkhoff, J., van Casteren, J., Hayakawa, H., Fujimoto, M., Laakso, H., Novara, M., Ferri, P., Middleton, H. R., Ziethe, R. BepiColombo—Comprehensive exploration of Mercury: Mission overview and science goals. Planetary and Space Science, 2010, 58(1–2): 2–20.
Gao, Y., Li, X. Optimization of low-thrust many-revolution transfers and Lyapunov-based guidance. Acta Astronautica, 2010, 66(1–2): 117–129.
Shannon, J. L., Ozimek, M. T., Atchison, J. A., Christine, M. Q-law aided direct trajectory optimization for the high-fidelity, many-revolution low-thrust orbit transfer problem. Advances in the Astronautical Sciences, 2019, 168: 781–800.
Shannon, J., Ozimek, M., Atchison, J., Hartzell, C. Rapid design and exploration of high-fidelity low-thrust transfers to the moon. In: Proceedings of the IEEE Aerospace Conference, 2020: 1–12.
Shannon, J. L., Ellison, D., Hartzell, C. M. Analytical partial derivatives of the Q-law guidance algorithm. In: Proceedings of the 31st Space Flight Mechanics Meeting, 2021: AAS 21–274.
Jagannatha, B. B., Bouvier, J. B. H., Ho, K. Preliminary design of low-energy, low-thrust transfers to halo orbits using feedback control. Journal of Guidance, Control, and Dynamics, 2019, 42(2): 260–271.
Epenoy, R., Pérez-Palau, D. Lyapunov-based low-energy low-thrust transfers to the Moon. Acta Astronautica, 2019, 162: 87–97.
Shannon, J. L., Ellison, D., Hartzell, C. M. Exploration of low-thrust lunar swingby escape trajectories. In: Proceedings of the 31st Space Flight Mechanics Meeting, 2021: AAS 21–273.
Peterson, J. T., Singh, S. K., Junkins, J. L., Taheri, E. Lyapunov guidance in orbit element space for low-thrust cislunar trajectories. In: Proceedings of the AAS Guidance, Navigation and Control Conference, 2020: AAS 20–115.
Lee, S., Petropoulos, A. E., von Allmen, P. Low-thrust orbit transfer optimization with refined Q-law and multi-objective genetic algorithm. In: Proceedings of the AAS/AIAA Astrodynamics Specialist Conference, 2005: AAS 05–393.
Varga, G. I., Pérez, J. S. Many-revolution low-thrust orbit transfer computation using equinoctial Q-law including J2 and eclipse effects. In: Proceedings of the 6th International Conference on Astrodynamics Tools and Techniques, 2016: 29–42.
Holt, H., Armellin, R., Baresi, N., Hashida, Y., Turconi, A., Scorsoglio, A., Furfaro, R. Optimal Q-laws via reinforcement learning with guaranteed stability. Acta Astronautica, 2021, 187: 511–528.
Kwon, H., Oghim, S., Bang, H. Autonomous guidance for multi-revolution lowthrust orbit transfer via reinforcement learning. In: Proceedings of the AAS/AIAA Astrodynamics Specialist Conference, 2021: AAS 21–315.
Miller, D., Englander, J. A., Linares, R. Interplanetary low-thrust design using proximal policy optimization. In: Proceedings of the AAS/AIAA Astrodynamics Specialist Conference, 2019: AAS 19–779.
Zavoli, A., Federici, L. Reinforcement learning for robust trajectory design of interplanetary missions. Journal of Guidance, Control, and Dynamics, 2021, 44(8): 1440–1453.
Yanagida, K., Ozaki, N., Funase, R.. Exploration of long time-of-flight three-body transfers using deep reinforcement learning. In: Proceedings of the AIAA Scitech Forum, 2020: AIAA 2020–0460.
Miller, D., Linares, R. Low-thrust optimal control via reinforcement learning. In: Proceedings of the 29th AAS/AIAA Space Flight Mechanics Meeting, 2019: 1817–1834.
LaFarge, N. B., Miller, D., Howell, K. C., Linares, R. Guidance for closed-loop transfers using reinforcement learning with application to libration point orbits. In: Proceedings of the AIAA Scitech Forum, 2020: AIAA 2020–0458.
LaFarge, N. B., Miller, D., Howell, K. C., Linares, R. Autonomous closed-loop guidance using reinforcement learning in a low-thrust, multi-body dynamical environment. Acta Astronautica, 2021, 186: 1–23.
LaFarge, N. B., Howell, K. C., Linares, R. A hybrid closed-loop guidance strategy for low-thrust spacecraft enabled by neural networks. In: Proceedings of the 31st AAS/AIAA Spaceflight Mechanics Meeting, 2021: AAS 21–302.
Sullivan, C. J., Bosanac, N. Using reinforcement learning to design a low-thrust approach into a periodic orbit in a multi-body system. In: Proceedings of the AIAA Scitech Forum, 2020: AIAA 2020–1914.
LaFarge, N. B., Howell, K. C., Folta, D. C. An autonomous stationkeeping strategy for multi-body orbits leveraging reinforcement learning. In: Proceedings of the AIAA SCITECH Forum, 2022: AIAA 2022–1764.
Bonasera, S., Elliott, I., Sullivan, C. J., Bosanac, N., Ahmed, N., McMahon, J. Designing impulsive station-keeping maneuvers near a Sun-Earth l2 halo orbit via reinforcement learning. In: Proceedings of the 31st AAS/AIAA Space Flight Mechanics Meeting, 2021: AAS 21–216.
Bosanac, N., Bonasera, S., Sullivan, C., McMahon, J., Ahmed, N. Reinforcement learning for reconfiguration maneuver design in multi-body systems. In: Proceedings of the AAS/AIAA Astrodynamics Specialist Conference, 2021.
Sutton, R. S., Barto, A. G. Reinforcement Learning: An Introduction. MIT Press, 2018.
Shirobokov, M., Trofimov, S., Ovchinnikov, M. Survey of machine learning techniques in spacecraft control design. Acta Astronautica, 2021, 186: 87–97.
Izzo, D., Blazquez, E., Ferede, R., Origer, S., De Wagter, C., de Croon, G. C. Optimality principles in spacecraft neural guidance and control. Science Robotics, 2024, 9(91): eadi6421.
Malyuta, D., Yu, Y., Elango, P., Açıkmeşe, B. Advances in trajectory optimization for space vehicle control. Annual Reviews in Control, 2021, 52: 282–315.
Scorsoglio, A., Furfaro, R., Linares, R., Massari, M. Relative motion guidance for near-rectilinear lunar orbits with path constraints via actor-critic reinforcement learning. Advances in Space Research, 2023, 71(1): 316–335.
Federici, L., Zavoli, A., Furfaro, R. Comparative analysis of reinforcement learning algorithms for robust interplanetary trajectory design. In: Studies in Computational Intelligence. Ieracitano, C., Mammone, N., Di Clemente, M., Mahmud, M., Furfaro, R., Morabito, F. C., Eds. Springer Nature Switzerland, 2023: 133–149.
Holt, H., Baresi, N., Armellin, R. Towards optimal Lyapunov controllers for low-thrust lunar transfers via reinforcement learning. In: Proceedings of the AAS Astrodynamics Specialist Conference, 2021: AAS 21–615.
Holt, H., Bernardini, N., Baresi, N., Armellin, R. Reinforced Lyapunov controllers and convex optimisation for low-thrust lunar transfers. In: Proceedings of the AAS Astrodynamics Specialist Conference, 2023: AAS 23–247.
Petropoulos, A. E. Simple control laws for low-thrust orbit transfers. In: Proceedings of the AAS/AIAA Astrodynamics Specialists Conference, 2003.
Petropoulos, A. E. Refinements to the Q-law for the low-thrust orbit transfers. Advances in the Astronautical Sciences, 2005, 120: 963–982.
Koon, W. S., Lo, M. W., Marsden, J. E., Ross, S. D. Dynamical systems, the three-body problem and space mission design. In: Equadiff 99. World Scientific Publishing Company, 2000: 1167–1181.
Wakker, K. F. Fundamentals of Astrodynamics. TU Delft, 2015.
Holt, H. J. Trajectory design using Lyapunov control laws and reinforcement learning. Doctoral Dissertation. University of Surrey, 2023.
Pontani, M., Pustorino, M. Nonlinear Earth orbit control using low-thrust propulsion. Acta Astronautica, 2021, 179: 296–310.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal policy optimization algorithms. arXiv preprint, 2017, arXiv:1707.06347.
Scorsoglio, A. Adaptive ZEM/ZEV feedback guidance for rendezvous in lunar NRO with collision avoidance. Master Thesis. Politecnico Di Milano, University of Arizona, 2018.
Miller, D. D. M. Low-thrust spacecraft guidance and control using proximal policy optimization. Doctoral Dissertation. Massachusetts Institute of Technology, 2020.
Grondman, I., Busoniu, L., Lopes, G. A. D., Babuska, R. A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2012, 42(6): 1291–1307.
Sutton, R. S., McAllester, D., Singh, S., Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. In: Proceedings of the 12th International Conference on Neural Information Processing Systems, 1999: 1057–1063.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M. Deterministic policy gradient algorithms. In: Proceedings of the International Conference on Machine Learning, 2014: 387–395.
Locoche, S. An analytical method for evaluation of low-thrust multi-revolutions orbit transfer with perturbations and power constraint. In: Proceedings of the Conference on Guidance, Navigation and Control Systems, 2018: 10–20
DeMars, K. J., Jah, M. K. Probabilistic initial orbit determination using Gaussian mixture models. Journal of Guidance, Control, and Dynamics, 2013, 36(5): 1324–1335.
Betts, J. T., Erb, S. O. Optimal low thrust trajectories to the Moon. SIAM Journal on Applied Dynamical Systems, 2003, 2(2): 144–170.
Shannon, J. L., Ozimek, M. T., Atchison, J. A., Hartzell, C. M. Rapid design of high-fidelity low-thrust transfers to the Moon. Journal of Spacecraft and Rockets, 2022, 59(5): 1522–1535.
Ozaki, N., Hatakeyama, A., Ito, S., Chikazawa, T., Akiyama, Y., Yamamoto, T. Low-thrust many-revolution trajectory design under operational uncertainties. In: Proceedings of the AAS/AIAA Astrodynamics Specialist Conference, 2023: AAS 23–222.
Naasz, B. J. Classical element feedback control for spacecraft orbital maneuvers. Master Thesis. Virginia Tech, 2002.
Lantukh, D. V., Ranieri, C. L., DiPrinzio, M. D., Edelman, P. J. Enhanced Q-law Lyapunov control for low-thrust transfer and rendezvous design. In: Proceedings of the AAS/AIAA Astrodynamics Specialist Conference, 2017: AAS 17–589.
Locoche, S., Lagadec, K., Erb, S., Valles, C. Y. Reducing operation cost with autonomous guidance for electrical orbit raising. In: Proceedings of the 8th International Conference on Astrodynamics Tools and Techniques, 2021.
Narayanaswamy, S., Damaren, C. J. Equinoctial Lyapunov control law for low-thrust rendezvous. Journal of Guidance, Control, and Dynamics, 2023, 46(4): 781–795.
Acknowledgements
Harry Holt is currently funded through the generous financial support from the Warwick & Judy Smith Engineering Endowment Fund and the support of the University of Auckland Foundation and Faculty of Engineering. Part of this research was conducted whilst HH was completing their Ph.D. at the University of Surrey, funded by Surrey Satellite Technology Limited (SSTL). The authors also wish to acknowledge the use of both the University of Surrey’s Eureka and New Zealand eScience Infrastructure (NeSI) high performance computing facilities as part of this research. New Zealand’s national facilities are provided by NeSI and funded jointly by NeSI’s collaborator institutions and through the Ministry of Business, Innovation & Employment’s Research Infrastructure programme.
Funding
Open Access funding enabled and organized by CAUL and its Member Institutions.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Harry Holt received his M.Sc. degree in natural sciences (physics) from the University of Cambridge, UK, in 2018. He completed his Ph.D. degree at the University of Surrey, UK, in 2022, working on trajectory design using Lyapunov control laws and reinforcement learning. He is now working as a research fellow at Te Pūnaha Ātea- Space Institute, at the University of Auckland, New Zealand. His research interests include trajectory design, Lyapunov control, and reinforcement learning. E-mail: hho1678@aucklanduni.ac.nz
Nicola Baresi graduated from the University of Colorado Boulder, USA, in 2017 with his Ph.D. thesis on spacecraft formation flight and quasi-periodic invariant tori. He later moved to Japan working on the MMX and EQUULEUS missions as a postdoctoral fellow at the Institute of Space and Astronautical Sciences (ISAS) of JAXA. Starting from 2019, Dr. Baresi has joined the University of Surrey, UK, where he is now a lecturer in orbital mechanics at Surrey Space Centre. Nicola has authored more than 60 scientific contributions, including conference proceedings and peer-reviewed articles in top-ranked astrodynamics journals. He is an elected member of the Space Flight Mechanics Committee of the American Astronautical Society, as well as a Fellow of the UK Higher Education Academy. E-mail: n.baresi@surrey.ac.uk
Roberto Armellin received his M.Sc. and Ph.D. degrees in aerospace engineering from Politecnico di Milano, Italy, in 2003 and 2007, respectively. Since November 2020, he has been a professor at Te Pūnaha Ātea - Space Institute at the University of Auckland, New Zealand. His current research interests include space trajectory optimization, spacecraft navigation and guidance, and space situational awareness. E-mail: roberto.armellin@auckland.ac.nz
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Holt, H., Baresi, N. & Armellin, R. Reinforced Lyapunov controllers for low-thrust lunar transfers. Astrodyn (2024). https://doi.org/10.1007/s42064-024-0212-x
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s42064-024-0212-x