Abstract
Reinforcement learning is an active research area with a vast number of applications in robotics, and the RoboCup competition is an interesting environment for studying and evaluating reinforcement learning methods. A known difficulty in applying reinforcement learning to robotics is the high number of experience samples required, being the use of simulated environments for training the agents followed by transfer learning to real-world (sim-to-real) a viable path. This article introduces an open-source simulator for the IEEE Very Small Size Soccer and the Small Size League optimized for reinforcement learning experiments. We also propose a framework for creating OpenAI Gym environments with a set of benchmarks tasks for evaluating single-agent and multi-agent robot soccer skills. We then demonstrate the learning capabilities of two state-of-the-art reinforcement learning methods as well as their limitations in certain scenarios introduced in this framework. We believe this will make it easier for more teams to compete in these categories using end-to-end reinforcement learning approaches and further develop this research area.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Code available at https://github.com/robocin/rSoccer.
- 2.
Code available at https://github.com/robocin/rSim.
- 3.
References
Andrychowicz, M., et al.: Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.00177 (2018)
Arnold, L.: Stochastic Differential Equations. New York (1974)
Bassani, H.F., et al.: A framework for studying reinforcement learning and sim-to-real in robot soccer (2020)
Brockman, G., et al.: OpenAI gym (2016)
Christiano, P., et al.: Transfer from simulation to real world through learning deep inverse dynamics model. arXiv preprint arXiv:1610.03518 (2016)
Duan, Y., Liu, Q., Xu, X.: Application of reinforcement learning in robot soccer. Eng. Appl. Artif. Intell. 20(7), 936–950 (2007)
Haarnoja, T., et al.: Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 (2018)
Kim, J.H., Kim, D.H., Kim, Y.J., Seow, K.T.: Soccer Robotics, vol. 11. Springer, Heidelberg (2004). https://doi.org/10.1007/b95999
Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I., Osawa, E.: RoboCup: the robot world cup initiative. In: Proceedings of the First International Conference on Autonomous Agents, AGENTS 1997, pp. 340–347. Association for Computing Machinery, New York (1997). https://doi.org/10.1145/267658.267738
Kurach, K., et al.: Google research football: a novel reinforcement learning environment. arXiv preprint arXiv:1907.11180 (2019)
Levine, S., Finn, C., Darrell, T., Abbeel, P.: End-to-end training of deep visuomotor policies. J. Mach. Learn. Res. 17(1), 1334–1373 (2016)
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
Liu, S., Lever, G., Merel, J., Tunyasuvunakool, S., Heess, N., Graepel, T.: Emergent coordination through competition (2019)
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, O.P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. In: Advances in Neural Information Processing Systems, pp. 6379–6390 (2017)
Monajjemi, V., Koochakzadeh, A.: FIRASim (2020). https://github.com/fira-simurosot/FIRASim. Accessed 28 Apr 2021
Monajjemi, V., Koochakzadeh, A., Ghidary, S.S.: grSim – RoboCup small size robot soccer simulator. In: Röfer, T., Mayer, N.M., Savage, J., Saranlı, U. (eds.) RoboCup 2011. LNCS (LNAI), vol. 7416, pp. 450–460. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-32060-6_38
RoboCup: Robocup small size league (SSL) hardware challenges 2021. https://robocup-ssl.github.io/ssl-hardware-challenge-rules/rules.html. Accessed 04 Aug 2021
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)
SDK, V.: VSS SDK (2019). https://vss-sdk.github.io/book/general.html. Accessed 5 Apr 2021
Shi, H., Lin, Z., Hwang, K.S., Yang, S., Chen, J.: An adaptive strategy selection method with reinforcement learning for robotic soccer games. IEEE Access 6, 8376–8386 (2018)
Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359 (2017)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)
Tassa, Y., et al.: DeepMind control suite (2018)
Vinyals, O., et al.: AlphaStar: mastering the real-time strategy game starcraft ii. DeepMind Blog (2019)
Volodymyr, M., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I.: Playing atari with deep reinforcement learning. In: NIPS Deep Learning Workshop (2013)
Yoon, M.: Developing basic soccer skills using reinforcement learning for the RoboCup small size league. Ph.D. thesis, Stellenbosch: Stellenbosch University (2015)
Zhu, Y., Schwab, D., Veloso, M.: Learning primitive skills for mobile robots. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 7597–7603 (2019)
Zolanvari, A., Shirazi, M., Menhaj, M.: A Q-learning approach for controlling a robotic goalkeeper during penalty procedure. In: 2019 II International Congress on Science and Engineering, Hamburg-Germany, pp. 1–12 (2019)
Acknowledgments
The authors would like to thank RoboCIn - UFPE Team and Mila - Quebec Artificial Intelligence Institute for the collaboration and resources provided; Conselho Nacional de Desenvolvimento Cientifico e Tecnológico (CNPq), and Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) for financial support. Moreover, the authors also gratefully acknowledge the support of NVIDIA Corporation with the donation of the RTX 2080 Ti GPU used for this research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Martins, F.B., Machado, M.G., Bassani, H.F., Braga, P.H.M., Barros, E.S. (2022). rSoccer: A Framework for Studying Reinforcement Learning in Small and Very Small Size Robot Soccer. In: Alami, R., Biswas, J., Cakmak, M., Obst, O. (eds) RoboCup 2021: Robot World Cup XXIV. RoboCup 2021. Lecture Notes in Computer Science(), vol 13132. Springer, Cham. https://doi.org/10.1007/978-3-030-98682-7_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-98682-7_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-98681-0
Online ISBN: 978-3-030-98682-7
eBook Packages: Computer ScienceComputer Science (R0)