Skip to main content
SpringerLink
Log in

A survey of research on several problems in the RoboCup3D simulation environment

  • Published:
Autonomous Agents and Multi-Agent Systems Aims and scope Submit manuscript

Abstract

In the process of robot research and development, due to the vulnerability of hardware, simulation environment is often used to verify and test algorithms first. RoboCup3D simulation environment is developed based on open dynamic engine, and the humanoid robot NAO is modeled as the main robot, which provides a simulation platform for humanoid robot researchers to study robot movements. At the same time, it is also the official platform of RoboCup 3D events. Under the rules of soccer robot competition, it is helpful for the research of multi-robots, especially multi-humanoid robots’ cooperation strategy. This paper summarizes the related research in RoboCup3D simulation environment, and first introduces the basic problems existing in this simulation environment. Secondly, the research of robot motion generation and optimization based on model and non-model in simulation environment is introduced respectively. Then, it introduces the related research of cooperation strategy design of multi-humanoid robots under RoboCup3D rules, including positioning, dynamic role assignment, etc. And sort out a typical practical solution to the above problems; Finally, the future development trend of related research in RoboCup3D simulation environment is analyzed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31
Fig. 32

Similar content being viewed by others

Notes

  1. http://simspark.sourceforge.net/.

  2. The official rules can be found at https://ssim.robocup.org/3d-simulation/3d-rules/.

  3. The official recorded game scores can be found at https://archive.robocup.info/Soccer/Simulation/3D/replays/RoboCup/.

  4. https://github.com/m-abr/FCPCodebase.

  5. https://github.com/LARG/utaustinvilla3d.

  6. https://github.com/magmaOffenburg/magmaRelease.

References

  1. Li, X. (2019). Research and Implementation of RoboCup3D Soccer Humanoid Robot Gait Optimization. Hefei University Of Technology.

  2. Hong, C., Jeong, I., Vecchietti, L. F., Har, D., & Kim, J.-H. (2021). AI world cup: Robot-Soccer-based competitions. IEEE Transactions on Games, 13(4), 330–341. https://doi.org/10.1109/TG.2021.3065410

    Article  Google Scholar 

  3. Depinet, M., MacAlpine, P., & Stone, P. (2014). Keyframe sampling, optimization, and behavior integration: Towards long-distance kicking in the robocup 3d simulation league (pp. 571–582). Robot Soccer World Cup. Springer: Cham.

  4. Hanna, J. P., Desai, S., Karnan, H., et al. (2021). Grounded action transformation for sim-to-real reinforcement learning. Machine Learning, 110(9), 2469–2499.

    Article  MathSciNet  Google Scholar 

  5. Kajita, S., Kanehiro, F., Kaneko, K., et al. (2003). Biped walking pattern generation by using preview control of zero-moment point. In 2003 IEEE international conference on robotics and automation (Cat. No. 03CH37422), Vol. 2. IEEE, pp. 1620–1626.

  6. Vukobratovic, M., & Borovac, B. (2004). Zero-moment point—thirty five years of its life. International Journal of Humanoid Robotics, 1, 157–173. https://doi.org/10.1142/S0219843604000083

    Article  Google Scholar 

  7. Hirai, K., Hirose, M., Haikawa, Y., et al. (1998). The development of Honda humanoid robot. Proceedings of 1998 IEEE international conference on robotics and automation (Cat. No. 98CH36146), Vol. 2. IEEE, pp. 1321–1326.

  8. Kajita, S., & Guan, Y. (2007). Humanoid Robots. Tsinghua University publishing house co., ltd.

  9. Wang, S., Hu, M., Shi, H., Zhang, S., Li, X., & Li, W. (2015). Humanoid robot’s omnidirectional walking. In: IEEE international conference on information and automation, Vol. 2015, pp. 381–385. https://doi.org/10.1109/ICInfA.2015.7279317

  10. Muniz, F., Maximo, M. R. O. A., & Ribeiro, C. H. C. (2016) Keyframe movement optimization for simulated humanoid robot using a parallel optimization framework. In: 2016 XIII Latin American robotics symposium and IV Brazilian robotics symposium (LARS/SBR), pp. 79–84. https://doi.org/10.1109/LARS-SBR.2016.20.

  11. Nezami, O. M., & Meybodi, M. R. (2012). Biped Robot walking using a combination of truncated Fourier series and GALA (Genetic algorithm parameters adaption using learning automata). International Journal of Machine Learning and Computing, 2(5), 598.

    Article  Google Scholar 

  12. Haider, S., Abidi, S. R., & Williams, M. (2012). On evolving a dynamic bipedal walk using Partial Fourier Series. In: IEEE international conference on robotics and biomimetics (ROBIO), Vol. 2012, pp. 8–13. https://doi.org/10.1109/ROBIO.2012.6490935

  13. Shafii, N., Javadi, M. H. S., & Kimiaghalam, B. (2009). A truncated fourier series with genetic algorithm for the control of biped locomotion. 2009 IEEE/ASME international conference on advanced intelligent mechatronics. IEEE, pp. 1781–1785.

  14. Shafii, N., Aslani, S., Nezami, O. M., et al. (2009). Evolution of biped walking using truncated fourier series and particle swarm optimization (pp. 344–354). Robot Soccer World Cup. Springer: Berlin.

  15. Braun, H. (2021). Analyse domnenseitiger Optimierungen für Deep reinforcement learning in der RoboCup Umgebung. Hochschule Offenburg.

  16. Tao, C., Xue, J., Zhang, Z., et al. (2022). Parallel deep reinforcement learning method for gait control of Biped Robot. IEEE transactions on circuits and systems—II: Express briefs, Vol. 69, Issue 6.

  17. Spitznagel, M. (2020). Analyse des Deep Reinforcement Learning Algorithmus PPO2 in der RoboCup Umgebung.

  18. Liu, J. (2015). The motion planning and cooperation mechanism of biped robots in RoboCup3D simulation environment. Nanjing University Of Posts And Telecommunications.

  19. Bavani, A. M., Ahmadi, H., & Nasrinpour, H. R. (2011). A closed-loop Central Pattern Generator approach to control NAO humanoid robots’ walking. The 2nd international conference on control, instrumentation and automation, pp. 1036–1041.https://doi.org/10.1109/ICCIAutom.2011.6356804.

  20. Holland, J. H. (1992). Genetic algorithms. Scientific American, 267(1), 66–73.

    Article  Google Scholar 

  21. Hansen, N. (2006). The CMA evolution strategy: A comparing review. Towards a new evolutionary computation, pp. 75–102.

  22. MacAlpine, P., Urieli, D., Barrett, S., et al. (2011). UT Austin Villa 2011: 3D simulation team report. University of Texas at Austin Austin United States.

  23. MacAlpine, P. M. (2017). Multilayered skill learning and movement coordination for autonomous robotic agents.

  24. Hemami, H. (1978). Reduced order models for biped locomotion. IEEE Transactions on Systems, Man, and Cybernetics, 8(4), 321–351.

    Article  MathSciNet  Google Scholar 

  25. Seekircher, A., & Visser, U. (2016). An adaptive LIPM-based dynamic walk using model parameter optimization on humanoid robots. Künstl Intelligenz, 30, 233–244.

    Article  Google Scholar 

  26. Shafii, N., Lau, N., & Reis, L. P. (2015). Learning to walk fast: Optimized hip height movement for simulated and real humanoid robots. Journal of Intelligent & Robotic Systems, 80(3), 555–571.

    Article  Google Scholar 

  27. Sui, Z., Yu, W., Tian, Y., & Xu, M. (2017). Gait planning of biped robot based on reference trajectory and COM balance. Journal of Jilin University (Information Science Edition), 35(2), 175–182.

    Google Scholar 

  28. Urieli, D., MacAlpine, P., Kalyanakrishnan, S., Bentor, Y., & Stone, P. (2011). On optimizing interdependent skills: A case study in simulated 3D humanoid robot soccer. International conference on autonomous agents and multiagent systems, pp. 769–776.

  29. Shen, P. (2015). Omni-directional walking skill and cooperation mechanism of biped robots in RoboCup3D simulation environment.

  30. Liang, Z., Zhao, H., & Yue, H. (2014). An omnidirectional walk for a biped robot based on gyroscope-accelerometer measurement. In: IEEE international conference on mechatronics & automation. IEEE.

  31. Hugel, V. & Jouandeau, N. (2012). Walking patterns for real time path planning simulation of humanoids. 2012 IEEE RO-MAN: The 21st IEEE international symposium on robot and human interactive communication, pp. 424–430, https://doi.org/10.1109/ROMAN.2012.6343789.

  32. Xu, J. (2014). Gait research and implementation of soccer humanoid robot based on CMA-ES algorithm. Hefei University Of Technology.

  33. Li, C. (2015). Research on motion planning and flocking control for humanoid robot. Jiangnan University.

  34. Snafii, N., Abdolmaleki, A., Lau, N., & Reis, L. P. (2015). Development of an omnidirectional walk engine for soccer humanoid robots. International Journal of Advanced Robotic Systems.

  35. Popovic, M. B., Goswami, A., & Herr, H. (2005). Ground reference points in legged locomotion: Definitions, biological trajectories and control implications. The International Journal of Robotics Research, 24(12), 1013–1032.

    Article  Google Scholar 

  36. Sato, T., Sakaino, S., & Ohnishi, K. (2011). Real-Time walking trajectory generation method with three-mass models at constant body height for three-dimensional biped robots. IEEE Transactions on Industrial Electronics, 58(2), 376–383.

    Article  Google Scholar 

  37. Li, C. (2014). Shooting method for humanoid robot based on three-mass model. Journal of Computer Applications, 34(6), 1657.

    Google Scholar 

  38. Yang, L., Chew, C. M. & Poo, A. N. (2006). Adjustable bipedal gait generation using Genetic algorithm optimized Fourier Series formulation. In: Proceedings of IEEE/RSJ international conference on intelligent robots and systems, pp. 4435–4440.

  39. Huang, C. L. (2011). Research of gait planning for biped robot. Guangdong University Of Technology. https://doi.org/10.7666/d.y1941788

    Article  Google Scholar 

  40. Feng, H. (2017). The optimization of skills and cooperation with machine learning in RoboCup3D. Nanjing University Of Posts And Telecommunications.

  41. Hecheng, Z., Zhiwei, L., & Qingyuan, W. (2015). Long range kick for RoboCup3D—A practical approach. In: 2015 34th Chinese control conference (CCC). IEEE

  42. MacAlpine, P., Urieli, D., Barrett, S., Austin Villa, U. T., et al. (2011). a champion agent in the RoboCup 3D soccer simulation competition. AAMAS, 2012, 129–136.

    Google Scholar 

  43. Shi, H., Li, X., Chen, H., & Wang, S. (2016). Adaptive omni-directional walking method with fuzzy interpolation for biped robots. International Journal of Networked and Distributed Computing, 4(3), 145–158.

    Article  Google Scholar 

  44. Shi, H., Li, X., Liang, W., Dang, M., Chen, H., & Wang, S. (2016). A novel fuzzy omni-directional gait planning algorithm for biped robot. 2016 17th IEEE/ACIS international conference on software engineering, artificial intelligence, networking and parallel/distributed computing (SNPD), pp. 71–76. https://doi.org/10.1109/SNPD.2016.7515880.

  45. Depinet, M., MacAlpine, P., & Stone, P. (2015). Keyframe sampling, optimization, and behavior integration: Towards long-distance kicking in the RoboCup 3D Simulation League. In R. Bianchi, H. Akin, S. Ramamoorthy, & K. Sugiura (Eds.), RoboCup 2014: Robot World Cup XVIII. RoboCup 2014. Lecture Notes in Computer Science(). (Vol. 8992). Cham: Springer.

  46. Kasaei, M., Lau, N., & Pereira, A. (2019). A fast and stable omnidirectional walking engine for the nao humanoid robot. In S. Chalup, T. Niemueller, J. Suthakorn, & M. A. Williams (Eds.), RoboCup 2019: Robot World Cup XXIII. RoboCup 2019. Lecture Notes in Computer Science(). (Vol. 11531). Cham: Springer.

  47. Seekircher, A. (2015). Adaptive dynamic walking and motion optimization for humanoid robots (Order No. 3720027). Available from ProQuest dissertations and theses global A & I The Sciences and Engineering Collection. (1718489352).

  48. Simoes, M. A. C., Ramos, C. E., Argollo, E., et al. (2017). Bahiart 2018: Team description paper for RoboCup 3D soccer simulation league. RoboCup.

  49. MacAlpine, P., & Stone, P. (2018). UT Austin Villa: RoboCup 2017 3D Simulation league competition and technical challenges champions. In H. Akiyama, O. Obst, C. Sammut, & F. Tonidandel (Eds.), RoboCup 2017: Robot World Cup XXI. RoboCup 2017. Lecture Notes in Computer Science. (Vol. 11175). Cham: Springer.

  50. Farchy, A. (2012). Learning in simulation for real robots.

  51. Li, X., Liang, Z., & Feng, H. (2015). Kicking motion planning of Nao robots based on CMA-ES. The 27th Chinese control and decision conference (2015 CCDC). IEEE, pp. 6158–6161.

  52. Dorer, K. (2018). Learning to use toes in a humanoid robot. In H. Akiyama, O. Obst, C. Sammut, & F. Tonidandel (Eds.), RoboCup 2017: Robot World Cup XXI. RoboCup 2017. Lecture notes in computer science. (Vol. 11175). Cham: Springer.

  53. MacAlpine, P., Collins, N., Lopez-Mobilia, A., & Stone, P. (2013). UT Austin Villa: RoboCup 2012 3D simulation league champion. In X. Chen, P. Stone, L. E. Sucar, & T. van der Zant (Eds.), RoboCup 2012: Robot Soccer World Cup XVI. RoboCup 2012. Lecture Notes in Computer Science. (Vol. 7500). Heidelberg: Springer.

  54. Hansen, N., & Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9(2), 159–195.

    Article  Google Scholar 

  55. Hansen, N., Müller, S. D., & Koumoutsakos, P. (2003). Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMAES). Evolutionary Computation, 11(1), 1–18.

    Article  Google Scholar 

  56. Hansen, N., & Kern, S. (2004). Evaluating the CMA evolution strategy on multimodal test functions. In International conference on parallel problem solving from nature, pp. 282–291. Springer.

  57. Kern, S., Müller, S. D., Hansen, N., Büche, D., Ocenasek, J., & Koumoutsakos, P. (2004). Learning probability distributions in continuous evolutionary algorithms—A comparative review. Natural Computing, 3(1), 77–112.

    Article  MathSciNet  Google Scholar 

  58. Hansen, N. (2006). The CMA evolution strategy: A comparing review. In Towards a new evolutionary computation, pp. 75–102. Springer

  59. Abdolmaleki, A., Simoes, D., Lau, N., et al. (2016). Learning a humanoid kick with controlled distance. Robot World Cup (pp. 45–57). Cham: Springer.

  60. Abdolmaleki, A., Simoes, D., Lau, N., et al. (2019). Contextual direct policy search. Journal of Intelligent & Robotic Systems, 96(2), 141–157.

    Article  Google Scholar 

  61. Lu, Y., Liang, Z., He, H., Xu, C., Yang, B., & Fang, F. (2019). 3D humanoid robot multi-gait switching and optimization. Chinese Control And Decision Conference (CCDC), 2019, 4196–4201. https://doi.org/10.1109/CCDC.2019.8832817

    Article  Google Scholar 

  62. Jouandeau, N., & Hugel, V. (2013). Simultaneous evolution of leg morphology and walking skills to build the best humanoid walker. IEEE-RAS international conference on humanoid robots, 8th workshop on humanoid soccer robots 2013.

  63. Uchitane, T., & Hatanaka, T. (2011). Applying evolution strategies for biped locomotion learning in RoboCup 3D soccer simulation. IEEE Congress of Evolutionary Computation (CEC), 2011, 179–185. https://doi.org/10.1109/CEC.2011.5949616

    Article  Google Scholar 

  64. Uchitane, T., Hatanaka, T., & Uosaki, K. (2010). Evolution strategies for biped locomotion learning using nonlinear oscillators. Proceedings of SICE annual conference 2010. IEEE, pp. 1458–1461.

  65. MacAlpine, P., Liebman, E., & Stone, P. (2016). Adaptation of surrogate tasks for bipedal walk optimization. In Proceedings of the 2016 on genetic and evolutionary computation conference companion (GECCO ’16 Companion). Association for Computing Machinery, New York, pp. 1275–1276.

  66. Urieli, D., MacAlpine, P., Kalyanakrishnan, S., et al. (2010). Optimizing interdependent skills for simulated 3D humanoid robot soccer. The fifth workshop on humanoid soccer robots at humanoids.

  67. Melo, L. C, Maximo, M. R. O. A., & da Cunha A. M. (2019). Bottom-up meta-policy search. arXiv:1910.10232.

  68. Zixuan, Z., Yu, Z., & Jiawen, W. (2016). Omnidirectional walk design of humanoid robots using layered learning method based on CMA-ES. 2016 IEEE advanced information management, communicates, electronic and automation control conference (IMCEC). IEEE, pp. 464–468.

  69. MacAlpine, P., Depinet, M., Liang, J., et al. (2014). UT Austin Villa: RoboCup 2014 3D simulation league competition and technical challenge champions. Robot Soccer World Cup (pp. 33–46). Cham: Springer.

  70. Urieli, D., MacAlpine, P., Kalyanakrishnan, S., et al. (2011). On optimizing interdependent skills: A case study in simulated 3D humanoid robot soccer. AAMAS, 11, 769.

    Google Scholar 

  71. Baur, M., Christmann, K., Dorer, K., et al. (2018). The magmaOffenburg 2018 RoboCup 3D simulation team. In: RoboCup 2018 symposium and competitions: Team description papers.

  72. Tao, C., Xue, J., Zhang, Z., et al. (2021). Gait optimization method for humanoid robots based on parallel comprehensive learning particle swarm optimizer algorithm. Frontiers in Neurorobotics, 14, 600885.

    Article  Google Scholar 

  73. Halataei, F., & Kayhani, A. K. (2015). Optimizing NAO humanoid walking using ABC algorithm. 2015 2nd international conference on knowledge-based engineering and innovation (KBEI) (pp. 1142–1144). https://doi.org/10.1109/KBEI.2015.7436208.

  74. Jouandeau, N., & Hugel, V. (2014). Optimization of parametrised kicking motion for humanoid soccer player. IEEE international conference on autonomous robot systems & competitions. IEEE.

  75. Cai, C., & Jiang, H. (2013). Performance comparisons of evolutionary algorithms for walking gait optimization. International Conference on Information Science and Cloud Computing Companion, 2013, 129–134. https://doi.org/10.1109/ISCC-C.2013.100

    Article  Google Scholar 

  76. Rei, J. L. M. (2010). Optimizing simulated humanoid robot skills.

  77. Picado, H., Gestal, M., Lau, N., et al. (2009). Automatic generation of biped walk behavior using genetic algorithms. International work-conference on artificial neural networks. Berlin: Springer, pp. 805–812.

  78. Cruz, L., Reis, L.P., Lau, N., & Sousa, A. (2012). Optimization approach for the development of humanoid robots’ behaviors. In: Pavón, J., Duque-Méndez, N.D., Fuentes-Fernández, R. (eds) Advances in artificial intelligence—IBERAMIA 2012. IBERAMIA 2012. Lecture Notes in Computer Science, Vol. 7637. Berlin: Springer.

  79. Lattarulo, V., & Dijk, S. G. (2011). Application of the “alliance algorithm” to energy constrained gait optimization (pp. 472–483). Robot Soccer World Cup. Springer: Berlin.

  80. He, H., Liang, Z., Lu, Y., Xu, C., Yang, B., & Fang, F. (2019). Dynamic kick optimization of humanoid robot based on options framework. Chinese Control And Decision Conference (CCDC), 2019, 5176–5181. https://doi.org/10.1109/CCDC.2019.8833269

    Article  Google Scholar 

  81. Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An introduction (2nd ed.). MIT Press.

    Google Scholar 

  82. Xing-Xing, L., Yang-He, F., Yang, M., Guang-Quan, C., Jin-Cai, H., Qi, W., Yu-Zhen, Z., & Zhong, L. (2020). Deep multi-agent reinforcement learning: A survey. Acta Automatica Sinica, 46(12), 2537–2557.

    Google Scholar 

  83. Nai-Jun, L. I. U., Tao, L. U., Ying-Hao, C. A. I., & Shuo, W. A. N. G. (2019). A review of robot manipulation skills learning methods. Acta Automatica Sinica, 45(3), 458–470.

    Google Scholar 

  84. Bellman, R. (1952). On the theory of dynamic programming. Proceedings of the National Academy of Sciences of the United States of America, 38(8), 716–719.

    Article  MathSciNet  Google Scholar 

  85. Wang, Q., Zhao, X., Huang, J. C., Feng, Y. H., Liu, Z., Su, Z. H., et al. (2017). Addressing complexities of machine learning in big data: Principles, trends and challenges from systematical perspectives. https://doi.org/10.20944/preprints201710.0076.v1

  86. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533.

    Article  Google Scholar 

  87. Dong-Bin, Z., Kun, S., Yuan-Heng, Z., Dong, L., Ya-Ran, C., Hai-Tao, W., et al. (2016). Review of deep reinforcement learning and discussions on the development of computer Go. Control Theory and Applications, 33(6), 701–717.

    Google Scholar 

  88. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., et al. (2017). Mastering the game of Go without human knowledge. Nature, 550(7676), 354–359.

    Article  Google Scholar 

  89. Graves, A., Wayne, G., Reynolds, M., et al. (2016). Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626), 471–476.

    Article  Google Scholar 

  90. Zhang, T. Y., Huang, M. L., & Zhao, L. (2018). Learning structured representation for text classification via reinforcement learning. in Proceedings of the 32nd AAAI conference on artificial intelligence (pp. 6053–6060). New Orleans: AAAI Press

  91. Su, P. H., Gasic, M., Mrksic, N., Rojas-Barahona, L. M., Ultes, S., Vandyke, D. et al. (2016). On-line active reward learning for policy optimisation in spoken dialogue systems. In Proceedings of the 54th annual meeting of the association for computational linguistics. Berlin: Association for Computational Linguistics.

  92. Zhi-Hua, Z. (2016). AlphaGo special session: An introduction. Acta Automatica Sinica, 42(5), 670.

    Google Scholar 

  93. Silver, D., Huang, A., Maddison, C. J. et al. Mastering the game of Go with deep neural networks and tree search. Nature.

  94. Silver, D., Schrittwieser, J., Simonyan, K., et al. (2017). Mastering the game of Go without human knowledge. Nature, 550, 354–359. https://doi.org/10.1038/nature24270

    Article  Google Scholar 

  95. Spitznagel, M., Weiler, D., & Dorer, K. (2021). Deep reinforcement multi-directional kick-learning of a simulated robot with toes. IEEE international conference on autonomous robot systems and competitions (ICARSC) (Vol. 2021, pp. 104–110). https://doi.org/10.1109/ICARSC52212.2021.9429811

  96. Abreu, M., Lau, N., Sousa, A., & Reis, L. P. (2019). Learning low level skills from scratch for humanoid robot soccer using deep reinforcement learning. IEEE international conference on autonomous robot systems and competitions (ICARSC) (Vo. 2019, pp. 1–8). https://doi.org/10.1109/ICARSC.2019.8733632

  97. Abreu, M., Reis, L. P., & Lau, N. (2019). Learning to run faster in a humanoid robot soccer environment through reinforcement learning (pp. 3–15). Robot World Cup. Springer: Cham.

  98. Melo, L. C., Melo, D. C., & Maximo, M. R. O. A. (2021). Learning humanoid robot running motions with symmetry incentive through proximal policy optimization. Journal of Intelligent & Robotic Systems, 102(3), 1–15.

    Article  Google Scholar 

  99. Melo, L. C., & Máximo, M. R. O. A. (2019). Learning Humanoid Robot Running Skills through Proximal Policy Optimization. 2019 Latin American robotics symposium (LARS), 2019 Brazilian symposium on robotics (SBR) and 2019 workshop on robotics in education (WRE) (pp. 37–42). https://doi.org/10.1109/LARS-SBR-WRE48964.2019.00015.

  100. Teixeira, H., Silva, T., Abreu, M., & Reis, L. P. (2020). Humanoid robot kick in motion ability for playing robotic soccer. IEEE International conference on autonomous robot systems and competitions (ICARSC), (Vol. 2020, pp. 34–39). https://doi.org/10.1109/ICARSC49921.2020.9096073

  101. Abreu, M., Silva, T., Teixeira, H., et al. (2021). 6D localization and kicking for humanoid robotic soccer. Journal of Intelligent & Robotic Systems, 102(2), 1–25.

    Article  Google Scholar 

  102. Kasaei, M., Abreu, M., Lau, N. et al. (2021). Robust biped locomotion using deep reinforcement learning on top of an analytical control approach. arXiv:2104.10592 .

  103. Muzio, A. F. V., Maximo, M. R. O. A. & Yoneyama, T. (2020). Deep reinforcement learning for humanoid robot dribbling. 2020 Latin American robotics symposium (LARS), 2020 Brazilian symposium on robotics (SBR) and 2020 workshop on robotics in education (WRE) (pp. 1–6). https://doi.org/10.1109/LARS/SBR/WRE51543.2020.9307084

  104. Muzio, A. F. V., Maximo, M. R. O. A., & Yoneyama, T. (2022). Deep reinforcement learning for humanoid robot behaviors. Journal of Intelligent & Robotic Systems, 105(1), 1–16.

    Article  Google Scholar 

  105. Rezaeipanah, A., Amiri, P., & Jafari, S. (2020). Performing the kick during walking for RoboCup 3D soccer simulation league using reinforcement learning algorithm. International Journal of Social Robotics, 1-18.

  106. de Almeida Martins, H. M. N. (2023). FCPortugal-machine learning for a flexible kicking robotic soccer skill.

  107. Wang, J., Liang, Z., Zhou, Z., & Zhang, Y. (2016). Kicking motion design of humanoid robots using gradual accumulation learning method based on Q-learning. Chinese Control and Decision Conference (CCDC), 2016, 5274–5279. https://doi.org/10.1109/CCDC.2016.7531941

    Article  Google Scholar 

  108. Melo, D. C., Máximo, M. R. O. A., & da Cunha, A. M. (2020). Push recovery strategies through deep reinforcement learning. 2020 Latin American robotics symposium (LARS), 2020 Brazilian symposium on robotics (SBR) and workshop on robotics in education (WRE), (Vol. 2020, pp. 1–6). https://doi.org/10.1109/LARS/SBR/WRE51543.2020.9306967

  109. Melo, D. C., Maximo, M. R. O. A., & da Cunha, A. M. (2022). Learning push recovery behaviors for humanoid walking using deep reinforcement learning. Journal of Intelligent & Robotic Systems, 106(1), 1–18.

    Article  Google Scholar 

  110. Liang, Z., Shen, P., & Li, X. (2015). Walking motion design of humanoid robots in RoboCup3D simulation platform. International Journal of Modelling and Simulation, 35(1), 35–42.

    Article  Google Scholar 

  111. Liang, Z., & Zhu, S. (2012). Walking parameters training algorithm of humanoid robot based on reinforcement learning. Computer Engineering, 38(8), 13–15.

    Google Scholar 

  112. Liang, Z., Zhu, S., & Jin, X. (2011). Walking parameters design of biped robots based on reinforcement learning. Proceedings of the 30th Chinese control conference. IEEE, pp. 4017–4022.

  113. Li, C., Li, M., & Tao, C. (2023). A parallel heterogeneous policy deep reinforcement learning algorithm for bipedal walking motion design. Frontiers in Neurorobotics, 17.

  114. Kiefer, J., & Dorer, K. (2023). Double Deep Reinforcement Learning. 2023 IEEE international conference on autonomous robot systems and competitions (ICARSC) (pp. 17–22). IEEE.

  115. Pavse, B. S., Torabi, F., Hanna, J., Warnell, G., & Stone, P. (2020). RIDM: Reinforced inverse dynamics modeling for learning from a single observed demonstration. IEEE Robotics and Automation Letters, 5(4), 6262–6269. https://doi.org/10.1109/LRA.2020.3010750

    Article  Google Scholar 

  116. Li, X. (2010). Study on the CMAC based coordinated control of multidrive systems. Hefei University Of Technology.

  117. Xu, Y. (2005). Agent localization in RoboCup3D. Proceedings of 2005 China robot competition.

  118. Chen, S., Yi, Y., & Tan, Y. (2009). RoboCup3D football localization method based on \(\alpha\)-\(\beta\)-\(\gamma\) filtering. Proceedings of the 2009 China conference on intelligent automation (Vol. 2).

  119. Shen, L., Ye, P., Ding, X., et al. (2010). Research on key technologies of RoboCup 3D simulation. Robot Technique and Application, 4, 54–56.

    Google Scholar 

  120. Shi, G. (2010). Implementation of omni-directional walking and high-level decision for humanoid Robots in RoboCup3D simulation system. Hefei University of Technology.

  121. Yin, Z., & Chen, W. H. Fantasia 2009 Team Description. Coordinates, 11(12):13.

  122. Vorst, P. (2006). Readylog agents for the robocup 3d soccer simulation league. RWTH Aachen University Thesis.

  123. Zhao, Y. (2010). Self localization under restricted vision in RoboCup3D. Electronic Test, 4, 81–85.

    Google Scholar 

  124. Sun, Z., Huang, H., & Ren, X. (2009). Application of an Enhanced Location System with the Virtual Multi-Sensor. In WRI global congress on intelligent systems (Vol. 1, pp. 36–41). IEEE.

  125. Wang, X., Yan, X., Zhang, Y., et al. (2012). Kalman filter in the robocup 3D positioning. 2012 international conference on computer science and electronics engineering (Vol. 3, pp. 47–52). IEEE.

  126. Seekircher, A., Abeyruwan, S., & Visser, U. (2011). Accurate ball tracking with extended Kalman filters as a prerequisite for a high-level behavior with reinforcement learning. The 6th Workshop on Humanoid Soccer Robots at Humanoid Conference, Bled (Slovenia).

  127. Fox, D., Burgard, W., Dellaert, F., et al. (1999). Monte Carlo localization: Efficient position estimation for mobile robots. AAAI/IAAI, 1999(343–349), 2–2.

    Google Scholar 

  128. Bustamante Horta C F. Probabilistic agent localization and fuzzy-bayesian pass evaluation for the RoboCup simulation 3D League-Edicion Unica.

  129. Hao, Y., Liang, Z., Liu, J. et al. (2013). The framework design of humanoid robots in the robocup 3D soccer simulation competition. 2013 10Th IEEE international conference on control and automation (ICCA). IEEE, 1423-1428.

  130. Wang, J. R., Yu, Y., Wei, J. G., et al. (2013). A Robot Simulation System For Self-location by using field visual information. Advanced Materials Research, 748, 690–694.

    Article  Google Scholar 

  131. Fu, H., & Cao, F. (2016). A survey of robot self-localization and the research in RoboCup3D. Computer Knowledge and Technology: Academic Edition, 4, 172–174.

    Google Scholar 

  132. Simoes, M. A. C., Ramos, C. E., Argollo, E. et al. (2017). Bahiart 2017: Team description paper for robocup 3D soccer simulation league. RoboCup.

  133. Fernandes, G. C. G., Dias, S. S., Maximo, M. R. O. A., et al. (2020). Cooperative localization for multiple soccer agents using factor graphs and sequential Monte Carlo. IEEE Access, 8, 213168–213184.

    Article  Google Scholar 

  134. Lu, W., Zhang, J., Zhao, X., et al. (2017). Multimodal sensory fusion for soccer robot self-localization based on long short-term memory recurrent neural network. Journal of Ambient Intelligence and Humanized Computing, 8(6), 885–893.

    Article  Google Scholar 

  135. Chen, T. (2014). Research and implement of RoboCup 3D simulation robot. Anhui University Of Technology.

  136. He, K., Liang, Z., Cui, T. et al. (2018). Formation optimization of RoboCup3D soccer robots using delaunay triangulation network. 2018 Chinese control and decision conference (CCDC) (pp. 224–229). IEEE

  137. Akiyama, H., & Noda, I. (2007). Multi-agent positioning mechanism in the dynamic environment (pp. 377–384). Robot Soccer World Cup. Springer: Berlin.

  138. Larik, A. S., & Haider, S. (2016). On using evolutionary computation approach for strategy optimization in robot soccer. 2016 2nd International Conference on Robotics and Artificial Intelligence (ICRAI). IEEE, 11-16.

  139. MacAlpine, P., Barrera, F., & Stone, P. (2012). Positioning to win: A dynamic role assignment and formation positioning system. In: Workshops at the twenty-sixth AAAI conference on artificial intelligence.

  140. Chen, W., & Chen, T. (2011). Multi-robot dynamic role assignment based on path cost. 2011 Chinese control and decision conference (CCDC) (pp. 3721–3724). IEEE.

  141. Ulusoy, O., & Talay, S. S. (2012). Distributed team formation for humanoid Robot Soccer. ICAART, 1, 605–613.

    Google Scholar 

  142. Li, L., & Fang, Y. (2015). Research on decision system model of Robocup3D robotics team. Computer Engineering and Applications, 6, 37–41.

    Google Scholar 

  143. MacAlpine, P., & Stone, P. (2016). Prioritized role assignment for marking (pp. 306–318). Robot World Cup. Springer: Cham.

  144. Chen, L., Qin, S., Chen, K. et al. (2020). Efficient role assignment with priority in Robocup3D. 2020 Chinese control and decision conference (CCDC) (pp. 2697–2702). IEEE.

  145. Abeyruwan, S., Seekircher, A., & Visser, U. (2014). Off-policy general value functions to represent dynamic role assignments in RoboCup 3D soccer simulation. arXiv:1402.4525.

  146. Abeyruwan, S., & Visser, U. (2014). A new real-time algorithm to extend DL assertional formalism to represent and deduce entities in robotic soccer (pp. 270–282). Robot Soccer World Cup. Springer: Cham.

  147. Chang, E. C., Choi, S. W., Kwon, D. Y., Park, H. & Yap, C. K. (2005). Shortest path amidst disc obstacles is computable. Proceedings of the annual symposium on computational Geometry (pp. 116–125).

  148. Zhen, C. (2013). The motion planning and intelligent decision of biped Robots in RoboCup3D simulation environment. Nanjing University Of Posts And Telecommunications.

  149. Su, L., & Liang, Z. (2013). Collision-avoidance planning of soccer robot in RoboCup3D simulation environment. Proceedings of the 32nd Chinese control conference (pp. 5739–5743). IEEE

  150. Rayermann, M., Cake, T.M. Cooperative pathfinding in 3D Robot Soccer.

  151. Muzio, A., Melo, D., Henrique, E. et al. (2016). Itandroids soccer3d team description paper 2016.

  152. Li, C., Tao, C., Liu, G., et al. (2016). Quantized flocking control for second-order multiple agents with obstacle avoidance. Advances in Mechanical Engineering, 8(1), 1687814015624646.

    Article  Google Scholar 

  153. Yao, Q. (2012). The Research of RoboCup 3D Simulation Robot. Guangdong University Of Technology.

  154. Shen, X., & Liu, G. (2011), Robocup vanguard’s goal-scoring ability based on Q-learning. Jisuanji Gongcheng yu Yingyong (Computer Engineering and Applications), 47(18):53–55.

  155. Rettinger, A., Zinkevich, M., & Bowling, M. (2006). Boosting expert ensembles for rapid concept recall. AAAI (pp. 464–469).

  156. Stone, P., & Sutton, R. S. (2001). Scaling reinforcement learning toward RoboCup soccer. ICML, 1, 537–544.

    Google Scholar 

  157. Li, X. (2016). Soccer robots local passing and tactics cooperation. Nanjing University Of Posts And Telecommunications.

  158. Xuanyu, C., Zhiwei, L., Yongyi, Y. et al. (2015). Multi-robot collaboration based on Markov decision process in Robocup3D soccer simulation game. The 27th Chinese control and decision conference (2015 CCDC) (pp. 4345–4349). IEEE.

  159. Zhao, Q., Liang, Z., Fang, F., et al. (2017). Local passing-ball tactics based on a Keepaway algorithm. 2017 29th Chinese control and decision conference (CCDC) (pp. 4884–4889). IEEE.

  160. Gupta, N., & Kalyanakrishnan, S. Learning complex behaviours and Keepaway in 3D Robocup environment.

  161. Huang, R., Xu, Y., & Tan, Y. (2008). Research and implementation of RoboCup goalkeeper action and strategy. China Science and Technology Information, 21, 36–37.

    Google Scholar 

  162. Masterjohn, J. G., Polceanu, M., Jarrett, J., et al. (2015). Regression and mental models for decision making on robotic biped goalkeepers (pp. 177–189). Robot Soccer World Cup. Springer: Cham.

  163. Polceanu M. (2015). ORPHEUS: Reasoning and prediction with heterogeneous representations using simulation. Université de Bretagne Occidentale (UBO).

  164. Wei, S., & Qin, H. (2022). Application of RoboCup 3D and intelligent technology in football simulation league. In; Computational intelligence and neuroscience.

  165. Yao, Q. (2015). Research on Robocup3D simulation robot interception technology. Shandong Industrial Technology, 11, 52–53.

    Google Scholar 

  166. Zhu, J. (2012). Prediction of the real-time interceptor algorithm for robot football match simulation. Electronic Test, 2, 31–34.

    Google Scholar 

  167. Mirmohammad, Y., Khorsandi, S., Shahsavari, M. N., et al. (2021). Ball path prediction for humanoid robots: Combination of k-NN regression and autoregression methods (pp. 3–14). Robot World Cup. Springer: Cham.

  168. Abadi, M. M. N., Lucas, C. Evolving Artificial Neural Networks for Prediction in Robocup Soccer.

  169. Yang, C., Chang, X., Chen, J. et al. (2019). Situation assessment for soccer robots using deep neural network. 2019 IEEE 9th international conference on electronics information and emergency communication (ICEIEC) (pp. 1–4). IEEE

  170. Larik, A. S., & Haider, S. (2012). Rule-based behavior prediction of opponent agents using robocup 3D soccer simulation league logfiles. In; IFIP international conference on artificial intelligence applications and innovations (pp. 285–295). Berlin: Springer.

  171. Raza, A., Sharif, U., Haider, S. (2012). On learning coordination among soccer agents. In: 2012 IEEE international conference on robotics and biomimetics (ROBIO) (pp. 699–703). IEEE.

  172. Simoes, M., & Nogueira, T. (2018). Towards setplays learning in a multiagent robotic soccer team. 2018 Latin American robotic symposium, 2018 Brazilian symposium on robotics (SBR) and 2018 Workshop on Robotics in Education (WRE) (pp. 277–282). IEEE.

  173. MacAlpine, P., Genter, K., Barrett, S., et al. (2014). The RoboCup 2013 drop-in player challenges: Experiments in ad hoc teamwork. In 2014 IEEE/RSJ international conference on intelligent robots and systems. (pp. 382–387). IEEE.

  174. Larik, A. S. (2013). Opponent modeling in RoboCup soccer simulation 3D. In: German conference on multiagent system technologies (pp. 416–419). Berlin: Springer.

  175. Larik, A. S., & Haider, S. (2015). Opponent classification in robot soccer. In International conference on industrial, engineering and other applications of applied intelligent systems (pp. 478–487). Cham: Springer

  176. Holmes, G., Donkin, A., & Witten, I. H. (1994). WEKA: A machine learning workbench. In Proceedings of second Australia and New Zealand conference on intelligent information systems, Brisbane, Australia

  177. Eibe, F., & Witten, I. H. (1998) Generating accurate rule sets without global optimization. In Proceedings of the 15th international conference on machine learning, San Francisco, USA

  178. Simoes, M. A. C., Nobre, J., Sousa, G. et al. (2020). Strategy planner: enhancements to support better defense and pass strategies within an LfD approach. In 2020 IEEE international conference on autonomous robot systems and competitions (ICARSC) (pp. 46–52). IEEE.

  179. de Sousa Pereira, V. (2020). FCPortugal-multi-robot action learning.

  180. da Silva R. M., de Souza J. R., Simoes, M. A. C. et al. (2018). Framework for modeling autonomous multi-robots systems. 2018 Latin American robotic symposium, 2018 Brazilian symposium on robotics (SBR) and 2018 workshop on robotics in education (WRE) (pp. 13–18). IEEE.

  181. Simōes, M. A. C., Mascarenhas, G., Fonseca, R., et al. (2022). Bahiart setplays collecting toolkit and BahiaRT gym. Software Impacts, 14, 100401.

    Article  Google Scholar 

  182. Simoes, M. A. C., & Nogueira, T. (2022). Learning by demonstration of coordinated plans in multiagent systems. Anais Estendidos do XIV Simpósio Brasileiro de Robótica e XIX Simpósio Latino-Americano de Robótica. SBC, pp. 121–132.

  183. Miikkulainen, R., & Grauman, K. Making friends on the Fly: Advances in ad hoc teamwork.

  184. MacAlpine, P., & Stone, P. (2012). Using dynamic rewards to learn a fully holonomic bipedal walk. In: Adaptive learning agents workshop.

  185. Lopez-Mobilia, A. (2012). Inverse kinematics kicking in the humanoid RoboCup simulation league. Master’s Thesis, University of Texas at Austin.

  186. MacAlpine, P., Depinet, M., & Stone, P. (2015). UT Austin Villa 2014: RoboCup 3D simulation league champion via overlapping layered learning. Proceedings of the AAAI conference on artificial intelligence, 29(1).

  187. MacAlpine, P., Price, E., & Stone, P. (2015). SCRAM: Scalable collision-avoiding role assignment with minimal-makespan for formational positioning. In: Twenty-ninth AAAI conference on artificial intelligence.

  188. MacAlpine, P., Hanna, J., Liang, J., et al. (2015) UT Austin Villa: RoboCup 2015 3D simulation league competition and technical challenges champions. In Robot soccer World Cup (pp. 118–131). Cham: Springer.

  189. MacAlpine, P., Torabi, F., Pavse, B., et al. (2018). UT Austin Villa: RoboCup 2018 3D simulation league champions (pp. 462–475), Robot World Cup. Cham: Springer.

  190. MacAlpine, P., & Stone, P. (2016). UT Austin Villa: RoboCup 2016 3D simulation league competition and technical challenges champions (pp. 515–528), Robot World Cup. Cham: Springer.

  191. Abreu, M., Reis, L. P., & Lau, N. (2023). Designing a Skilled Soccer Team for RoboCup: Exploring skill-set-primitives through reinforcement learning. arXiv:2312.14360.

  192. Oliveira, G. N., Maximo, M. R. O. A., & Curtis, V. V. (2022). Distributed optimization tool for RoboCup 3D soccer simulation league using intel DevCloud (pp. 152–163) Robot World Cup. Cham: Springer International Publishing.

  193. Lau, N. (2023). FC Portugal: RoboCup 2022 3D simulation league and technical challenge champions (Vol. 13561). RoboCup 2022: Robot World Cup XXV.

  194. Hart, P. E., Nilsson, N. J., & Raphael, B. (1968). A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2), 100–107. https://doi.org/10.1109/TSSC.1968.300136

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210000, China

    Zhongye Gao, Mengjun Yi, Ying Jin & Furao Shen

  2. Department of Computer Science and Technology, Nanjing University, Nanjing, 210000, China

    Zhongye Gao & Ying Jin

  3. School of Artificial Intelligence, Nanjing University, Nanjing, 210000, China

    Mengjun Yi, Hanwen Zhang, Yun Hao, Ming Yin, Ziwen Cai & Furao Shen

Authors
  1. Zhongye Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mengjun Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ying Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hanwen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yun Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ming Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ziwen Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Furao Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

ZG and MY wrote the main manuscript text. Others organize figures and tables and check for revisions. All authors reviewed the manuscript.

Corresponding author

Correspondence to Ying Jin.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, Z., Yi, M., Jin, Y. et al. A survey of research on several problems in the RoboCup3D simulation environment. Auton Agent Multi-Agent Syst 38, 13 (2024). https://doi.org/10.1007/s10458-024-09642-z

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10458-024-09642-z

Keywords

Search

Navigation