Abstract
Nowadays, humanoid soccer serves as a benchmark for artificial intelligence and robotic problems. The factors such as the kicking speed and the number of kicks by robot soccer players are the most significant aims that the participating teams are pursued in the RoboCup 3D Soccer Simulation League. The proposed method presents a kicking strategy during walking for humanoid soccer robots. Achieving an accurate and powerful kicking while robots are moving requires a dynamic optimization of the speed and motion parameters of the robot. In this paper, a curved motion path has been designed based on the robot position relative to the ball and the goal. Ultimately, the robot will be able to kick at the goal by walking along this curve path. The speed and angle of the walking robot are set towards the ball with regard to the robots curved motion path. After the final step of the robot, the accurate and effective adjustment of these two parameters ensures that the robot is located in the ideal position to perform the perfect kick. Due to the noise and walking condition of the robot, it is essential that the speed and angle of motion to be measured more accurately. For this purpose, we use a reinforcement learning model to adjust the robots step size and so does achieve the optimal value of two abovementioned parameters. Using reinforcement learning, robot would learn to pursue an optimal policy to correctly kick towards designated points. Therefore, the proposed method is a model-free and based on dynamic programming. The experiments reveal that the proposed method has significantly improved the team overall performance and robots ability to kick. Our proposed method has been 9.32% successful on average and outperformed the UTAustinVilla agent in terms of goal-scoring time in a non-opponent simulator.
Similar content being viewed by others
References
Kitano H, Okuno HG, Nakadai K, Sabisch T, Matsui T (2000) Design and architecture of SIG the humanoid: an experimental platform for integrated perception in RoboCup humanoid challenge. In: IEEE/RSJ international conference on intelligent robots and systems (IROS 2000), pp 181–190. https://doi.org/10.1109/IROS.2000.894602
Haddadin S, Laue T, Frese U, Wolf S, Albu-Schäffer A, Hirzinger G (2009) Kick it with elasticity: safety and performance in human–robot soccer. Robot Auton Syst 57:761–775. https://doi.org/10.1016/j.robot.2009.03.004
Carlson Z, Lemmon L, Higgins M, Frank D, Shahrezaie RS, Feil-Seifer D (2019) Perceived mistreatment and emotional capability following aggressive treatment of robots and computers. Int J Social Robot 11:1–13. https://doi.org/10.1007/s12369-019-00599-8
RoboCup, http://www.robocup.org/. Accessed 29 April 2020
Shi H, Xu L, Zhang L, Pan W, Xu G (2015) Research on self-adaptive decision-making mechanism for competition strategies in robot soccer. Front Comput Sci 9:485–494. https://doi.org/10.1007/s11704-015-4210-7
Colbaugh R, Glass K, Seraji H (1991) An adaptive inverse kinematics algorithm for robot manipulators. Int J Model Simul 11:33–38. https://doi.org/10.1080/02286203.1991.11760118
Jouandeau N, Hugel V (2014) Optimization of parametrised kicking motion for humanoid soccer player. In: IEEE international conference on autonomous robot systems and competitions, pp 241–246. https://doi.org/10.1109/ICARSC.2014.6849793
Hereid A, Cousineau EA, Hubicki CM, Ames AD (2016). 3D dynamic walking with underactuated humanoid robots: A direct collocation framework for optimizing hybrid zero dynamics. In: IEEE international conference on robotics and automation, pp 1447–1454. https://doi.org/10.1109/ICRA.2016.7487279
SimSpark, http://simspark.sourceforge.net/. Accessed 29 April 2020
Böckmann A, Laue T (2016) Kick motions for the nao robot using dynamic movement primitives. In: Robot world cup, Springer, Cham, pp 33–44. https://doi.org/10.1007/978-3-319-68792-6_3
Claret JA, Venture G, Basañez L (2017) Exploiting the robot kinematic redundancy for emotion conveyance to humans as a lower priority task. Int J Soc Robot 9:277–292. https://doi.org/10.1007/s12369-016-0387-2
Watkins CJ, Dayan P (1992) Q-learning. Mach Learn 8:279–292. https://doi.org/10.1007/BF00992698
Akgun B, Cakmak M, Jiang K, Thomaz AL (2012) Keyframe-based learning from demonstration. Int J Soc Robot 4:343–355. https://doi.org/10.1007/s12369-012-0160-0
MacAlpine P, Stone P (2016) UT Austin Villa robocup 3D simulation base code release. In: Robot World Cup, Springer, Cham, pp 135–143. https://doi.org/10.1007/978-3-319-68792-6_11
Müller J, Laue T, Röfer T (2010) Kicking a ball–modeling complex dynamic motions for humanoid robots. In: Robot soccer world cup, Springer, Berlin, Heidelberg, pp 109–120. https://doi.org/10.1007/978-3-642-20217-9_10
Fu B, Zhang PX, Wang CF (2012) A cooperation strategy for shooting in robot soccer competition based on the multi-suppose tree. Procedia Engineering 29:1462–1466. https://doi.org/10.1016/j.proeng.2012.01.155
Nasrollahi P, Jafari S, Ebrahimi M (2012) Action classification of humanoid soccer robots using machine learning. In: Artificial intelligence and signal processing, CSI, IEEE, pp 598–603. https://doi.org/10.1109/AISP.2012.6313816
Röfer T, Laue T, Hasselbring A, Richter-Klug J, Röhrig E (2017) B-Human 2017–team tactics and robot skills in the standard platform league. In: Robot world cup, Springer, Cham, pp. 461–472. https://doi.org/10.1007/978-3-030-00308-1_38
Röfer T, Laue T (2013) On B-human’s code releases in the standard platform league–software architecture and impact. In: Robot soccer world cup, Springer, Berlin, Heidelberg, pp 648–655. https://doi.org/10.1007/978-3-662-44468-9_61
Genter K, MacAlpine P, Menashe J, Hannah J, Liebman E, Narvekar S, Stone P (2016) UT Austin Villa: Project-driven research in ai and robotics. IEEE Intell Syst 31:94–101. https://doi.org/10.1109/MIS.2016.35
Liang Z, Shen P, Li X (2015) Walking motion design of humanoid robots in RoboCup3D simulation platform. Int J Model Simul 35:35–42. https://doi.org/10.1080/02286203.2015.1080453
Depinet M, MacAlpine P, Stone P (2014) Keyframe sampling, optimization, and behavior integration: towards long-distance kicking in the robocup 3d simulation league. In: Robot soccer world cup, Springer, Cham, pp 571–582. https://doi.org/10.1007/978-3-319-18615-3_47
Ficht G, Allgeuer P, Farazi H, Behnke S (2017) NimbRo-OP2: grown-up 3D printed open humanoid platform for research. In: IEEE-RAS 17th international conference on humanoid robotics (humanoids), pp 669–675. https://doi.org/10.1109/HUMANOIDS.2017.8246944
Brindza J, Lee A, Majumdar A, Scharfman B, Schneider A, Shor R, Lee D (2009) Upennalizers robocup standard platform league team report 2009. University of Pennsylvania, Technical Report http://www.ais.uni-bonn.de/humanoidsoccer/ws10/papers/HSR10_02.pdf. Accessed 20 August 2019
Li X, Liang Z, Feng H (2015) Kicking motion planning of Nao robots based on CMA-ES. In: IEEE 27th Chinese control and decision conference (CCDC), pp 6158–6161. https://doi.org/10.1109/CCDC.2015.7161918
Beyer HG, Sendhoff B (2017) Simplify your covariance matrix adaptation evolution strategy. IEEE Trans Evol Comput 21:746–759. https://doi.org/10.1109/TEVC.2017.2680320
Stone P, Sutton RS, Kuhlmann G (2005) Reinforcement learning for robocup soccer keepaway. Adapt Behav 13:165–188. https://doi.org/10.1177/105971230501300301
Leottau DL, Ruiz-del-Solar J, MacAlpine P, Stone P (2015) A study of layered learning strategies applied to individual behaviors in robot soccer. In: Robot soccer world cup, Springer, Cham, pp 290–302. https://doi.org/10.1007/978-3-319-29339-4_24
Hester T, Quinlan M, Stone P (2010) Generalized model learning for reinforcement learning on a humanoid robot. In: IEEE international conference on robotics and automation, pp 2369–2374. https://doi.org/10.1109/ROBOT.2010.5509181
Fahami MA, Roshanzamir M, Izadi NH (2017) A reinforcement learning approach to score goals in RoboCup 3D soccer simulation for nao humanoid robot. In: 7th international conference on computer and knowledge engineering (ICCKE), IEEE, pp 450–454. https://doi.org/10.1109/ICCKE.2017.8167920
Teixeira H, Silva T, Abreu M, Reis LP (2020) Humanoid robot kick in motion ability for playing robotic soccer. In: International conference on autonomous robot systems and competitions (ICARSC), IEEE, pp 34–39. https://doi.org/10.1109/ICARSC49921.2020.9096073
Lu W, Zhang J, Zhao X, Wang J, Dang J (2017) Multimodal sensory fusion for soccer robot self-localization based on long short-term memory recurrent neural network. J Ambient Intell Humaniz Comput 8:885–893. https://doi.org/10.1007/s12652-017-0483-7
Torabian S, HoseinAlipour S, Mirzargar A, Tavakkolian M (2013) Improving the localization of humanoid soccer robots in specified fields: a neural network approach. In: First RSI/ISM international conference on robotics and mechatronics (ICRoM), IEEE, pp 443–448. https://doi.org/10.1109/ICRoM.2013.6510148
Yazdi E, Haghighat AT (2010) Evolution of biped walking using neural oscillators controller and harmony search algorithm optimizer. J Comput 2:111–118. https://arxiv.org/abs/1006.4553v1
Ferreira R, Reis LP, Moreira AP, Lau N (2012) Development of an omnidirectional kick for a nao humanoid robot. In: Ibero-American conference on artificial intelligence, Springer, Berlin, Heidelberg, pp 571–580. https://doi.org/10.1007/978-3-642-34654-5_58
Uchitane T, Hatanaka T (2011) Applying evolution strategies for biped locomotion learning in RoboCup 3D Soccer Simulation. In: IEEE congress of evolutionary computation (CEC), pp 179–185. https://doi.org/10.1109/CEC.2011.5949616
Huang Q, Yokoi K, Kajita S, Kaneko K, Arai H, Koyachi N, Tanie K (2001) Planning walking patterns for a biped robot. IEEE Trans Robot Autom 17:280–289. https://doi.org/10.1109/70.938385
Shamsuddin S, Ismail LI, Yussof H, Zahari NI, Bahari S, Hashim H, Jaffar A (2011) Humanoid robot NAO: review of control and motion exploration. In: IEEE international conference on control system, computing and engineering (ICCSCE), pp 511–516. https://doi.org/10.1109/ICCSCE.2011.6190579
MacAlpine P, Barrett S, Urieli D, Vu V, Stone P (2012) Design and optimization of an omnidirectional humanoid walk: a winning approach at the RoboCup 2011 3D simulation competition. In: 26th AAAI conference on artificial intelligence, pp 1047–1053. https://www.cs.utexas.edu/~pstone/Papers/bib2html-links/AAAI12-MacAlpine.pdf. Accessed 29 Apr 2020
Han SC, Li HX (2004) Invertible incline matrices and Cramer’s rule over inclines. Linear Algebra Appl 389:121–138. https://doi.org/10.1016/j.laa.2004.03.025
Gouaillier D, Collette C, Kilner C (2010) Omni-directional closed-loop walk for NAO. In: 10th IEEE-RAS international conference on humanoid robots (humanoids), pp 448–454. https://doi.org/10.1109/ICHR.2010.5686291
Muniz F, Maximo MR, Ribeiro CH (2016) Keyframe movement optimization for simulated humanoid robot using a parallel optimization framework. In: Robotics symposium and IV Brazilian robotics symposium (LARS/SBR), XIII Latin American, IEEE, pp 79–84. https://doi.org/10.1109/LARS-SBR.2016.20
Zhang PY, Lü TS, Song LB (2004) Soccer robot path planning based on the artificial potential field approach with simulated annealing. Robotica 22:563–566. https://doi.org/10.1017/S0263574703005666
Sun K, Liu L, Qiu J, Feng G (2020) Fuzzy adaptive finite-time fault-tolerant control for strict-feedback nonlinear systems. IEEE Trans Fuzzy Syst. https://doi.org/10.1109/TFUZZ.2020.2965890
Sun K, Jianbin Q, Karimi HR, Fu Y (2020) Event-triggered robust fuzzy adaptive finite-time control of nonlinear systems with prescribed performance. IEEE Trans Fuzzy Syst. https://doi.org/10.1109/TFUZZ.2020.2979129
Ghalehgolabi M, Rezaeipanah A (2017) Intrusion detection system using genetic algorithm and data mining techniques based on the reduction. Int J Comput Appl Technol Res 6:461–466. https://doi.org/10.7753/IJCATR0611.1003
Andersson SA, Madigan D, Perlman MD (2001) Alternative Markov properties for chain graphs. Scand J Stat 28:33–85. https://doi.org/10.1111/1467-9469.00224
Sunehag P, Evans R, Dulac-Arnold G, Zwols Y, Visentin D, Coppin B (2015) Deep reinforcement learning with attention for slate Markov decision processes with high-dimensional states and actions. arXiv preprint arXiv:1512.01124
Szepesvári C (2010) Algorithms for reinforcement learning. Synth Lect Artif Intell Mach Learn 4:1–103. https://doi.org/10.2200/S00268ED1V01Y201005AIM009
Tokic M (2010) Adaptive ε-greedy exploration in reinforcement learning based on value differences. In: Annual conference on artificial intelligence, Springer, Berlin, Heidelberg, pp 203–210. https://doi.org/10.1007/978-3-642-16111-7_23
Park JH (2001) Impedance control for biped robot locomotion. IEEE Trans Robot Autom 17:870–882. https://doi.org/10.1109/70.976014
Jafarian S, Abshirini A, Jafari S (2013) Online stability and direction fall detection for robotic soccer players using feedback from server. In: The 5th conference on information and knowledge technology, IEEE, pp 323–327. https://doi.org/10.1109/IKT.2013.6620087
Goldenberg A, Benhabib B, Fenton R (1985) A complete generalized solution to the inverse kinematics of robots. IEEE J Robot Autom 1:14–20. https://doi.org/10.1109/JRA.1985.1086995
Kofinas N, Orfanoudakis E, Lagoudakis MG (2015) Complete analytical forward and inverse kinematics for the NAO humanoid robot. J Intell Robot Syst 77:251–264. https://doi.org/10.1007/s10846-013-0015-4
Binaries File, https://archive.robocup.info/Soccer/Simulation/3D/binaries/RoboCup/. Accessed 29 Apr 2020
Xu Y, Vatankhah H (2013) Simspark: an open source robot simulator developed by the robocup community. In: Robot soccer world cup, Springer, Berlin, Heidelberg, pp 632–639. https://doi.org/10.1007/978-3-662-44468-9_59
Allan K (2013) Introduction. In: Disability in science fiction. Palgrave Macmillan, New York, pp 1–15. https://doi.org/10.1057/9781137343437_1
Riedmiller M, Gabel T, Hafner R, Lange S (2009) Reinforcement learning for robot soccer. Autonomous Robots 27:55–73. https://doi.org/10.1007/s10514-009-9120-4
Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3:9–44. https://doi.org/10.1007/BF00115009
Fang NC, Tsai TN, Wu LF, Cheng CH, Huang CY, Liu CY, Li THS (2017) Multi-robot coordination strategy for 3 vs. 3 teen-sized humanoid robot soccer game. In: automatic control conference (CACS), IEEE, pp 1–6. https://doi.org/10.1109/CACS.2017.8284265
Nasrollahi P, Jafari S, Jamaseb M, Nikooee A (2013). Decision making of humanoid soccer robots using rule based expert systems. In: The 5th conference on information and knowledge technology, IEEE, pp 258–261. https://doi.org/10.1109/IKT.2013.6620075
Gao L, Wang C, Zhang Y, Liu J. AUA3D Team Description Paper For RoboCup (2012) Robocup archive. http://archive.robocup.info/Soccer/Simulation/3D/TDPs/RoboCup/2012/AUA_SS3D_RC2012_TDP.pdf. Accessed 29 Apr 2020
Li C, Liu R, Feng X, Chen Y, Lu S, CIT3D Soccer Simulation Team Description for RoboCup (2012) Robocup archive. https://hedayat.fedorapeople.org/misc/rc2011tdps/cit3d_tdp.pdf. Accessed 29 Apr 2020
Scheunemann MM, van Dijk SG, Miko R, Barry D, Evans GM, Rossi A, Polani D (2019) Bold hearts team description for RoboCup 2019 (humanoid kid size league). arXiv preprint arXiv:1904.10066
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Rezaeipanah, A., Amiri, P. & Jafari, S. Performing the Kick During Walking for RoboCup 3D Soccer Simulation League Using Reinforcement Learning Algorithm. Int J of Soc Robotics 13, 1235–1252 (2021). https://doi.org/10.1007/s12369-020-00712-2
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12369-020-00712-2