A Reinforcement Learning Approach with Spline-Fit Object Tracking for AIBO Robot’s High Level Decision Making

  • Subhasis Mukherjee
  • Shamsul Huda
  • John Yearwood
Conference paper
Part of the Studies in Computational Intelligence book series (SCI, volume 368)


Robocup is a popular test bed for AI programs around the world. Robosoccer is one of the two major parts of Robocup, in which AIBO entertainment robots take part in the middle sized soccer event. The three key challenges that robots need to face in this event are manoeuvrability, image recognition and decision making skills. This paper focuses on the decision making problem in Robosoccer- The goal keeper problem. We investigate whether reinforcement learning (RL) as a form of semi-supervised learning can effectively contribute to the goal keeper’s decision making process when penalty shot and two attacker problem are considered. Currently, the decision making process in Robosoccer is carried out using rule-base system. RL also is used for quadruped locomotion and navigation purpose in Robosoccer using AIBO. Moreover the ball distance is being calculated using IR sensors available at the nose of the robot. In this paper, we propose a reinforcement learning based approach that uses a dynamic state-action mapping using back propagation of reward and Q-learning along with spline fit (QLSF) for the final choice of high level functions in order to save the goal. The novelty of our approach is that the agent learns while playing and can take independent decision which overcomes the limitations of rule-base system due to fixed and limited predefined decision rules. The spline fit method used with the nose camera was also able to find out the location and the ball distance more accurately compare to the IR sensors. The noise source and near and far sensor dilemma problem with IR sensor was neutralized using the proposed spline fit method. Performance of the proposed method has been verified against the bench mark data set made with Upenn’03 code logic and a base line experiment with IR sensors. It was found that the efficiency of our QLSF approach in goalkeeping was better than the rule based approach in conjunction with the IR sensors. The QLSF develops a semi-supervised learning process over the rule-base system’s input-output mapping process, given in the Upenn’03 code.


Reinforcement Learning semi-supervised Robocup Aperius Robosoccer IR sensros camera 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Mackworth, A.: Computer Vision: System, Theory, and Applications. World Scientific Press, Singapore (1993)Google Scholar
  2. 2.
  3. 3.
    Honda, Asimo (2008), http://asimo.honda.com/
  4. 4.
  5. 5.
  6. 6.
    Operating system documentation project (2003), http://www.operating-system.org
  7. 7.
    Currie, A.: The history of roboticsGoogle Scholar
  8. 8.
    Fujita, M., Kitano, H.: Development of autonomous robot quadruped robot for robot entertainment. In: Autonomous Agents, pp. 7–18. Springer, Heidelberg (1998)Google Scholar
  9. 9.
    Coradeschi, S., Karlsson, L., Stone, P., Balch, T., Kraetzschmar, G., Asad, M., Veloso, M.: Overview of robocup 1999. Springer, Heidelberg (2000)Google Scholar
  10. 10.
    Stone, P., Balch, T., Kraetzschmar, G.K. (eds.): RoboCup 2000. LNCS (LNAI), vol. 2019. Springer, Heidelberg (2001)MATHGoogle Scholar
  11. 11.
    Birk, A., Coradeschi, S., Tadokoro, S. (eds.): RoboCup 2001. LNCS (LNAI), vol. 2377. Springer, Heidelberg (2002)MATHGoogle Scholar
  12. 12.
    Sammut, C., Uther, W., Hengst, B.: ”runswift 2003”, school of Computer Science and Engineering University of New South Wales and National ICT Australia (2003)Google Scholar
  13. 13.
    Kleiner, A.: Rescue simulation project (December 14, 2008), http://kaspar.informatik.uni-freiburg.de/rcr2005/
  14. 14.
    Niranjan, M., Rummery, G.A.: On-line q-learning using connectionist systems, Ph.D. dissertation, Cambridge University Engineering Department (1994)Google Scholar
  15. 15.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (1998)Google Scholar
  16. 16.
  17. 17.
    Negnevitsky, M.: Artificial Intelligence: Aguide to intelligent system, ch. 2. Pearson education, London (2002) ISBN 0-201-71159-1 Google Scholar
  18. 18.
    Mukherjee, S., Yearwood, J., Vamplew, P.: Applying Reinforcement Learning in playing Robosoccer using the AIBO. In: GSITMS, University of Ballarat, Victoria, Australia (2010)Google Scholar
  19. 19.
    Robocup, Humanoid league 2006, http://www.humanoidsoccer.org/
  20. 20.

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Subhasis Mukherjee
    • 1
  • Shamsul Huda
    • 1
  • John Yearwood
    • 1
  1. 1.Centre for Informatics and Applied optimization GSITMSUniversity of BallaratAustralia

Personalised recommendations