Abstract
RoboCup simulated soccer presents many challenges to machine learning (ML) methods, including a large state space, hidden and uncertain state, multiple agents, and long and variable delays in the effects of actions. While there have been many successful ML applications to portions of the robotic soccer task, it appears to be still beyond the capabilities of modern machine learning techniques to enable a team of 11 agents to successfully learn the full robotic soccer task from sensors to actuators. Because the successful applications to portions of the task have been embedded in different teams and have often addressed different sub-tasks, they have been difficult to compare. We put forth keepaway soccer as a domain suitable for directly comparing different machine learning approaches to robotic soccer. It is complex enough that it can’t be solved trivially, yet simple enough that complete machine learning approaches are feasible. In keepaway, one team, “the keepers,” tries to keep control of the ball for as long as possible despite the efforts of “the takers.” The keepers learn individually when to hold the ball and when to pass to a teammate, while the takers learn when to charge the ball-holder and when to cover possible passing lanes. We fully specify the domain and summarize some initial, successful learning results.
Chapter PDF
References
Tomohito Andou. Refinement of soccer agents’ positions using reinforcement learning. In Hiroaki Kitano, editor, RoboCup-97: Robot Soccer World Cup I, pages 373–388. Springer Verlag, Berlin, 1998.
David Andre and Astro Teller. Evolving team Darwin United. In Minoru Asada and Hiroaki Kitano, editors, RoboCup-98: Robot Soccer World Cup II. Springer Verlag, Berlin, 1999.
S. J. Bradtke and M. O. Duff. Reinforcement learning methods for continuous-time Markov decision problems. pages 393–400, San Mateo, CA, 1995. Morgan Kaufmann.
Hiroaki Kitano, Milind Tambe, Peter Stone, Manuela Veloso, Silvia Coradeschi, Eiichi Osawa, Hitoshi Matsubara, Itsuki Noda, and Minoru Asada. The RoboCup synthetic agent challenge 97. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, pages 24–29, San Francisco, CA, 1997. Morgan Kaufmann.
Sean Luke, Charles Hohn, Jonathan Farris, Gary Jackson, and James Hendler. Co-evolving soccer softbot team coordination with genetic programming. In Hiroaki Kitano, editor, RoboCup-97: Robot Soccer World Cup I, pages 398–411, Berlin, 1998. Springer Verlag.
Itsuki Noda, Hitoshi Matsubara, Kazuo Hiraki, and Ian Frank. Soccer server: A tool for research on multiagent systems. Applied Artificial Intelligence, 12:233–250, 1998.
M. L. Puterman. Markov Decision Problems. Wiley, NY, 1994.
J. Ross Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, CA, 1993.
M. Riedmiller, A. Merke, D. Meier, A. Hoffman, A. Sinner, O. Thate, and R. Ehrmann. Karlsruhe brainstormers—a reinforcement learning approach to robotic soccer. In Peter Stone, Tucker Balch, and Gerhard Kraetszchmar, editors, RoboCup-2000: Robot Soccer World Cup IV. Springer Verlag, Berlin, 2001.
Peter Stone. Layered Learning in Multiagent Systems: A Winning Approach to Robotic Soccer. MIT Press, 2000.
Peter Stone and David McAllester. An architecture for action selection in robotic soccer. In Proceedings of the Fifth International Conference on Autonomous Agents, 2001.
Peter Stone and Richard S. Sutton. Scaling reinforcement learning toward robocup soccer. In Proceedings of the Eighteenth International Conference on Machine Learning, 2001.
Peter Stone and Manuela Veloso. Team-partitioned, opaque-transition reinforcement learning. In Minoru Asada and Hiroaki Kitano, editors, RoboCup-98: Robot Soccer World Cup II. Springer Verlag, Berlin, 1999. Also in Proceedings of the Third International Conference on Autonomous Agents, 1999.
R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12, pages 1057–1063. The MIT Press, 2000.
Eiji Uchibe. Cooperative Behavior Acquisition by Learning and Evolution in a Multi-Agent Environment for Mobile Robots. PhD thesis, Osaka University, January 1999.
Manuela Veloso, Peter Stone, and Michael Bowling. Anticipation as a key for collaboration in a team of agents: A case study in robotic soccer. In Proceedings of SPIE Sensor Fusion and Decentralized Control in Robotic Systems II, volume 3839, Boston, September 1999.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Stone, P., Sutton, R.S. (2002). Keepaway Soccer: A Machine Learning Test bed. In: Birk, A., Coradeschi, S., Tadokoro, S. (eds) RoboCup 2001: Robot Soccer World Cup V. RoboCup 2001. Lecture Notes in Computer Science(), vol 2377. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45603-1_22
Download citation
DOI: https://doi.org/10.1007/3-540-45603-1_22
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-43912-7
Online ISBN: 978-3-540-45603-2
eBook Packages: Springer Book Archive