Keepaway Soccer: From Machine Learning Testbed to Benchmark

  • Peter Stone
  • Gregory Kuhlmann
  • Matthew E. Taylor
  • Yaxin Liu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4020)


Keepaway soccer has been previously put forth as a testbed for machine learning. Although multiple researchers have used it successfully for machine learning experiments, doing so has required a good deal of domain expertise. This paper introduces a set of programs, tools, and resources designed to make the domain easily usable for experimentation without any prior knowledge of RoboCup or the Soccer Server. In addition, we report on new experiments in the Keepaway domain, along with performance results designed to be directly comparable with future experimental results. Combined, the new infrastructure and our concrete demonstration of its use in comparative experiments elevate the domain to a machine learning benchmark, suitable for use by researchers across the field.


Radial Basis Function Multiagent System Radial Basis Function Network Function Approximator Episode Duration 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Albus, J.S.: Brains, Behavior, and Robotics. Byte Books, Peterborough (1981)Google Scholar
  2. 2.
    Balch, T.: Teambots (2000),
  3. 3.
    Balch, T.: Teambots domain: Soccerbots (2000),
  4. 4.
    Blake, C.L., Merz, C.J.: UCI repository of machine learning databases (1998)Google Scholar
  5. 5.
    Bradtke, S.J., Duff, M.O.: Reinforcement learning methods for continuous-time Markov decision problems. In: Leen, T., Tesauro, G., Touretzky, D. (eds.) Advances in Neural Information Processing Systems, San Mateo, CA, vol. 7, pp. 393–400. Morgan Kaufmann, San Francisco (1995)Google Scholar
  6. 6.
    Chen, M., Foroughi, E., Heintz, F., Kapetanakis, S., Kostiadis, K., Kummeneje, J., Noda, I., Obst, O., Riley, P., Steffens, T., Wang, Y., Yin, X.: Users manual: RoboCup soccer server manual for soccer server version 7.07 and later (2003), available at:
  7. 7.
    Crites, R.H., Barto, A.G.: Improving elevator performance using reinforcement learning. In: Touretzky, D.S., Mozer, M.C., Hasselmo, M.E. (eds.) Advances in Neural Information Processing Systems, vol. 8. MIT Press, Cambridge (1996)Google Scholar
  8. 8.
    de Boer, R., Kok, J.R.: The incremental development of a synthetic multi-agent system: The uva trilearn 2001 robotic soccer simulation team. Master’s thesis, University of Amsterdam, The Netherlands (February 2002)Google Scholar
  9. 9.
    Hsu, W.H., Gustafson, S.M.: Genetic programming and multi-agent layered learning by reinforcements. In: Genetic and Evolutionary Computation Conference, New York (July 2002)Google Scholar
  10. 10.
    Kuhlmann, G., Stone, P.: Progress in learning 3 vs. 2 keepaway. In: Polani, D., Browning, B., Bonarini, A., Yoshida, K. (eds.) RoboCup 2003. LNCS, vol. 3020, pp. 694–702. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  11. 11.
    Noda, I., Matsubara, H., Hiraki, K., Frank, I.: Soccer server: A tool for research on multiagent systems. Applied Artificial Intelligence 12, 233–250 (1998)CrossRefGoogle Scholar
  12. 12.
    Di Pietro, A., While, L., Barone, L.: Learning in RoboCup keepaway using evolutionary algorithms. In: Langdon, W.B., Cantú-Paz, E., Mathias, K., Roy, R., Davis, D., Poli, R., Balakrishnan, K., Honavar, V., Rudolph, G., Wegener, J., Bull, L., Potter, M.A., Schultz, A.C., Miller, J.F., Burke, E., Jonoska, N. (eds.) GECCO 2002: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1065–1072. Morgan Kaufmann, San Francisco (2002)Google Scholar
  13. 13.
    Puterman, M.L.: Markov Decision Processes. Wiley, Chichester (1994)MATHCrossRefGoogle Scholar
  14. 14.
    Rummery, G.A., Niranjan, M.: On-line Q-learning using connectionist systems. Technical Report CUED/F-INFENG/TR 166, Cambridge University Engineering Department (1994)Google Scholar
  15. 15.
    Stone, P., Sutton, R.S.: Keepaway soccer: A machine learning testbed. In: Birk, A., Coradeschi, S., Tadokoro, S. (eds.) RoboCup 2001. LNCS, vol. 2377, pp. 214–223. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  16. 16.
    Stone, P., Sutton, R.S., Kuhlmann, G.: Reinforcement learning for RoboCup-soccer keepaway. In: Adaptive Behavior (to appear, 2005)Google Scholar
  17. 17.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  18. 18.
    Taylor, M.E., Stone, P.: Behavior transfer for value-function-based reinforcement learning. In: The Fourth International Joint Conference on Autonomous Agents and Multiagent Systems (July 2005) (to appear)Google Scholar
  19. 19.
    Tesauro, G.: TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Computation 6(2), 215–219 (1994)CrossRefGoogle Scholar
  20. 20.
    Walker, T., Shavlik, J., Maclin, R.: Relational reinforcement learning via sampling the space of first-order conjunctive features. In: Proceedings of the ICML Workshop on Relational Reinforcement Learning, Banff, Canada (July 2004)Google Scholar
  21. 21.
    Whiteson, S., Kohl, N., Miikkulainen, R., Stone, P.: Evolving keepaway soccer players through task decomposition. Machine Learning 59(1), 5–30 (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Peter Stone
    • 1
  • Gregory Kuhlmann
    • 1
  • Matthew E. Taylor
    • 1
  • Yaxin Liu
    • 1
  1. 1.Department of Computer SciencesThe University of Texas at AustinAustin

Personalised recommendations