Argumentation Accelerated Reinforcement Learning for RoboCup Keepaway-Takeaway

  • Yang Gao
  • Francesca Toni
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8306)


Multi-Agent Learning (MAL) is a complex problem, especially in real-time systems where both cooperative and competitive learning are involved. We study this problem in the RoboCup Soccer Keepaway-Takeaway game and propose Argumentation Accelerated Reinforcement Learning (AARL) for this game. AARL incorporates heuristics, represented by arguments in Value-Based Argumentation, into Reinforcement Learning (RL) by using Heuristically Accelerated RL techniques. We empirically study for a specific setting of the Keepaway-Takeaway game the suitability of AARL, in comparison with standard RL and hand-coded strategies, to meet the challenges of MAL.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bench-Capon, T.: Persuasion in practical argument using value-based argumentation frameworks. J. Log. Comput. 13(3), 429–448 (2003)CrossRefMATHMathSciNetGoogle Scholar
  2. 2.
    Bianchi, R.: Using heuristics to accelerate reinforcement learning algorithms. Ph.D. thesis, University of São Paulo (2004) (in Portuguese)Google Scholar
  3. 3.
    Bianchi, R., Ribeiro, C., Costa, A.: Accelerated autonumous learning by using heuristic selection of actions. Journal of Heuristics 14, 135–168 (2008)CrossRefGoogle Scholar
  4. 4.
    Bradtke, S., Duff, M.: Reinforcement learning methods for continuous-time markov decision problems. Advances in Neural Information Processing Systems 7, 393–400 (1995)Google Scholar
  5. 5.
    Claus, C., Boutilier, C.: The dynamics of reinforcement learning in cooperative multiagent systems. In: The Proc. of AAAI (1998)Google Scholar
  6. 6.
    Devlin, S., Grzes, M., Kudenko, D.: An empirical study of potential-based reward shaping and advice in complex, multi-agent systems. Advances in Complex Systems 14, 251–278 (2011)CrossRefMathSciNetGoogle Scholar
  7. 7.
    Dung, P.M.: On the acceptability of arugments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artificial Intelligence 77(2), 321–357 (1995)CrossRefMATHMathSciNetGoogle Scholar
  8. 8.
    Fan, X., Toni, F.: Argumentation dialogues for two-agent conflict resolution. In: Proc. of COMMA (2012)Google Scholar
  9. 9.
    Ferretti, E., Errecalde, M., García, A., Simari, G.: An application of defeasible logic programming to decision making in a robotic environment. In: LPNMR (2007)Google Scholar
  10. 10.
    Gao, Y., Toni, F., Craven, R.: Argumentation-based reinforcement learning for robocup soccer keepaway. In: Proc. of ECAI (2012)Google Scholar
  11. 11.
    Ghavamzadeh, M., Mahadevan, S., Makar, R.: Hierarchical multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems 13, 197–229 (2006)CrossRefGoogle Scholar
  12. 12.
    Guestrin, C., Lagoudakis, M., Parr, R.: Coordinated reinforcement learning. In: Machine Learning International Workshop Then Conference (2002)Google Scholar
  13. 13.
    Hu, J., Wellman, M.P.: Multiagent reinforcement learning: Theoretical framework and an algorithm. In: Proc. of ICML (1998)Google Scholar
  14. 14.
    Iscen, A., Erogul, U.: A new perspective to the keepaway soccer: The takers (short paper). In: Proc. of AAMAS (2008)Google Scholar
  15. 15.
    Lau, Q.P., Lee, M.L., Hsu, W.: Coordination guided reinforcement learning. In: Proc. of AAMAS (2012)Google Scholar
  16. 16.
    Littman, M.L.: Markov games as a framework for multi-agent reinforcement learning. In: Proc. of ICML (1994)Google Scholar
  17. 17.
    Min, H.Q., Zeng, J.A., Chen, J., Zhu, J.H.: A study of reinforcement learning in a new multiagent domain. In: 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (2008)Google Scholar
  18. 18.
    Mozina, M., Zabkar, J., Bratko, I.: Argument based machine learning. Artificial Intelligence 171, 922–937 (2007)CrossRefMATHMathSciNetGoogle Scholar
  19. 19.
    Sen, S., Sekaran, M., Hale, J.: Learning to coordinate without sharing information. In: Proc. of AAAI (1994)Google Scholar
  20. 20.
    Singh, S.P., Sutton, R.S.: Reinforcement learning with replacing eligibility traces. Machine Learning 22, 123–158 (1996)MATHGoogle Scholar
  21. 21.
    Stone, P., Sutton, R., Kuhlmann, G.: Reinforcement learning for robocup soccer keepaway. Adaptive Behavior 13, 165–188 (2005)CrossRefGoogle Scholar
  22. 22.
    Sutton, R., Barto, A.: Reinforcement Learning. MIT Press (1998)Google Scholar
  23. 23.
    Tambe, M., Jung, H.: The benefits of arguing in a team. AI Magzine 20(4), 85–92 (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Yang Gao
    • 1
  • Francesca Toni
    • 1
  1. 1.Department of ComputingImperial College LondonUK

Personalised recommendations