Argumentation Accelerated Reinforcement Learning for RoboCup Keepaway-Takeaway

  • Yang Gao
  • Francesca Toni
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8306)


Multi-Agent Learning (MAL) is a complex problem, especially in real-time systems where both cooperative and competitive learning are involved. We study this problem in the RoboCup Soccer Keepaway-Takeaway game and propose Argumentation Accelerated Reinforcement Learning (AARL) for this game. AARL incorporates heuristics, represented by arguments in Value-Based Argumentation, into Reinforcement Learning (RL) by using Heuristically Accelerated RL techniques. We empirically study for a specific setting of the Keepaway-Takeaway game the suitability of AARL, in comparison with standard RL and hand-coded strategies, to meet the challenges of MAL.


Reinforcement Learn Multiagent System Markov Decision Process Argumentation Framework Strict Partial Order 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bench-Capon, T.: Persuasion in practical argument using value-based argumentation frameworks. J. Log. Comput. 13(3), 429–448 (2003)CrossRefzbMATHMathSciNetGoogle Scholar
  2. 2.
    Bianchi, R.: Using heuristics to accelerate reinforcement learning algorithms. Ph.D. thesis, University of São Paulo (2004) (in Portuguese)Google Scholar
  3. 3.
    Bianchi, R., Ribeiro, C., Costa, A.: Accelerated autonumous learning by using heuristic selection of actions. Journal of Heuristics 14, 135–168 (2008)CrossRefGoogle Scholar
  4. 4.
    Bradtke, S., Duff, M.: Reinforcement learning methods for continuous-time markov decision problems. Advances in Neural Information Processing Systems 7, 393–400 (1995)Google Scholar
  5. 5.
    Claus, C., Boutilier, C.: The dynamics of reinforcement learning in cooperative multiagent systems. In: The Proc. of AAAI (1998)Google Scholar
  6. 6.
    Devlin, S., Grzes, M., Kudenko, D.: An empirical study of potential-based reward shaping and advice in complex, multi-agent systems. Advances in Complex Systems 14, 251–278 (2011)CrossRefMathSciNetGoogle Scholar
  7. 7.
    Dung, P.M.: On the acceptability of arugments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artificial Intelligence 77(2), 321–357 (1995)CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Fan, X., Toni, F.: Argumentation dialogues for two-agent conflict resolution. In: Proc. of COMMA (2012)Google Scholar
  9. 9.
    Ferretti, E., Errecalde, M., García, A., Simari, G.: An application of defeasible logic programming to decision making in a robotic environment. In: LPNMR (2007)Google Scholar
  10. 10.
    Gao, Y., Toni, F., Craven, R.: Argumentation-based reinforcement learning for robocup soccer keepaway. In: Proc. of ECAI (2012)Google Scholar
  11. 11.
    Ghavamzadeh, M., Mahadevan, S., Makar, R.: Hierarchical multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems 13, 197–229 (2006)CrossRefGoogle Scholar
  12. 12.
    Guestrin, C., Lagoudakis, M., Parr, R.: Coordinated reinforcement learning. In: Machine Learning International Workshop Then Conference (2002)Google Scholar
  13. 13.
    Hu, J., Wellman, M.P.: Multiagent reinforcement learning: Theoretical framework and an algorithm. In: Proc. of ICML (1998)Google Scholar
  14. 14.
    Iscen, A., Erogul, U.: A new perspective to the keepaway soccer: The takers (short paper). In: Proc. of AAMAS (2008)Google Scholar
  15. 15.
    Lau, Q.P., Lee, M.L., Hsu, W.: Coordination guided reinforcement learning. In: Proc. of AAMAS (2012)Google Scholar
  16. 16.
    Littman, M.L.: Markov games as a framework for multi-agent reinforcement learning. In: Proc. of ICML (1994)Google Scholar
  17. 17.
    Min, H.Q., Zeng, J.A., Chen, J., Zhu, J.H.: A study of reinforcement learning in a new multiagent domain. In: 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (2008)Google Scholar
  18. 18.
    Mozina, M., Zabkar, J., Bratko, I.: Argument based machine learning. Artificial Intelligence 171, 922–937 (2007)CrossRefzbMATHMathSciNetGoogle Scholar
  19. 19.
    Sen, S., Sekaran, M., Hale, J.: Learning to coordinate without sharing information. In: Proc. of AAAI (1994)Google Scholar
  20. 20.
    Singh, S.P., Sutton, R.S.: Reinforcement learning with replacing eligibility traces. Machine Learning 22, 123–158 (1996)zbMATHGoogle Scholar
  21. 21.
    Stone, P., Sutton, R., Kuhlmann, G.: Reinforcement learning for robocup soccer keepaway. Adaptive Behavior 13, 165–188 (2005)CrossRefGoogle Scholar
  22. 22.
    Sutton, R., Barto, A.: Reinforcement Learning. MIT Press (1998)Google Scholar
  23. 23.
    Tambe, M., Jung, H.: The benefits of arguing in a team. AI Magzine 20(4), 85–92 (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Yang Gao
    • 1
  • Francesca Toni
    • 1
  1. 1.Department of ComputingImperial College LondonUK

Personalised recommendations