Learning to Coordinate Using Commitment Sequences in Cooperative Multi-agent Systems

  • Spiros Kapetanakis
  • Daniel Kudenko
  • Malcolm J. A. Strens
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3394)

Abstract

We report on an investigation of the learning of coordination in cooperative multi-agent systems. Specifically, we study solutions that are applicable to independent agents i.e. agents that do not observe one another’s actions. In previous research [5] we have presented a reinforcement learning approach that converges to the optimal joint action even in scenarios with high miscoordination costs. However, this approach failed in fully stochastic environments. In this paper, we present a novel approach based on reward estimation with a shared action-selection protocol. The new technique is applicable in fully stochastic environments where mutual observation of actions is not possible. We demonstrate empirically that our approach causes the agents to converge almost always to the optimal joint action even in difficult stochastic scenarios with high miscoordination penalties.

Keywords

Reinforcement Learning Joint Action Multiagent System Stochastic Game Average Reward 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Boutilier, C.: Sequential optimality and coordination in multiagent systems. In: Proceedings of the Sixteenth International Joint Conference on Articial Intelligence (IJCAI 1999), pp. 478–485 (1999)Google Scholar
  2. 2.
    Claus, C., Boutilier, C.: The dynamics of reinforcement learning in cooperative multiagent systems. In: Proceedings of the Fifteenth National Conference on Articial Intelligence, pp. 746–752 (1998)Google Scholar
  3. 3.
    Fudenberg, D., Levine, D.K.: The Theory of Learning in Games. MIT Press, Cambridge (1998)MATHGoogle Scholar
  4. 4.
    Hu, J., Wellman, M.P.: Multiagent q-learning. Machine Learning Research (2002)Google Scholar
  5. 5.
    Kapetanakis, S., Kudenko, D.: Reinforcement learning of coordination in cooperative multi-agent systems. In: Proceedings of the Eighteenth National Conference on Artificial Intelligence, AAAI 2002 (2002)Google Scholar
  6. 6.
    Lauer, M., Riedmiller, M.: An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In: Proceedings of the Seventeenth International Conference in Machine Learning (2000)Google Scholar
  7. 7.
    Nowé, A., Parent, J., Verbeeck, K.: Social agents playing a periodical policy. In: Proceedings of the 12th European Conference on Machine Learning, Freiburg, Germany (2001)Google Scholar
  8. 8.
    Peshkin, L., Kim, K.-E., Meuleau, N., Kaelbling, L.: Learning to cooperate via policy search. In: Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence (2000)Google Scholar
  9. 9.
    Sen, S., Sekaran, M., Hale, J.: Learning to coordinate without sharing information. In: Proceedings of the Twelfth National Conference on Artificial Intelligence, Seattle, WA, pp. 426–431 (1994)Google Scholar
  10. 10.
    Wang, X., Sandholm, T.: Reinforcement learning to play an optimal nash equilibrium in team markov games. In: Proceedings of the 16th Neural Information Processing Systems: Natural and Synthetic (NIPS) conference, Vancouver, Canada (2002)Google Scholar
  11. 11.
    Weiss, G.: Learning to coordinate actions in multi-agent systems. In: Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, vol. 1, pp. 311–316. Morgan Kaufmann Publ., San Francisco (1993)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Spiros Kapetanakis
    • 1
  • Daniel Kudenko
    • 1
  • Malcolm J. A. Strens
    • 2
  1. 1.Department of Computer ScienceUniversity of YorkHeslington, YorkUK
  2. 2.Guidance and Imaging Solutions,QinetiQFarnborough, HampshireUK

Personalised recommendations