Implicit Negotiation in Repeated Games

  • Michael L. Littman
  • Peter Stone
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2333)


In business-related interactions such as the on-going high-stakes FCC spectrum auctions, explicit communication among participants is regarded as collusion, and is therefore illegal. In this paper, we consider the possibility of autonomous agents engaging in implicit negotiation via their tacit interactions. In repeated general-sum games, our testbed for studying this type of interaction, an agent using a “best response” strategy maximizes its own payoff assuming its behavior has no effect on its opponent. This notion of best response requires some degree of learning to determine the fixed opponent behavior. Against an unchanging opponent, the best-response agent performs optimally, and can be thought of as a “follower,” since it adapts to its opponent. However, pairing two best-response agents in a repeated game can result in sub-optimal behavior. We demonstrate this suboptimality in several different games using variants of Q-learning as an example of a best-response strategy. We then examine two “leader” strategies that induce better performance from opponent followers via stubbornness and threats. These tactics are forms of implicit negotiation in that they aim to achieve a mutually beneficial outcome without using explicit communication outside of the game.


Nash Equilibrium Multiagent System Repeated Game Targetable Pair Mutual Cooperation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Robert Axelrod. The Evolution of Cooperation. Basic Books, 1984.Google Scholar
  2. 2.
    Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. In Proceedings of the Fifteenth National Conference on Artificial Intelligence, pages 746–752, 1998.Google Scholar
  3. 3.
    Peter Cramton and Jesse Schwartz. Collusive bidding: Lessons from the FCC spectrum auctions. Journal of Regulatory Economics, 17:229–252, 2000.CrossRefGoogle Scholar
  4. 4.
    János A. Csirik, Michael L. Littman, Satinder Singh, and Peter Stone. FAucS: An FCC spectrum auction simulator for autonomous bidding agents. In Ludger Fiege, Gero Mühl, and Uwe Wilhelm, editors, Electronic Commerce: Proceedings of the Second International Workshop, pages 139–151, Heidelberg, Germany, 2001. Springer Verlag.Google Scholar
  5. 5.
    Maier Fenster, Sarit Kraus, and Jeffrey S. Rosenschein. Coordination without communication: Experimental validation of focal point techniques. In Proceedings of the First International Conference on Multi-Agent Systems, pages 102–108, Menlo Park, California, June 1995. AAAI Press.Google Scholar
  6. 6.
    Drew Fudenberg and David K. Levine. The Theory of Learning in Games.The MIT Press, 1998.Google Scholar
  7. 7.
    Amy Greenwald, Eric Friedman, and Scott Shenker. Learning in network contexts: Experimental results from simulations. Journal of Games and Economic Behavior, 35(1/2):80–123, 2001.zbMATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Junling Hu and Michael P. Wellman. Multiagent reinforcement learning: Theoretical framework and an algorithm. In Jude Shavlik, editor, Proceedings of the Fifteenth International Conference on Machine Learning, pages 242–250, 1998.Google Scholar
  9. 9.
    Amir Jafari, Amy Greenwald, David Gondek, and Gunes Ercal. On no-regret learning, fictitious play, and Nash equilibria. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 226–233. Morgan Kaufmann, 2001.Google Scholar
  10. 10.
    Michael L. Littman. Friend-or-foe Q-learning in general-sum games. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 322–328. Morgan Kaufmann, 2001.Google Scholar
  11. 11.
    Michael L. Littman and Peter Stone. A polynomial-time algorithm for computing Nash equilibria in repeated bimatrix games. Research note, 2001.Google Scholar
  12. 12.
    Manisha Mundhe and Sandip Sen. Evaluating concurrent reinforcement learners. In Proceedings of the Fourth International Conference on Multiagent Systems, pages 421–422. IEEE Press, 2000.Google Scholar
  13. 13.
    J. F. Nash. Non-cooperative games. Annals of Mathematics, 54:286–295, 1951.CrossRefMathSciNetGoogle Scholar
  14. 14.
    Martin J. Osborne and Ariel Rubinstein. A Course in Game Theory. The MIT Press, 1994.Google Scholar
  15. 15.
    Christos H. Papadimitriou. On players with a bounded number of states. Games and Economic Behavior, 4:122–131, 1992.zbMATHCrossRefMathSciNetGoogle Scholar
  16. 16.
    Paul S. A. Reitsma, Peter Stone, Janos A. Csirik, and Michael L. Littman. Randomized strategic demand reduction: Getting more by asking for less. Submitted, 2002.Google Scholar
  17. 17.
    Tuomas Sandholm and Victor Lesser. Issues in automated negotiation and electronic commerce: Extending the contract net framework. In Proceedings of the First International Conference on Multi-Agent Systems, pages 328–335, Menlo Park, California, June 1995. AAAI Press.Google Scholar
  18. 18.
    Peter Stone and Manuela Veloso. Task decomposition, dynamic role assignment, and low-bandwidth communication for real-time strategic teamwork. Artificial Intelligence, 110(2):241–273, June 1999.Google Scholar
  19. 19.
    J. vonNeumann and O. Morgenstern. Theory of Games and Economic Behavior. Princeton University Press, Princeton, NJ, 1947.Google Scholar
  20. 20.
    Christopher J.C.H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3):279–292, 1992.zbMATHGoogle Scholar
  21. 21.
    Robert J. Weber. Making more from less: Strategic demand reduction in the FCC spectrum auctions. Journal of Economics and Management Strategy, 6(3):529–548, 1997.CrossRefGoogle Scholar
  22. 22.
    Gilad Zlotkin and Jeffrey S. Rosenschein. Mechanisms for automated negotiation in state oriented domains. Journal of Artificial Intelligence Research, 5:163–238, 1996.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Michael L. Littman
    • 1
  • Peter Stone
    • 1
  1. 1.AT&T Labs ResearchFlorham Park

Personalised recommendations