Anticipatory Behavior of Software Agents in Self-organizing Negotiations

  • Jan Ole BerndtEmail author
  • Otthein Herzog
Part of the Cognitive Systems Monographs book series (COSMOS, volume 29)


Software agents are a well-established approach for modeling autonomous entities in distributed artificial intelligence. Iterated negotiations allow for coordinating the activities of multiple autonomous agents by means of repeated interactions. However, if several agents interact concurrently, the participants’ activities can mutually influence each other. This leads to poor coordination results. In this paper, we discuss these interrelations and propose a self-organization approach to cope with that problem. To that end, we apply distributed reinforcement learning as a feedback mechanism to the agents’ decision-making process. This enables the agents to use their experiences from previous activities to anticipate the results of potential future actions. They mutually adapt their behaviors to each other which results in the emergence of social order within the multiagent system. We empirically evaluate the dynamics of that process in a multiagent resource allocation scenario. The results show that the agents successfully anticipate the reactions to their activities in that dynamic and partially observable negotiation environment. This enables them to maximize their payoffs and to drastically outperform non-anticipating agents.


Nash Equilibrium Team Manager Combinatorial Auction Acceptance Level Member Agent 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Berndt, J.O.: Self-organizing supply networks: autonomous agent coordination based on expectations. In: Filipe, J., Fred, A. (eds.) ICAART 2011, vol. 2, pp. 104–113. SciTePress, Rome (2011)Google Scholar
  2. 2.
    Berndt, J.O.: Self-organizing logistics process control: an agent-based approach. In: Filipe, J., Fred, A. (eds.) Agents and Artificial Intelligence, pp. 397–412. Springer, Berlin (2013)CrossRefGoogle Scholar
  3. 3.
    Berndt, J.O., Herzog, O.: Efficient multiagent coordination in dynamic environments. In: Boissier, O., Bradshaw, J., Cao, L., Fischer, K., Hacid, M.S. (eds.) WI-IAT 2011, pp. 188–195. IEEE Computer Society, Lyon (2011)Google Scholar
  4. 4.
    Berndt, J.O., Herzog, O.: Distributed learning of best response behaviors in concurrent iterated many-object negotiations. In: Timm, I.J., Guttmann, C. (eds.) MATES 2012, pp. 15–29. Springer, Berlin (2012)Google Scholar
  5. 5.
    Berndt, J.O., Herzog, O.: Distributed reinforcement learning for optimizing resource allocation in autonomous logistics processes. In: Kreowski, H.J., Scholz-Reiter, B., Thoben, K.D. (eds.) LDIC 2012, pp. 429–439. Springer, Berlin (2013)Google Scholar
  6. 6.
    Buşoniu, L., Babuška, R., De Schutter, B.: Multi-agent reinforcement learning: an overview. In: Srinivasan, D., Jain, L. (eds.) Innovations in Multi-Agent Systems and Applications—1, pp. 183–221. Springer, Heidelberg (2010)Google Scholar
  7. 7.
    Claus, C., Boutilier, C.: The dynamics of reinforcement learning in cooperative multiagent systems. In: AAAI 1998. pp. 746–752. Madison, USA (1998)Google Scholar
  8. 8.
    Cramton, P., Shoham, Y., Steinberg, R. (eds.): Combinatorial Auctions. The MIT Press, Cambridge (2006)zbMATHGoogle Scholar
  9. 9.
    Endriss, U., Maudet, N., Sadri, F., Toni, F.: Negotiating socially optimal allocations of resources. J. Artif. Intell. Res. 25, 315–348 (2006)MathSciNetGoogle Scholar
  10. 10.
    Faratin, P., Sierra, C., Jennings, N.R.: Negotiation decision functions for autonomous agents. Robot. Auton. Syst. 24(3–4), 159–182 (1998)CrossRefGoogle Scholar
  11. 11.
    Foundation for Intelligent Physical Agents: FIPA Iterated Contract Net Interaction Protocol Specification. Standard (2002), document No. SC00030HGoogle Scholar
  12. 12.
    Gjerstad, S., Dickhaut, J.: Price formation in double auctions. Game. Econ. Behav. 22(1), 1–29 (1998)zbMATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Jennings, N.R., Faratin, P., Lomuscio, A.R., Parsons, S., Wooldridge, M.J., Sierra, C.: Automated negotiation: prospects. Methods Chall. Group Decis. Negoti. 10, 199–215 (2001)CrossRefGoogle Scholar
  14. 14.
    Luckhart, C., Irani, K.B.: An algorithmic solution of N-person games. In: AAAI 1986. vol. 1, pp. 158–162. Morgan Kaufmann, Philadelphia, USA (1986)Google Scholar
  15. 15.
    Luhmann, N.: Soziale Systeme. Grundriß einer allgemeinen Theorie. Suhrkamp, Frankfurt (1984)Google Scholar
  16. 16.
    Luhmann, N.: Probleme mit operativer Schließung. In: Luhmann, N. (ed.) Die Soziologie und der Mensch, Soziologische Aufklärung, vol. 6, pp. 12–24. Westdeutscher Verlag, Opladen (1995)Google Scholar
  17. 17.
    Luhmann, N.: Social Systems. Stanford University Press, Stanford (1995)Google Scholar
  18. 18.
    Mazur, D.R.: Combinatorics. A guided tour. MAA Textbooks, The Mathematical Association of America, Washington (2010)zbMATHGoogle Scholar
  19. 19.
    Nash, J.: Non-cooperative Games. Ann. Math. 54(2), 286–295 (1950)CrossRefMathSciNetGoogle Scholar
  20. 20.
    Porter, R., Nudelman, E., Shoham, Y.: Simple search methods for finding a Nash equilibrium. Game. Econ. Behav. 63(2), 642–662 (2008)zbMATHCrossRefMathSciNetGoogle Scholar
  21. 21.
    Ramezani, S., Endriss, U.: Nash social welfare in multiagent resource allocation. In: David, E., Gerding, E., Sarne, D., Shehory, O. (eds.) Agent-Mediated Electronic Commerce, pp. 117–131. Springer, Heidelberg (2010)Google Scholar
  22. 22.
    Schuldt, A.: Multiagent coordination enabling autonomous logistics. Springer, Heidelberg (2011)zbMATHCrossRefGoogle Scholar
  23. 23.
    Schuldt, A., Berndt, J.O., Herzog, O.: The interaction effort in autonomous logistics processes: potential and limitations for cooperation. In: Hülsmann, M., Scholz-Reiter, B., Windt, K. (eds.) Autonomous Cooperation and Control in Logistics, pp. 77–90. Springer, Berlin (2011)CrossRefGoogle Scholar
  24. 24.
    Schuldt, A., Gehrke, J.D., Werner, S.: Designing a simulation middleware for FIPA multiagent systems. In: Jain, L., Gini, M., Faltings, B.B., Terano, T., Zhang, C., Cercone, N., Cao, L. (eds.) WI-IAT 2008, pp. 109–113. IEEE Computer Society Press, Sydney (2008)Google Scholar
  25. 25.
    Schuldt, A., Werner, S.: Distributed Clustering of Autonomous Shipping Containers by Concept, Location, and Time. In: Müller, J.P., Petta, P., Klusch, M., Georgeff, M. (eds.) MATES 2007, pp. 121–132. Springer, Berlin (2007)Google Scholar
  26. 26.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (1998)Google Scholar
  27. 27.
    van Bragt, D.D.B., La Poutré, J.A.: Why Agents for Automated Negotiations Should Be Adaptive. Netnomics 5(2), 101–118 (2003)CrossRefGoogle Scholar
  28. 28.
    von Neumann, J.: Zur Theorie der Gesellschaftsspiele. Math. Ann. 100, 295–320 (1928)zbMATHCrossRefMathSciNetGoogle Scholar
  29. 29.
    von Neumann, J., Morgenstern, O.: Theory of Games and Economic Behavior. Princeton University Press, Princeton (1944)zbMATHGoogle Scholar
  30. 30.
    Watkins, C.J.C.H., Dayan, P.: Q-learning. Mach. Learn. 8(3–4), 279–292 (1992)zbMATHGoogle Scholar
  31. 31.
    Wooldridge, M., Jennings, N.R.: Intelligent agents: theory and practice. Knowl. Eng. Rev. 10(2), 115–152 (1995)CrossRefGoogle Scholar
  32. 32.
    Wooldridge, M., Jennings, N.R.: The cooperative problem-solving process. J. Logic Comput. 9(4), 563–592 (1999)CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Center for Computing and Communication Technologies (TZI)Universität BremenBremenGermany

Personalised recommendations