Autonomous Agents and Multi-Agent Systems

, Volume 28, Issue 5, pp 749–778 | Cite as

A decentralized approach for convention emergence in multi-agent systems

  • Mihail MihaylovEmail author
  • Karl Tuyls
  • Ann Nowé


The field of convention emergence studies how agents involved in repeated coordination games can reach consensus through only local interactions. The literature on this topic is vast and is motivated by human societies, mainly addressing coordination problems between human agents, such as who gets to redial after a dropped telephone call. In contrast, real-world engineering problems, such as coordination in wireless sensor networks, involve agents with limited resources and knowledge and thus pose certain restrictions on the complexity of the coordination mechanisms. Due to these restrictions, strategies proposed for human coordination may not be suitable for engineering applications and need to be further explored in the context of real-world application domains. In this article we take the role of designers of large decentralized multi-agent systems. We investigate the factors that speed up the convergence process of agents arranged in different static and dynamic topologies and under different interaction models, typical for engineering applications. We also study coordination problems both under partial observability and in the presence of faults (or noise). The main contributions of this article are that we propose an approach for emergent coordination, motivated by highly constrained devices, such as wireless nodes and swarm bots, in the absence of a central entity and perform extensive theoretical and empirical studies. Our approach is called Win-Stay Lose-probabilistic-Shift, generalizing two well-known strategies in game theory that have been applied in other domains. We demonstrate that our approach performs well in different settings under limited information and imposes minimal system requirements, due to its simplicity. Moreover, our technique outperforms state-of-the-art coordination mechanisms, guarantees full convergence in any topology and has the property that all convention states are absorbing.


Coordination games Convention emergence Decentralized systems 


  1. 1.
    Axelrod, R. (1984). The evolution of cooperation. New York: Basic Books.Google Scholar
  2. 2.
    Axelrod, R. (1986). An evolutionary approach to norms. The American Political Science Review, 80(4), 1095–1111.CrossRefGoogle Scholar
  3. 3.
    Barabasi, A. L., Albert, R., & Jeong, H. (1999). Mean-field theory for scale-free random networks. Physica A: Statistical Mechanics and its Applications, 272(1–2), 19.Google Scholar
  4. 4.
    Barrett, J., & Zollman, K. J. S. (2009). The role of forgetting in the evolution and learning of language. Journal of Experimental & Theoretical Artificial Intelligence, 21(4), 293–309.CrossRefzbMATHGoogle Scholar
  5. 5.
    Bendor, J., Mookherjee, D., & Ray, D. (1994). Aspirations, adaptive learning and cooperation in repeated games. Tech. rep., Tilburg University, Center for Economic Research.Google Scholar
  6. 6.
    Bowling, M., & Veloso, M. (2002). Multiagent learning using a variable learning rate. Artificial Intelligence, 136(2), 215–250.CrossRefzbMATHMathSciNetGoogle Scholar
  7. 7.
    Bramoullé, Y., López-Pintado, D., Goyal, S., & Vega-Redondo, F. (2004). Network formation and anti-coordination games. International Journal of Games Theory, 33(1), 1–19.CrossRefzbMATHGoogle Scholar
  8. 8.
    Brooks, L., Iba, W., & Sen, S. (2011). Modeling the emergence and convergence of norms. In Proceedings of the Twenty-Second international joint conference on Artificial Intelligence (Vol. 1, pp. 97–102). Menlo Park, CA: AAAI Press.Google Scholar
  9. 9.
    Castellano, C., Fortunato, S., & Loreto, V. (2009). Statistical physics of social dynamics. Reviews of modern physics, 81(2), 591.CrossRefGoogle Scholar
  10. 10.
    De Hauwere, Y.M. (2011). Sparse interactions in multi-agent reinforcement learning (Ph.D. thesis, Vrije Universiteit Brussel, 2011).Google Scholar
  11. 11.
    De Vylder B. (2007). The evolution of conventions in multi-agent systems (Ph.D. thesis, Vrije Universiteit Brussel, 2007).Google Scholar
  12. 12.
    Delgado, J., Pujol, J., & Sanguesa, R. (2003). Emergence of coordination in scale-free networks. Web Intelligence and Agent Systems, 1(2), 131–138.Google Scholar
  13. 13.
    Farinelli, A., Rogers, A., Petcu, A., & Jennings, N. R. (2008). Decentralised coordination of lowpower embedded devices using the max-sum algorithm. In Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent System (pp. 639–646). Richland, SC.Google Scholar
  14. 14.
    Fischer, M. J., Na, Lynch, & Paterson, M. S. (1985). Impossibility of distributed consensus with one faulty process. Journal of the ACM, 32(2), 374–382.CrossRefzbMATHGoogle Scholar
  15. 15.
    Franks, H., Griffiths, N., & Jhumka, A. (2013). Manipulating convention emergence using influencer agents. Journal of Autonomous Agents and Multi-Agent Systems, 26(3), 315–353.CrossRefGoogle Scholar
  16. 16.
    Grenager, T., Powers, R., & Shoham, Y. (2002). Dispersion games: general definitions and some specific learning results. In Proceedings of the Eighteenth National Conference on Artificial Intelligence (pp. 398–403). Alpern: AAAI Press.Google Scholar
  17. 17.
    de Jong, S., Uyttendaele, S., & Tuyls, K. (2008). Learning to reach agreement in a continuous ultimatum game. Journal of Artificial Intelligence Research (JAIR), 33, 551–574.zbMATHGoogle Scholar
  18. 18.
    Karandikar, R., Mookherjee, D., Ray, D., & Vega-Redondo, F. (1998). Evolving aspirations and cooperation. Journal of Economic Theory, 80(2), 292–331.CrossRefzbMATHMathSciNetGoogle Scholar
  19. 19.
    Kelley, H., Thibaut, J., Radloff, R., & Mundy, D. (1962). The development of cooperation in the minimal social situation. Psychological Monographs: General and Applied, 76(19), 1–19.CrossRefGoogle Scholar
  20. 20.
    Kemeny, J., & Snell, J. (1969). Finite Markov chains. New York: VanNostrand.Google Scholar
  21. 21.
    Kittock, J. (1993). Emergent conventions and the structure of multi-agent systems. In Proceedings of the 1993 Santa Fe Institute Complex Systems Summer School (Vol. 5, pp. 1–14). Citeseer.Google Scholar
  22. 22.
    Knoester, D. B., & McKinley, P. K. (2009). Evolving virtual fireflies. In Proceedings of the 10th European Conference on Artificial Life, Budapest, Hungary.Google Scholar
  23. 23.
    Kojima, F., & Takahashi, S. (2007). Anti-coordination games and dynamic stability. International Game Theory Review, 9(4), 667–688.CrossRefzbMATHMathSciNetGoogle Scholar
  24. 24.
    Lemmens, B., Steenhaut, K., Ruckebusch, P., Moerman, I., & Nowé, A. (2012). Network-wide synchronization in wireless sensor networks. In Proceedings of the 19th IEEE Symposium on Communications and Vehicular Technology in the Benelux.Google Scholar
  25. 25.
    Lewis, D. (1969). Convention: A philosophical study. Cambridge: Harvard University Press.Google Scholar
  26. 26.
    Lu, G., Krishnamachari, B., & Raghavendra, C. (2004). An adaptive energy-efficient and low-latency MAC for data gathering in wireless sensor networks. In Proceedings of the 18th International Symposium on Parallel and Distributed Processing (p. 224).Google Scholar
  27. 27.
    Mihaylov, M. (2012). Decentralized coordination in multi-agent systems (Ph.D. thesis, Vrije Universiteit Brussel, 2012).Google Scholar
  28. 28.
    Mihaylov, M., Le Borgne, Y. A., Tuyls, K., & Nowé, A. (2013). Reinforcement learning for self-organizing wake-up scheduling in wireless sensor networks. Agents and artificial intelligence (Vol. 271, pp. 382–396). Berlin, Heidelberg: Springer.Google Scholar
  29. 29.
    Mobilia, M. (2003). Does a single zealot affect an infinite group of voters? Physical Review Letters, 91(2), 028,701.CrossRefGoogle Scholar
  30. 30.
    Mukherjee, P., Sen, S., & Airiau, S. (2008). Norm emergence under constrained interactions in diverse societies. In Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems (Vol. 2, pp. 779–786). International Foundation for Autonomous Agents and Multiagent Systems.Google Scholar
  31. 31.
    Nowak, M., & Sigmund, K. (1993). A strategy of Win-Stay, Lose-Shift that outperforms tit-for-tat in the Prisoner’s Dilemma game. Nature, 364, 56–58.CrossRefGoogle Scholar
  32. 32.
    Savarimuthu, B. T. R., Cranefield, S., Purvis, M. K., & Purvis, M. A. (2009). Norm emergence in agent societies formed by dynamically changing networks. Web Intelligence and Agent Systems, 7(3), 223–232.Google Scholar
  33. 33.
    Schelling, T. C. (1960). The strategy of conflict. Cambridge: Harvard University Press.Google Scholar
  34. 34.
    Segbroeck, S.V., Santos, F.C., Lenaerts, T., Pacheco, J.M. (2009) Emergence of cooperation in adaptive social networks with behavioral diversity. In Proceedings of the 10th European Conference on Artificial Life (ECAL) (pp. 434–441).Google Scholar
  35. 35.
    Sen, S., & Airiau, S. (2007). Emergence of norms through social learning. In Proceedings of the Twentieth International Joint Conference on Artificial Intelligence (pp. 1507–1512).Google Scholar
  36. 36.
    Shoham, Y., & Tennenholtz, M. (1993). Co-learning and the evolution of social activity. Tech. rep. Stanford University.Google Scholar
  37. 37.
    Shoham, Y., & Tennenholtz, M. (1997). On the emergence of social conventions: Modeling, analysis, and simulations. Artificial Intelligence, 94, 139–166.CrossRefzbMATHGoogle Scholar
  38. 38.
    Steels, L. (1997). The synthetic modeling of language origins. Evolution of Communication, 1(1), 1–34.CrossRefGoogle Scholar
  39. 39.
    Traag, V. (2011). Indirect reciprocity through gossiping can lead to cooperative clusters. In IEEE ALIFE (pp. 154–161).Google Scholar
  40. 40.
    Urbano, P., Ja, B., Antunes, L., & Moniz, L. (2009). Force versus majority: A comparison in convention emergence efficiency. In Coordination, Organizations, Institutions and Norms in Agent Systems IV (pp. 48–63).Google Scholar
  41. 41.
    Villatoro, D., Sabater-Mir, J., & Sen, S. (2011a). Social Instruments for Robust Convention Emergence. In Twenty-Second International Joint Conference On Artificial Intelligence (IJCAI) (p. 6). Barcelona, Spain.Google Scholar
  42. 42.
    Villatoro, D., Sen, S., & Sabater-Mir, J. (2011b). Exploring the dimensions of convention emergence in multiagent systems. Advances in Complex Systems, 14(02), 201–227.CrossRefMathSciNetGoogle Scholar
  43. 43.
    Ye, W., Heidemann, J., & Estrin, D. (2004). Medium access control with coordinated adaptive sleeping for wireless sensor networks. IEEE/ACM Transactions on Networking, 12(3), 493–506.CrossRefGoogle Scholar
  44. 44.
    Young, H. P. (1993). The evolution of conventions. Econometrica, 61(1), 57–84.CrossRefzbMATHMathSciNetGoogle Scholar

Copyright information

© The Author(s) 2013

Authors and Affiliations

  1. 1.Vrije Universiteit BrusselBrusselsBelgium
  2. 2.Maastricht UniversityMaastrichtThe Netherlands
  3. 3.University of LiverpoolLiverpoolUK

Personalised recommendations